Re: High Interrupt After 7.3 Upgrade

[Boyd Stephens]

On 6/1/23 13:16, Samuel Jayden wrote:

Hi Boyd,

I noted the uptime values when I received notifications like "Internet is
slow", "Intranet is too slow" from users. In all of them, the load average
was 5 and above.
This is what I mean with the firewall just slowed down.

Also there were more error messages like these:
pmap_unwire: wiring for pmap 0xfd8e8c2f8bd8 va 0xc000a44000 didn't
change!
In Fact I live with pmap_unwire messages but that time those messages were
increased so much.

That's nearly all I have at the moment. Thanks.




On Thu, Jun 1, 2023 at 8:39 PM Boyd Stephens 
wrote:


On 5/2/23 13:24, Samuel Jayden wrote:


My firewall just slowed down after upgrading from 7.2 to 7.3.


Hello Samuel,

When you mention that your "firewall just slowed down" specifically what
metric and/or anecdotal data are/were you using to determine this
particular status of its operation(s)?

Secondly what type of network load is present when you experience these
issues?


Boyd Stephens
I85Cyber.org



Samuel,

I did note those particulars metrics listed in your previous 
correspondence but when I refer to network loads I am specifically 
referencing data that show cases


- Volume of traffic that your network appliance is processes when the 
gremlins show themselves
- Types of network services being offered by and traffic flowing through 
the device at those particular times
- Looking at basic appliance hardware/OS system load readings vs network 
traffic loads ratio graphs/data sets


From you dmesg, it appears that the hardware about which we are having 
a conversation is a


Lanner Inc. NCA4010D
- 
https://www.lannerinc.com/products/telecom-datacenter-appliances/vcpe-ucpe-platforms/nca-4010


is this the case?

Finally, are you utilizing any of the 10Gbps Intel X710 SFP+ ports.  As 
Mr Stuart will bear witness, a number of months back he assisted me in 
tracking down some resource utilization issues around a similar but 
different 10Gbps Intel based ethernet controller.


Over the last 45 days our small team has begun rolling out 7.3 stable in 
both our lab and across a few of our customers' infrastructures and have 
yet to experience these issues but our upgrading efforts are still young.


Thank you much.


Boyd Stephens
I85Cyber.org

Re: High Interrupt After 7.3 Upgrade

[Samuel Jayden]

Hi Boyd,

I noted the uptime values when I received notifications like "Internet is
slow", "Intranet is too slow" from users. In all of them, the load average
was 5 and above.
This is what I mean with the firewall just slowed down.

Also there were more error messages like these:
pmap_unwire: wiring for pmap 0xfd8e8c2f8bd8 va 0xc000a44000 didn't
change!
In Fact I live with pmap_unwire messages but that time those messages were
increased so much.

That's nearly all I have at the moment. Thanks.




On Thu, Jun 1, 2023 at 8:39 PM Boyd Stephens 
wrote:

> On 5/2/23 13:24, Samuel Jayden wrote:
> >
> > My firewall just slowed down after upgrading from 7.2 to 7.3.
>
> Hello Samuel,
>
> When you mention that your "firewall just slowed down" specifically what
> metric and/or anecdotal data are/were you using to determine this
> particular status of its operation(s)?
>
> Secondly what type of network load is present when you experience these
> issues?
>
> 
> Boyd Stephens
> I85Cyber.org
>

Re: High Interrupt After 7.3 Upgrade

[Boyd Stephens]

On 5/2/23 13:24, Samuel Jayden wrote:


My firewall just slowed down after upgrading from 7.2 to 7.3.


Hello Samuel,

When you mention that your "firewall just slowed down" specifically what 
metric and/or anecdotal data are/were you using to determine this 
particular status of its operation(s)?


Secondly what type of network load is present when you experience these 
issues?



Boyd Stephens
I85Cyber.org

Re: High Interrupt After 7.3 Upgrade

[Stuart Henderson]

On 2023-05-31, Sven F.  wrote:
> On Wed, May 31, 2023 at 5:27 PM Stuart Henderson 
> wrote:
>
>> On 2023-05-31, Mark (obsd)  wrote:
>> >>
>> > I'm not the OP, but that's interesting to me because I'm wondering if it's 
>> > why Prometheus'
>> > node_exporter from packages is reporting wildly wrong CPU stats on 7.3 that
>> > don't at all match what you'd expect when comparing top/htop output? It 
>> > was fine prior
>> > to upgrading to 7.3, but I've just left digging into it on the back burner 
>> > due to other
>> > priorities.
>>
>> That's a different issue, it was fixed in -current - I've just merged it to
>> -stable so updated packages should show up in a day or two.
>>
>> 7.3 interrupt ( Intel(R) Celeron(R) J6412 )
>
> v6-fw# vmstat -i
> interrupt   total rate

The node_exporter issue is not related to the number of interrupts, it is 
because
programs written in go keep using static copies of information converted from OS
C headers, and then hardly ever get round to updating them when things change
in the OS.

Re: High Interrupt After 7.3 Upgrade

[Valdrin MUJA]

Hi,

I hit the same case too.
It looks like there's something wrong with the ipi:
I have a system where I am running the current OpenBSD kernel dated May 21.
The systat output and the vmstat -i output do not match, and there are serious 
differences between them.
For example, while the ip in vmstat -i output is below 5000, the ip in systat 
output can go above 65000.

I don't know if it's a coincidence, but I received complaints from users on a 
firewall I upgraded to 7.3 and then I've downgraded the system when I saw the 
systat values. Maybe the notifications from the user were not correct and I was 
in a hurry. It can be both; I am not sure.

On the other hand, when the ix(4) tso code is fully committed(*), I wanna make 
detailed tests with Cisco Trex and share it.

(*) I think the ix(4) tso code is partially committed, but I guess it's not 
completely finished yet, right?

From: owner-m...@openbsd.org  on behalf of Sven F. 

Sent: Thursday, June 1, 2023 00:35
To: misc@openbsd.org 
Subject: Re: High Interrupt After 7.3 Upgrade

On Wed, May 31, 2023 at 5:27 PM Stuart Henderson 
wrote:

> On 2023-05-31, Mark (obsd)  wrote:
> > Hi Chris,
> >
> > On Tue, May 30, 2023 at 8:59 AM Chris Cappuccio 
> wrote:
> >
> >> Samuel Jayden [samueljaydan1...@gmail.com] wrote:
> >> > Hi again,
> >> >
> >> > Just for the record:
> >> > I've downgraded to OpenBSD 7.2 (reinstalled) and everything is working
> >> like
> >> > a charm again.
> >> > I don't know what is wrong with 7.3 but ipi interrupt rate is too much
> >> and
> >> > somehow OpenBSD performance is too bad..
> >> > Thanks for reading.
> >> >
> >>
> >> Sounds like you are using 'systat' to measure interrupts. This is a bug
> >> in systat was was fixed in 7.3. Here is Scott Cheloha's message from
> that
> >> fix:
> >>
> >> "systat(1): vmstat: measure elapsed time with clock_gettime(2) instead
> of
> >> ticks
> >>
> >> The vmstat view in systat(1) should not use statclock() ticks to count
> >> elapsed time.  First, ticks are low resolution.  Second, the statclock
> >> is sometimes randomized, so each tick is not necessarily of equal
> >> length.  Third, we're counting ticks from every CPU on the system, so
> >> every rate in the view is divided by the number of CPUs.  For example,
> >> on an amd64 system with 8 CPUs you currently see:
> >>
> >>  200 clock
> >>
> >> ... when the true clock interrupt rate on that system is 1600.
> >>
> >> Instead, measure elapsed time with clock_gettime(2).  Use CLOCK_UPTIME
> >> here so we exclude time when the system is suspended.  With this
> >> change we no longer need "stathz" or "hertz".  We can also get rid of
> >> the anachronistic secondary clock failure test.
> >>
> >>
> >>
> > I'm not the OP, but that's interesting to me because I'm wondering if
> it's
> > why Prometheus'
> > node_exporter from packages is reporting wildly wrong CPU stats on 7.3
> that
> > don't at all
> > match what you'd expect when comparing top/htop output? It was fine prior
> > to upgrading
> > to 7.3, but I've just left digging into it on the back burner due to
> other
> > priorities.
>
> That's a different issue, it was fixed in -current - I've just merged it to
> -stable so updated packages should show up in a day or two.
>
>
> 7.3 interrupt ( Intel(R) Celeron(R) J6412 )

v6-fw# vmstat -i
interrupt   total rate
irq96/acpi0 10
irq145/inteldrm0  4970
irq97/xhci0 30
irq98/ahci0   18738060
irq114/igc0:0   157799531   50
irq115/igc0:1   194120194   61
irq116/igc0:2   148272908   47
irq117/igc0:3   159077128   50
irq118/igc0 20
irq119/igc1:0   158925348   50
irq120/igc1:1   181916246   58
irq121/igc1:2   155586734   49
irq122/igc1:3   170737329   54
irq123/igc1 20
irq129/igc3:021260
irq130/igc3:1   540117832  172
irq131/igc3:2  5688860
irq132/igc3:3   909270099  290
irq133/igc3130
irq0/clock 2505321992  799
irq0/ipi   5601964631 1788
Total 1088308 3475

I did not notice performance issue here,
but maybe irq0/ipi   5601964631 1788
is bad
i did noticed some unexpected kernel_lock jittering the traffic ~15ms

--
--
-
Knowing is not enough; we must apply. Willing is not enough; we must do

Re: High Interrupt After 7.3 Upgrade

[Sven F.]

On Wed, May 31, 2023 at 5:27 PM Stuart Henderson 
wrote:

> On 2023-05-31, Mark (obsd)  wrote:
> > Hi Chris,
> >
> > On Tue, May 30, 2023 at 8:59 AM Chris Cappuccio 
> wrote:
> >
> >> Samuel Jayden [samueljaydan1...@gmail.com] wrote:
> >> > Hi again,
> >> >
> >> > Just for the record:
> >> > I've downgraded to OpenBSD 7.2 (reinstalled) and everything is working
> >> like
> >> > a charm again.
> >> > I don't know what is wrong with 7.3 but ipi interrupt rate is too much
> >> and
> >> > somehow OpenBSD performance is too bad..
> >> > Thanks for reading.
> >> >
> >>
> >> Sounds like you are using 'systat' to measure interrupts. This is a bug
> >> in systat was was fixed in 7.3. Here is Scott Cheloha's message from
> that
> >> fix:
> >>
> >> "systat(1): vmstat: measure elapsed time with clock_gettime(2) instead
> of
> >> ticks
> >>
> >> The vmstat view in systat(1) should not use statclock() ticks to count
> >> elapsed time.  First, ticks are low resolution.  Second, the statclock
> >> is sometimes randomized, so each tick is not necessarily of equal
> >> length.  Third, we're counting ticks from every CPU on the system, so
> >> every rate in the view is divided by the number of CPUs.  For example,
> >> on an amd64 system with 8 CPUs you currently see:
> >>
> >>  200 clock
> >>
> >> ... when the true clock interrupt rate on that system is 1600.
> >>
> >> Instead, measure elapsed time with clock_gettime(2).  Use CLOCK_UPTIME
> >> here so we exclude time when the system is suspended.  With this
> >> change we no longer need "stathz" or "hertz".  We can also get rid of
> >> the anachronistic secondary clock failure test.
> >>
> >>
> >>
> > I'm not the OP, but that's interesting to me because I'm wondering if
> it's
> > why Prometheus'
> > node_exporter from packages is reporting wildly wrong CPU stats on 7.3
> that
> > don't at all
> > match what you'd expect when comparing top/htop output? It was fine prior
> > to upgrading
> > to 7.3, but I've just left digging into it on the back burner due to
> other
> > priorities.
>
> That's a different issue, it was fixed in -current - I've just merged it to
> -stable so updated packages should show up in a day or two.
>
>
> 7.3 interrupt ( Intel(R) Celeron(R) J6412 )

v6-fw# vmstat -i
interrupt   total rate
irq96/acpi0 10
irq145/inteldrm0  4970
irq97/xhci0 30
irq98/ahci0   18738060
irq114/igc0:0   157799531   50
irq115/igc0:1   194120194   61
irq116/igc0:2   148272908   47
irq117/igc0:3   159077128   50
irq118/igc0 20
irq119/igc1:0   158925348   50
irq120/igc1:1   181916246   58
irq121/igc1:2   155586734   49
irq122/igc1:3   170737329   54
irq123/igc1 20
irq129/igc3:021260
irq130/igc3:1   540117832  172
irq131/igc3:2  5688860
irq132/igc3:3   909270099  290
irq133/igc3130
irq0/clock 2505321992  799
irq0/ipi   5601964631 1788
Total 1088308 3475

I did not notice performance issue here,
but maybe irq0/ipi   5601964631 1788
is bad
i did noticed some unexpected kernel_lock jittering the traffic ~15ms

-- 
--
-
Knowing is not enough; we must apply. Willing is not enough; we must do

Re: High Interrupt After 7.3 Upgrade

[Stuart Henderson]

On 2023-05-31, Mark (obsd)  wrote:
> Hi Chris,
>
> On Tue, May 30, 2023 at 8:59 AM Chris Cappuccio  wrote:
>
>> Samuel Jayden [samueljaydan1...@gmail.com] wrote:
>> > Hi again,
>> >
>> > Just for the record:
>> > I've downgraded to OpenBSD 7.2 (reinstalled) and everything is working
>> like
>> > a charm again.
>> > I don't know what is wrong with 7.3 but ipi interrupt rate is too much
>> and
>> > somehow OpenBSD performance is too bad..
>> > Thanks for reading.
>> >
>>
>> Sounds like you are using 'systat' to measure interrupts. This is a bug
>> in systat was was fixed in 7.3. Here is Scott Cheloha's message from that
>> fix:
>>
>> "systat(1): vmstat: measure elapsed time with clock_gettime(2) instead of
>> ticks
>>
>> The vmstat view in systat(1) should not use statclock() ticks to count
>> elapsed time.  First, ticks are low resolution.  Second, the statclock
>> is sometimes randomized, so each tick is not necessarily of equal
>> length.  Third, we're counting ticks from every CPU on the system, so
>> every rate in the view is divided by the number of CPUs.  For example,
>> on an amd64 system with 8 CPUs you currently see:
>>
>>  200 clock
>>
>> ... when the true clock interrupt rate on that system is 1600.
>>
>> Instead, measure elapsed time with clock_gettime(2).  Use CLOCK_UPTIME
>> here so we exclude time when the system is suspended.  With this
>> change we no longer need "stathz" or "hertz".  We can also get rid of
>> the anachronistic secondary clock failure test.
>>
>>
>>
> I'm not the OP, but that's interesting to me because I'm wondering if it's
> why Prometheus'
> node_exporter from packages is reporting wildly wrong CPU stats on 7.3 that
> don't at all
> match what you'd expect when comparing top/htop output? It was fine prior
> to upgrading
> to 7.3, but I've just left digging into it on the back burner due to other
> priorities.

That's a different issue, it was fixed in -current - I've just merged it to
-stable so updated packages should show up in a day or two.

Re: High Interrupt After 7.3 Upgrade

[Mark (obsd)]

Hi Chris,

On Tue, May 30, 2023 at 8:59 AM Chris Cappuccio  wrote:

> Samuel Jayden [samueljaydan1...@gmail.com] wrote:
> > Hi again,
> >
> > Just for the record:
> > I've downgraded to OpenBSD 7.2 (reinstalled) and everything is working
> like
> > a charm again.
> > I don't know what is wrong with 7.3 but ipi interrupt rate is too much
> and
> > somehow OpenBSD performance is too bad..
> > Thanks for reading.
> >
>
> Sounds like you are using 'systat' to measure interrupts. This is a bug
> in systat was was fixed in 7.3. Here is Scott Cheloha's message from that
> fix:
>
> "systat(1): vmstat: measure elapsed time with clock_gettime(2) instead of
> ticks
>
> The vmstat view in systat(1) should not use statclock() ticks to count
> elapsed time.  First, ticks are low resolution.  Second, the statclock
> is sometimes randomized, so each tick is not necessarily of equal
> length.  Third, we're counting ticks from every CPU on the system, so
> every rate in the view is divided by the number of CPUs.  For example,
> on an amd64 system with 8 CPUs you currently see:
>
>  200 clock
>
> ... when the true clock interrupt rate on that system is 1600.
>
> Instead, measure elapsed time with clock_gettime(2).  Use CLOCK_UPTIME
> here so we exclude time when the system is suspended.  With this
> change we no longer need "stathz" or "hertz".  We can also get rid of
> the anachronistic secondary clock failure test.
>
>
>
I'm not the OP, but that's interesting to me because I'm wondering if it's
why Prometheus'
node_exporter from packages is reporting wildly wrong CPU stats on 7.3 that
don't at all
match what you'd expect when comparing top/htop output? It was fine prior
to upgrading
to 7.3, but I've just left digging into it on the back burner due to other
priorities.

Thanks!
Mark

Re: High Interrupt After 7.3 Upgrade

[Chris Cappuccio]

Samuel Jayden [samueljaydan1...@gmail.com] wrote:
> Hi again,
> 
> Just for the record:
> I've downgraded to OpenBSD 7.2 (reinstalled) and everything is working like
> a charm again.
> I don't know what is wrong with 7.3 but ipi interrupt rate is too much and
> somehow OpenBSD performance is too bad..
> Thanks for reading.
> 

Sounds like you are using 'systat' to measure interrupts. This is a bug
in systat was was fixed in 7.3. Here is Scott Cheloha's message from that fix:

"systat(1): vmstat: measure elapsed time with clock_gettime(2) instead of ticks

The vmstat view in systat(1) should not use statclock() ticks to count
elapsed time.  First, ticks are low resolution.  Second, the statclock
is sometimes randomized, so each tick is not necessarily of equal
length.  Third, we're counting ticks from every CPU on the system, so
every rate in the view is divided by the number of CPUs.  For example,
on an amd64 system with 8 CPUs you currently see:

 200 clock

... when the true clock interrupt rate on that system is 1600.

Instead, measure elapsed time with clock_gettime(2).  Use CLOCK_UPTIME
here so we exclude time when the system is suspended.  With this
change we no longer need "stathz" or "hertz".  We can also get rid of
the anachronistic secondary clock failure test.

Prompted by dlg@ and jmatthew@.  deraadt@ says this has been in snaps
since 2022-11-21; no complaints.

Link: https://marc.info/?l=openbsd-tech=166898960831136=2

ok dlg@ deraadt@"

Re: High Interrupt After 7.3 Upgrade

[Samuel Jayden]

Hi again,

Just for the record:
I've downgraded to OpenBSD 7.2 (reinstalled) and everything is working like
a charm again.
I don't know what is wrong with 7.3 but ipi interrupt rate is too much and
somehow OpenBSD performance is too bad..
Thanks for reading.


On Tue, May 2, 2023 at 9:24 PM Samuel Jayden 
wrote:

> Hello misc,
>
>
> My firewall just slowed down after upgrading from 7.2 to 7.3.
>
> When I look at some values on the system I’ve realized there are high
> interrupts on it.
>
>
> Total Interrupts are over 40.000
>
> em1 is over 4000
>
> em2 is over 3000
>
> Clock is nearly 2000
>
> ipi over 30.000
>
>
> But there are no Ierrs, Oerrs or Colls on those interfaces.
>
>
> You can find some hardware information and my question is where to start
> for debugging?
>
> Why IPI is so heavy?
>
> Can it be related via this notice from 73.html
>
> ‘’Added support for per-CPU counters to evcount(9)
> . Useful for counting events that are
> prone to occur simultaneously across multiple CPUs, like clock interrupts
> and IPIs.’’
>
> Thanks.
>
>
>
> # sysctl hw.model
>
> hw.model=Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz
>
> # sysctl hw.ncpuonline
>
>
>
> hw.ncpuonline=8
>
>
> # uptime
>
>
>
>  9:08PM  up 13 mins, 1 user, load averages: 2.06, 1.98, 1.38
>
>
> OpenBSD 7.3 (GENERIC.MP) #1125: Sat Mar 25 10:36:29 MDT 2023
>
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> real mem = 34237882368 (32651MB)
>
> avail mem = 33180819456 (31643MB)
>
> random: good seed from bootblocks
>
> mpath0 at root
>
> scsibus0 at mpath0: 256 targets
>
> mainbus0 at root
>
> bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7bd0 (101 entries)
>
> bios0: vendor American Megatrends Inc. version "AHBA" date 10/03/2018
>
> bios0: Lanner Inc. NCA4010D
>
> efi0 at bios0: UEFI 2.4
>
> efi0: American Megatrends rev 0x5000b
>
> acpi0 at bios0: ACPI 5.0
>
> acpi0: sleep states S0 S5
>
> acpi0: tables DSDT FACP APIC FPDT FIDT TCPA MCFG UEFI DBG2 HPET MSCT SLIT
> SRAT WDDT SSDT SSDT SSDT PRAD DMAR
>
> acpi0: wakeup devices XHCI(S0) EHC1(S0) EHC2(S0) RP01(S0) RP02(S0)
> RP03(S0) RP04(S0) RP05(S0) RP06(S0) RP07(S0) RP08(S0) BR1A(S0) BR1B(S0)
> BR2A(S0) BR2B(S0) BR2C(S0) [...]
>
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
>
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
>
> cpu0 at mainbus0: apid 0 (boot processor)
>
> cpu0: Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz, 1995.41 MHz, 06-56-03
>
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>
> cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
>
> cpu0: smt 0, core 0, package 0
>
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
>
> cpu0: apic clock running at 99MHz
>
> cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
>
> cpu1 at mainbus0: apid 2 (application processor)
>
> cpu1: Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz, 1995.45 MHz, 06-56-03
>
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>
> cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
>
> cpu1: smt 0, core 1, package 0
>
> cpu2 at mainbus0: apid 4 (application processor)
>
> cpu2: Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz, 1995.49 MHz, 06-56-03
>
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>
> cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
>
> cpu2: smt 0, core 2, package 0
>
> cpu3 at mainbus0: apid 6 (application processor)
>
> cpu3: Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz, 1995.58 MHz, 06-56-03
>
> cpu3:
> 

High Interrupt After 7.3 Upgrade

[Samuel Jayden]

Hello misc,


My firewall just slowed down after upgrading from 7.2 to 7.3.

When I look at some values on the system I’ve realized there are high
interrupts on it.


Total Interrupts are over 40.000

em1 is over 4000

em2 is over 3000

Clock is nearly 2000

ipi over 30.000


But there are no Ierrs, Oerrs or Colls on those interfaces.


You can find some hardware information and my question is where to start
for debugging?

Why IPI is so heavy?

Can it be related via this notice from 73.html

‘’Added support for per-CPU counters to evcount(9)
. Useful for counting events that are
prone to occur simultaneously across multiple CPUs, like clock interrupts
and IPIs.’’

Thanks.



# sysctl hw.model

hw.model=Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz

# sysctl hw.ncpuonline



hw.ncpuonline=8


# uptime



 9:08PM  up 13 mins, 1 user, load averages: 2.06, 1.98, 1.38


OpenBSD 7.3 (GENERIC.MP) #1125: Sat Mar 25 10:36:29 MDT 2023

dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

real mem = 34237882368 (32651MB)

avail mem = 33180819456 (31643MB)

random: good seed from bootblocks

mpath0 at root

scsibus0 at mpath0: 256 targets

mainbus0 at root

bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7bd0 (101 entries)

bios0: vendor American Megatrends Inc. version "AHBA" date 10/03/2018

bios0: Lanner Inc. NCA4010D

efi0 at bios0: UEFI 2.4

efi0: American Megatrends rev 0x5000b

acpi0 at bios0: ACPI 5.0

acpi0: sleep states S0 S5

acpi0: tables DSDT FACP APIC FPDT FIDT TCPA MCFG UEFI DBG2 HPET MSCT SLIT
SRAT WDDT SSDT SSDT SSDT PRAD DMAR

acpi0: wakeup devices XHCI(S0) EHC1(S0) EHC2(S0) RP01(S0) RP02(S0) RP03(S0)
RP04(S0) RP05(S0) RP06(S0) RP07(S0) RP08(S0) BR1A(S0) BR1B(S0) BR2A(S0)
BR2B(S0) BR2C(S0) [...]

acpitimer0 at acpi0: 3579545 Hz, 24 bits

acpimadt0 at acpi0 addr 0xfee0: PC-AT compat

cpu0 at mainbus0: apid 0 (boot processor)

cpu0: Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz, 1995.41 MHz, 06-56-03

cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN

cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache

cpu0: smt 0, core 0, package 0

mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges

cpu0: apic clock running at 99MHz

cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE

cpu1 at mainbus0: apid 2 (application processor)

cpu1: Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz, 1995.45 MHz, 06-56-03

cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN

cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache

cpu1: smt 0, core 1, package 0

cpu2 at mainbus0: apid 4 (application processor)

cpu2: Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz, 1995.49 MHz, 06-56-03

cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN

cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache

cpu2: smt 0, core 2, package 0

cpu3 at mainbus0: apid 6 (application processor)

cpu3: Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz, 1995.58 MHz, 06-56-03

cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN

cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 12MB 64b/line