Re: Please test: ix(4)+em(4) MSI-X towards multiqueues
On 17.3.2020. 13:16, Martin Pieuchot wrote:
> Diff below allows ix(4) and em(4) to establish two MSI-X handlers.
> Multiqueue support is intentionally not included in this diff.
>
Hi all,
if someone is willing to make em multiqueue i have
em0 at pci6 dev 0 function 0 "Intel 82576" rev 0x01, msix, 1 queue
em2 at pci7 dev 0 function 0 "Intel I210" rev 0x03, msix, 1 queue
em4 at pci11 dev 0 function 0 "Intel I350" rev 0x01, msix, 1 queue
for testing and i will look for 82574 and 82575
Thank you
> One handler is for the single queue (index ":0") and one for the
> link interrupt:
>
> # vmstat -i |grep ix0
> irq114/ix0:0 73178178 3758
> irq115/ix0 20
>
> A per-driver toggle allows to switch MSI-X support on/off. My plan is
> to commit with the switch "off" for the moment. Switching it to "on"
> should be better done in the beginning of a release cycle. However it
> is "on" in the diff below for testing purposes.
>
> The em(4) diff doesn't include support for 82575 nor 82574. If somebody
> has such hardware and is interested I can lift the necessary bits from
> FreeBSD. That said em_setup_queues_msix() contains references to 82575
> to help diffing with FreeBSD.
>
> The ix(4) diff cannot be committed right now as it breaks the build on
> architectures that define pci_intr_map_msix() to (-1). The compiler
> complains that the arguments of the macro are unused.
>
> /sys/dev/pci/if_ix.c:1789:21: error: unused variable 'dummy'
> [-Werror,-Wunused-variable]
> pci_intr_handle_tdummy;
>^
> /sys/dev/pci/if_ix.c:1788:26: error: unused variable 'pa'
> [-Werror,-Wunused-variable]
> struct pci_attach_args *pa = >os_pa;
>
> So I'd like to know if that's the way to test if MSI-X can be used:
>
>/*
> * Try a dummy map, maybe this bus doesn't like MSI, this function
> * has no side effects.
> */
>if (pci_intr_map_msix(pa, 0, ))
>return (0);
>
> I'd appreciate tests on many hardware as possible, especially for em(4).
> If you have been testing an earlier version of these diffs, please do
> it again :)
>
> Thanks,
> Martin
>
> Index: dev/pci/if_em.c
> ===
> RCS file: /cvs/src/sys/dev/pci/if_em.c,v
> retrieving revision 1.347
> diff -u -p -r1.347 if_em.c
> --- dev/pci/if_em.c 8 Mar 2020 11:43:43 - 1.347
> +++ dev/pci/if_em.c 17 Mar 2020 11:18:01 -
> @@ -233,6 +233,7 @@ void em_defer_attach(struct device*);
> int em_detach(struct device *, int);
> int em_activate(struct device *, int);
> int em_intr(void *);
> +int em_allocate_legacy(struct em_softc *);
> void em_start(struct ifqueue *);
> int em_ioctl(struct ifnet *, u_long, caddr_t);
> void em_watchdog(struct ifnet *);
> @@ -290,6 +291,13 @@ void em_flush_tx_ring(struct em_queue *)
> void em_flush_rx_ring(struct em_queue *);
> void em_flush_desc_rings(struct em_softc *);
>
> +/* MSIX/Multiqueue functions */
> +int em_allocate_msix(struct em_softc *);
> +int em_setup_queues_msix(struct em_softc *);
> +int em_queue_intr_msix(void *);
> +int em_link_intr_msix(void *);
> +void em_enable_queue_intr_msix(struct em_queue *);
> +
> /*
> * OpenBSD Device Interface Entry Points
> */
> @@ -304,6 +312,7 @@ struct cfdriver em_cd = {
> };
>
> static int em_smart_pwr_down = FALSE;
> +int em_enable_msix = 1;
>
> /*
> * Device identification routine
> @@ -919,6 +928,14 @@ em_init(void *arg)
> }
> em_initialize_receive_unit(sc);
>
> + if (sc->msix) {
> + if (em_setup_queues_msix(sc)) {
> + printf("%s: Can't setup msix queues\n", DEVNAME(sc));
> + splx(s);
> + return;
> + }
> + }
> +
> /* Program promiscuous mode and multicast filters. */
> em_iff(sc);
>
> @@ -1617,10 +1634,7 @@ int
> em_allocate_pci_resources(struct em_softc *sc)
> {
> int val, rid;
> - pci_intr_handle_t ih;
> - const char *intrstr = NULL;
> struct pci_attach_args *pa = >osdep.em_pa;
> - pci_chipset_tag_t pc = pa->pa_pc;
> struct em_queue*que = NULL;
>
> val = pci_conf_read(pa->pa_pc, pa->pa_tag, EM_MMBA);
> @@ -1691,6 +1705,9 @@ em_allocate_pci_resources(struct em_soft
> }
> }
>
> + sc->osdep.dev = (struct device *)sc;
> + sc->hw.back = >osdep;
> +
> /* Only one queue for the moment. */
> que = malloc(sizeof(struct em_queue), M_DEVBUF, M_NOWAIT | M_ZERO);
> if (que == NULL) {
> @@ -1703,29 +1720,10 @@ em_allocate_pci_resources(struct
Re: [External] : Re: pfsync mutex
On 28.2.2022. 10:49, Alexandr Nedvedicky wrote:
> Hello,
>
>
>>> i will still play with sasyncd setup, maybe something comes up
>>>
>>
>> And when I thought everything is fine.. after 2 days of hitting sasyncd
>> setup I've got this panic .. will stay in ddb if some other information
>> is needed
>>
>
> I think you are hitting a different issue, which could be hiding there
> for a while.
>
>
>>
>>
>> ddb{1}> trace
>> pfsync_sendout() at pfsync_sendout+0x5a5
>> pfsync_update_state(fd8425b77030) at pfsync_update_state+0x15b
>> pf_test(2,3,800c2048,800022c71dd8) at pf_test+0xd61
>> ip_output(fd8007117b00,0,800022c71f68,1,0,0,e436e0ec669ea3ac) at
>> ip_out
>> put+0x6b7
>> ip_forward(fd8007117b00,80099048,fd842d596cb8,0) at
>> ip_forward+
>> 0x2da
>> ip_input_if(800022c720a8,800022c720b4,4,0,80099048) at
>> ip_input
>> _if+0x353
>> ipv4_input(80099048,fd8007117b00) at ipv4_input+0x39
>> ether_input(80099048,fd8007117b00) at ether_input+0x3ad
>> if_input_process(80099048,800022c72198) at if_input_process+0x92
>> ifiq_process(80098800) at ifiq_process+0x69
>> taskq_thread(8002f200) at taskq_thread+0x100
>> end trace frame: 0x0, count: -11
>
> we die here in pfsync_sendout() at linet 1871 (given I can trust gdb):
>
> 1861 /* walk the queues */
> 1862 for (q = 0; q < PFSYNC_S_COUNT; q++) {
> 1863 if (TAILQ_EMPTY(_qs[q]))
> 1864 continue;
> 1865
> 1866 subh = (struct pfsync_subheader *)(m->m_data + offset);
> 1867 offset += sizeof(*subh);
> 1868
> 1869 count = 0;
> 1870 while ((st = TAILQ_FIRST(_qs[q])) != NULL) {
> 1871 TAILQ_REMOVE(_qs[q], st, sync_list);
> 1872 #ifdef PFSYNC_DEBUG
> 1873 KASSERT(st->sync_state == q);
> 1874 #endif
> 1875 st->sync_state = PFSYNC_S_NONE;
> 1876 pfsync_qs[q].write(st, m->m_data + offset);
> 1877 offset += pfsync_qs[q].len;
> 1878
> 1879 pf_state_unref(st);
> 1880 count++;
> 1881 }
> 1882
>
> I need to poke to disasm more closely to figure out, which queue (`q`)
> we are processing when we die, this might get us closer to culprit.
> those queues belong to pf(4), so I see tdb/ipsec to be innocent here.
>
> I'll try to take a look later today.
>
> thanks and
> regards
> sashan
>
do you need ddb console ? if you want i can try to reproduce this panic
without isakmpd, just hitting pfsync setup?
Re: pfsync mutex
On 26.2.2022. 21:36, Hrvoje Popovski wrote:
> On 25.2.2022. 23:22, Alexander Bluhm wrote:
>> On Fri, Feb 25, 2022 at 10:21:50PM +0100, Hrvoje Popovski wrote:
>>> On 24.2.2022. 22:42, Alexander Bluhm wrote:
>>>> Hi,
>>>>
>>>> Hrvoje reported some crashes with pfsync, IPsec and parallel
>>>> forwarding. Some locks were missing around the tdb flags in pfsync.
>>>
>>> before trying this diff i wanted to see if i could still trigger panic
>>> with sasyncd setup and parallel forwarding diff. and here's panic
>>>
>>> r620-2# uvm_fault(0x822c8f70, 0xe7, 0, 2) -> e
>>> kernel: page fault trap, code=0
>>> Stopped at ipsp_delete_acquire+0x61: movq%rax,0xe8(%rcx)
>>> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
>>> 486149 44310 68 0x110 03 sasyncd
>>> 330850 23209 0 0x14000 0x2004 softnet
>>> 48580 70158 0 0x14000 0x2001 softnet
>>>4459 11329 0 0x14000 0x2002 softnet
>>> 70953 79831 0 0x14000 0x2005 softnet
>>> *303117 65399 0 0x14000 0x42000 softclock
>>> ipsp_delete_acquire(fd837c504c30) at ipsp_delete_acquire+0x61
>>> ipsp_delete_acquire_timo(fd837c504c30) at ipsp_delete_acquire_timo+0x20
>>> softclock_thread(8000f500) at softclock_thread+0x13b
>>> end trace frame: 0x0, count: 12
>>> https://www.openbsd.org/ddb.html describes the minimum info required in bug
>>> reports. Insufficient info makes it difficult to find and fix bugs.
>>> ddb{0}>
>>>
>>>
>>> now i will try all the same with this diff ..
>>
>> I just commited it before I saw your mail. I hope the best.
>>
>
> I'm hitting committed diff for a sometime and by now boxes should panic,
> but they running just fine ..
>
> i will still play with sasyncd setup, maybe something comes up
>
And when I thought everything is fine.. after 2 days of hitting sasyncd
setup I've got this panic .. will stay in ddb if some other information
is needed
r620-2# uvm_fault(0xfufvfm_ffafulftf(f08xf2f3f9f1ff2fef802,3 91 2 e 0,
0 x 17 ,0 ,
2 )-> e
ke0rxne7e,l:p0ag,e 2 ) - > f a eu
t kterrnape,l: p caogde e = 0 l
Stopped at pfsync_sendout+0x5a5: movq%rax,0(%rcx)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
190670 3268 68 0x110 05 isakmpd
*304096 4574 0 0x14000 0x2001K softnet
129724 5880 0 0x14000 0x2004 softnet
73765 66549 0 0x14000 0x2003 softnet
358805 73199 0 0x14000 0x2002 softnet
388277 75259 0 0x14000 0x42000 softclock
pfsync_sendout() at pfsync_sendout+0x5a5
pfsync_update_state(fd8425b77030) at pfsync_update_state+0x15b
pf_test(2,3,800c2048,800022c71dd8) at pf_test+0xd61
ip_output(fd8007117b00,0,800022c71f68,1,0,0,e436e0ec669ea3ac) at
ip_output+0x6b7
ip_forward(fd8007117b00,80099048,fd842d596cb8,0) at
ip_forward+0x2da
ip_input_if(800022c720a8,800022c720b4,4,0,80099048) at
ip_input_if+0x353
ipv4_input(80099048,fd8007117b00) at ipv4_input+0x39
ether_input(80099048,fd8007117b00) at ether_input+0x3ad
if_input_process(80099048,800022c72198) at if_input_process+0x92
ifiq_process(80098800) at ifiq_process+0x69
taskq_thread(8002f200) at taskq_thread+0x100
end trace frame: 0x0, count: 4
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{1}>
ddb{1}> show panic
*cpu0: uvm_fault(0x823912e0, 0xe7, 0, 2) -> e
cpu1: uvm_fault(0x823912e0, 0x17, 0, 2) -> e
ddb{1}> trace
pfsync_sendout() at pfsync_sendout+0x5a5
pfsync_update_state(fd8425b77030) at pfsync_update_state+0x15b
pf_test(2,3,800c2048,800022c71dd8) at pf_test+0xd61
ip_output(fd8007117b00,0,800022c71f68,1,0,0,e436e0ec669ea3ac) at
ip_out
put+0x6b7
ip_forward(fd8007117b00,80099048,fd842d596cb8,0) at
ip_forward+
0x2da
ip_input_if(800022c720a8,800022c720b4,4,0,80099048) at
ip_input
_if+0x353
ipv4_input(80099048,fd8007117b00) at ipv4_input+0x39
ether_input(80099048,fd8007117b00) at ether_input+0x3ad
if_input_process(80099048,800022c72198) at if_input_process+0x92
ifiq_process(80098800) at ifiq_process+0x69
taskq_thread(8002f200) at taskq_thread+0x100
end trace frame: 0x0, count: -11
ddb{1}> show reg
rdi 0xfd80a3151c00
rsi 0xf
Re: pfsync mutex
On 25.2.2022. 23:22, Alexander Bluhm wrote:
> On Fri, Feb 25, 2022 at 10:21:50PM +0100, Hrvoje Popovski wrote:
>> On 24.2.2022. 22:42, Alexander Bluhm wrote:
>>> Hi,
>>>
>>> Hrvoje reported some crashes with pfsync, IPsec and parallel
>>> forwarding. Some locks were missing around the tdb flags in pfsync.
>>
>> before trying this diff i wanted to see if i could still trigger panic
>> with sasyncd setup and parallel forwarding diff. and here's panic
>>
>> r620-2# uvm_fault(0x822c8f70, 0xe7, 0, 2) -> e
>> kernel: page fault trap, code=0
>> Stopped at ipsp_delete_acquire+0x61: movq%rax,0xe8(%rcx)
>> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
>> 486149 44310 68 0x110 03 sasyncd
>> 330850 23209 0 0x14000 0x2004 softnet
>> 48580 70158 0 0x14000 0x2001 softnet
>>4459 11329 0 0x14000 0x2002 softnet
>> 70953 79831 0 0x14000 0x2005 softnet
>> *303117 65399 0 0x14000 0x42000 softclock
>> ipsp_delete_acquire(fd837c504c30) at ipsp_delete_acquire+0x61
>> ipsp_delete_acquire_timo(fd837c504c30) at ipsp_delete_acquire_timo+0x20
>> softclock_thread(8000f500) at softclock_thread+0x13b
>> end trace frame: 0x0, count: 12
>> https://www.openbsd.org/ddb.html describes the minimum info required in bug
>> reports. Insufficient info makes it difficult to find and fix bugs.
>> ddb{0}>
>>
>>
>> now i will try all the same with this diff ..
>
> I just commited it before I saw your mail. I hope the best.
>
I'm hitting committed diff for a sometime and by now boxes should panic,
but they running just fine ..
i will still play with sasyncd setup, maybe something comes up
Re: pfsync mutex
On 24.2.2022. 22:42, Alexander Bluhm wrote:
> Hi,
>
> Hrvoje reported some crashes with pfsync, IPsec and parallel
> forwarding. Some locks were missing around the tdb flags in pfsync.
before trying this diff i wanted to see if i could still trigger panic
with sasyncd setup and parallel forwarding diff. and here's panic
r620-2# uvm_fault(0x822c8f70, 0xe7, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at ipsp_delete_acquire+0x61: movq%rax,0xe8(%rcx)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
486149 44310 68 0x110 03 sasyncd
330850 23209 0 0x14000 0x2004 softnet
48580 70158 0 0x14000 0x2001 softnet
4459 11329 0 0x14000 0x2002 softnet
70953 79831 0 0x14000 0x2005 softnet
*303117 65399 0 0x14000 0x42000 softclock
ipsp_delete_acquire(fd837c504c30) at ipsp_delete_acquire+0x61
ipsp_delete_acquire_timo(fd837c504c30) at ipsp_delete_acquire_timo+0x20
softclock_thread(8000f500) at softclock_thread+0x13b
end trace frame: 0x0, count: 12
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
now i will try all the same with this diff ..
Re: [v2] amd64: simplify TSC sync testing
On 23.2.2022. 3:58, Scott Cheloha wrote: > Please test! In particular: > > - I'd love retests on systems that failed the test using the previous > patch. Almost all of these were AMD Ryzen CPUs. It's hard to say > what the issue is there. My vague guess is a firmware bug. > > One would hope that AMD's QA would catch an issue with the #RESET > signal, which is supposed to start all TSCs on all CPUs from zero > simultaneously. I am unsure how you would diagnose that it was, > in fact, a firmware bug though. > > - Multisocket systems > > - Multiprocessor VMs > > Please include your dmesg. Hi, Lenovo Thinkpad e14gen2 tsc: cpu0/cpu1: sync test round 1/2 failed tsc: cpu0/cpu1: cpu1: 1 lags 23 cycles tsc: cpu0/cpu2: sync test round 1/2 failed tsc: cpu0/cpu2: cpu2: 1 lags 24 cycles tsc: cpu0/cpu3: sync test round 1/2 failed tsc: cpu0/cpu3: cpu3: 19985 lags 4702 cycles tsc: cpu0/cpu4: sync test round 1/2 failed tsc: cpu0/cpu4: cpu4: 24670 lags 4655 cycles tsc: cpu0/cpu5: sync test round 1/2 failed tsc: cpu0/cpu5: cpu5: 47653 lags 4252 cycles OpenBSD 7.1-beta (GENERIC.MP) #15: Wed Feb 23 17:29:23 CET 2022 hrv...@e14gen2.srce.hr:/sys/arch/amd64/compile/GENERIC.MP real mem = 7713394688 (7356MB) avail mem = 7462412288 (7116MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.2 @ 0xbf913000 (63 entries) bios0: vendor LENOVO version "R1AET41W (1.17 )" date 11/11/2021 bios0: LENOVO 20T6000TSC acpi0 at bios0: ACPI 6.3 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP SSDT SSDT IVRS SSDT SSDT TPM2 SSDT MSDM BATB HPET APIC MCFG SBST WSMT VFCT SSDT CRAT CDIT FPDT SSDT SSDT SSDT BGRT UEFI SSDT SSDT acpi0: wakeup devices GPP3(S3) GPP4(S4) GPP5(S3) XHC0(S3) XHC1(S3) GP19(S3) LID_(S4) SLPB(S3) acpitimer0 at acpi0: 3579545 Hz, 32 bits acpihpet0 at acpi0: 14318180 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 5 4500U with Radeon Graphics, 2370.91 MHz, 17-60-01 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu0: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu0: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=1.1, IBE cpu1 at mainbus0: apid 1 (application processor) tsc: cpu0/cpu1: sync test round 1/2 failed tsc: cpu0/cpu1: cpu1: 1 lags 23 cycles cpu1: AMD Ryzen 5 4500U with Radeon Graphics, 2370.55 MHz, 17-60-01 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu1: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu1: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 2 (application processor) tsc: cpu0/cpu2: sync test round 1/2 failed tsc: cpu0/cpu2: cpu2: 1 lags 24 cycles cpu2: AMD Ryzen 5 4500U with Radeon Graphics, 2370.55 MHz, 17-60-01 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu2: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu2: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 4 (application processor) tsc: cpu0/cpu3: sync test round 1/2 failed tsc: cpu0/cpu3: cpu3: 19985 lags 4702 cycles cpu3: AMD Ryzen 5 4500U with
Re: parallel ip forwarding
On 25.12.2021. 18:52, Alexander Bluhm wrote:
> On Sat, Dec 25, 2021 at 09:24:07AM +0100, Hrvoje Popovski wrote:
>> On 24.12.2021. 0:55, Alexander Bluhm wrote:
>>> I think we can remove the ipsec_in_use workaround now. The IPsec
>>> path is protected with the kernel lock.
>>>
>>> There are some issues left:
>>> - npppd l2pt ipsecflowinfo is not MP safe
>>> - the acquire SA feature is not MP safe
>>> - Hrvoje has seen a panic with sasync
>>>
>>> If you use one of these, the diff below should trigger crashes faster.
>>> If you use only regular IPsec or forwarding, I hope it is stable.
>> Hi,
>>
>> after hitting sasyncd setup with this diff for some time i've run
>> ipsecctl -sa just to see what's going on and box panic
> According to ddb output ipsecctl is running in userland and the
> panic is triggered by a kernel timeout. Could it be coincidence?
> When I run with 4 softnet threads, I see userland starvation. Maybe
> both timeout and ipsecctl wait for the exclusive netlock and are
> executed shortly after each other.
>
>> r620-1# ipsecctl -sa
>> uvm_fault(0x82200c18, 0x417, 0, 2) -> e
>> kernel: page fault trap, code=0
>> Stopped at pfsync_delete_tdb+0x84: movq%rcx,0x8(%rsi)
>> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
>> 290490 40316 0 0x3 03 ipsecctl
>> 10869 22801 680x10 05 sasyncd
>> 504041 13202 680x10 0x801 isakmpd
>> 476980 6400 00x10 02 ntpd
>> 224100 72648 0 0x14000 0x2004 reaper
>> * 75659 10211 0 0x14000 0x42000K softclock
>> pfsync_delete_tdb(812e8008) at pfsync_delete_tdb+0x84
>> tdb_free(812e8008) at tdb_free+0x67
>> tdb_timeout(812e8008) at tdb_timeout+0x7e
>> softclock_thread(8000f260) at softclock_thread+0x13b
>> end trace frame: 0x0, count: 11
>> https://www.openbsd.org/ddb.html describes the minimum info required in
>> bug reports. Insufficient info makes it difficult to find and fix bugs.
>> ddb{0}>
> /home/bluhm/openbsd/cvs/src/sys/net/if_pfsync.c:2519
> 5f30: 49 8b 86 10 04 00 00mov0x410(%r14),%rax
> 5f37: 49 8b 8e 18 04 00 00mov0x418(%r14),%rcx
> 5f3e: 49 8d 94 24 e8 09 00lea0x9e8(%r12),%rdx
> 5f45: 00
> 5f46: 48 8d b0 10 04 00 00lea0x410(%rax),%rsi
> 5f4d: 48 85 c0test %rax,%rax
> * 5f50: 48 0f 44 f2 cmove %rdx,%rsi
> 5f54: 48 89 4e 08 mov%rcx,0x8(%rsi)
> 5f58: 49 8b 86 10 04 00 00mov0x410(%r14),%rax
> 5f5f: 49 8b 8e 18 04 00 00mov0x418(%r14),%rcx
> 5f66: 48 89 01mov%rax,(%rcx)
> 5f69: 49 c7 86 18 04 00 00movq $0x,0x418(%r14)
> 5f70: ff ff ff ff
> 5f74: 49 c7 86 10 04 00 00movq $0x,0x410(%r14)
> 5f7b: ff ff ff ff
> /home/bluhm/openbsd/cvs/src/sys/net/if_pfsync.c:2520
>
> 2508 void
> 2509 pfsync_delete_tdb(struct tdb *t)
> 2510 {
> 2511 struct pfsync_softc *sc = pfsyncif;
> 2512 size_t nlen;
> 2513
> 2514 if (sc == NULL || !ISSET(t->tdb_flags, TDBF_PFSYNC))
> 2515 return;
> 2516
> 2517 mtx_enter(>sc_tdb_mtx);
> 2518
> * 2519 TAILQ_REMOVE(>sc_tdb_q, t, tdb_sync_entry);
> 2520 CLR(t->tdb_flags, TDBF_PFSYNC);
> 2521
> 2522 nlen = sizeof(struct pfsync_tdb);
> 2523 if (TAILQ_EMPTY(>sc_tdb_q))
> 2524 nlen += sizeof(struct pfsync_subheader);
> 2525 atomic_sub_long(>sc_len, nlen);
> 2526
> 2527 mtx_leave(>sc_tdb_mtx);
> 2528 }
>
> Most sc_tdb_q are protected by sc_tdb_mtx. But pfsync_drop_snapshot()
> and pfsync_sendout() do not have it.
>
>> rsi0x40f
>> rcx 0x
>> rax 0x
> tqe_next and tqe_prev are -1. Looks like a double remove. Can
> pfsync_delete_tdb() be called twice? The tdb_refcnt should enforce
> that tdb_free() is only called once per TDB.
>
>> flags: d1040
> Flags look reasonable. But a problem could be that they are not
> protected in if_pfsync. Then set or cleared bits may be lost.
>
> Fix both missing mutexes . Also put the tdb_sync_entry
> modification and TDBF_PFSYNC flag in the same sc_tdb_mtx.
>
> I still have no pf sync setup. Hrvoje, can you try this?
Hi,
i think that this panic is coincidence, but i can't prove it because
it's not so easy to reproduce. I'm hitting sasync setup with this diff,
for 2 days and i can panic boxes...
If i try to reproduce panic without this diff, would it give you some
more information or hints regarding ipsec and mp forwarding?
Re: parallel ip forwarding
On 24.12.2021. 0:55, Alexander Bluhm wrote:
> I think we can remove the ipsec_in_use workaround now. The IPsec
> path is protected with the kernel lock.
>
> There are some issues left:
> - npppd l2pt ipsecflowinfo is not MP safe
> - the acquire SA feature is not MP safe
> - Hrvoje has seen a panic with sasync
>
> If you use one of these, the diff below should trigger crashes faster.
> If you use only regular IPsec or forwarding, I hope it is stable.
Hi,
after hitting sasyncd setup with this diff for some time i've run
ipsecctl -sa just to see what's going on and box panic
r620-1# ipsecctl -sa
uvm_fault(0x82200c18, 0x417, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at pfsync_delete_tdb+0x84: movq%rcx,0x8(%rsi)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
290490 40316 0 0x3 03 ipsecctl
10869 22801 680x10 05 sasyncd
504041 13202 680x10 0x801 isakmpd
476980 6400 00x10 02 ntpd
224100 72648 0 0x14000 0x2004 reaper
* 75659 10211 0 0x14000 0x42000K softclock
pfsync_delete_tdb(812e8008) at pfsync_delete_tdb+0x84
tdb_free(812e8008) at tdb_free+0x67
tdb_timeout(812e8008) at tdb_timeout+0x7e
softclock_thread(8000f260) at softclock_thread+0x13b
end trace frame: 0x0, count: 11
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
ddb{0}> show reg
rdi 0x4
rsi0x40f
rbp 0x800022c4e390
rbx0
rdx 0x806b39e8
rcx 0x
rax 0x
r8 0x1f
r9 0
r10 0xbaf844ce8eec335d
r11 0xb9858c0c287d2c4d
r12 0x806b3000
r13 0x821c5e60timeout_proc
r14 0x812e8008
r15 0x806b39f8
rip 0x8131a254pfsync_delete_tdb+0x84
cs 0x8
rflags 0x10286__ALIGN_SIZE+0xf286
rsp 0x800022c4e360
ss 0x10
pfsync_delete_tdb+0x84: movq%rcx,0x8(%rsi)
ddb{0}>
ddb{0}> show all tdb
0x812e8008: f9f247f0 192.168.42.100->192.168.42.112:50 #0 000d1040
0x812e8428: 959c114b 192.168.42.112->192.168.42.100:50 #2 1002
0x812e8848: b7eb65bc 192.168.42.113->192.168.42.100:50 #2 1002
0x812e9ce8: 55495192 192.168.42.100->192.168.42.113:50 #3 000d1002
ddb{0}>
ddb{0}> show tdb /f 0x812e8008
tdb at 0x812e8008
hnext: 0x0
dnext: 0x0
snext: 0x0
inext: 0x0
onext: 0x0
xform: 0x0
refcnt: 0
encalgxform: 0x81f36090
authalgxform: 0x81f36380
compalgxform: 0x0
flags: d1040
seq: 8
exp_allocations: 0
soft_allocations: 0
cur_allocations: 0
exp_bytes: 0
soft_bytes: 0
cur_bytes: 1272048336570
exp_timeout: 1200
soft_timeout: 1080
established: 1640372736
first_use: 1640372754
soft_first_use: 0
exp_first_use: 0
last_used: 1640419926
last_marked: 0
cryptoid: 0
tdb_spi: f9f247f0
amxkeylen: 20
emxkeylen: 20
ivlen: 8
sproto: 50
wnd: 16
satype: 2
updates: 158
dst: 192.168.42.112
src: 192.168.42.100
amxkey: 0x0
emxkey: 0x0
rpl: 5256398086
ids: 0x812d8800
ids_swapped: 0
mtu: 0
mtutimeout: 0
udpencap_port: 0
tag: 0
tap: 0
rdomain: 0
rdomain_post: 0
ddb{0}>
PID TID PPIDUID S FLAGS WAIT COMMAND
40316 290490 5050 0 7 0x3ipsecctl
22801 10869 51239 68 70x10sasyncd
51239 39174 1 0 30x80 kqreadsasyncd
13202 504041 43160 68 70x90isakmpd
43160 53413 1 0 30x80 netio isakmpd
5050 12722 1 0 30x10008b sigsusp ksh
64387 345990 1 0 30x100098 poll cron
58819 219224 16427 95 30x100092 kqreadsmtpd
67207 340852 16427103 30x100092 kqreadsmtpd
97983 457473 16427 95 30x100092 kqreadsmtpd
6608 99059 16427 95 30x100092 kqreadsmtpd
91670 313820 16427 95 30x100092 kqreadsmtpd
7571 97218 16427 95 30x100092
Re: ipsec kernel lock
On 22.12.2021. 14:52, Alexander Bluhm wrote: > Hi, > > IPsec is not MP safe yet. To allow forwarding in parallel without > dirty hacks, it is better to protect IPsec input and output with > kernel lock. We do not loose much as crypto needs the kernel lock > anyway. From here we can refine the lock later. > > Note that there is no kernel lock in the SPD lockup path. I want > to keep that lock free to allow fast forwarding with non IPsec > traffic. > > There are still some races in special cases, but in general it works > with parallel IP input. > > ok? Hi, i'm trying to panic sasyncd setup with this and parallel forwarding diff and i just can't :)
Re: boot stuck with latest cvs checkout
On 22.12.2021. 17:20, Hrvoje Popovski wrote: > On 22.12.2021. 17:07, Hrvoje Popovski wrote: >> Hi all, >> >> i've sysupgrade box and reboot it and everything seems fine. then cvs >> checkout it, compile and then box stuck at boot >> >>>> OpenBSD/amd64 BOOT 3.53 >> boot> >> booting hd0a:/bsd: 10246939+2425860+258068+0+1130496 >> [97+573872+611580]=0xe8c924 >> entry point at 0xd0201000 >> >> please forget this .. i did something very very stupid ...
Re: boot stuck with latest cvs checkout
On 22.12.2021. 17:07, Hrvoje Popovski wrote: > Hi all, > > i've sysupgrade box and reboot it and everything seems fine. then cvs > checkout it, compile and then box stuck at boot > >>> OpenBSD/amd64 BOOT 3.53 > boot> > booting hd0a:/bsd: 10246939+2425860+258068+0+1130496 > [97+573872+611580]=0xe8c924 > entry point at 0xd0201000 > > r620-1# dmesg OpenBSD 7.0-current (GENERIC.MP) #188: Mon Dec 20 22:32:56 MST 2021 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17115840512 (16322MB) avail mem = 16581124096 (15812MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries) bios0: vendor Dell Inc. version "2.9.0" date 12/06/2019 bios0: Dell Inc. PowerEdge R620 acpi0 at bios0: ACPI 3.0 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WDAT SLIC ERST HEST BERT EINJ TCPA PC__ SRAT SSDT acpi0: wakeup devices PCI0(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 4 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.59 MHz, 06-3e-04 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 2, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 6 (application processor) cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 3, package 0 cpu2 at mainbus0: apid 8 (application processor) cpu2: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 4, package 0 cpu3 at mainbus0: apid 16 (application processor) cpu3: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 8, package 0 cpu4 at mainbus0: apid 18 (application processor) cpu4: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04 cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu4: 256KB 64b/line 8-way L2 cache cpu4: smt 0, core 9, package 0 cpu5 at mainbus0: apid 20 (application processor) cpu5: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04 cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu5: 256KB 64b/line 8-way L2 cache cpu5: smt 0, core 10, package 0 ioapic0 at mainbus0: apid 0 pa 0xfec0, version 20, 24 pins ioapic1 at mainbus0: apid 1 pa 0xfec3f000, version 20, 24 pins, remapped acpihpet0 at acpi0: 14318179 Hz acpimcfg0 at acpi0 acpimcfg0: addr 0xe000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0)
boot stuck with latest cvs checkout
Hi all, i've sysupgrade box and reboot it and everything seems fine. then cvs checkout it, compile and then box stuck at boot >> OpenBSD/amd64 BOOT 3.53 boot> booting hd0a:/bsd: 10246939+2425860+258068+0+1130496 [97+573872+611580]=0xe8c924 entry point at 0xd0201000
Re: ipsec ipo tdb mutex
On 12.12.2021. 16:37, Hrvoje Popovski wrote: > On 11.12.2021. 20:03, Alexander Bluhm wrote: >> On Sat, Dec 11, 2021 at 12:53:35AM +0100, Alexander Bluhm wrote: >>> To cache lookups, the policy ipo is linked to its SA tdb. There >>> is a list of SAs that belong to a policy. To make it MP safe we >>> need a mutex around these pointers. >>> >>> Hrvoje: Can you test this alone and together with the ip forwarding >>> diff. I protects the data structure where the latter crashed. >>> Wonder if this helps. >> updated diff, merged with -current > > > Hi, > > i'm hitting this and ip forwarding diff for 20 hours and boxes didn't panic. > Will try now with plain source .. > Same with clean source, i can't trigger panic ... tnx ..
Re: ipsec ipo tdb mutex
On 11.12.2021. 20:03, Alexander Bluhm wrote: > On Sat, Dec 11, 2021 at 12:53:35AM +0100, Alexander Bluhm wrote: >> To cache lookups, the policy ipo is linked to its SA tdb. There >> is a list of SAs that belong to a policy. To make it MP safe we >> need a mutex around these pointers. >> >> Hrvoje: Can you test this alone and together with the ip forwarding >> diff. I protects the data structure where the latter crashed. >> Wonder if this helps. > updated diff, merged with -current Hi, i'm hitting this and ip forwarding diff for 20 hours and boxes didn't panic. Will try now with plain source ..
Re: em(4) vlan tagging support for 82576
On 6.12.2021. 4:22, Yury Shefer wrote: > Hi all, > > I have quad-port Intel ET2 NIC based on 82576[1] controller. The manual > says that hardware VLAN tagging should be supported but ifconfig output > shows VLAN_MTU only in hwfeatures on OpenBSD 7.0. How do I check if 802.1Q > tagging is offloaded or not? And if it's not - does it matter at 1Gbps > speeds on 3 Ghz CPU? > > $ dmesg | grep em0 > em0 at pci11 dev 0 function 0 "Intel 82576" rev 0x01: msi, address > 90:e2:ba:84:64:14 > > $ ifconfig em0 hwfeatures > em0: flags=808843 mtu 1500 > hwfeatures=10 hardmtu 9216 > lladdr 90:e2:ba:84:64:14 > index 1 priority 0 llprio 3 > media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause) > status: active > inet 192.168.0.143 netmask 0xff00 broadcast 192.168.0.255 > > > > [1] > https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/gigabit-et-et2-ef-multi-port-server-adapters-brief.pdf > > > Thanks, > Yury. > Hi, man ifconfig and search for hwfeatures hwfeatures Display the interface hardware features: CSUM_IPv4 The device supports IPv4 checksum offload. CSUM_TCPv4 As above, for TCP in IPv4 datagrams. CSUM_UDPv4 As above, for UDP. VLAN_MTUThe device can handle full sized frames, plus the size of the vlan(4) tag. VLAN_HWTAGGING On transmit, the device can add the vlan(4) tag. CSUM_TCPv6 As CSUM_TCPv4, but supports IPv6 datagrams. CSUM_UDPv6 As above, for UDP. WOL The device supports Wake on LAN (WoL). hardmtu The maximum MTU supported. so i would say that your card can offload vlan header ..
Re: parallel ip forwarding
On 4.12.2021. 10:41, Hrvoje Popovski wrote:
> On 3.12.2021. 20:35, Alexander Bluhm wrote:
>> Hi,
>>
>> After implementing MP safe refcounting for IPsec TDB, I wonder how
>> stable my diff for forwarding on multiple CPU is.
>>
>> Note that IPsec still has the workaround to disable multiple queues.
>> We do not have a mutex that protects the TDB fields yet. We have
>> only fixed the memory management.
>>
>> My goal is to get real MP pressure on the lower part of the IP
>> network stack. Only this will show remaining bugs.
> Hi,
>
> with only plain forwarding it seems that everything works just fine, but
> with ipsec i'm getting this panic. i will leave ddb console active if
> something else will be needed
Now with WITNESS
r620-2# uvm_fault(0x8226ee78, 0x137, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at ipsp_spd_lookup+0xa2f: movq%rax,0(%rcx)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
258332 43705 0 0x14000 0x2004 softnet
*440260 99913 0 0x14000 0x2003K softnet
63340 24364 0 0x14000 0x2002 softnet
ipsp_spd_lookup(fd80a46ed700,2,14,2,0,0,dcfe3c78989b6cd1,fd80a46ed700)
at ipsp_spd_lookup+0xa2f
ip_output_ipsec_lookup(fd80a46ed700,14,0,800022c71248,0) at
ip_output_ipsec_lookup+0x4c
ip_output(fd80a46ed700,0,800022c71408,1,0,0,6db6075421f27eb8) at
ip_output+0x39d
ip_forward(fd80a46ed700,80087048,fd83b4374cb0,0) at
ip_forward+0x26a
ip_input_if(800022c71548,800022c71554,4,0,80087048) at
ip_input_if+0x353
ipv4_input(80087048,fd80a46ed700) at ipv4_input+0x39
ether_input(80087048,fd80a46ed700) at ether_input+0x3aa
if_input_process(80087048,800022c71638) at if_input_process+0x92
ifiq_process(80086e00) at ifiq_process+0x69
taskq_thread(80030200) at taskq_thread+0x9f
end trace frame: 0x0, count: 5
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{3}>
ddb{3}> show reg
rdi0
rsi 0x8145d018
rbp 0x800022c71150
rbx0
rdx 0x1
rcx0x137
rax 0x
r80xfd839a222d30
r90x81453904
r10 0x8142f240
r11 0xebc2ae563992db8e
r12 0x800022c71178
r130
r14 0x1
r150
rip 0x811f56cfipsp_spd_lookup+0xa2f
cs 0x8
rflags 0x10213__ALIGN_SIZE+0xf213
rsp 0x800022c71040
ss 0x10
ipsp_spd_lookup+0xa2f: movq%rax,0(%rcx)
ddb{3}>
ddb{3}> show locks
exclusive kernel_lock _lock r = 0 (0x822b3ce8)
#0 witness_lock+0x333
#1 kpageflttrap+0x172
#2 kerntrap+0x91
#3 alltraps_kern_meltdown+0x7b
#4 ipsp_spd_lookup+0xa2f
#5 ip_output_ipsec_lookup+0x4c
#6 ip_output+0x39d
#7 ip_forward+0x26a
#8 ip_input_if+0x353
#9 ipv4_input+0x39
#10 ether_input+0x3aa
#11 if_input_process+0x92
#12 ifiq_process+0x69
#13 taskq_thread+0x9f
#14 proc_trampoline+0x1c
shared rwlock netlock r = 0 (0x8217f598)
#0 witness_lock+0x333
#1 rw_enter+0x27f
#2 if_input_process+0x75
#3 ifiq_process+0x69
#4 taskq_thread+0x9f
#5 proc_trampoline+0x1c
shared rwlock softnet r = 0 (0x80030270)
#0 witness_lock+0x333
#1 taskq_thread+0x92
#2 proc_trampoline+0x1c
ddb{3}>
ddb{3}> show all locks
Process 43705 (softnet) thread 0x8000e540 (258332)
shared rwlock netlock r = 0 (0x8217f598)
#0 witness_lock+0x333
#1 rw_enter+0x27f
#2 if_input_process+0x75
#3 ifiq_process+0x69
#4 taskq_thread+0x9f
#5 proc_trampoline+0x1c
shared rwlock softnet r = 0 (0x80030370)
#0 witness_lock+0x333
#1 taskq_thread+0x92
#2 proc_trampoline+0x1c
Process 99913 (softnet) thread 0x8000e000 (440260)
exclusive kernel_lock _lock r = 0 (0x822b3ce8)
#0 witness_lock+0x333
#1 kpageflttrap+0x172
#2 kerntrap+0x91
#3 alltraps_kern_meltdown+0x7b
#4 ipsp_spd_lookup+0xa2f
#5 ip_output_ipsec_lookup+0x4c
#6 ip_output+0x39d
#7 ip_forward+0x26a
#8 ip_input_if+0x353
#9 ipv4_input+0x39
#10 ether_input+0x3aa
#11 if_input_process+0x92
#12 ifiq_process+0x69
#13 taskq_thread+0x9f
#14 proc_trampoline+0x1c
shared rwlock netlock r = 0 (0x8217f598)
#0 witness_lock+0x333
#1 rw_enter+0x27f
#2 if_input_process+0x75
#3 ifiq_process+0x69
#4 taskq_thread+0x9f
#5 proc_trampoline+0x1c
shared rwlock softnet r = 0 (0x80030270)
#0 witness_lock+0x333
#1 taskq_thread+0x92
#2 proc_trampoline+0x1c
Process 39040 (softnet) thread 0x8000f
Re: parallel ip forwarding
On 4.12.2021. 10:41, Hrvoje Popovski wrote: > Hi, > > with only plain forwarding it seems that everything works just fine, but > with ipsec i'm getting this panic. i will leave ddb console active if > something else will be needed regarding performance ... without diff i'm getting 1.1Mpps and with this diff i'm getting 2Mpps of plain forwarding ..
Re: parallel ip forwarding
On 3.12.2021. 20:35, Alexander Bluhm wrote:
> Hi,
>
> After implementing MP safe refcounting for IPsec TDB, I wonder how
> stable my diff for forwarding on multiple CPU is.
>
> Note that IPsec still has the workaround to disable multiple queues.
> We do not have a mutex that protects the TDB fields yet. We have
> only fixed the memory management.
>
> My goal is to get real MP pressure on the lower part of the IP
> network stack. Only this will show remaining bugs.
Hi,
with only plain forwarding it seems that everything works just fine, but
with ipsec i'm getting this panic. i will leave ddb console active if
something else will be needed
r620-2# uvm_fault(0x8229d4e0, 0x137, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at ipsp_spd_lookup+0xa2f: movq%rax,0(%rcx)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
419237 67407 0 0x14000 0x2000 softnet
*157694 94649 0 0x14000 0x2002K softnet
ipsp_spd_lookup(fd80a4139800,2,14,2,0,0,5b815d966b14b44b,fd80a4139800)
at ipsp_spd_lookup+0xa2f
ip_output_ipsec_lookup(fd80a4139800,14,0,800022c60228,0) at
ip_output_ipsec_lookup+0x4c
ip_output(fd80a4139800,0,800022c603e8,1,0,0,3ada3367ffb43fe1) at
ip_output+0x39d
ip_forward(fd80a4139800,80087048,fd8394511078,0) at
ip_forward+0x26a
ip_input_if(800022c60528,800022c60534,4,0,80087048) at
ip_input_if+0x353
ipv4_input(80087048,fd80a4139800) at ipv4_input+0x39
ether_input(80087048,fd80a4139800) at ether_input+0x3aa
if_input_process(80087048,800022c60618) at if_input_process+0x92
ifiq_process(80087458) at ifiq_process+0x69
taskq_thread(8002f080) at taskq_thread+0x81
end trace frame: 0x0, count: 5
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{2}>
ddb{2}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
84832 15199 94097 68 30x10 netlock isakmpd
94097 209788 1 0 30x80 netio isakmpd
15214 351483 1 0 30x100083 ttyin ksh
12905 119865 1 0 30x100098 poll cron
84388 106290 61660 95 30x100092 kqreadsmtpd
49054 464016 61660103 30x100092 kqreadsmtpd
24623 388600 61660 95 30x100092 kqreadsmtpd
25783 71877 61660 95 30x100092 kqreadsmtpd
44044 451515 61660 95 30x100092 kqreadsmtpd
74629 163500 61660 95 30x100092 kqreadsmtpd
61660 91566 1 0 30x100080 kqreadsmtpd
44941 436617 1 0 30x88 poll sshd
63116 236112 1 0 30x100080 poll ntpd
67685 455461 13478 83 30x100092 poll ntpd
13478 271069 1 83 30x100092 poll ntpd
4556 497201 93102 73 30x100090 kqreadsyslogd
93102 715 1 0 30x100082 netio syslogd
75807 17005 0 0 3 0x14200 bored smr
19891 296875 0 0 3 0x14200 pgzerozerothread
85805 493633 0 0 3 0x14200 aiodoned aiodoned
30502 291595 0 0 3 0x14200 syncerupdate
70346 429811 0 0 3 0x14200 cleaner cleaner
2149 206582 0 0 3 0x14200 reaperreaper
34585 457270 0 0 3 0x14200 pgdaemon pagedaemon
68729 397903 0 0 3 0x14200 usbtskusbtask
65328 269730 0 0 3 0x14200 usbatsk usbatsk
69367 498053 0 0 3 0x40014200 acpi0 acpi0
5156 405582 0 0 7 0x40014200idle5
53211 338561 0 0 7 0x40014200idle4
19386 131626 0 0 7 0x40014200idle3
22779 421169 0 0 3 0x40014200idle2
27190 363359 0 0 7 0x40014200idle1
1080 80711 0 0 3 0x14200 bored sensors
67407 419237 0 0 7 0x14200softnet
34617 10042 0 0 3 0x14200 netlock softnet
50133 404397 0 0 3 0x14200 netlock softnet
*94649 157694 0 0 7 0x14200softnet
51030 188067 0 0 3 0x14200 bored systqmp
46228 294212 0 0 3 0x14200 bored systq
62154 451682 0 0 2 0x40014200softclock
34808 227380 0 0 3 0x40014200idle0
1 24458 0 0 30x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb{2}> trace /t 0t157694
80087048(1,8122bc39,800022c60550,800022c60528,80002
2c60534,4) at 0x80087048
Re: ipsp_spd_lookup tdb refcount
On 2.12.2021. 0:49, Alexander Bluhm wrote: > Hi, > > This adds TDB ref counting to ipsp_spd_lookup(). > > While there make ip6_output() look a bit more like ip_output(). > > ok? Hi, i'm hitting this diff for one day and sasyncd boxes didn't panic.
Re: ipsec: refactor TDBF_DELETED
On 25.11.2021. 17:13, Tobias Heider wrote: > On Thu, Nov 25, 2021 at 03:50:29PM +0100, Tobias Heider wrote: >> As discussed in the previous thread we can simplify the tdb cleanup >> code by removing the TDBF_DELETED flag and instead checking if the >> tdb was already unlinked. >> >> ok? >> > > Now with the missing parts from pfkeyv2.c as noticed by Hrvoje. > I'm hitting this diff for 24 hours and i can't get panic ...
Re: IPsec tdb ref counting
On 25.11.2021. 9:52, Alexander Bluhm wrote: > On Sat, Nov 13, 2021 at 06:04:07PM +0100, Alexander Bluhm wrote: >> When testing, please check for tdb leaks. > The diff below was running on my performance setup for more than > 10 hours. iked SA lifetime was about 10 seconds. ipsecctl -F; > vmstat -m showed no leak. Running regress passed. > > Hrvoje is also testing this diff. So I would recommend to commit > what we have after positive feedback from Hrvoje and some OKs. > > Things like removing TDBF_DELETED flag from tobhe@ or concerns from > mvs@ that we need more ref count protection will be added later. > > ok? Hi, i'm hitting this diff on sasyncd setup for some time and boxes didn't panic...
Re: IPsec tdb ref counting
On 23.11.2021. 14:18, Alexander Bluhm wrote:
> On Tue, Nov 23, 2021 at 06:54:59AM +0100, Hrvoje Popovski wrote:
>> after 24 hours hitting sasyncd setup one box panic
>
> Thanks for testing.
>
> I have reduced my iked lifetime to about 10 seconds and got the
> same panic on my new 8 core test machine.
>
> ddb{2}> trace
> db_enter() at db_enter+0x10
> panic(81eaa8e3) at panic+0xbf
> pool_do_get(821e64d8,9,8000238b0524) at pool_do_get+0x35c
> pool_get(821e64d8,9) at pool_get+0x93
> tdb_alloc(0) at tdb_alloc+0x62
> reserve_spi(0,100,,80d41254,80d41238,32,cbd2b00c6d3d3ec
> d) at reserve_spi+0xfc
> pfkeyv2_send(fd8739174900,81b3ba80,50) at pfkeyv2_send+0x19c6
> pfkeyv2_output(fd80948cea00,fd8739174900,0,0) at pfkeyv2_output+0x8a
> pfkeyv2_usrreq(fd8739174900,9,fd80948cea00,0,0,8000238857b0) at
> pfk
> eyv2_usrreq+0x1b0
> sosend(fd8739174900,0,8000238b0b60,0,0,0) at sosend+0x3a9
> dofilewritev(8000238857b0,3,8000238b0b60,0,8000238b0c60) at
> dofilew
> ritev+0x14d
> sys_writev(8000238857b0,8000238b0c00,8000238b0c60) at
> sys_writev+0x
> d2
> syscall(8000238b0cd0) at syscall+0x3a9
> Xsyscall() at Xsyscall+0x128
>
>> ddb{3}> show tdb
>
> You have to add the pool item addr to this command.
>
> I additionally have refcount tracing diff on my machine. With that
> I see this result:
>
> ddb{2}> show panic
> *cpu2: pool_do_get: tdb free list modified: page 0x8801; item
> addr 0
> x8801c998; offset 0x28=0xdeadbeee
>
> ddb{2}> show tdb /f 0x8801c998
> tdb at 0x8801c998
> hnext: 0x4c38c8f8ffb0cab5
> dnext: 0xff2c2a5ac7964242
> snext: 0xdeadbeefdeadbeef
> ...
> tdb_trace[78]: 350309838: refs 5 -1 cpu2 ipsec_forward_check:1081
> tdb_trace[79]: 350309839: refs 4 +1 cpu2 gettdb_dir:358
> tdb_trace[80]: 350309840: refs 5 -1 cpu2 ipsec_common_input:355
> tdb_trace[81]: 350309841: refs 4 +1 cpu2 gettdb_dir:358
> tdb_trace[82]: 350309842: refs 5 -1 cpu2 ipsec_forward_check:1081
> tdb_trace[83]: 350310888: refs 4 -1 cpu2 ipsp_spd_lookup:529
> tdb_trace[84]: 350816099: refs 3 -1 cpu0 tdb_soft_timeout:726
> tdb_trace[85]: 351266117: refs 2 +1 cpu2 gettdb_dir:358
> tdb_trace[86]: 351266118: refs 3 +0 cpu2 pfkeyv2_send:1599
> tdb_trace[87]: 351266119: refs 3 -1 cpu2 tdb_delete0:997
> tdb_trace[88]: 351271898: refs 2 -1 cpu2 pfkeyv2_send:2143
> tdb_trace[89]: 351300368: refs 1 +0 cpu0 tdb_timeout:688
> tdb_trace[90]: 351300369: refs 1 -1 cpu0 tdb_delete0:997
> tdb_trace[91]: 351300370: refs 3735928559 -1 cpu0 tdb_timeout:691
>
> I will try mvs@ IPL_NET fix and think a bit more about the problem.
>
> bluhm
>
Hi,
i've got panic with mvs@ diff
bluhm@ thank you for tips ..
r620-2# panic: pool_do_get: tdb free list modified: page
0x812ee000; item addr 0
x812f1bb0; offset 0x28=0xdeafbeac
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
263347 98359 680x10 02 sasyncd
*136177 87522 680x10 03 isakmpd
282035451 0 0x14000 0x2001 softnet
db_enter() at db_enter+0x10
panic(81ea6d34) at panic+0xbf
pool_do_get(822308b8,9,800022da0f94) at pool_do_get+0x35c
pool_get(822308b8,9) at pool_get+0x93
tdb_alloc(0) at tdb_alloc+0x62
reserve_spi(0,100,,812c4254,812c4238,32,3f96bc02a5ef3ac
f) at reserve_spi+0xff
pfkeyv2_send(fd83b1902a90,812c3400,50) at pfkeyv2_send+0x146a
pfkeyv2_output(fd80a5358c00,fd83b1902a90,0,0) at pfkeyv2_output+0x8a
pfkeyv2_usrreq(fd83b1902a90,9,fd80a5358c00,0,0,800022d03a48)
at pfk
eyv2_usrreq+0x1b0
sosend(fd83b1902a90,0,800022da15e0,0,0,0) at sosend+0x3a9
dofilewritev(800022d03a48,7,800022da15e0,0,800022da16e0) at
dofilew
ritev+0x14d
sys_writev(800022d03a48,800022da1680,800022da16e0) at
sys_writev+0x
d2
syscall(800022da1750) at syscall+0x3a9
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7c6cc0, count: 1
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{3}> show tdb
0x812ee000: (unknown address family)->(unknown address
family)
:0 #-2135853982 dead4110
ddb{3}> show all tdb
0x812ee8b0: fac0dfe4 192.168.42.113->192.168.42.100:50 #3 1082
0x812efdf0: 4e927a9b 192.168.42.112->192.168.42.100:50 #2 0012
0x812f0ab0: c630e737 192.168.42.100->192.168.42.113:50 #4 000d1082
ddb{3}>
Re: IPsec tdb ref counting
On 21.11.2021. 23:36, Alexander Bluhm wrote:
> Updated tdb refcounting diff after merging with mvs@'s commit.
Hi,
after 24 hours hitting sasyncd setup one box panic
r620-2# panic: pool_do_get: tdb free list modified: page
0x812e; item addr 0
x812e2a88; offset 0x28=0xdead410f
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
272153 56403 680x10 01 sasyncd
* 32102 91875 680x10 03 isakmpd
471006 1037 730x100010 0x800 syslogd
484026 49900 0 0x14000 0x2004 softnet
106465 49485 0 0x14000 0x2002 systq
db_enter() at db_enter+0x10
panic(81ea6a83) at panic+0xbf
pool_do_get(822bd5a0,9,800022d80e34) at pool_do_get+0x35c
pool_get(822bd5a0,9) at pool_get+0x93
tdb_alloc(0) at tdb_alloc+0x62
reserve_spi(0,100,,812c6254,812c6238,32,73d60b71a1c10a4
9) at reserve_spi+0xff
pfkeyv2_send(fd8394d5f550,812c5c80,50) at pfkeyv2_send+0x146a
pfkeyv2_output(fd800a5c5100,fd8394d5f550,0,0) at pfkeyv2_output+0x8a
pfkeyv2_usrreq(fd8394d5f550,9,fd800a5c5100,0,0,800022cd5268)
at pfkeyv2_usrreq+0x1b0
sosend(fd8394d5f550,0,800022d81480,0,0,0) at sosend+0x3a9
dofilewritev(800022cd5268,7,800022d81480,0,800022d81580) at
dofilewritev+0x14d
sys_writev(800022cd5268,800022d81520,800022d81580) at
sys_writev+0xd2
syscall(800022d815f0) at syscall+0x3a9
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7b2430, count: 1
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{3}>
ddb{3}> show tdb
0x8198ba60: 0041 (unknown address family)->(unknown address
family)
:139 #1428429875 02c08348
ddb{3}> show all tdb
0x812e2a88: f5c0089f 192.168.42.112->192.168.42.100:50
#-559070961
1000
ddb{3}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
56403 272153 15076 68 70x10sasyncd
15076 149191 1 0 30x80 kqreadsasyncd
*91875 32102 21647 68 70x10isakmpd
21647 46093 1 0 30x80 netio isakmpd
62911 252487 1 0 30x100083 ttyin ksh
27760 230059 1 0 30x100098 kqreadcron
64991 237179 48935 95 30x100092 kqreadsmtpd
39524 451490 48935103 30x100092 kqreadsmtpd
41945 372539 48935 95 30x100092 kqreadsmtpd
71764 444337 48935 95 30x100092 kqreadsmtpd
71639 125115 48935 95 30x100092 kqreadsmtpd
28372 471050 48935 95 30x100092 kqreadsmtpd
48935 320655 1 0 30x100080 kqreadsmtpd
82840 422286 1 0 30x88 kqreadsshd
75675 375246 1 0 30x100080 kqreadntpd
3482 28773 19227 83 30x100092 kqreadntpd
19227 110748 1 83 30x100092 kqreadntpd
42341 280950 33183 74 30x100092 bpf pflogd
33183 127564 1 0 30x80 netio pflogd
1037 471006 30335 73 70x100090syslogd
30335 62561 1 0 30x100082 netio syslogd
41504 427392 0 0 3 0x14200 bored smr
21187 243711 0 0 3 0x14200 pgzerozerothread
48815 395094 0 0 3 0x14200 aiodoned aiodoned
48275 488591 0 0 3 0x14200 syncerupdate
38791 457271 0 0 3 0x14200 cleaner cleaner
30177 510262 0 0 3 0x14200 reaperreaper
434605699 0 0 3 0x14200 pgdaemon pagedaemon
13390 114146 0 0 3 0x14200 usbtskusbtask
75817 456494 0 0 3 0x14200 usbatsk usbatsk
49408 32535 0 0 3 0x40014200 acpi0 acpi0
34775 312507 0 0 7 0x40014200idle5
48439 117613 0 0 3 0x40014200idle4
36384 294756 0 0 3 0x40014200idle3
91987 363872 0 0 3 0x40014200idle2
91389 296087 0 0 3 0x40014200idle1
7900 284285 0 0 3 0x14200 bored sensors
49900 484026 0 0 7 0x14200softnet
31157 45925 0 0 3 0x14200 bored systqmp
49485 106465 0 0 7 0x14200systq
52790 213052 0 0 3 0x40014200 bored softclock
88741 290928 0 0 3 0x40014200idle0
1 245546 0 0 30x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb{3}>
Re: IPsec tdb ref counting
On 16.11.2021. 0:08, Hrvoje Popovski wrote:
> On 15.11.2021. 16:44, Hrvoje Popovski wrote:
>> On 15.11.2021. 15:04, Vitaliy Makkoveev wrote:
>>> On Mon, Nov 15, 2021 at 02:51:16PM +0100, Hrvoje Popovski wrote:
>>>
>>> And you don'n see "> tdb_free() killing ourself" in dmesg
>>> output?
>>
>>
>> I couldn't find that message anywhere
>>
>>
>
> with new panic i see this message ... this time i have mvs@ + "IPsec tdb
> ddb print" diff ...
>
> r620-1# > tdb_free() killing ourself
> panic: kernel diagnostic assertion "refcnt != ~0" failed: file
> "/sys/kern/kern_synch.c", line 824
> Stopped at db_enter+0x10: popq%rbp
> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
> 142769 73130 680x10 01 sasyncd
> *407867 7095 0 0x14000 0x42000K softclock
> db_enter() at db_enter+0x10
> panic(81e456ab) at panic+0xbf
> __assert(81eb4ec9,81e1c0b3,338,81e4cbf5) at
> __assert+0x25
> refcnt_rele(813f3478) at refcnt_rele+0x6f
> tdb_free(813f3450) at tdb_free+0x14f
> tdb_timeout(813f3450) at tdb_timeout+0x39
> softclock_thread(8000efc0) at softclock_thread+0x16e
> end trace frame: 0x0, count: 8
> https://www.openbsd.org/ddb.html describes the minimum info required in
> bug reports. Insufficient info makes it difficult to find and fix bugs.
> ddb{0}>
>
panic with show tdb feature in ddb and witness
r620-2# panic: kernel diagnostic assertion "refcnt != ~0" failed: file
"/sys/kern/kern_
synch.c", line 824
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
219343 43291 680x10 02 sasyncd
*321579 50760 680x10 04 isakmpd
187492 98224 0 0x14000 0x2003 smr
166953 9856 0 0x14000 0x2001 systq
33216 41335 0 0x14000 0x42000 softclock
db_enter() at db_enter+0x10
panic(81e4f74f) at panic+0xbf
__assert(81ebf7a0,81e233bf,338,81e54f8e) at
__assert+0x25
refcnt_rele(814829a8) at refcnt_rele+0x6f
tdb_unref(81482980) at tdb_unref+0x26
pfkeyv2_send(fd83b0ddd968,81439d00,50) at pfkeyv2_send+0x662
pfkeyv2_output(fd80a4b0c200,fd83b0ddd968,0,0) at pfkeyv2_output+0x8a
pfkeyv2_usrreq(fd83b0ddd968,9,fd80a4b0c200,0,0,800022cd5510)
at pfkeyv2_usrreq+0x1b0
sosend(fd83b0ddd968,0,800022dc9000,0,0,0) at sosend+0x3a9
dofilewritev(800022cd5510,7,800022dc9000,0,800022dc9100) at
dofilew
ritev+0x14d
sys_writev(800022cd5510,800022dc90a0,800022dc9100) at
sys_writev+0xd2
syscall(800022dc9170) at syscall+0x3a9
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7e90a0, count: 2
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{4}>
ddb{4}> show tdb
0x8188e110: 2403 (unknown address family)->(unknown address
family)
:128 02c08348
ddb{4}> show all tdb
0x81481880: 38a3d1b0 192.168.42.100->192.168.42.112:50 000d1082
0x81482540: dc955248 192.168.42.113->192.168.42.100:50 1002
0x81482980: 344ab07c 192.168.42.100->192.168.42.113:50 00091000
0x81483a80: 06e97cf1 192.168.42.112->192.168.42.100:50 1082
0x81483ec0: 064e7b16 192.168.42.100->192.168.42.113:50 000d1082
0x81484b80: d213635f 192.168.42.113->192.168.42.100:50 1082
ddb{4}>
ddb{4}> show locks
exclusive rwlock netlock r = 0 (0x8216c5e0)
#0 witness_lock+0x333
#1 rw_enter+0x27f
#2 pfkeyv2_send+0x659
#3 pfkeyv2_output+0x8a
#4 pfkeyv2_usrreq+0x1b0
#5 sosend+0x3a9
#6 dofilewritev+0x14d
#7 sys_writev+0xd2
#8 syscall+0x3a9
#9 Xsyscall+0x128
ddb{4}>
ddb{4}> show all locks
CPU 3:
exclusive mutex /sys/kern/kern_malloc.c:105 r = 0 (0x82180d60)
#0 witness_lock+0x333
#1 mtx_enter_try+0x95
#2 mtx_enter+0x48
#3 free+0x74
#4 aesni_free+0x3c
#5 smr_thread+0x21e
#6 proc_trampoline+0x1c
Process 43291 (sasyncd) thread 0x800022cd4550 (219343)
exclusive kernel_lock _lock r = 0 (0x82255630)
#0 witness_lock+0x333
#1 __mp_acquire_count+0x38
#2 mi_switch+0x299
#3 sleep_finish+0x11c
#4 rw_enter+0x223
#5 solock+0x4b
#6 filt_soreadmodify+0x32
#7 kqueue_register+0x6f1
#8 pselregister+0x1f1
#9 dopselect+0x300
#10 sys_pselect+0xdb
#11 syscall+0x3a9
#12 Xsyscall+0x128
Process 50760 (isakmpd) thread 0x800022cd5510 (321579)
exclusive rwlock netlock r = 0 (0x8216c5e0)
#0 witness_lock+0x333
#1 rw_enter+0x27f
#2 pfkeyv2_send+0x659
#3 pfkeyv2_output+0x8a
#4 pfkeyv2_usrreq+0x1b0
#5 sosend+0x3a9
#6 dofilewritev+0
Re: IPsec tdb ddb print
On 15.11.2021. 17:23, Alexander Bluhm wrote:
> Hi,
>
> To debug IPsec and tdb refcounting it may be useful to have "show
> tdb" and "show all tdbs" in ddb.
Here's panic with this and mvs@ version of "IPsec tdb ref counting" diff
r620-1# > tdb_free() killing ourself
panic: kernel diagnostic assertion "refcnt != ~0" failed: file
"/sys/kern/kern_synch.c", line 824
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
142769 73130 680x10 01 sasyncd
*407867 7095 0 0x14000 0x42000K softclock
db_enter() at db_enter+0x10
panic(81e456ab) at panic+0xbf
__assert(81eb4ec9,81e1c0b3,338,81e4cbf5) at
__assert+0x25
refcnt_rele(813f3478) at refcnt_rele+0x6f
tdb_free(813f3450) at tdb_free+0x14f
tdb_timeout(813f3450) at tdb_timeout+0x39
softclock_thread(8000efc0) at softclock_thread+0x16e
end trace frame: 0x0, count: 8
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
ddb{0}> show tdb
0x81333460: b774bf02 (unknown address family)->(unknown address
family)
ddb{0}> show all tdb
0x813f3450: 1712f629 192.168.42.112->192.168.42.100:50 1002
0x813f5210: 5f9890f7 192.168.42.100->192.168.42.112:50 1082
0x813f5a90: 6eebe767 192.168.42.100->192.168.42.112:50 1000
0x813f5ed0: d7874fc6 192.168.42.112->192.168.42.100:50 1082
Re: IPsec tdb ref counting
On 15.11.2021. 16:44, Hrvoje Popovski wrote:
> On 15.11.2021. 15:04, Vitaliy Makkoveev wrote:
>> On Mon, Nov 15, 2021 at 02:51:16PM +0100, Hrvoje Popovski wrote:
>>
>> And you don'n see "> tdb_free() killing ourself" in dmesg
>> output?
>
>
> I couldn't find that message anywhere
>
>
with new panic i see this message ... this time i have mvs@ + "IPsec tdb
ddb print" diff ...
r620-1# > tdb_free() killing ourself
panic: kernel diagnostic assertion "refcnt != ~0" failed: file
"/sys/kern/kern_synch.c", line 824
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
142769 73130 680x10 01 sasyncd
*407867 7095 0 0x14000 0x42000K softclock
db_enter() at db_enter+0x10
panic(81e456ab) at panic+0xbf
__assert(81eb4ec9,81e1c0b3,338,81e4cbf5) at
__assert+0x25
refcnt_rele(813f3478) at refcnt_rele+0x6f
tdb_free(813f3450) at tdb_free+0x14f
tdb_timeout(813f3450) at tdb_timeout+0x39
softclock_thread(8000efc0) at softclock_thread+0x16e
end trace frame: 0x0, count: 8
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
Re: IPsec tdb ref counting
On 15.11.2021. 15:04, Vitaliy Makkoveev wrote: > On Mon, Nov 15, 2021 at 02:51:16PM +0100, Hrvoje Popovski wrote: > > And you don'n see "> tdb_free() killing ourself" in dmesg > output? I couldn't find that message anywhere
Re: IPsec tdb ref counting
On 15.11.2021. 13:11, Vitaliy Makkoveev wrote:
> Hi,
>
> Could you try this diff? It should still panic, but I suspect to see
> "> tdb_free() killing ourself" string.
panic with your diff
r620-1# panic: kernel diagnostic assertion "refcnt != ~0" failed: file
"/sys/kern/kern_synch.c", line 824
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
229354 54144 680x10 02 sasyncd
*119032 22019 680x10 01 isakmpd
491600 50358 0 0x14000 0x2003 softnet
db_enter() at db_enter+0x10
panic(81e49a8f) at panic+0xbf
__assert(81eb660d,81e20855,338,81e518ae) at
__assert+0x25
refcnt_rele(812e7470) at refcnt_rele+0x6f
tdb_unref(812e7448) at tdb_unref+0x26
pfkeyv2_send(fd83ae8761f0,812d5900,50) at pfkeyv2_send+0x662
pfkeyv2_output(fd80a555bc00,fd83ae8761f0,0,0) at pfkeyv2_output+0x8a
pfkeyv2_usrreq(fd83ae8761f0,9,fd80a555bc00,0,0,800022cdc7f0)
at pfkeyv2_usrreq+0x1b0
sosend(fd83ae8761f0,0,800022cff160,0,0,0) at sosend+0x3a9
dofilewritev(800022cdc7f0,7,800022cff160,0,800022cff260) at
dofilewritev+0x14d
sys_writev(800022cdc7f0,800022cff200,800022cff260) at
sys_writev+0xd2
syscall(800022cff2d0) at syscall+0x3a9
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7eb0d0, count: 2
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{1}>
ddb{1}> mach ddbcpu 0
Stopped at x86_ipi_db+0x12:leave
x86_ipi_db(82162ff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
_kernel_lock() at _kernel_lock+0xa9
softintr_dispatch(0) at softintr_dispatch+0x4a
Xsoftclock() at Xsoftclock+0x1f
acpicpu_idle() at acpicpu_idle+0x281
sched_idle(82162ff0) at sched_idle+0x27e
end trace frame: 0x0, count: 7
ddb{0}>
ddb{0}> mach ddbcpu 2
Stopped at x86_ipi_db+0x12:leave
x86_ipi_db(800022412ff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
_kernel_lock() at _kernel_lock+0xb2
syscall(800022d83450) at syscall+0x29e
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7c7320, count: 9
ddb{2}> mach ddbcpu 3
Stopped at x86_ipi_db+0x12:leave
x86_ipi_db(80002241bff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
pf_find_state_byid(800022c606f8) at pf_find_state_byid+0x41
pfsync_in_upd_c(fd8003f5c2f0,54,10,2) at pfsync_in_upd_c+0xff
pfsync_input(800022c60988,800022c60994,f0,2) at pfsync_input+0x33e
ip_deliver(800022c60988,800022c60994,f0,2) at ip_deliver+0x103
ip_ours(800022c60988,800022c60994,f0e0,0) at ip_ours+0x31d
ip_input_if(800022c60988,800022c60994,4,0,800a0048) at
ip_input_if+0x19d
ipv4_input(800a0048,fd8002ea5c00) at ipv4_input+0x39
ether_input(800a0048,fd8002ea5c00) at ether_input+0x39f
if_input_process(800a0048,800022c60a78) at if_input_process+0x6f
ifiq_process(8009df00) at ifiq_process+0x69
taskq_thread(8002f080) at taskq_thread+0x81
end trace frame: 0x0, count: 1
ddb{3}>
ddb{3}> mach ddbcpu 4
Stopped at x86_ipi_db+0x12:leave
x86_ipi_db(800022424ff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
acpicpu_idle() at acpicpu_idle+0x281
sched_idle(800022424ff0) at sched_idle+0x27e
end trace frame: 0x0, count: 10
ddb{4}> mach ddbcpu 5
Stopped at x86_ipi_db+0x12:leave
x86_ipi_db(80002242dff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
acpicpu_idle() at acpicpu_idle+0x281
sched_idle(80002242dff0) at sched_idle+0x27e
end trace frame: 0x0, count: 10
ddb{5}>
Re: IPsec tdb ref counting
On 14.11.2021. 22:50, Alexander Bluhm wrote:
> New diff with fix from mvs@. Please continue testing with this one.
Hi,
i've applied this diff on sasyncd setup with two ipsec sessions and i'm
getting this panic. Box didn't panic instantly but after some time. I
will leave ddb console active...
r620-1# panic: kernel diagnostic assertion "refcnt != ~0" failed: file
"/sys/kern/kern_synch.c", line 824
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
382266 23173 680x10 0x803 sasyncd
407096 64559 680x10 0x801 isakmpd
*287846 76389 0 0x14000 0x42000K softclock
db_enter() at db_enter+0x10
panic(81e52fc8) at panic+0xbf
__assert(81ebf6f1,81e23497,338,81e54ff7) at
__assert+0x25
refcnt_rele(81488038) at refcnt_rele+0x6f
tdb_free(81488010) at tdb_free+0x116
tdb_timeout(81488010) at tdb_timeout+0x39
timeout_run(81488068) at timeout_run+0x93
softclock_thread(8000f260) at softclock_thread+0x11d
end trace frame: 0x0, count: 7
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
ddb{0}> show locks
exclusive rwlock netlock r = 0 (0x8219c1b8)
#0 witness_lock+0x333
#1 tdb_timeout+0x18
#2 timeout_run+0x93
#3 softclock_thread+0x11d
#4 proc_trampoline+0x1c
shared rwlock timeout r = 0 (0x82160700)
#0 witness_lock+0x333
#1 timeout_run+0x88
#2 softclock_thread+0x11d
#3 proc_trampoline+0x1c
exclusive kernel_lock _lock r = 0 (0x8233cbd0)
#0 witness_lock+0x333
#1 __mp_acquire_count+0x38
#2 mi_switch+0x299
#3 sleep_finish+0x11c
#4 softclock_thread+0xd4
#5 proc_trampoline+0x1c
ddb{0}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
23173 382266 23482 68 70x90sasyncd
23482 270620 1 0 30x80 kqreadsasyncd
64559 407096 82882 68 70x90isakmpd
82882 269197 1 0 30x80 netio isakmpd
37653 133205 65408 0 30x100083 ttyin ksh
65408 289926 97192 1000 30x10008b sigsusp ksh
97192 251184 8671 1000 30x98 kqreadsshd
8671 56983 39827 0 30x82 kqreadsshd
46017 298252 1 0 30x100083 ttyin ksh
1861 295220 1 0 30x100098 kqreadcron
16854 177115 71819 95 30x100092 kqreadsmtpd
1393 225046 71819103 30x100092 kqreadsmtpd
64691 272570 71819 95 30x100092 kqreadsmtpd
83230 208366 71819 95 30x100092 kqreadsmtpd
97497 421023 71819 95 30x100092 kqreadsmtpd
88432 457526 71819 95 30x100092 kqreadsmtpd
71819 188076 1 0 30x100080 kqreadsmtpd
39827 214126 1 0 30x88 kqreadsshd
29655 495924 1 0 30x100080 kqreadntpd
36076 376711 37441 83 30x100092 kqreadntpd
37441 106087 1 83 30x100092 kqreadntpd
68059 409459 2456 74 30x100092 bpf pflogd
2456 325385 1 0 30x80 netio pflogd
10369 56599 82476 73 30x100090 kqreadsyslogd
824764579 1 0 30x100082 netio syslogd
40946 324347 0 0 3 0x14200 bored smr
77501 384609 0 0 3 0x14200 pgzerozerothread
78741 290337 0 0 3 0x14200 aiodoned aiodoned
45215 268650 0 0 3 0x14200 syncerupdate
62355 93014 0 0 3 0x14200 cleaner cleaner
44276 256950 0 0 3 0x14200 reaperreaper
30498 450293 0 0 3 0x14200 pgdaemon pagedaemon
43809 11307 0 0 3 0x14200 usbtskusbtask
58116 68425 0 0 3 0x14200 usbatsk usbatsk
25415 496397 0 0 3 0x40014200 acpi0 acpi0
9902 65516 0 0 7 0x40014200idle5
23605 371050 0 0 7 0x40014200idle4
96607 441776 0 0 3 0x40014200idle3
5176 230617 0 0 7 0x40014200idle2
19030 267534 0 0 3 0x40014200idle1
32317 110469 0 0 3 0x14200 bored sensors
77670 69914 0 0 3 0x14200 bored softnet
76476 412718 0 0 3 0x14200 bored systqmp
2878 385617 0 0 3 0x14200 bored systq
*76389 287846 0 0 7 0x40014200softclock
23236 11183 0 0 3 0x40014200idle0
1 95380 0 0 30x82 wait init
0 0 -1
Re: Dell R620 - latest snapshot
On 25.10.2021. 22:58, Theo de Raadt wrote: > Can you checkout -current, and try this diff: > > It has one additional check for cpu_setperf being NULL, near the top of > setperf_auto() Hi, yes, with this diff box boot normally Tnx ..
Dell R620 - latest snapshot
Hi all,
with latest snaphost from few minutes ago Dell R620 box panic while booting
vmm0 at mainbus0: VMX/EPT
dt: 445 probes
attempt to execute user address 0x0 in supervisor mode
kernel: page fault trap, code=0
Stopped at 0TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
0(64,8216f720,de001fffc258,821b1568,821ffc80,1)
at 0
setperf_auto(0,0,d986fb49e8e1e7b2,0,81593260,1) at setperf_auto+0x4d
softclock_process_tick_timeout(8216de78,0,e2e5855cd0aeac94,821b
1b20,0,821b1b20) at softclock_process_tick_timeout+0x106
softclock(0,0,f1deb9b688ef9df3,0,821b1578,821b1560) at
softclock+0xb8
softintr_dispatch(0,0,ec7e785c93841f4b,0,0,8152d414) at
softintr_dispatch+0xeb
Xsoftclock(0,0,1388,0,8002e6c0,8215a700) at Xsoftclock+0x1f
acpicpu_idle(cf980a4d1289648c,8215a010,82159ff0,8000f7a
0,0,0) at acpicpu_idle+0x281
sched_idle(82159ff0,82159ff0,0,0,82159ff0,81e13
660) at sched_idle+0x27e
end trace frame: 0x0, count: 8
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
ddb{0}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
95902 409912 0 0 3 0x14200 usbdlyusbtask
76026 40696 0 0 3 0x14200 usbatsk usbatsk
73092 189429 0 0 3 0x40014200 acpi0 acpi0
48658 523253 0 0 1 0x14200idle5
10829 25129 0 0 1 0x14200idle4
94971 504025 0 0 1 0x14200idle3
59213 299094 0 0 1 0x14200idle2
53939 241990 0 0 1 0x14200idle1
97462 152756 0 0 3 0x14200 bored sensors
57912 119825 0 0 3 0x14200 bored softnet
73318 325240 0 0 3 0x14200 bored systqmp
83926 221734 0 0 3 0x14200 bored systq
98228 59299 0 0 3 0x40014200 bored softclock
*28692 166452 0 0 7 0x40014200idle0
1 332959 0 0 3 0 initexec swapper
0 0 -1 0 3 0x10200 cfpendswapper
ddb{0}> trace /t 0t166452
sched_idle(82159ff0,82159ff0,0,0,82159ff0,81e13
660) at sched_idle+0x27e
end trace frame: 0x0, count: -1
ddb{0}> show reg
rdi 0x64
rsi 0x8216f720timeout_mutex
rbp 0x800022c48440
rbx 0x821b1568x86_soft_intrs+0x8
rdx 0xde001fffc258
rcx0x202
rax 0xd
r8 0xff
r90x112e0be826d694b3
r10 0x1
r110
r12 0x821ffc80timeout_todo
r13 0x1
r14 0x81593260setperf_auto
r150
rip0
cs 0x8
rflags 0x10202__ALIGN_SIZE+0xf202
rsp 0x800022c483f8
ss 0x10
0
OpenBSD 7.0-current (GENERIC.MP) #27: Sat Oct 23 22:57:48 CEST 2021
hrv...@r620-2.srce.hr:/sys/arch/amd64/compile/GENERIC.MP
real mem = 17115840512 (16322MB)
avail mem = 16581156864 (15813MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries)
bios0: vendor Dell Inc. version "2.9.0" date 12/06/2019
bios0: Dell Inc. PowerEdge R620
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WDAT SLIC ERST HEST
BERT EINJ TCPA PC__ SRAT SSDT
acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 4 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.49 MHz, 06-3e-04
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 2, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 100MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 6 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.02 MHz, 06-3e-04
cpu1:
Re: ixl(4): add checksum receive offloading
On 22.10.2021. 16:57, Theo de Raadt wrote: > Claudio Jeker wrote: > >> For ospfd tests you want to make sure that some of the ospf packets need >> fragmenting. So this needs a sizeable network to hit this. > > Yes, as I remember, the problems only related to fragments of large > packets. > > These tests are too narrow and the results inconclusive. > > Is this any better? .. i'm sending 30k /32 routes to ixl box .. smc24# ospfctl show Router ID: 192.168.11.1 Uptime: 00:05:13 RFC1583 compatibility flag is disabled SPF delay is 1000 msec(s), hold time between two SPFs is 5000 msec(s) Number of external LSA(s) 29802 (Checksum sum 0x3a65c06e) Number of areas attached to this router: 1 Area ID: 0.0.0.5 Number of interfaces in this area: 1 Number of fully adjacent neighbors in this area: 1 SPF algorithm executed 4 time(s) Number LSA(s) 4 (Checksum sum 0x2496e)
Re: ixl(4): add checksum receive offloading
On 22.10.2021. 16:53, Claudio Jeker wrote: > On Fri, Oct 22, 2021 at 04:45:09PM +0200, Hrvoje Popovski wrote: >> On 22.10.2021. 16:09, Florian Obser wrote: >>> >>> >>> On 22 October 2021 13:55:20 CEST, Stuart Henderson >>> wrote: >>>> On 2021/10/22 11:25, Jan Klemkow wrote: >>>>> this diff add hardware checksum offloading for the receive path of >>>>> ixl(4) interfaces. >>>> >>>> Would be good to have this tested with NFS if anyone has a way to do so. >>>> nics are probably better now but I'm pretty sure we have had problems >>>> with NFS and offloading in the past. >>> >>> ospf as well. >>> >> >> ospf seems to work >> >> >> smc24# ospfctl show nei >> ID Pri StateDeadTime Address Iface Uptime >> 10.1.1.11 FULL/BCKUP 00:00:30 192.168.15.123 ixl0 00:01:20 >> >> >> >> 10.11.0.0/16 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 10.12.0.0/16 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 10.13.0.0/16 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 10.14.0.0/16 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 10.15.0.0/16 192.168.15.11 Type 1 ext Network 110 >> 00:00:13 >> 10.16.0.0/16 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 10.17.0.0/16 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 10.18.0.0/16 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> >> 192.168.11.0/24 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 192.168.12.0/24 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 192.168.13.0/24 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 192.168.14.0/24 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 192.168.16.0/24 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 192.168.17.0/24 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> 192.168.18.0/24 192.168.15.1 Type 1 ext Network 110 >> 00:00:13 >> > > For ospfd tests you want to make sure that some of the ospf packets need > fragmenting. So this needs a sizeable network to hit this. > Thank you .. i didn't know that ..
Re: ixl(4): add checksum receive offloading
On 22.10.2021. 16:22, Sebastian Benoit wrote: > Stuart Henderson(s...@spacehopper.org) on 2021.10.22 12:55:20 +0100: >> On 2021/10/22 11:25, Jan Klemkow wrote: >>> this diff add hardware checksum offloading for the receive path of >>> ixl(4) interfaces. >> >> Would be good to have this tested with NFS if anyone has a way to do so. >> nics are probably better now but I'm pretty sure we have had problems >> with NFS and offloading in the past. > > And ospf(6)d... mikeb hit that i believe? > and ospf6d seems to work smc24# ospf6ctl show nei de Neighbor 10.1.1.1, interface address fe80::eef4:bbff:feda:f7f8 Area 0.0.0.5, interface ixl0 Neighbor priority is 5, State is FULL, 7 state changes DR is 161.53.253.247, BDR is 10.1.1.1 Options *|*|-|R|-|*|E|V6 Dead timer due in 00:00:17 Uptime 00:02:28 Database Summary List 0 Link State Request List 0 Link State Retransmission List 0 Destination Nexthop Path TypeType CostUptime ::10.1.1.1 fe80::eef4:bbff:feda:f7f8%ixl0 Inter-Area Router 10 00:01:54 2001:db8:15::/64 :: C Intra-Area Network 10 00:01:59 ::/96fe80::eef4:bbff:feda:f7f8%ixl0 Type 1 ext Network 110 00:01:54 2002::/24fe80::eef4:bbff:feda:f7f8%ixl0 Type 1 ext Network 110 00:01:54 2002:db8:15::/64 fe80::eef4:bbff:feda:f7f8%ixl0 Type 1 ext Network 110 00:00:26 2002:7f00::/24 fe80::eef4:bbff:feda:f7f8%ixl0 Type 1 ext Network 110 00:01:54 2002:e000::/20 fe80::eef4:bbff:feda:f7f8%ixl0 Type 1 ext Network 110 00:01:54 2002:ff00::/24 fe80::eef4:bbff:feda:f7f8%ixl0 Type 1 ext Network 110 00:01:54 2003:db8:15::/64 fe80::eef4:bbff:feda:f7f8%ixl0 Type 1 ext Network 110 00:00:18 2004:db8:15::/64 fe80::eef4:bbff:feda:f7f8%ixl0 Type 1 ext Network 110 00:00:13 2005:db8:15::/64 fe80::eef4:bbff:feda:f7f8%ixl0 Type 1 ext Network 110 00:00:04
Re: ixl(4): add checksum receive offloading
On 22.10.2021. 16:09, Florian Obser wrote: > > > On 22 October 2021 13:55:20 CEST, Stuart Henderson > wrote: >> On 2021/10/22 11:25, Jan Klemkow wrote: >>> this diff add hardware checksum offloading for the receive path of >>> ixl(4) interfaces. >> >> Would be good to have this tested with NFS if anyone has a way to do so. >> nics are probably better now but I'm pretty sure we have had problems >> with NFS and offloading in the past. > > ospf as well. > ospf seems to work smc24# ospfctl show nei ID Pri StateDeadTime Address Iface Uptime 10.1.1.11 FULL/BCKUP 00:00:30 192.168.15.123 ixl0 00:01:20 10.11.0.0/16 192.168.15.1 Type 1 ext Network 110 00:00:13 10.12.0.0/16 192.168.15.1 Type 1 ext Network 110 00:00:13 10.13.0.0/16 192.168.15.1 Type 1 ext Network 110 00:00:13 10.14.0.0/16 192.168.15.1 Type 1 ext Network 110 00:00:13 10.15.0.0/16 192.168.15.11 Type 1 ext Network 110 00:00:13 10.16.0.0/16 192.168.15.1 Type 1 ext Network 110 00:00:13 10.17.0.0/16 192.168.15.1 Type 1 ext Network 110 00:00:13 10.18.0.0/16 192.168.15.1 Type 1 ext Network 110 00:00:13 192.168.11.0/24 192.168.15.1 Type 1 ext Network 110 00:00:13 192.168.12.0/24 192.168.15.1 Type 1 ext Network 110 00:00:13 192.168.13.0/24 192.168.15.1 Type 1 ext Network 110 00:00:13 192.168.14.0/24 192.168.15.1 Type 1 ext Network 110 00:00:13 192.168.16.0/24 192.168.15.1 Type 1 ext Network 110 00:00:13 192.168.17.0/24 192.168.15.1 Type 1 ext Network 110 00:00:13 192.168.18.0/24 192.168.15.1 Type 1 ext Network 110 00:00:13
Re: ixl(4): add checksum receive offloading
On 22.10.2021. 13:55, Stuart Henderson wrote: > On 2021/10/22 11:25, Jan Klemkow wrote: >> this diff add hardware checksum offloading for the receive path of >> ixl(4) interfaces. > > Would be good to have this tested with NFS if anyone has a way to do so. > nics are probably better now but I'm pretty sure we have had problems > with NFS and offloading in the past. > > (there's a chance I might be able to test if my c27xx atom box still > lives but you might be able to do it more easily) > I've created nfs server on ixl box, mount it from another and on that other box i'm doing "cvs -d /home/cvs checkout -P src" without any problem ...
Re: ixl(4): add checksum receive offloading
On 22.10.2021. 13:39, Jan Klemkow wrote: > Hi Hrvoje, > > Thats because, you only see this flags, if the checksum offloading is > enabled for "sending". I'm still working/debugging on the sending side. > Thus, I just send a diff with the receiving part for now. > > You can see if its working for your card with the netstat(8) statistics. > > # netstat -s | grep software-checksummed > > These counters should not raise much on the receive side if you put some > traffic over the interface. > > Thanks for testing, > Jan Thank you for explanation... I'm sending 8 tcp streams with iperf3 from some box to openbsd ixl box and here are results: without diff smc24# netstat -s | grep software-checksummed 5039250 input datagrams software-checksummed 2592718 output datagrams software-checksummed 2592709 packets software-checksummed 5039250 packets software-checksummed 0 input packets software-checksummed 0 output packets software-checksummed cca 6.12 Gbits/sec with diff smc24# netstat -s | grep software-checksummed 0 input datagrams software-checksummed 2956546 output datagrams software-checksummed 2956537 packets software-checksummed 0 packets software-checksummed 0 input packets software-checksummed 0 output packets software-checksummed cca 6.70 Gbits/sec are result like those expected? is forwarding testing any good for checksum offload diffs?
Re: ixl(4): add checksum receive offloading
On 22.10.2021. 11:25, Jan Klemkow wrote:
> Hi,
>
> this diff add hardware checksum offloading for the receive path of
> ixl(4) interfaces.
>
> Tested on:
> ixl1 at pci3 dev 0 function 1 "Intel X710 SFP+" rev 0x02: port 1, FW
> 6.0.48442 API 1.7, msix, 8 queues, address 40:a6:b7:02:38:3d
>
> OK?
>
Hi,
I've applied this diff and i can't see anything regarding offload with
ifconfig ixl hwfeatures?
smc24# ifconfig ixl0 hwfeatures
ixl0: flags=8843 mtu 1500
hwfeatures=10 hardmtu 9712
lladdr 3c:fd:fe:04:0d:64
index 7 priority 0 llprio 3
media: Ethernet autoselect (10GSFP+Cu full-duplex)
status: active
inet 192.168.15.1 netmask 0xff00 broadcast 192.168.15.255
smc24# dmesg | grep ixl
ixl0 at pci21 dev 0 function 0 "Intel X710 SFP+" rev 0x01: port 0, FW
8.2.64244 API 1.13, msix, 8 queues, address 3c:fd:fe:04:0d:64
ixl1 at pci21 dev 0 function 1 "Intel X710 SFP+" rev 0x01: port 1, FW
8.2.64244 API 1.13, msix, 8 queues, address 3c:fd:fe:04:0d:66
> Index: dev/pci/if_ixl.c
> ===
> RCS file: /cvs/src/sys/dev/pci/if_ixl.c,v
> retrieving revision 1.75
> diff -u -p -r1.75 if_ixl.c
> --- dev/pci/if_ixl.c 23 Jul 2021 00:29:14 - 1.75
> +++ dev/pci/if_ixl.c 22 Oct 2021 09:20:59 -
> @@ -1388,6 +1388,7 @@ static int ixl_rxeof(struct ixl_softc *,
> static void ixl_rxfill(struct ixl_softc *, struct ixl_rx_ring *);
> static void ixl_rxrefill(void *);
> static int ixl_rxrinfo(struct ixl_softc *, struct if_rxrinfo *);
> +static void ixl_rx_checksum(struct mbuf *, uint64_t);
>
> #if NKSTAT > 0
> static void ixl_kstat_attach(struct ixl_softc *);
> @@ -3190,6 +3191,7 @@ ixl_rxeof(struct ixl_softc *sc, struct i
> m->m_pkthdr.csum_flags |= M_FLOWID;
> }
>
> + ixl_rx_checksum(m, word);
> ml_enqueue(, m);
> } else {
> ifp->if_ierrors++; /* XXX */
> @@ -3320,6 +3322,23 @@ ixl_rxrinfo(struct ixl_softc *sc, struct
> free(ifr, M_TEMP, ixl_nqueues(sc) * sizeof(*ifr));
>
> return (rv);
> +}
> +
> +static void
> +ixl_rx_checksum(struct mbuf *m, uint64_t word)
> +{
> + if (!ISSET(word, IXL_RX_DESC_L3L4P))
> + return;
> +
> + if (ISSET(word, IXL_RX_DESC_IPE))
> + return;
> +
> + m->m_pkthdr.csum_flags |= M_IPV4_CSUM_IN_OK;
> +
> + if (ISSET(word, IXL_RX_DESC_L4E))
> + return;
> +
> + m->m_pkthdr.csum_flags |= M_TCP_CSUM_IN_OK | M_UDP_CSUM_IN_OK;
> }
>
> static int
>
Re: iwx(4) 40MHz channel support
On 12.10.2021. 16:29, Hrvoje Popovski wrote: > On 12.10.2021. 14:47, Stefan Sperling wrote: >> This patch adds support for 40MHz channels to iwx(4). >> >> Please sync your source tree before attempting to apply this patch. >> I have committed some changes to this driver today which this patch >> is based on. >> >> Works for me on AX200/AX201. Does anyone else want to do a pre-commit test? > > with this diff i'm getting 150Mbps vs 100Mbps without it here at home > ... will test it at work ... > > Thank you .. > > > Hi, on work i'm getting 170Mbps iwx0 at pci3 dev 0 function 0 "Intel Wi-Fi 6 AX200" rev 0x1a, msix tcpbench: Conn: 1 Mbps:166.087 Peak Mbps:172.858 Avg Mbps:166.087 Conn: 1 Mbps:171.568 Peak Mbps:172.858 Avg Mbps:171.568 Conn: 1 Mbps:169.212 Peak Mbps:172.858 Avg Mbps:169.212 Conn: 1 Mbps:167.841 Peak Mbps:172.858 Avg Mbps:167.841 Conn: 1 Mbps:169.302 Peak Mbps:172.858 Avg Mbps:169.302 tcpdump -n -i iwx0 -v -y IEEE802_11_RADIO -s 4096 type mgt and subtype beacon 12:06:52.870418 802.11 flags=0<>: beacon, caps=421, ssid (eduroam), rates 12M* 18M 24M* 36M 48M 54M 12M* 12M*, ds (chan 36), tim 0x0001, country 'HR ', channel 36 limit 23dB, channel 40 limit 23dB, channel 44 limit 23dB, channel 48 limit 23dB, channel 52 limit 23dB, channel 56 limit 23dB, channel 60 limit 23dB, channel 64 limit 23dB, channel 100 limit 30dB, channel 104 limit 30dB, channel 108 limit 30dB, channel 112 limit 30dB, channel 116 limit 30dB, channel 120 limit 30dB, channel 124 limit 30dB, channel 128 limit 30dB, channel 132 limit 30dB, channel 136 limit 30dB, channel 140 limit 30dB, channel 149 limit 23dB, channel 153 limit 23dB, channel 157 limit 23dB, channel 161 limit 23dB, channel 165 limit 23dB, power constraint 0dB, tpcreport 0x0b00, rsn=, 9 stations, 14% utilization, admission capacity 0us/s, 71:1 0x04, htcaps=<20/40MHz,LDPC,SGI@20MHz,SGI@40MHz,TXSTBC,RXSTBC 1 stream,A-MSDU 7935,A-MPDU max 65535,A-MPDU spacing 4.00us,RxMCS 0xff000100>, htop=<40MHz chan 36:40,RIFS,htprot none,non-greenfield STA,basic MCS set 0x>, 127:8 0x04000140, 191:12 0x92018003eaffeaff, 192:5 0x002600, 195:3 0x010202, vendor 0x00904c0407, vendor 0x0010180209001c, vendor 0x0050f202000100, vendor 0x001977010603140d04175cf482000a8894ae7735c413e21d478026d8c413e21d47a01803085af60227e13e0808030427a309120064000100, vendor 0x0019772101000c535243452d41502d53313500,
Re: iwx(4) 40MHz channel support
On 12.10.2021. 14:47, Stefan Sperling wrote: > This patch adds support for 40MHz channels to iwx(4). > > Please sync your source tree before attempting to apply this patch. > I have committed some changes to this driver today which this patch > is based on. > > Works for me on AX200/AX201. Does anyone else want to do a pre-commit test? with this diff i'm getting 150Mbps vs 100Mbps without it here at home ... will test it at work ... Thank you ..
Re: WIP iwx(4) Tx aggregation
On 29.7.2021. 18:05, Stefan Sperling wrote: > This is an updated patch which has been rebased on top of -current. > Make sure that your tree is fully synced up to r1.86 of if_iwx.c before > applying this patch. > > This patch includes the small change to ieee80211_input.c I sent here: > https://marc.info/?l=openbsd-tech=162757206707535=2 > > Tests would be very welcome. I cannot test laptop use cases right now > because my AX200 is currently plugged into an APU board. With this diff i'm getting same as before. 25/5 Mbps without and 100/75 with diff ... iwx0 at pci3 dev 0 function 0 "Intel Wi-Fi 6 AX200" rev 0x1a, msix Thank you
Re: forwarding in parallel ipsec workaround
On 23.7.2021. 16:20, Vitaliy Makkoveev wrote: > On Thu, Jul 22, 2021 at 11:30:02PM +0200, Hrvoje Popovski wrote: >> On 22.7.2021. 22:52, Vitaliy Makkoveev wrote: >>> On Thu, Jul 22, 2021 at 08:38:04PM +0200, Hrvoje Popovski wrote: >>>> On 22.7.2021. 12:21, Hrvoje Popovski wrote: >>>>> Thank you for explanation.. >>>>> >>>>> after hitting box all night, box panic and i was able to reproduce it >>>>> this morning ... I'm not sure but box panic after hour or more of >>>>> sending traffic through iked tunnel .. >>>>> I will try to reproduce it through ipsec tunnel .. >>>> >>>> with isakmpd i've got panic after 5 or more hours >>>> >>>> r620-1# uvm_fault(0x822b2158, 0x137, 0, 2) -> e >>>> kernel: page fault trap, code=0 >>>> Stopped at tdb_free+0x9c: movq%rsi,0(%rdi) >>>> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND >>>> 469292 75215 680x10 02 isakmpd >>>> *142487 42745 0 0x14000 0x42000K softclock >>>> tdb_free(8118f6e0) at tdb_free+0x9c >>>> tdb_timeout(8118f6e0) at tdb_timeout+0x77 >>>> softclock_thread(8000efc0) at softclock_thread+0x16e >>>> end trace frame: 0x0, count: 12 >>>> https://www.openbsd.org/ddb.html describes the minimum info required in >>>> bug reports. Insufficient info makes it difficult to find and fix bugs. >>>> >>> There is the bluhm@'s diff with serialized `ipsec_in_use' access. Can >>> you try it? >> >> Hi, >> >> I'm running this diff with "ipsec crypto no queue" and i'm seeing >> traffic drops. This is little strange because "ipsec crypto no queue" >> diff take care of those drops... >> > > Thanks! > > Did you caught panics? > Could you share "netstat -s -p ipsec" output? > Hi, box didn't panic, just stopped forwarding traffic through tunnel. ipsec: 0 input IPsec packets 3299331 output IPsec packets 0 input bytes 1781638740 output bytes 0 input bytes, decompressed 2235858152 output bytes, uncompressed 0 packets dropped on input 1030761 packets dropped on output 0 packets that failed crypto processing 0 packets for which no XFORM was set in TDB received 447309 packets for which no TDB was found it still little strange to me that with "ipsec crypto no queue" diff i see traffic drops ... so i will do all over again just to be sure that i'm not doing something wrong ..
Re: forwarding in parallel ipsec workaround
On 22.7.2021. 22:52, Vitaliy Makkoveev wrote: > On Thu, Jul 22, 2021 at 08:38:04PM +0200, Hrvoje Popovski wrote: >> On 22.7.2021. 12:21, Hrvoje Popovski wrote: >>> Thank you for explanation.. >>> >>> after hitting box all night, box panic and i was able to reproduce it >>> this morning ... I'm not sure but box panic after hour or more of >>> sending traffic through iked tunnel .. >>> I will try to reproduce it through ipsec tunnel .. >> >> with isakmpd i've got panic after 5 or more hours >> >> r620-1# uvm_fault(0x822b2158, 0x137, 0, 2) -> e >> kernel: page fault trap, code=0 >> Stopped at tdb_free+0x9c: movq%rsi,0(%rdi) >> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND >> 469292 75215 680x10 02 isakmpd >> *142487 42745 0 0x14000 0x42000K softclock >> tdb_free(8118f6e0) at tdb_free+0x9c >> tdb_timeout(8118f6e0) at tdb_timeout+0x77 >> softclock_thread(8000efc0) at softclock_thread+0x16e >> end trace frame: 0x0, count: 12 >> https://www.openbsd.org/ddb.html describes the minimum info required in >> bug reports. Insufficient info makes it difficult to find and fix bugs. >> > There is the bluhm@'s diff with serialized `ipsec_in_use' access. Can > you try it? Hi, I'm running this diff with "ipsec crypto no queue" and i'm seeing traffic drops. This is little strange because "ipsec crypto no queue" diff take care of those drops...
Re: forwarding in parallel ipsec workaround
On 22.7.2021. 12:21, Hrvoje Popovski wrote:
> Thank you for explanation..
>
> after hitting box all night, box panic and i was able to reproduce it
> this morning ... I'm not sure but box panic after hour or more of
> sending traffic through iked tunnel ..
> I will try to reproduce it through ipsec tunnel ..
with isakmpd i've got panic after 5 or more hours
r620-1# uvm_fault(0x822b2158, 0x137, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at tdb_free+0x9c: movq%rsi,0(%rdi)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
469292 75215 680x10 02 isakmpd
*142487 42745 0 0x14000 0x42000K softclock
tdb_free(8118f6e0) at tdb_free+0x9c
tdb_timeout(8118f6e0) at tdb_timeout+0x77
softclock_thread(8000efc0) at softclock_thread+0x16e
end trace frame: 0x0, count: 12
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
75215 469292 76137 68 70x10isakmpd
76137 414524 1 0 30x80 netio isakmpd
96813 295173 1 0 30x100083 ttyin ksh
19120 299871 1 0 30x100098 poll cron
1532 13552 38673 95 30x100092 kqreadsmtpd
26046 463899 38673103 30x100092 kqreadsmtpd
36828 266802 38673 95 30x100092 kqreadsmtpd
70875 110170 38673 95 30x100092 kqreadsmtpd
94519 66859 38673 95 30x100092 kqreadsmtpd
82071 306235 38673 95 30x100092 kqreadsmtpd
38673 196258 1 0 30x100080 kqreadsmtpd
32975 338908 1 0 30x88 selectsshd
83970 285872 1 0 30x100080 poll ntpd
13501 277269 28135 83 30x100092 poll ntpd
28135 154371 1 83 30x100092 poll ntpd
46277 200739 37910 73 30x100090 kqreadsyslogd
37910 178152 1 0 30x100082 netio syslogd
15169 243641 0 0 3 0x14200 bored smr
72867 54384 0 0 3 0x14200 pgzerozerothread
96747 464994 0 0 3 0x14200 aiodoned aiodoned
21420 278055 0 0 3 0x14200 syncerupdate
4733 35757 0 0 3 0x14200 cleaner cleaner
65184 477103 0 0 3 0x14200 reaperreaper
74689 248130 0 0 3 0x14200 pgdaemon pagedaemon
63567 184653 0 0 3 0x14200 bored crynlk
25576 473224 0 0 3 0x14200 bored crypto
59824 205269 0 0 3 0x14200 usbtskusbtask
79880 326022 0 0 3 0x14200 usbatsk usbatsk
92636 507494 0 0 3 0x40014200 acpi0 acpi0
1561 284019 0 0 7 0x40014200idle5
71732 313587 0 0 7 0x40014200idle4
50021 333287 0 0 7 0x40014200idle3
87783 242752 0 0 3 0x40014200idle2
48193 69868 0 0 7 0x40014200idle1
80238 196627 0 0 3 0x14200 bored sensors
36233 88931 0 0 3 0x14200 netlock softnet
76181 390400 0 0 3 0x14200 netlock softnet
12911 400662 0 0 3 0x14200 netlock softnet
23428 46363 0 0 3 0x14200 netlock softnet
22304 109508 0 0 3 0x14200 bored systqmp
97244 473960 0 0 3 0x14200 bored systq
*42745 142487 0 0 7 0x40014200softclock
7369 374311 0 0 3 0x40014200idle0
1 360880 0 0 30x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb{0}>
ddb{0}> trace /t 0t142487
alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b
tdb_free(8118f6e0) at tdb_free+0x9c
tdb_timeout(8118f6e0) at tdb_timeout+0x77
softclock_thread(8000efc0) at softclock_thread+0x16e
end trace frame: 0x0, count: -4
ddb{0}>
ddb{0}> trace /t 0t469292
sleep_finish(800023906db0,1) at sleep_finish+0x11c
rw_enter(82187ca8,1) at rw_enter+0x1c2
solock(fd83b1019e10) at solock+0x4b
soo_poll(fd83b1989a68,1,8000238fb7a0) at soo_poll+0x34
selscan(8000238fb7a0,800023906fd0,800023906fdc,14,4,8000239071b
0) at selscan+0x14b
dopselect(8000238fb7a0,15,bf1473583e0,bf14736a140,0,8000239070e0) at
dopselect+0x487
sys_pselect(8000238fb7a0,800023907150,8000239071b0) at
sys_pselect+0xdb
syscall(800023907220) at syscall+0x3a9
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7d45b0, count: -9
ddb{0}>
Re: forwarding in parallel ipsec workaround
On 22.7.2021. 0:39, Alexander Bluhm wrote:
> On Thu, Jul 22, 2021 at 12:06:09AM +0200, Hrvoje Popovski wrote:
>> I'm combining this and last parallel diff and i can't see any drops in
>> traffic. Even sending at high rate, traffic through iked or isakmpd is
>> stable at 150Kpps, which is good ..
>
> Thanks, good news.
>
>> One funny thing is that with top -SHs1 crypto CPU usage is always at 0.00%
>> Could it be because of aes-ni?
>
> I don't use the crypto thread anymore. It is idle. All crypto is
> done in softnet between handling the network part of ipsec.
>
> Here is the flame graph. Crypto is running on top of network.
> http://bluhm.genua.de/perform/results/current/patch-sys-ipsec-noqueue.1/btrace/ssh_perform%40lt13_iperf3_-c10.4.56.36_-P10_-t10-btrace-kstack.0.svg
>
> bluhm
>
Thank you for explanation..
after hitting box all night, box panic and i was able to reproduce it
this morning ... I'm not sure but box panic after hour or more of
sending traffic through iked tunnel ..
I will try to reproduce it through ipsec tunnel ..
r620-1# uvm_fault(0x821a82d8, 0x137, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at tdb_free+0x9c: movq%rsi,0(%rdi)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
*310284 77290 0 0x14000 0x2001K softnet
tdb_free(811832a8) at tdb_free+0x9c
esp_output(fd80a5c2f300,811832a8,0,14,9) at esp_output+0x44f
ipsp_process_packet(fd80a5c2f300,811832a8,2,0) at
ipsp_process_packet+0x466
ip_output_ipsec_send(811832a8,fd80a5c2f300,800023871cf8,1)
at ip_output_ipsec_send+0x161
ip_output(fd80a5c2f300,0,800023871cf8,1,0,0) at ip_output+0x8bd
ip_forward(fd80a5c2f300,80087048,fd83b3454bd0,0) at
ip_forward+0x26a
ip_input_if(800023871e38,800023871e44,4,0,80087048) at
ip_input_if+0x353
ipv4_input(80087048,fd80a5c2f300) at ipv4_input+0x39
if_input_process(80087048,800023871eb8) at if_input_process+0x92
ifiq_process(80086c00) at ifiq_process+0x69
taskq_thread(8002f200) at taskq_thread+0x81
end trace frame: 0x0, count: 4
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{1}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
94310 423919 1 0 30x100083 ttyin ksh
77465 202466 1 0 30x100098 poll cron
65594 481620 35139 95 30x100092 kqreadsmtpd
2326 44364 35139103 30x100092 kqreadsmtpd
1225 215234 35139 95 30x100092 kqreadsmtpd
69590 400378 35139 95 30x100092 kqreadsmtpd
7267 26043 35139 95 30x100092 kqreadsmtpd
38120 331881 35139 95 30x100092 kqreadsmtpd
35139 66391 1 0 30x100080 kqreadsmtpd
79402 86611 1 0 30x88 selectsshd
43297 495028 53834101 30x100010 netlock iked
73854 239338 53834101 30x100090 kqreadiked
79322 243271 53834101 30x100090 kqreadiked
53834 169269 1 0 30x100080 kqreadiked
98497 240479 1 0 30x100080 poll ntpd
93883 433035 46155 83 30x100092 poll ntpd
46155 241594 1 83 30x100092 poll ntpd
60135 75740 14081 73 30x100090 kqreadsyslogd
14081 390346 1 0 30x100082 netio syslogd
95646 201039 0 0 3 0x14200 bored smr
54746 191914 0 0 3 0x14200 pgzerozerothread
42039 120601 0 0 3 0x14200 aiodoned aiodoned
23202 404058 0 0 3 0x14200 syncerupdate
32492 255945 0 0 3 0x14200 cleaner cleaner
63792 183728 0 0 3 0x14200 reaperreaper
85251 251516 0 0 3 0x14200 pgdaemon pagedaemon
70650 399091 0 0 3 0x14200 bored crynlk
93527 443884 0 0 3 0x14200 bored crypto
98145 39889 0 0 3 0x14200 usbtskusbtask
67334 434779 0 0 3 0x14200 usbatsk usbatsk
86238 520690 0 0 3 0x40014200 acpi0 acpi0
52180 314140 0 0 7 0x40014200idle5
23611 433449 0 0 7 0x40014200idle4
40949 492122 0 0 7 0x40014200idle3
1758 355637 0 0 7 0x40014200idle2
24269 99037 0 0 3 0x40014200idle1
26630 422993 0 0 3 0x14200 bored sensors
*77290 310284 0 0 7 0x14200softnet
68084 469615 0 0 3 0x1420
Re: forwarding in parallel ipsec workaround
On 21.7.2021. 22:21, Alexander Bluhm wrote: > Ahh, to many diffs in my tree. I have forgotten the cunk > crp->crp_flags = ... | CRYPTO_F_NOQUEUE > > Try this. Still testing it myself, it looks a bit faster. I'm combining this and last parallel diff and i can't see any drops in traffic. Even sending at high rate, traffic through iked or isakmpd is stable at 150Kpps, which is good .. One funny thing is that with top -SHs1 crypto CPU usage is always at 0.00% Could it be because of aes-ni? r620-1# cat /etc/ipsec.conf ike active esp from 192.168.232.0/24 to 192.168.123.0/24 \ local 192.168.42.1 peer 192.168.42.111 \ main auth hmac-sha1 enc aes group modp1024 \ quick enc aes-128-gcm group modp1024 \ psk "123" r620-1# cat /etc/iked.conf ikev2 active esp from 192.168.232.0/24 to 192.168.123.0/24 \ local 192.168.42.1 peer 192.168.42.111 \ ikesa enc aes-128-gcm group modp1024 prf hmac-sha1 \ childsa enc aes-128-gcm group modp1024 \ psk "123"
Re: forwarding in parallel ipsec workaround
On 21.7.2021. 18:41, Alexander Bluhm wrote: > On Mon, Jul 19, 2021 at 07:33:55PM +0300, Vitaliy Makkoveev wrote: >> Hi, pipex(4) is also not ready for parallel access. In the chunk below >> it will be accessed through (*ifp->if_input)() -> ether_input() -> >> pipex_pppoe_input(). This looks not fatal but makes at least session >> statistics inconsistent. > For pipex and pppoe we can put a kernel lock into ether_resolve(). > I also put locks in mpls. > > I am not aware of any other issues. Hi, i've applied this and ipsec crypto no queue diff and i'm getting splasserts below ... maybe it's something obvious, if not, i will try diff by diff .. I'm running iked tunnel ... log on box where traffic goes into tunnel splassert: ipsp_process_done: want 2 have 0 Starting stack trace... ipsp_process_done(fd80a8647400,811832a0) at ipsp_process_done+0x59 esp_output_cb(811832a0,81194e00,fd80a8647400,21c,0) at esp_output_cb+0x41 ipsec_output_cb(fd8397dad440) at ipsec_output_cb+0x177 taskq_thread(8013f480) at taskq_thread+0x81 end trace frame: 0x0, count: 253 End of stack trace. splassert: ip_output: want 2 have 0 Starting stack trace... ip_output(fd80a8647400,0,0,2,0,0) at ip_output+0x88 ipsp_process_done(fd80a8647400,811832a0) at ipsp_process_done+0x384 esp_output_cb(811832a0,81194e00,fd80a8647400,21c,0) at esp_o utput_cb+0x41 ipsec_output_cb(fd8397dad440) at ipsec_output_cb+0x177 taskq_thread(8013f480) at taskq_thread+0x81 end trace frame: 0x0, count: 252 End of stack trace. logs on box where traffic goes from tunnel out splassert: tdb_hash: want 2 have 0 Starting stack trace... tdb_hash(36b50c1c,81199200,32) at tdb_hash+0x5c gettdb_dir(0,36b50c1c,81199200,32,0) at gettdb_dir+0x83 ipsec_input_cb(fd839e0c6ca0) at ipsec_input_cb+0x7a taskq_thread(80140d80) at taskq_thread+0x81 end trace frame: 0x0, count: 253 End of stack trace. splassert: esp_input_cb: want 2 have 0 Starting stack trace... esp_input_cb(8118a5f8,81199200,fd80026f5500,0) at esp_input_cb+0x73 ipsec_input_cb(fd839e0c6ca0) at ipsec_input_cb+0x162 taskq_thread(80140d80) at taskq_thread+0x81 end trace frame: 0x0, count: 254 End of stack trace. splassert: enc_getif: want 2 have 0 Starting stack trace... enc_getif(0,0) at enc_getif+0x54 ipsec_common_input_cb(fd80026f5500,8118a5f8,14,9) at ipsec_common_input_cb+0x934 esp_input_cb(8118a5f8,81199200,fd80026f5500,0) at esp_input_cb+0x456 ipsec_input_cb(fd839e0c6ca0) at ipsec_input_cb+0x162 taskq_thread(80140d80) at taskq_thread+0x81 end trace frame: 0x0, count: 252 End of stack trace.
Re: forwarding in parallel ipsec workaround
On 19.7.2021. 17:53, Alexander Bluhm wrote: > Hi, > > I found why the IPsec workaround did not work. > > At init time we set ifiq->ifiq_softnet = net_tq(ifp->if_index + > idx), but the workaround modifies net_tq() at runtime. Modifying > net_tq() at runtime is bad anyway as task_add() and task_del() could > be called with different task queues. > > So better use exclusive lock if IPsec is in use. For me this is > running stable. Hi, i can't trigger panic with this diff. I've tried with isakmpd and with iked ... With this diff traffic through tunnel seems little slower ...
Re: WIP iwx(4) Tx aggregation
On 30.6.2021. 13:28, Stefan Sperling wrote: > On Mon, Jun 21, 2021 at 08:37:11PM +0200, Stefan Sperling wrote: >> This patch attempts to implement Tx aggregation support for iwx(4). >> >> It is not yet ready to be committed because of outstanding problems: >> >> - Under load the firmware throws a fatal firmware error every few minutes. >> >> - Starting a background scan under load can cause firmware errors and >> might error out when the driver attempts to flush Tx queues. >> However, roaming seems to be generally working while traffic is light. >> >> - Sometimes traffic seems to get stuck for no apparent reason and the driver >> won't recover without down/up. This is independent from the rx_reorder() >> fix which was committed today. > Following my commits to iwx from today, here is a rebased txagg patch. > The above issues are still present, unfortunately. Hi, without this diff i'm getting 25/5 Mb and with that diff 100/75Mb .. Thank you .. e14gen2# ifconfig iwx0 iwx0: flags=8847 mtu 1500 lladdr b0:7d:64:1e:60:e0 index 2 priority 4 llprio 3 groups: wlan egress media: IEEE802.11 autoselect (HT-MCS3 mode 11n) status: active ieee80211: join hrvojewless chan 2 bssid 30:e9:8e:ad:90:3c 67% wpakey wpaprotos wpa2 wpaakms psk wpaciphers ccmp wpagroupcipher ccmp inet 192.168.100.2 netmask 0xff00 broadcast 192.168.100.255 OpenBSD 6.9-current (GENERIC.MP) #2: Tue Jul 13 22:10:50 CEST 2021 hrv...@e14gen2.srce.hr:/sys/arch/amd64/compile/GENERIC.MP real mem = 7742496768 (7383MB) avail mem = 7491899392 (7144MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.2 @ 0xbf913000 (63 entries) bios0: vendor LENOVO version "R1AET37W (1.13 )" date 04/09/2021 bios0: LENOVO 20T6000TSC acpi0 at bios0: ACPI 6.3 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP SSDT SSDT IVRS SSDT SSDT TPM2 SSDT MSDM BATB HPET APIC MCFG SBST WSMT VFCT SSDT CRAT CDIT FPDT SSDT SSDT SSDT BGRT UEFI SSDT SSDT acpi0: wakeup devices GPP3(S3) GPP4(S4) GPP5(S3) XHC0(S3) XHC1(S3) GP19(S3) LID_(S4) SLPB(S3) acpitimer0 at acpi0: 3579545 Hz, 32 bits acpihpet0 at acpi0: 14318180 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 5 4500U with Radeon Graphics, 2370.88 MHz, 17-60-01 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu0: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu0: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=1.1, IBE cpu1 at mainbus0: apid 1 (application processor) cpu1: AMD Ryzen 5 4500U with Radeon Graphics, 2370.55 MHz, 17-60-01 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu1: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu1: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu1: disabling user TSC (skew=-189) cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: AMD Ryzen 5 4500U with Radeon Graphics, 2370.55 MHz, 17-60-01 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu2: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu2: DTLB 64 4KB entries fully associative, 64 4MB entries fully
Re: forwarding in parallel with ipsec panic
On 8.7.2021. 0:10, Vitaliy Makkoveev wrote:
> On Wed, Jul 07, 2021 at 11:07:08PM +0200, Hrvoje Popovski wrote:
>> On 7.7.2021. 22:36, Vitaliy Makkoveev wrote:
>>> Thanks. ipsp_spd_lookup() stopped panic in pool_get(9).
>>>
>>> I guess the panics continue because simultaneous modifications of
>>> 'tdbp->tdb_policy_head' break it. Could you try the diff below? It
>>> introduces `tdb_polhd_mtx' mutex(9) and uses it to protect
>>> 'tdbp->tdb_policy_head' modifications. I don't propose this diff for
>>> commit but to check my suggestion.
>>
>>
>> Hi,
>>
>> with this diff i'm getting this panic
>>
>> r620-1# panic: acquiring blockable sleep lock with spinlock or critical
>> section held (kernel_lock) _lock
>> Stopped at db_enter+0x10: popq%rbp
>> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
>> 375321 87823 0 0x14000 0x2005 crynlk
>> 455594 99250 0 0x14000 0x2000 crypto
>> 124997 16472 0 0x14000 0x2001 softnet
>> 409214 30226 0 0x14000 0x2003 softnet
>> 347403 66039 0 0x14000 0x2004 softnet
>> *345146 25512 0 0x14000 0x2002 softnet
>> db_enter() at db_enter+0x10
>> panic(81e7ce76) at panic+0xbf
>> witness_checkorder(82348dc0,9,0) at witness_checkorder+0xbce
>> __mp_lock(82348bb8) at __mp_lock+0x5f
>> kpageflttrap(800023864a30,147) at kpageflttrap+0x178
>> kerntrap(800023864a30) at kerntrap+0x91
>> alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b
>> ipsp_spd_lookup(fd80a05e9200,2,14,800023864d0c,2,0) at
>> ipsp_spd_lookup+0x9fd
>> ip_output_ipsec_lookup(fd80a05e9200,14,800023864d0c,0,0) at
>> ip_output_ipsec_lookup+0x4d
>> ip_output(fd80a05e9200,0,800023864e98,1,0,0) at ip_output+0x42a
>> ip_forward(fd80a05e9200,80087048,fd83b39799a8,0) at
>> ip_forward+0x26a
>> ip_input_if(800023864fd8,800023864fe4,4,0,80087048) at
>> ip_input_if+0x365
>> ipv4_input(80087048,fd80a05e9200) at ipv4_input+0x39
>> if_input_process(80087048,800023865058) at if_input_process+0x6f
>> end trace frame: 0x8000238650a0, count: 0
>> https://www.openbsd.org/ddb.html describes the minimum info required in
>> bug reports. Insufficient info makes it difficult to find and fix bugs.
>> ddb{2}>
>>
>> ddb{2}> show locks
>> shared rwlock netlock r = 0 (0x8219ce60)
>> #0 witness_lock+0x339
>> #1 if_input_process+0x43
>> #2 ifiq_process+0x69
>> #3 taskq_thread+0x9f
>> #4 proc_trampoline+0x1c
>> shared rwlock softnet r = 0 (0x80030070)
>> #0 witness_lock+0x339
>> #1 taskq_thread+0x92
>> #2 proc_trampoline+0x1c
>> exclusive mutex /sys/netinet/ip_ipsp.c:95 r = 0 (0x82192398)
>> #0 witness_lock+0x339
>> #1 mtx_enter_try+0x95
>> #2 mtx_enter+0x48
>> #3 ipsp_spd_lookup+0x961
>> #4 ip_output_ipsec_lookup+0x4d
>> #5 ip_output+0x42a
>> #6 ip_forward+0x26a
>> #7 ip_input_if+0x365
>> #8 ipv4_input+0x39
>> #9 if_input_process+0x6f
>> #10 ifiq_process+0x69
>> #11 taskq_thread+0x9f
>> #12 proc_trampoline+0x1c
>>
>
> Thanks.
>
> Now panics only in ipsp_spd_lookup() and never in pfkeyv2_send() or in
> tdb_free() called from pfkeyv2_send(), right?
>
Yes,
i can only trigger this panic
r620-1# panic: acquiring blockable sleep lock with spinlock or critical
section held (kernel_lock) _lock
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
340098 10711 0 0x14000 0x2003 crynlk
*173315 77636 0 0x14000 0x2001 softnet
211687 60430 0 0x14000 0x2002 softnet
db_enter() at db_enter+0x10
panic(81e79af6) at panic+0xbf
witness_checkorder(8237dbb8,9,0) at witness_checkorder+0xbce
__mp_lock(8237d9b0) at __mp_lock+0x5f
kpageflttrap(800023871010,147) at kpageflttrap+0x178
kerntrap(800023871010) at kerntrap+0x91
alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b
ipsp_spd_lookup(fd80a1bde300,2,14,8000238712ec,2,0) at
ipsp_spd_lookup+0x9fd
ip_output_ipsec_lookup(fd80a1bde300,14,8000238712ec,0,0) at
ip_output_ipsec_lookup+0x4d
ip_output(fd80a1bde300,0,800023871478,1,0,0) at ip_output+0x42a
ip_forward(fd80a1bde300,80087048,fd83b4060cb8,0) at
ip_forward+0x26a
ip_input_if(8000238715b8,8000238715c4,4,0,80087048) at
ip_input_if+0x365
ipv4_input(80087048,fd80a1bde300) at ipv4_input+0x39
if_input_process(80087048,800023871638) at if_input_process+0x6f
end trace frame: 0x800023871680, count: 0
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
Re: forwarding in parallel with ipsec panic
On 7.7.2021. 22:36, Vitaliy Makkoveev wrote:
> Thanks. ipsp_spd_lookup() stopped panic in pool_get(9).
>
> I guess the panics continue because simultaneous modifications of
> 'tdbp->tdb_policy_head' break it. Could you try the diff below? It
> introduces `tdb_polhd_mtx' mutex(9) and uses it to protect
> 'tdbp->tdb_policy_head' modifications. I don't propose this diff for
> commit but to check my suggestion.
Hi,
with this diff i'm getting this panic
r620-1# panic: acquiring blockable sleep lock with spinlock or critical
section held (kernel_lock) _lock
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
375321 87823 0 0x14000 0x2005 crynlk
455594 99250 0 0x14000 0x2000 crypto
124997 16472 0 0x14000 0x2001 softnet
409214 30226 0 0x14000 0x2003 softnet
347403 66039 0 0x14000 0x2004 softnet
*345146 25512 0 0x14000 0x2002 softnet
db_enter() at db_enter+0x10
panic(81e7ce76) at panic+0xbf
witness_checkorder(82348dc0,9,0) at witness_checkorder+0xbce
__mp_lock(82348bb8) at __mp_lock+0x5f
kpageflttrap(800023864a30,147) at kpageflttrap+0x178
kerntrap(800023864a30) at kerntrap+0x91
alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b
ipsp_spd_lookup(fd80a05e9200,2,14,800023864d0c,2,0) at
ipsp_spd_lookup+0x9fd
ip_output_ipsec_lookup(fd80a05e9200,14,800023864d0c,0,0) at
ip_output_ipsec_lookup+0x4d
ip_output(fd80a05e9200,0,800023864e98,1,0,0) at ip_output+0x42a
ip_forward(fd80a05e9200,80087048,fd83b39799a8,0) at
ip_forward+0x26a
ip_input_if(800023864fd8,800023864fe4,4,0,80087048) at
ip_input_if+0x365
ipv4_input(80087048,fd80a05e9200) at ipv4_input+0x39
if_input_process(80087048,800023865058) at if_input_process+0x6f
end trace frame: 0x8000238650a0, count: 0
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{2}>
ddb{2}> show locks
shared rwlock netlock r = 0 (0x8219ce60)
#0 witness_lock+0x339
#1 if_input_process+0x43
#2 ifiq_process+0x69
#3 taskq_thread+0x9f
#4 proc_trampoline+0x1c
shared rwlock softnet r = 0 (0x80030070)
#0 witness_lock+0x339
#1 taskq_thread+0x92
#2 proc_trampoline+0x1c
exclusive mutex /sys/netinet/ip_ipsp.c:95 r = 0 (0x82192398)
#0 witness_lock+0x339
#1 mtx_enter_try+0x95
#2 mtx_enter+0x48
#3 ipsp_spd_lookup+0x961
#4 ip_output_ipsec_lookup+0x4d
#5 ip_output+0x42a
#6 ip_forward+0x26a
#7 ip_input_if+0x365
#8 ipv4_input+0x39
#9 if_input_process+0x6f
#10 ifiq_process+0x69
#11 taskq_thread+0x9f
#12 proc_trampoline+0x1c
Re: forwarding in parallel with ipsec panic
On 7.7.2021. 19:38, Vitaliy Makkoveev wrote:
> Hi,
>
> It seems the first the first panic occured because ipsp_spd_lookup()
> modifies tdbp->tdb_policy_head and simultaneous execution breaks it.
> I guess at least mutex(9) should be used to protect `tdb_policy_head'.
>
> The second panic occured because ipsp_acquire_sa() does
> `ipsec_acquire_pool' initialization in runtime so parallel execution
> breaks it. It's easy to fix.
>
> Could you try the diff below? It moves `ipsec_acquire_pool'
> initialization to pfkey_init() just after `ipsec_policy_pool'
> initialization. This should fix the second panic.
Hi,
with this diff i manage to get this panics ..
r620-1# uvm_fault(0xfd842f83e170, 0x147, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at tdb_free+0x9c: movq%rsi,0(%rdi)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
* 38448 89106 680x10 01K isakmpd
tdb_free(8131e688) at tdb_free+0x9c
pfkeyv2_send(fd83b2512d30,812f2680,50) at pfkeyv2_send+0x1191
pfkeyv2_output(fd80a4b37400,fd83b2512d30,0,0) at pfkeyv2_output+0x8a
pfkeyv2_usrreq(fd83b2512d30,9,fd80a4b37400,0,0,800023908d20)
at pfkeyv2_usrreq+0x1b0
sosend(fd83b2512d30,0,800023959130,0,0,0) at sosend+0x3a9
dofilewritev(800023908d20,7,800023959130,0,800023959230) at
dofilewritev+0x14d
sys_writev(800023908d20,8000239591d0,800023959230) at
sys_writev+0xe2
syscall(8000239592a0) at syscall+0x3b9
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7b27c0, count: 6
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{1}>
r620-1# uvm_fault(0x82379d80, 0x147, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at ipsp_spd_lookup+0x9a4: movq%rax,0(%rcx)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
434446 37326 0 0x14000 0x2004 softnet
405134 41722 0 0x14000 0x2001 softnet
106389 88948 0 0x14000 0x2003 softnet
*262295 93659 0 0x14000 0x2002 softnet
ipsp_spd_lookup(fd80a44f6f00,2,14,8000238651bc,2,0) at
ipsp_spd_lookup+0x9a4
ip_output_ipsec_lookup(fd80a44f6f00,14,8000238651bc,0,0) at
ip_output_ipsec_lookup+0x4d
ip_output(fd80a44f6f00,0,800023865348,1,0,0) at ip_output+0x42a
ip_forward(fd80a44f6f00,80087048,fd83b3ea6d98,0) at
ip_forward+0x26a
ip_input_if(800023865488,800023865494,4,0,80087048) at
ip_input_if+0x365
ipv4_input(80087048,fd80a44f6f00) at ipv4_input+0x39
if_input_process(80087048,800023865508) at if_input_process+0x6f
ifiq_process(80086c00) at ifiq_process+0x69
taskq_thread(8003) at taskq_thread+0x9f
end trace frame: 0x0, count: 6
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
Re: forwarding in parallel with ipsec panic
On 7.7.2021. 12:46, Hrvoje Popovski wrote:
> Panic can be triggered when i have parallel diff and sending traffic
> over ipsec tunnel and on other side while traffic is flowing i'm
> restarting isakmpd daemon and while negotiating ipsec doing ifconfig ix1
> down && ifconfig ix1 up ... sometimes it panic all by itself.. it's not
> that i know exactly what's going on ..
Different panic on same setup ...
uvm_fault(0x8221de20, 0x8, 0, 1) -> e
kernel: page fault trap, code=0
Stopped at pool_get+0x6e: movq0x8(%rax),%r11
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
288284 51828 0 0x3 00 ifconfig
329904 31282 0 0x14000 0x2004 softnet
157417 71744 0 0x14000 0x2002 softnet
*152664 84791 0 0x14000 0x2003K softnet
58812 4615 0 0x14000 0x2001 softnet
pool_get(82153918,a) at pool_get+0x6e
ipsp_acquire_sa(fd842ae97580,fd842ae9766c,fd842ae97650,80002386b068,ff84)
at ipsp_acquire_sa+0x124
ipsp_spd_lookup(fd80b842db00,2,14,80002386b24c,2,0) at
ipsp_spd_lookup+0xb4e
ip_output_ipsec_lookup(fd80b842db00,14,80002386b24c,0,0) at
ip_output_ipsec_lookup+0x4d
ip_output(fd80b842db00,0,80002386b3d8,1,0,0) at ip_output+0x42a
ip_forward(fd80b842db00,80087048,fd83b3fecb68,0) at
ip_forward+0x26a
ip_input_if(80002386b518,80002386b524,4,0,80087048) at
ip_input_if+0x365
ipv4_input(80087048,fd80b842db00) at ipv4_input+0x39
if_input_process(80087048,80002386b598) at if_input_process+0x6f
ifiq_process(80086d00) at ifiq_process+0x69
taskq_thread(80030100) at taskq_thread+0x9f
end trace frame: 0x0, count: 4
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{3}> show reg
rdi 0x82153918ipsec_acquire_pool
rsi 0xa
rbp 0x80002386af70
rbx0
rdx 0xfd842ae97650
rcx 0x1
rax0
r80xff84
r90x8016a604
r10 0x812c7d40
r11 0x4f581b7438ceb9e9
r12 0xfd842ae9766c
r13 0xfd842ae97580
r14 0x82153918ipsec_acquire_pool
r15 0xa
rip 0x8133e97epool_get+0x6e
cs 0x8
rflags 0x10246__ALIGN_SIZE+0xf246
rsp 0x80002386aed0
ss 0
pool_get+0x6e: movq0x8(%rax),%r11
ddb{3}>
ddb{3}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
51828 288284 1009 0 7 0x3ifconfig
1009 461325 1 0 30x10008b sigsusp ksh
56847 290195 1 0 30x100098 poll cron
83452 510449 26068 95 30x100092 kqreadsmtpd
41700 201644 26068103 30x100092 kqreadsmtpd
26028 130559 26068 95 30x100092 kqreadsmtpd
11949 518693 26068 95 30x100092 kqreadsmtpd
4520 211282 26068 95 30x100092 kqreadsmtpd
82047 205282 26068 95 30x100092 kqreadsmtpd
26068 82309 1 0 30x100080 kqreadsmtpd
46745 316858 1 0 30x88 selectsshd
5198 371736 71443 68 30x90 selectisakmpd
71443 13600 1 0 30x80 netio isakmpd
71184 265525 1 0 30x100080 poll ntpd
26181 322162 3615 83 30x100092 poll ntpd
3615 143908 1 83 30x100092 poll ntpd
82091 209375 43348 73 30x100090 kqreadsyslogd
43348 363890 1 0 30x100082 netio syslogd
16351 58282 0 0 3 0x14200 bored smr
27233 372715 0 0 3 0x14200 pgzerozerothread
68321 430498 0 0 3 0x14200 aiodoned aiodoned
10629 99914 0 0 3 0x14200 syncerupdate
74747 455079 0 0 3 0x14200 cleaner cleaner
67178 305272 0 0 3 0x14200 reaperreaper
54113 244787 0 0 3 0x14200 pgdaemon pagedaemon
88516 52910 0 0 3 0x14200 bored crynlk
84865 523006 0 0 3 0x14200 bored crypto
87886 398410 0 0 3 0x14200 usbtskusbtask
79482 516951 0 0 3 0x14200 usbatsk usbatsk
7554 475208 0 0 3 0x40014200 acpi0 acpi0
55853 38718 0 0 7 0x40014200idle5
13965 462123 0 0 3 0x4
forwarding in parallel with ipsec panic
Hi,
i don't want to pollute bluhm@ parallel forwarding mail on tech@ so i'm
sending this report as a separate thread. this panic it's dependent on
bluhm@ parallel diff ... and i found it yesterday
I'm having ipsec tunnel between two hosts without pf and i'm sending
traffic over that tunnel .. i'm sending 200Kpps
With plain kernel every 20 sec or so traffic stops and i need to run
ifconfig ix1 down && ifconfig ix1 up to flow again ..
When traffic stops with "vmstat -m" i'm seeing Fail mcl2k2,mcl2k
counters ...
And same behavior is with parallel diff ...
It seems that this is not big problem but i just wanted to describe what
i'm seeing ..
Panic can be triggered when i have parallel diff and sending traffic
over ipsec tunnel and on other side while traffic is flowing i'm
restarting isakmpd daemon and while negotiating ipsec doing ifconfig ix1
down && ifconfig ix1 up ... sometimes it panic all by itself.. it's not
that i know exactly what's going on ..
i couldn't trigger panic without parallel diff ..
r620-1 is box that panic and ddb output is in attachment
ipsec configuration:
r620-1# cat /etc/ipsec.conf
ike esp from 192.168.232.0/24 to 192.168.123.0/24 \
local 192.168.42.1 peer 192.168.42.111 \
main auth hmac-sha1 enc aes group modp1024 \
quick enc aes-128-gcm group modp1024 \
psk "123"
r620-1# ipsecctl -sa
FLOWS:
flow esp in from 192.168.123.0/24 to 192.168.232.0/24 peer
192.168.42.111 srcid 192.168.4
2.1/32 dstid 192.168.42.111/32 type require
flow esp out from 192.168.232.0/24 to 192.168.123.0/24 peer
192.168.42.111 srcid 192.168.
42.1/32 dstid 192.168.42.111/32 type require
SAD:
esp tunnel from 192.168.42.111 to 192.168.42.1 spi 0x9710a75f enc
aes-128-gcm
esp tunnel from 192.168.42.1 to 192.168.42.111 spi 0xa5484a26 enc
aes-128-gcm
r620-1# ifconfig ix
ix0: flags=8843 mtu 1500
lladdr ec:f4:bb:da:f7:f8
index 1 priority 0 llprio 3
media: Ethernet autoselect (10GSFP+Cu full-duplex,rxpause,txpause)
status: active
inet 192.168.42.1 netmask 0xff00 broadcast 192.168.42.255
inet6 fe80::eef4:bbff:feda:f7f8%ix0 prefixlen 64 scopeid 0x1
inet6 2001:db8:42::1 prefixlen 64
ix1: flags=8843 mtu 1500
lladdr ec:f4:bb:da:f7:fa
index 2 priority 0 llprio 3
media: Ethernet autoselect (10GSFP+Cu full-duplex,rxpause,txpause)
status: active
inet 192.168.232.1 netmask 0xff00 broadcast 192.168.232.255
x3550m4# cat /etc/ipsec.conf
ike esp from 192.168.123.0/24 to 192.168.232.0/24 \
local 192.168.42.111 peer 192.168.42.1 \
main auth hmac-sha1 enc aes group modp1024 \
quick enc aes-128-gcm group modp1024 \
psk "123"
x3550m4# ipsecctl -sa
FLOWS:
flow esp in from 192.168.232.0/24 to 192.168.123.0/24 peer 192.168.42.1
srcid 192.168.42.111/32 dstid 192.168.42.1/32 type require
flow esp out from 192.168.123.0/24 to 192.168.232.0/24 peer 192.168.42.1
srcid 192.168.42.111/32 dstid 192.168.42.1/32 type require
x3550m4# ifconfig ix
ix0: flags=8843 mtu 1500
lladdr a0:36:9f:2e:96:a0
index 5 priority 0 llprio 3
media: Ethernet autoselect (10GSFP+Cu full-duplex,rxpause,txpause)
status: active
inet 192.168.42.111 netmask 0xff00 broadcast 192.168.42.255
ix1: flags=8843 mtu 1500
lladdr a0:36:9f:2e:96:a1
index 6 priority 0 llprio 3
media: Ethernet autoselect (10GSFP+Cu full-duplex,rxpause,txpause)
status: active
inet 192.168.123.1 netmask 0xff00 broadcast 192.168.123.255
r620-1# uvm_fault(0xfd842f83e450, 0x147, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at tdb_free+0x9c: movq%rsi,0(%rdi)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
*238511 80709 680x10 04 isakmpd
391606 55892 0 0x14000 0x2005 softnet
135192 35596 0 0x14000 0x2002 softnet
211135 99090 0 0x14000 0x2000 softnet
tdb_free(812e6ad8) at tdb_free+0x9c
pfkeyv2_send(fd83b2ab4970,812bc600,50) at pfkeyv2_send+0x1191
pfkeyv2_output(fd80a19cf200,fd83b2ab4970,0,0) at pfkeyv2_output+0x8a
pfkeyv2_usrreq(fd83b2ab4970,9,fd80a19cf200,0,0,800023908d20) at
pfkeyv2_usrreq+0x1b0
sosend(fd83b2ab4970,0,800023959890,0,0,0) at sosend+0x3a9
dofilewritev(800023908d20,7,800023959890,0,800023959990) at
dofilewritev+0x14d
sys_writev(800023908d20,800023959930,800023959990) at
sys_writev+0xe2
syscall(800023959a00) at syscall+0x3b9
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7aeea0, count: 6
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{4}>
ddb{4}> show reg
rdi0x147
rsi 0x
rbp 0x800023959360
rbx 0x812bc610
rdx
Re: ifnewlladdr spl
On 29.6.2021. 23:05, Alexander Bluhm wrote: > On Tue, Jun 29, 2021 at 10:39:14PM +0200, Hrvoje Popovski wrote: >> with this diff without any traffic through aggr if i destroy aggr >> interface i'm getting log below ... log can't be reproduced after first >> destroy.. you need to reboot box and then destroy aggr ... >> i can't reproduce it with kernel from snapshot .. > Thanks for testing. I tried same stuff with trunk(4) and changing mac addr on physical interfaces and with or without traffic and machine seems just fine ..
Re: ifnewlladdr spl
On 29.6.2021. 19:19, Alexander Bluhm wrote: > So what to do with this diff? > > - OK to commit? > - Test it in snaps? > - Call for testers? > > I it would be interesting if the kernel is stable when trunk or > aggr interfaces are created or destroyed while the machine is under > network load. Changing the MAC address of a physical interface > should also run through this code path. Hi, with this diff without any traffic through aggr if i destroy aggr interface i'm getting log below ... log can't be reproduced after first destroy.. you need to reboot box and then destroy aggr ... i can't reproduce it with kernel from snapshot .. r620-1# ifconfig aggr0 destroy witness: lock order reversal: 1st 0x821442a8 netlock (netlock) 2nd 0x8211d2e8 timeout (timeout) lock order "timeout"(rwlock) -> "netlock"(rwlock) first seen at: #0 rw_enter_write+0x43 #1 mld6_fasttimeo+0x14 #2 pffasttimo+0x97 #3 timeout_run+0x93 #4 softclock_thread+0x11d #5 proc_trampoline+0x1c lock order "netlock"(rwlock) -> "timeout"(rwlock) first seen at: #0 timeout_del_barrier+0x41 #1 aggr_p_dtor+0x17b #2 aggr_clone_destroy+0x91 #3 if_clone_destroy+0xd8 #4 ifioctl+0x1d2 #5 soo_ioctl+0x167 #6 sys_ioctl+0x2c4 #7 syscall+0x3b9 #8 Xsyscall+0x128 i will play with mac addresses and if if anything comes up, i'll let you know
Re: [External] : Re: if_etherbridge.c vs. parallel forwarding
On 25.6.2021. 10:02, Alexandr Nedvedicky wrote: > Hello David, > > >> >> during the drive to work it occurred to me that we should basically have >> the same logic around whether we should insert or replace or do nothing >> in both the smr and mutex critical sections. >> >> it at least makes the code easier to understand. i think? > > yes, the new diff is easier to reads. I would not call it dumb, just > 'easier to read' sounds more appropriate. > > I think combination of ebt_find() and ebt_insert() (price doubles, when > inserting a new entry) is acceptable cost for extra diagnostic panic. > We can always change it later. I'm OK if it will get committed as-is. > > OK sashan > Hi, with this diff i can't trigger panic what ever i do... but it could be that i'm not very creative right now :)
Re: limit MSR_INT_PEN_MSG use to < family 16h
On 10.6.2021. 8:17, Jonathan Gray wrote:
> On Wed, Jun 09, 2021 at 10:35:48PM -0700, Mike Larkin wrote:
>> On Thu, Jun 10, 2021 at 03:19:43PM +1000, Jonathan Gray wrote:
>>> Ilya Voronin sent a diff to misc to limit MSR_INT_PEN_MSG use to
>>> < AMD family 17h prompted by a problem with an AWS t3a instance.
>>>
>>> https://marc.info/?l=openbsd-misc=162120066715633=2
>>>
>>> Digging some more the 16h bkdgs have it as RAZ/non-functional as well.
>>> Bits are documented in 15h.
>>>
>>> BKDG for AMD Family 16h Models 00h-0Fh Processors
>>> MSRC001_0055 Interrupt Pending
>>> 63:0 RAZ.
>>>
>>> BKDG for AMD Family 16h Models 30h-3Fh Processors
>>> MSRC001_0055 Interrupt Pending
>>> 63:0 RAZ
>>>
>>> PPR for AMD Family 17h Model 71h B0
>>> MSRC001_0055 [Reserved.] (Core::X86::Msr::IntPend)
>>> Read-only. Reset: Fixed,___h.
>>>
>>> Change the test to use extended family id while here.
>>>
>>
>> I'd be ok with this if someone reported that it works on a bare metal EPYC,
>> since the fix here is for a virtualized environment (and we don't know what
>> AWS is doing here).
>
> I don't have an epyc but this boots with a mp kernel on
> cpu0: AMD Ryzen 5 2600X Six-Core Processor, 3593.93 MHz, 17-08-02
>
> The c1e timer stop problem / erratum 400 should only be a problem
> on families 0xf and 0x10.
>
> Index: sys/arch/amd64/amd64/lapic.c
> ===
> RCS file: /cvs/src/sys/arch/amd64/amd64/lapic.c,v
> retrieving revision 1.57
> diff -u -p -r1.57 lapic.c
> --- sys/arch/amd64/amd64/lapic.c 6 Sep 2020 20:50:00 - 1.57
> +++ sys/arch/amd64/amd64/lapic.c 10 Jun 2021 06:13:38 -
> @@ -299,8 +299,7 @@ lapic_set_lvt(void)
>*Family 0Fh Processors"
>* #32559 revision 3.00
>*/
> - if ((cpu_id & 0x0f00) == 0x0f00 &&
> - (cpu_id & 0x0fff) >= 0x0004) {
> + if (ci->ci_family == 0xf || ci->ci_family == 0x10) {
> uint64_t msr;
>
> msr = rdmsr(MSR_INT_PEN_MSG);
> Index: sys/arch/i386/i386/lapic.c
> ===
> RCS file: /cvs/src/sys/arch/i386/i386/lapic.c,v
> retrieving revision 1.47
> diff -u -p -r1.47 lapic.c
> --- sys/arch/i386/i386/lapic.c30 Jul 2018 14:19:12 - 1.47
> +++ sys/arch/i386/i386/lapic.c10 Jun 2021 06:13:21 -
> @@ -160,8 +160,7 @@ lapic_set_lvt(void)
>*Family 0Fh Processors"
>* #32559 revision 3.00
>*/
> - if ((cpu_id & 0x0f00) == 0x0f00 &&
> - (cpu_id & 0x0fff) >= 0x0004) {
> + if (ci->ci_family == 0xf || ci->ci_family == 0x10) {
> uint64_t msr;
>
> msr = rdmsr(MSR_INT_PEN_MSG);
>
Hi all,
i've applied this diff on Dell r7515 and it seems to work just fine ..
dmesg after diff
OpenBSD 6.9-current (GENERIC.MP) #17: Thu Jun 10 08:33:42 CEST 2021
hrv...@r7515.asd.dsa:/sys/arch/amd64/compile/GENERIC.MP
real mem = 549314162688 (523866MB)
avail mem = 532650774528 (507975MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.3 @ 0x697a5000 (72 entries)
bios0: vendor Dell Inc. version "2.1.6" date 04/09/2021
bios0: Dell Inc. PowerEdge R7515
acpi0 at bios0: ACPI 6.0
acpi0: sleep states S0 S5
acpi0: tables DSDT FACP BERT HEST HPET APIC MCFG WSMT SLIC SSDT SSDT
EINJ SSDT CRAT CDIT IVRS SSDT
acpi0: wakeup devices PC00(S5) XHCI(S3) PC01(S5) XHCI(S3) PC02(S5)
XHCI(S3) PC03(S5) XHCI(S3)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihpet0 at acpi0: 14318180 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
ioapic0 at mainbus0: apid 240 pa 0xfec0, version 21, 24 pins, can't
remap
ioapic1 at mainbus0: apid 241 pa 0xe010, version 21, 32 pins, can't
remap
ioapic2 at mainbus0: apid 242 pa 0xc510, version 21, 32 pins, can't
remap
ioapic3 at mainbus0: apid 243 pa 0xaa10, version 21, 32 pins, can't
remap
ioapic4 at mainbus0: apid 244 pa 0xfd10, version 21, 32 pins, can't
remap
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD EPYC 7702P 64-Core Processor, 1996.52 MHz, 17-31-00
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu0: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB
64b/line 8-way L2 cache
cpu0: ITLB 64 4KB entries fully associative, 64 4MB entries fully
associative
cpu0: DTLB 64 4KB entries fully associative,
Re: [External] : Re: parallel forwarding vs. bridges
On 7.6.2021. 9:25, Alexandr Nedvedicky wrote:
> Hello,
>
> On Sun, Jun 06, 2021 at 09:54:50PM +0200, Hrvoje Popovski wrote:
>
>>
>> this one?
> yes, exactly this one.
Ok, great ..
this and similar panics are products of accidental stp loops. I have two
same boxes with the same network setup connected to the same switch. At
one point i've tested bluhm@ and sashan@ diffs and forget that i have
tpmr(4) or veb(4) configured on both boxes, which is loop ...
So now, when i understood what i've done i can easily reproduce panic
with yours and dlg@ diff ... i just start to send traffic over one box
and other box panic :)
all those panics are without pf and with tpmr or veb
here's few panics .. i can send details if you find some panics
interesting ..
login: papnapiniacc:n: i kc e: r n e l di a g n o s ti
ca s s e
r t i o n " r v e b e = =N UL L "f a i
l e d :f
i l e"
p /okseyoslr/_ndenole t_ /piu ft : _ e t h e rb r i dg e .
c " , li
n e22 6e b t_ r e p la c ee b 0 x
f f f f 8
0e0bd0eipa0lg0n6:8 o2 s te i6 c8n e b e 0 x f f f ff
d 8 3 ab 7
7 b d 9 8 r v e b e 0 x f f f ff d 8 3 a b
7 7 b e 1 0
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
91272 9 0 0x14000 0x2001 softnet
*410720 68055 0 0x14000 0x2000 softnet
444532 44174 0 0x14000 0x2002 softnet
149723 5636 0 0x14000 0x2004 softnet
db_enter() at db_enter+0x10
panic(81e85119) at panic+0xbf
etherbridge_map(80682e68,8016ad00,90e2ba33b4a0) at
etherbridge_map+0x37d
veb_port_input(80082048,fd80bcb8f600,90e2ba33b4a1,8016ad00)
at veb_port_input+0x2fa
ether_input(80082048,fd80bcb8f600) at ether_input+0xf5
if_input_process(80082048,800023871988) at if_input_process+0x6f
ifiq_process(80080f00) at ifiq_process+0x69
taskq_thread(80030200) at taskq_thread+0x9f
end trace frame: 0x0, count: 7
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}> show panic
cpu0: kernel diagnostic assertion "rvebe == NULL" failed: file
"/sys/net/if_etherbridge.c", line 226 ebt_replace eb 0x80682e68
nebe 0xfd83ab77bd98 rvebe 0xfd83ab77be10
cpu2: pool_do_put: ebepl: double pool_put: 0xfd83ab77b8e8
*cpu1: kernel diagnostic assertion "smr->smr_func == NULL" failed: file
"/sys/kern/kern_smr.c", line 247
ddb{0}>
r620-1# papnainicc: : k er n eld ia g n o st i c a s s e r ti
o n " r e f c n
t ! =~0 "f a i le d : f il e" /s y s / k e r
n/kkeernrenl_ s yn c h . c ",l i ne 8 2 6
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
73487 28509 0 0x14000 0x2001 softnet
99803 26490 0 0x14000 0x2005 softnet
*372376 10060 0 0x14000 0x2004 softnet
128194 68537 0 0x14000 0x2003 softnet
105333 38464 0 0x14000 0x42000 softclock
db_enter() at db_enter+0x10
panic(81e07adc) at panic+0xbf
__assert(81e7558b,81de2e62,33a,81e13185) at
__assert+0x2b
refcnt_rele(fd839484c768) at refcnt_rele+0x6f
etherbridge_map(80682e68,812c6f00,90e2ba33b4a0) at
etherbridge_map+0x1ca
veb_port_input(80087048,fd80a6545200,90e2ba33b4a1,812c6f00)
at veb_port_input+0x2fa
ether_input(80087048,fd80a6545200) at ether_input+0xf5
if_input_process(80087048,80002386bba8) at if_input_process+0x6f
ifiq_process(80086d00) at ifiq_process+0x69
taskq_thread(80030100) at taskq_thread+0x9f
end trace frame: 0x0, count: 5
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{4}> show panic
cpu4: kernel diagnostic assertion "refcnt != ~0" failed: file
"/sys/kern/kern_synch.c", line 826
*cpu1: kernel diagnostic assertion "smr->smr_func == NULL" failed: file
"/sys/kern/kern_smr.c", line 247
ddb{4}>
r620-1# papppnpppannnanniiniiciicc::c:cc : :: poo
l_d
o _ge t: mc l 2k
2:p
a ge emp ty
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
* 13147 52870 0 0x14000 0x2002 softnet
db_enter() at db_enter+0x10
pani
Re: [External] : Re: parallel forwarding vs. bridges
On 5.6.2021. 18:43, Alexandr Nedvedicky wrote:
> According to early tests it works well. Currently there is a one
> mysterious panic, which Hrvoje may be able to comment on
> how to trigger it. The stack looks as follows:
>
> kernel diagnostic assertion "smr->smr_func == NULL" failed: file
> "/sys/kern/kern_smr.c", line 247
> panic() at panic+0x12a
> __assert() at
> __assert+0x2b
> smr_call_impl() at smr_call_impl+0xd4
> veb_port_input() at veb_port_input+0x2fa
> ether_input() at ether_input+0x10b
> if_input_process() at if_input_process+0x6f
> ifiq_process() at ifiq_process+0x69
> taskq_thread() at taskq_thread+0x81
this one?
r620-1# ppaanniicc: : k e r ne ld ia g n o s t icas s e r t
io n" rv e b e
= = N U LL "f ai l e d : f i l e " / s y s/ n e
tk/eirnfe_el t he r b r i d g e. c " , l i ne 2 2 6eb t _ r
ep l a ce e b
0 x f f f f 80 0 0 00 6 8 2 e6 8n eb e 0 x f
fdifaffgdno8s39t1iec 9 16 3 0r v eb e 0 x f ff f f d 83 9 1 e
9 1 4c 8
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
218522 97075 0 0x14000 0x2001 softnet
*210501 76472 0 0x14000 0x2005 softnet
192964556 0 0x14000 0x2002 softnet
312196 98971 0 0x14000 0x2003 softnet
db_enter() at db_enter+0x10
panic(81e84b7f) at panic+0xbf
etherbridge_map(80682e68,8016a000,90e2ba33b4a0) at
etherbridge_map+0x37d
veb_port_input(80082048,fd80a6ec8800,90e2ba33b4a1,8016a000)
at veb_port_input+0x2fa
ether_input(80082048,fd80a6ec8800) at ether_input+0xf5
if_input_process(80082048,800023871bf8) at if_input_process+0x6f
ifiq_process(80086300) at ifiq_process+0x69
taskq_thread(80030200) at taskq_thread+0x9f
end trace frame: 0x0, count: 7
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{5}> show panic
cpu5: kernel diagnostic assertion "rvebe == NULL" failed: file
"/sys/net/if_etherbridge.c", line 226 ebt_replace eb 0x80682e68
nebe 0xfd8391e91630 rvebe 0xfd8391e914c8
*cpu2: kernel diagnostic assertion "smr->smr_func == NULL" failed: file
"/sys/kern/kern_smr.c", line 247
ddb{5}>
Re: move copyout() in DIOCGETSTATES outside of NET_LOCK() and state_lcok
On 20.5.2021. 3:23, Alexandr Nedvedicky wrote:
> Hello,
>
> Hrvoje gave a try to experimental diff, which trades rw-locks in pf(4)
> for mutexes [1]. Hrvoje soon discovered machine panics, when doing 'pfctl -ss'
> The callstack looks as follows:
>
> panic: acquiring blockable sleep lock with spinlock or critical section
> held (rwlock) vmmaplk
> Stopped at db_enter+0x10: popq%rbp
> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
> *512895 28841 0 0x3 03K pfctl
> db_enter() at db_enter+0x10
> panic(81e19411) at panic+0x12a
> witness_checkorder(fd83b09b4d18,1,0) at witness_checkorder+0xbce
> rw_enter_read(fd83b09b4d08) at rw_enter_read+0x38
> uvmfault_lookup(8000238e3418,0) at uvmfault_lookup+0x8a
> uvm_fault_check(8000238e3418,8000238e3450,8000238e3478) at
> uvm_fault_check+0x32
> uvm_fault(fd83b09b4d00,e36553c000,0,2) at uvm_fault+0xfc
> kpageflttrap(8000238e3590,e36553c000) at kpageflttrap+0x131
> kerntrap(8000238e3590) at kerntrap+0x91
> alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b
> copyout() at copyout+0x53
> VOP_IOCTL(fd83b24340e0,c0104419,8000238e3930,1,fd842f7d7120,800
> 0239597a8) at VOP_IOCTL+0x68
> vn_ioctl(fd83b294edc0,c0104419,8000238e3930,8000239597a8) at
> vn_ioctl+0x75
> sys_ioctl(8000239597a8,8000238e3a40,8000238e3aa0) at
> sys_ioctl+0x2c4
> end trace frame: 0x8000238e3b00, count: 0
> https://www.openbsd.org/ddb.html
> reports. Insufficient info makes it difficult to find and fix bugs.
> ddb{3}>
>
> The problem comes from DIOCGETSTATES in pfioctl() here:
> 1775
> 1776 NET_LOCK();
> 1777 PF_STATE_ENTER_READ();
> 1778 state = TAILQ_FIRST(_list);
> 1779 while (state) {
> 1780 if (state->timeout != PFTM_UNLINKED) {
> 1781 if ((nr+1) * sizeof(*p) > ps->ps_len)
> 1782 break;
> 1783 pf_state_export(pstore, state);
> 1784 error = copyout(pstore, p, sizeof(*p));
> 1785 if (error) {
> 1786 free(pstore, M_TEMP,
> sizeof(*pstore));
> 1787 PF_STATE_EXIT_READ();
> 1788 NET_UNLOCK();
> 1789 goto fail;
> 1790 }
> 1791 p++;
> 1792 nr++;
> 1793 }
> 1794 state = TAILQ_NEXT(state, entry_list);
> 1795 }
> 1796 PF_STATE_EXIT_READ();
> 1797 NET_UNLOCK();
> 1798
>
> at line 1784 we do copyout() while holding mutex. As I've mentioned glitch is
> a
> specific to experimental diff [1]. However this made me thinking, that
> it's not a good idea to do copyout() while holding NET_LOCK() and state_lock.
>
> Diff below moves copyout() at line 1784 outside of protection of both locks.
> The approach I took is relatively straightforward:
>
> let DIOCGETSTATES to allocate hold_states array, which will keep
> references to states.
>
> grab locks and take references, keep those references in hold
> array.
>
> drop locks, export states and do copyout, while walking
> array of references.
>
> drop references, release hold_states array.
>
> does it make sense? If we agree that this approach makes sense
> I'll commit this diff and revisit other such places we currently
> have in pfioctl().
Hi,
with this diff i can't reproduce panic as before. i tried pf with
routing, veb, tpmr and bridge.
Do you want me to test this diff with parallel diff?
Re: parallel forwarding vs. bridges
On 17.5.2021. 16:24, Alexandr Nedvedicky wrote:
> Hrvoje,
>
> managed to trigger diagnostic panics with diff [1] sent by bluhm@
> some time ago. The panic Hrvoje sees comes from ether_input() here:
> [1] https://marc.info/?l=openbsd-tech=161903387904923=2
>
> [2] https://marc.info/?l=openbsd-tech=162099270106067=2
for those interested, I sent emails to a couple of developers with the
following panics
c/p
Hi guys,
i'm testing bluhm@ parallel diff with pf and tpmr/veb/bridge.
if i'm sending traffic over tmpr or veb, the same second i enable pf,
box panic.. see attachment ..
with bridge i can't panic box, but it seems that performance is even
worse than without parallel diff ... i will play with it more, maybe
it's just some glitch ...
pf and tpmr
r620-1# panic: kernel diagnostic assertion
"curcpu()->ci_schedstate.spc_smrdepth == 0" failed: file "/sys/kern/sub
r_xxx.c", line 163
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
11538 95266 0 0x14000 0x2002 softnet
239565 4172 0 0x14000 0x2001 softnet
db_enter() at db_enter+0x10
panic(81e0198a) at panic+0x12a
__assert(81e6f4ee,81e93713,a3,81ea877c) at __assert+0x2b
assertwaitok() at assertwaitok+0xdc
mi_switch() at mi_switch+0x40
sleep_finish(800023877740,1) at sleep_finish+0x11c
rw_enter(8212e900,1) at rw_enter+0x229
pf_test(2,1,80082048,800023877978) at pf_test+0x1055
tpmr_pf(80082048,1,fd80cccb8c00) at tpmr_pf+0x7e
tpmr_input(80082048,fd80cccb8c00,90e2ba1d4f89,80680f00) at
tpmr_input+0x124
ether_input(80082048,fd80cccb8c00) at ether_input+0xf5
if_input_process(80082048,800023877af8) at if_input_process+0x6f
ifiq_process(80086000) at ifiq_process+0x69
taskq_thread(80030300) at taskq_thread+0x9f
end trace frame: 0x0, count: 1
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
ddb{0}> show panic
kernel diagnostic assertion "curcpu()->ci_schedstate.spc_smrdepth == 0" failed:
file "/sys/kern/subr_xxx.c", line 163
ddb{0}> show locks
shared rwlock netlock r = 0 (0x8212eab0)
#0 witness_lock+0x339
#1 if_input_process+0x43
#2 ifiq_process+0x69
#3 taskq_thread+0x9f
#4 proc_trampoline+0x1c
shared rwlock softnet r = 0 (0x80030370)
#0 witness_lock+0x339
#1 taskq_thread+0x92
#2 proc_trampoline+0x1c
exclusive sched_lock _lock r = 0 (0x8230a100)
#0 witness_lock+0x339
#1 sleep_setup+0xa5
#2 rw_enter+0x208
#3 pf_test+0x1055
#4 tpmr_pf+0x7e
#5 tpmr_input+0x124
#6 ether_input+0xf5
#7 if_input_process+0x6f
#8 ifiq_process+0x69
#9 taskq_thread+0x9f
#10 proc_trampoline+0x1c
ddb{0}> show reg
rdi 0x820f1c80kprintf_mutex
rsi 0x5
rbp 0x8000238775a0
rbx 0x8000238775b0
rdx 0x8000
rcx0x206
rax 0x1
r80x800023877560
r9 0
r100
r11 0x80ef1bc6ac87b49d
r12 0x38
r13 0x800023877650
r140x100
r15 0x81e0198acmd0646_9_tim_udma+0x24a93
rip 0x81106070db_enter+0x10
cs 0x8
rflags 0x282
rsp 0x8000238775a0
ss 0x10
db_enter+0x10: popq%rbp
ddb{0}> show mbuf
mbuf 0x81106070
m_type: -13108 m_flags: c3cc
m_next: 0x1d3b4c241c334c5d m_nextpkt: 0x117400fdffac
m_data: 0x m_len: 3435973836
m_dat: 0x81106090 m_pktdat: 0x811060e0
ddb{0}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
80810 268029 1 0 30x100083 ttyin ksh
90433 180245 1 0 30x100098 poll cron
921768747 42701 95 30x100092 kqreadsmtpd
83989 36449 42701103 30x100092 kqreadsmtpd
82331 325465 42701 95 30x100092 kqreadsmtpd
28903 422168 42701 95 30x100092 kqreadsmtpd
5781 10814 42701 95 30x100092 kqreadsmtpd
70395 59130 42701 95 30x100092 kqreadsmtpd
42701 329509 1 0 30x100080 kqreadsmtpd
47357 342015 1 0 30x80 selectsshd
91828 134960 1 0 30x100080 poll ntpd
42264 230567 72970 83 30x100092 poll ntpd
72970 150962 1 83 30x100092 poll ntpd
23464 496327 92870 73 30x100090 kqreadsyslogd
92870 181949 1 0 30x100082 netio syslogd
15150 95273
Re: reposync error
On 17.5.2021. 13:55, Theo Buehler wrote: > On Mon, May 17, 2021 at 11:24:25AM +0200, Hrvoje Popovski wrote: >> Hi all, >> >> today after sysupgrade i' getting this error with reposync >> >> r620-1# su -m cvs -c "reposync -s src rsync://ftp.hostserver.de/cvs >> /home/cvs" >> reposync: rsync error: rsync: did not see server greeting >> rsync error: error starting client-server protocol (code 5) at >> main.c(1814) [Receiver=3.2.3] >> >> >> i tried reposync on a few hosts and error is the same .. >> > This is a bug introduced in ssh.c -r1.554. Fixed in ssh.c -r1.556: tnx .. .
reposync error
Hi all, today after sysupgrade i' getting this error with reposync r620-1# su -m cvs -c "reposync -s src rsync://ftp.hostserver.de/cvs /home/cvs" reposync: rsync error: rsync: did not see server greeting rsync error: error starting client-server protocol (code 5) at main.c(1814) [Receiver=3.2.3] i tried reposync on a few hosts and error is the same ..
Re: running network stack forwarding in parallel
On 13.5.2021. 1:25, Vitaliy Makkoveev wrote: > It seems this lock order issue is not parallel diff specific. Yes, you are right ... it seemed familiar but i couldn't reproduce it on lapc trunk or without this diff so i thought that parallel diff is one to blame .. sorry for noise ..
Re: running network stack forwarding in parallel
On 21.4.2021. 21:36, Alexander Bluhm wrote: > We need more MP preassure to find such bugs and races. I think now > is a good time to give this diff broader testing and commit it. > You need interfaces with multiple queues to see a difference. Hi, while forwarding ip4 traffic over box with parallel diff and aggr interfaces, and then aggr is destroyed, i'm getting witness log below... i can't reproduce this log without parallel diff and i'm getting this log only first time destroying aggr interface .. witness: lock order reversal: 1st 0x82139de0 netlock (netlock) 2nd 0x82120bb8 timeout (timeout) lock order "timeout"(rwlock) -> "netlock"(rwlock) first seen at: #0 rw_enter_write+0x43 #1 mld6_fasttimeo+0x14 #2 pffasttimo+0x67 #3 timeout_run+0x93 #4 softclock_thread+0x11d #5 proc_trampoline+0x1c lock order "netlock"(rwlock) -> "timeout"(rwlock) first seen at: #0 timeout_del_barrier+0x41 #1 aggr_p_dtor+0x17b #2 aggr_clone_destroy+0x91 #3 if_clone_destroy+0xd8 #4 ifioctl+0x1d2 #5 soo_ioctl+0x171 #6 sys_ioctl+0x2c4 #7 syscall+0x3b9 #8 Xsyscall+0x128 r620-1# cat /etc/hostname.aggr0 trunkport ix0 lladdr ec:f4:bb:da:f7:f8 inet 192.168.42.1 255.255.255.0 !route add 16/8 192.168.42.11 up r620-1# cat /etc/hostname.aggr1 trunkport ix1 inet 192.168.43.1 255.255.255.0 !route add 48/8 192.168.43.11 up OpenBSD 6.9-current (GENERIC.MP) #166: Wed May 12 10:18:11 CEST 2021 hrv...@r620-1.srce.hr:/sys/arch/amd64/compile/GENERIC.MP real mem = 17115840512 (16322MB) avail mem = 16450007040 (15687MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries) bios0: vendor Dell Inc. version "2.9.0" date 12/06/2019 bios0: Dell Inc. PowerEdge R620 acpi0 at bios0: ACPI 3.0 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WDAT SLIC ERST HEST BERT EINJ TCPA PC__ SRAT SSDT acpi0: wakeup devices PCI0(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 4 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.44 MHz, 06-3e-04 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 2, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 6 (application processor) cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.04 MHz, 06-3e-04 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 3, package 0 cpu2 at mainbus0: apid 8 (application processor) cpu2: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.03 MHz, 06-3e-04 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 4, package 0 cpu3 at mainbus0: apid 16 (application processor) cpu3: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.03 MHz, 06-3e-04 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 8, package 0 cpu4 at mainbus0: apid 18 (application processor) cpu4: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.03 MHz, 06-3e-04 cpu4:
Re: iwx and sysupgrade
On 4.5.2021. 12:15, Raf Czlonka wrote: > On Tue, May 04, 2021 at 10:55:37AM BST, Stefan Sperling wrote: >> On Tue, May 04, 2021 at 11:47:43AM +0200, Hrvoje Popovski wrote: >>> I'm not sure that with iwx and eduroam, sysupgrade can finish. Maybe i >>> need to wait longer, will try that ... sysupgrade will finish if iwx is >>> disabled or hostname.iwx0 is removed from etc before doing sysupgrade >> >> If iwx is disabled, dhclient won't be started. The problem for sysupgrade >> is that dhclient never goes into the background when it gets started. This >> has nothing to with iwx. It is about dhclient and 802.1x. >> >> I suspect proper link state reporting doesn't occur with 802.1x and no >> wpa_supplicant running. But I don't know enough. Locate someone who >> understands dhclient, and/or someone who understands our 802.1x hack, >> and they will hopefully find a solution for you. >> > > Unless I'm missing something, Hrvoje's not suing dhclient(8) - as > there's no 'dhcp' entry in the /etc/hostname.iwx0 - but dhcpleased(8), > most likely. > > P.S. Commenting only as an 'eduroam', not iwx(4), user. > > Raf > Hi, i tried with dhclient right now and results are the same .. This isn't big problem for me, i know what i need to do I've sent mail to tech@ because i thought that it's some problem with iwx
Re: iwx and sysupgrade
On 4.5.2021. 11:55, Stefan Sperling wrote: > On Tue, May 04, 2021 at 11:47:43AM +0200, Hrvoje Popovski wrote: >> I'm not sure that with iwx and eduroam, sysupgrade can finish. Maybe i >> need to wait longer, will try that ... sysupgrade will finish if iwx is >> disabled or hostname.iwx0 is removed from etc before doing sysupgrade > > If iwx is disabled, dhclient won't be started. The problem for sysupgrade > is that dhclient never goes into the background when it gets started. This > has nothing to with iwx. It is about dhclient and 802.1x. > > I suspect proper link state reporting doesn't occur with 802.1x and no > wpa_supplicant running. But I don't know enough. Locate someone who > understands dhclient, and/or someone who understands our 802.1x hack, > and they will hopefully find a solution for you. > Thank you for information and sorry for noise ..
Re: iwx and sysupgrade
On 4.5.2021. 11:44, Stefan Sperling wrote: > On Tue, May 04, 2021 at 11:36:01AM +0200, Hrvoje Popovski wrote: >> On 4.5.2021. 11:02, Hrvoje Popovski wrote: >>> i've disabled and stopped wpa_supplicant and reboot laptop and iwx0 >>> didn't get ip but laptop did boot normally .. >>> >>> iwx0: flags=808843 mtu >>> 1500 >>> lladdr xx:xx:xx:xx:xx:xx >>> index 2 priority 4 llprio 3 >>> groups: wlan >>> media: IEEE802.11 autoselect (HT-MCS11 mode 11n) >>> status: no network >>> ieee80211: join eduroam wpaprotos wpa2 wpaakms 802.1x wpaciphers >>> ccmp wpagroupcipher ccmp >>> >>> >> >> sorry, yes, yes same logs appears but not same second as i thought > > Ok, thanks. This is something I might take a look at myself. > > Looks like the firmware is failing to send an association request in > this broken configuration (802.1x enabled but no wpa_supplicant running). > Not that it would help much if it did send the frame, since you cannot get > link anyway without the wpa_supplicant running. But a firmware crash is > not pretty. > > You are in range of an eduroam AP when this happens? > Yes I am. AP is above my head staring at me :)
Re: iwx and sysupgrade
On 4.5.2021. 11:38, Stefan Sperling wrote: > On Tue, May 04, 2021 at 11:02:49AM +0200, Hrvoje Popovski wrote: >> On 4.5.2021. 10:47, Stefan Sperling wrote: >>> On Tue, May 04, 2021 at 10:32:02AM +0200, Hrvoje Popovski wrote: >>>> Hi all, >>>> >>>> today i tried to do sysupgrade and it wouldn't finish because of iwx >>>> errors. >>>> iwx is working just fine with with snapshots, even with eduroam :) >>>> >>>> e14gen2# cat /etc/hostname.iwx0 >>>> debug >>>> join "eduroam" wpa wpaakms 802.1x >>>> autoconf >>> >>> You probably have wpa_supplicant running when it works, right? >> >> yes ... >> >>> Does this issue really occur only in bsd.rd, or can this be reproduced >>> on any kernel while wpa_supplicant is not running? >> >> i've disabled and stopped wpa_supplicant and reboot laptop and iwx0 >> didn't get ip but laptop did boot normally .. >> >> iwx0: flags=808843 mtu >> 1500 >> lladdr xx:xx:xx:xx:xx:xx >> index 2 priority 4 llprio 3 >> groups: wlan >> media: IEEE802.11 autoselect (HT-MCS11 mode 11n) >> status: no network >> ieee80211: join eduroam wpaprotos wpa2 wpaakms 802.1x wpaciphers >> ccmp wpagroupcipher ccmp > > Hmm. OK. > > I have no idea how exactly 802.1x is supposed to work. > I am not the person who added support for 802.1x. I have never used it, > and I don't have a wpa enterprise setup. > > Nevertheless, I can reproduce the following: > > I think your problem is that dhclient is waiting for link to come up in > the foreground, long enough that the sysupgrade watchdog timeout triggers > and the system reboots. > > (I)nstall, (U)pgrade, (A)utoinstall or (S)hell? s > # ifconfig iwx0 join "eduroam" wpa wpaakms 802.1x > # cd /dev > # sh MAKEDEV sd0 > # fsck -p /dev/sd0a > /dev/rsd0a: file system is clean; not checking > # mount /dev/sd0a /mnt > # route monitor & > [1] 26254 > # dhclient iwx0 > iwx0: hw rev 0x340, fw ver 48.1335886879.0, address d0:ab:d5:f4:6b:31 > got message of size 200 on Tue May 4 09:32:35 2021 > RTM_IFINFO: iface status change: len 200, if# 5, name iwx0, link: no carrier, > m> > sockaddrs: > d0:ab:d5:f4:6b:31 > got message of size 96 on Tue May 4 09:32:35 2021 > RTM_PROPOSAL: config proposal: len 96, source dhcp, table 0, if# 5, name > iwx0, 0 > flags: > fmask: > use:0 mtu:0expire:0 > locks: inits: > > > At this point dhclient stays blocked in the foreground, seemingly forever. > The expected behaviour would be that it eventually times out and prints > its "...sleeping" message. > > I don't see any of the iwx firmware errors you are seeing. > But would fixing such iwx errors even help you, given that eduroam is an > entirely useless wifi network for bsd.rd? I'm not sure that with iwx and eduroam, sysupgrade can finish. Maybe i need to wait longer, will try that ... sysupgrade will finish if iwx is disabled or hostname.iwx0 is removed from etc before doing sysupgrade
Re: iwx and sysupgrade
On 4.5.2021. 11:02, Hrvoje Popovski wrote: > On 4.5.2021. 10:47, Stefan Sperling wrote: >> On Tue, May 04, 2021 at 10:32:02AM +0200, Hrvoje Popovski wrote: >>> Hi all, >>> >>> today i tried to do sysupgrade and it wouldn't finish because of iwx errors. >>> iwx is working just fine with with snapshots, even with eduroam :) >>> >>> e14gen2# cat /etc/hostname.iwx0 >>> debug >>> join "eduroam" wpa wpaakms 802.1x >>> autoconf >> >> You probably have wpa_supplicant running when it works, right? > > yes ... > >> Does this issue really occur only in bsd.rd, or can this be reproduced >> on any kernel while wpa_supplicant is not running? > > i've disabled and stopped wpa_supplicant and reboot laptop and iwx0 > didn't get ip but laptop did boot normally .. > > iwx0: flags=808843 mtu > 1500 > lladdr xx:xx:xx:xx:xx:xx > index 2 priority 4 llprio 3 > groups: wlan > media: IEEE802.11 autoselect (HT-MCS11 mode 11n) > status: no network > ieee80211: join eduroam wpaprotos wpa2 wpaakms 802.1x wpaciphers > ccmp wpagroupcipher ccmp > > sorry, yes, yes same logs appears but not same second as i thought May 4 10:56:37 e14gen2 reorder_kernel: kernel relinking done May 4 10:57:05 e14gen2 /bsd: iwx0: dumping device error log May 4 10:57:05 e14gen2 /bsd: iwx0: Start Error Log Dump: May 4 10:57:05 e14gen2 /bsd: iwx0: Status: 0x9, count: 6 May 4 10:57:05 e14gen2 /bsd: iwx0: 0x0071 | NMI_INTERRUPT_UMAC_FATAL May 4 10:57:05 e14gen2 /bsd: iwx0: 00A0A200 | trm_hw_status0 May 4 10:57:05 e14gen2 /bsd: iwx0: | trm_hw_status1 May 4 10:57:05 e14gen2 /bsd: iwx0: 004F8E3C | branchlink2 May 4 10:57:05 e14gen2 /bsd: iwx0: 00016BFE | interruptlink1 May 4 10:57:05 e14gen2 /bsd: iwx0: 00016BFE | interruptlink2 May 4 10:57:05 e14gen2 /bsd: iwx0: 004F4B14 | data1 May 4 10:57:05 e14gen2 /bsd: iwx0: 1000 | data2 May 4 10:57:05 e14gen2 /bsd: iwx0: F008 | data3 May 4 10:57:05 e14gen2 /bsd: iwx0: 64410D2E | beacon time May 4 10:57:05 e14gen2 /bsd: iwx0: 2485C522 | tsf low May 4 10:57:05 e14gen2 /bsd: iwx0: 0057 | tsf hi May 4 10:57:05 e14gen2 /bsd: iwx0: | time gp1 May 4 10:57:05 e14gen2 /bsd: iwx0: 02BAFA49 | time gp2 May 4 10:57:05 e14gen2 /bsd: iwx0: 0001 | uCode revision type May 4 10:57:05 e14gen2 /bsd: iwx0: 0030 | uCode version major May 4 10:57:05 e14gen2 /bsd: iwx0: 4FA0041F | uCode version minor May 4 10:57:05 e14gen2 /bsd: iwx0: 0340 | hw version May 4 10:57:05 e14gen2 /bsd: iwx0: 18489000 | board version May 4 10:57:05 e14gen2 /bsd: iwx0: 809BFC14 | hcmd May 4 10:57:05 e14gen2 /bsd: iwx0: 2402 | isr0 May 4 10:57:05 e14gen2 /bsd: iwx0: 0100 | isr1 May 4 10:57:05 e14gen2 /bsd: iwx0: 08B2 | isr2 May 4 10:57:05 e14gen2 /bsd: iwx0: 04C1DFCC | isr3 May 4 10:57:05 e14gen2 /bsd: iwx0: | isr4 May 4 10:57:05 e14gen2 /bsd: iwx0: 021C | last cmd Id May 4 10:57:05 e14gen2 /bsd: iwx0: 004F4B14 | wait_event May 4 10:57:05 e14gen2 /bsd: iwx0: 00C0 | l2p_control May 4 10:57:05 e14gen2 /bsd: iwx0: 00018014 | l2p_duration May 4 10:57:05 e14gen2 /bsd: iwx0: 003F | l2p_mhvalid May 4 10:57:05 e14gen2 /bsd: iwx0: | l2p_addr_match May 4 10:57:05 e14gen2 /bsd: iwx0: 000B | lmpm_pmg_sel May 4 10:57:05 e14gen2 /bsd: iwx0: | timestamp May 4 10:57:05 e14gen2 /bsd: iwx0: E808 | flow_handler May 4 10:57:05 e14gen2 /bsd: iwx0: Start UMAC Error Log Dump: May 4 10:57:05 e14gen2 /bsd: iwx0: Status: 0x9, count: 7 May 4 10:57:05 e14gen2 /bsd: iwx0: 0x20003421 | ADVANCED_SYSASSERT May 4 10:57:05 e14gen2 /bsd: iwx0: 0x | umac branchlink1 May 4 10:57:05 e14gen2 /bsd: iwx0: 0xC008CC3C | umac branchlink2 May 4 10:57:05 e14gen2 /bsd: iwx0: 0xC0084696 | umac interruptlink1 May 4 10:57:05 e14gen2 /bsd: iwx0: 0x | umac interruptlink2 May 4 10:57:05 e14gen2 /bsd: iwx0: 0x | umac data1 May 4 10:57:05 e14gen2 /bsd: iwx0: 0x0001 | umac data2 May 4 10:57:05 e14gen2 /bsd: iwx0: 0xDEADBEEF | umac data3 May 4 10:57:05 e14gen2 /bsd: iwx0: 0x0030 | umac major May 4 10:57:05 e14gen2 /bsd: iwx0: 0x4FA0041F | umac minor May 4 10:57:05 e14gen2 /bsd: iwx0: 0x02BAFA3B | frame pointer May 4 10:57:05 e14gen2 /bsd: iwx0: 0xC0885DB0 | stack pointer May 4 10:57:05 e14gen2 /bsd: iwx0: 0x002D0028 | last host cmd May 4 10:57:05 e14gen2 /bsd: iwx0: 0x | isr status reg May 4 10:57:05 e14gen2 /bsd: driver status: May 4 10:57:05 e14gen2 /bsd: tx ring 0: qid=0 cur=46 queued=1 May 4 10:57:05 e14gen2 /bsd: tx ring 1: qid=1 cur=0 queued=0 May 4 10:57:05 e14gen2 /bsd: tx ring 2: qid=2 cur=1 queued=0 May 4 10:57:05 e14gen2 /bsd: tx ring 3: qid=3 cur=0 queued=0 May 4 10:57:05 e14gen2 /bsd: tx ring 4: qid=4 cur=0 queued=0 May 4 10:57:05 e14gen2 /bsd: tx ring
Re: iwx and sysupgrade
On 4.5.2021. 10:47, Stefan Sperling wrote: > On Tue, May 04, 2021 at 10:32:02AM +0200, Hrvoje Popovski wrote: >> Hi all, >> >> today i tried to do sysupgrade and it wouldn't finish because of iwx errors. >> iwx is working just fine with with snapshots, even with eduroam :) >> >> e14gen2# cat /etc/hostname.iwx0 >> debug >> join "eduroam" wpa wpaakms 802.1x >> autoconf > > You probably have wpa_supplicant running when it works, right? yes ... > Does this issue really occur only in bsd.rd, or can this be reproduced > on any kernel while wpa_supplicant is not running? i've disabled and stopped wpa_supplicant and reboot laptop and iwx0 didn't get ip but laptop did boot normally .. iwx0: flags=808843 mtu 1500 lladdr xx:xx:xx:xx:xx:xx index 2 priority 4 llprio 3 groups: wlan media: IEEE802.11 autoselect (HT-MCS11 mode 11n) status: no network ieee80211: join eduroam wpaprotos wpa2 wpaakms 802.1x wpaciphers ccmp wpagroupcipher ccmp
iwx and sysupgrade
Hi all, today i tried to do sysupgrade and it wouldn't finish because of iwx errors. iwx is working just fine with with snapshots, even with eduroam :) e14gen2# cat /etc/hostname.iwx0 debug join "eduroam" wpa wpaakms 802.1x autoconf e14gen2# ifconfig iwx0 iwx0: flags=808847 mtu 1500 lladdr xx:xx:xx:xx:xx:xx index 2 priority 4 llprio 3 groups: wlan egress media: IEEE802.11 autoselect (HT-MCS11 mode 11n) status: active ieee80211: join eduroam chan 40 bssid yy:yy:yy:yy:yy:yy 83% wpaprotos wpa2 wpaakms 802.1x wpaciphers ccmp wpagroupcipher ccmp inet xxx.xxx.xxx.xxx netmask 0xff00 broadcast xxx.xxx.xxx.xxx if i disable iwx0 or remove hostname.iwx0 from etc, sysupgrade successfully finish dmesg and logs from ramdisk when iwx error happened OpenBSD 6.9-current (RAMDISK_CD) #1: Mon May 3 11:11:38 MDT 2021 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/RAMDISK_CD real mem = 7742496768 (7383MB) avail mem = 7503826944 (7156MB) random: good seed from bootblocks mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.2 @ 0xbf913000 (62 entries) bios0: vendor LENOVO version "R1AET36W (1.12 )" date 03/15/2021 bios0: LENOVO 20T6000TSC acpi0 at bios0: ACPI 6.3 acpi0: tables DSDT FACP SSDT SSDT IVRS SSDT SSDT TPM2 SSDT MSDM BATB HPET APIC MCFG SBST WSMT VFCT SSDT CRAT CDIT FPDT SSDT SSDT SSDT BGRT UEFI SSDT SSDT acpihpet0 at acpi0: 14318180 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 5 4500U with Radeon Graphics, 2370.85 MHz, 17-60-01 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu0: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu0: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=1.1, IBE cpu at mainbus0: not configured cpu at mainbus0: not configured cpu at mainbus0: not configured cpu at mainbus0: not configured cpu at mainbus0: not configured ioapic0 at mainbus0: apid 32 pa 0xfec0, version 21, 24 pins, can't remap ioapic1 at mainbus0: apid 33 pa 0xfec01000, version 21, 32 pins, can't remap acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus -1 (GPP3) acpiprt2 at acpi0: bus 1 (GPP4) acpiprt3 at acpi0: bus 2 (GPP5) acpiprt4 at acpi0: bus 3 (GPP6) acpiprt5 at acpi0: bus 4 (GP17) acpiprt6 at acpi0: bus -1 (GP18) acpiprt7 at acpi0: bus -1 (GP19) acpiec0 at acpi0 "ACPI0010" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "PNP0C0C" at acpi0 not configured acpipci0 at acpi0 PCI0: 0x0010 0x0011 0x acpicmos0 at acpi0 "PNP0C0A" at acpi0 not configured "ACPI0003" at acpi0 not configured "LEN0268" at acpi0 not configured "LEN0130" at acpi0 not configured "LEN0100" at acpi0 not configured "SMB0001" at acpi0 not configured "PNP0C0D" at acpi0 not configured "PNP0C0E" at acpi0 not configured "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured amdgpio0 at acpi0 GPIO uid 0 addr 0xfed81500/0x400 irq 7, 184 pins "USBC000" at acpi0 not configured "STM0125" at acpi0 not configured "PNP0C14" at acpi0 not configured acpipwrres at acpi0 not configured acpipwrres at acpi0 not configured acpipwrres at acpi0 not configured acpipwrres at acpi0 not configured pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "AMD 17h/6xh Root Complex" rev 0x00 "AMD 17h/6xh IOMMU" rev 0x00 at pci0 dev 0 function 2 not configured pchb1 at pci0 dev 1 function 0 "AMD 17h/6xh Host" rev 0x00 pchb2 at pci0 dev 2 function 0 "AMD 17h/6xh Host" rev 0x00 ppb0 at pci0 dev 2 function 2 "AMD 17h/6xh PCIE" rev 0x00: msi pci1 at ppb0 bus 1 re0 at pci1 dev 0 function 0 "Realtek 8168" rev 0x10: RTL8168GU/8111GU (0x5080), msi, address 8c:8c:aa:1a:b2:35 rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0 ppb1 at pci0 dev 2 function 3 "AMD 17h/6xh PCIE" rev
Re: [External] : Re: running network stack forwarding in parallel
On 22.4.2021. 13:08, Hrvoje Popovski wrote: > On 22.4.2021. 12:38, Alexander Bluhm wrote: >> It is not clear why the lock helps. Is it a bug in routing or ARP? >> Or is it just different timing introduced by the additional kernel >> lock? The parallel network task stress the subsystems of the kernel >> more than before with MP load. Having more results from machines >> with different timing would help. >> >> bluhm >> > > > I'm running this diff on Dell r620 with 6 x Intel(R) Xeon(R) CPU E5-2643 > v2 @ 3.50GHz, 3600.52 MHz, 06-3e-04 and ix and forwarding went up from > 1Mpps to 1.8Mpps fun fact, with parallel diff on top of fresh source i'm getting 2.2Mpps of plain forwarding and 3.5Mpps over veb(4) with NET_TASKQ=4 on 6 x Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.03 MHz
Re: pf_state_key_link_reverse() is prone to race on parallel forwarding
On 21.4.2021. 22:19, Alexandr Nedvedicky wrote:
> Hello,
>
> people who will be running pf(4) with bluhm's diff [1], may trip
> one of the asserts triggered by pf_state_key_link_reverse() here:
>
> 7366 void
> 7367 pf_state_key_link_reverse(struct pf_state_key *sk, struct pf_state_key
> *skrev)
> 7368 {
> 7369 /* Note that sk and skrev may be equal, then we refcount twice.
> */
> 7370 KASSERT(sk->reverse == NULL);
> 7371 KASSERT(skrev->reverse == NULL);
> 7372 sk->reverse = pf_state_key_ref(skrev);
> 7373 skrev->reverse = pf_state_key_ref(sk);
> 7374 }
>
> pf(4) currently does not provide a mutual access to state instances.
> so it may happen pf(4) will be handling more packets, which will be
> updating the same state. This is the situation of one winner and
> more losers. At this point I'm suggesting to change those asserts
> to take the race into account. diff below makes pf_state_key_link_reverse()
> happy when pf(4) runs in parallel.
>
> would it be OK to commit it once bluhm's diff [1] will be in?
>
> thanks and
> regards
> sashan
I'm running this diff with latest version of bluhm@ network parallel
diff and i'm not seeing and problems .. i'm not running pfsync ..
Re: [External] : Re: running network stack forwarding in parallel
On 22.4.2021. 12:38, Alexander Bluhm wrote:
> It is not clear why the lock helps. Is it a bug in routing or ARP?
> Or is it just different timing introduced by the additional kernel
> lock? The parallel network task stress the subsystems of the kernel
> more than before with MP load. Having more results from machines
> with different timing would help.
>
> bluhm
>
I'm running this diff on Dell r620 with 6 x Intel(R) Xeon(R) CPU E5-2643
v2 @ 3.50GHz, 3600.52 MHz, 06-3e-04 and ix and forwarding went up from
1Mpps to 1.8Mpps
Will play with this and contact you if something pops up ..
tnx ..
>
> Index: net/if.c
> ===
> RCS file: /data/mirror/openbsd/cvs/src/sys/net/if.c,v
> retrieving revision 1.640
> diff -u -p -r1.640 if.c
> --- net/if.c 26 Mar 2021 22:41:06 - 1.640
> +++ net/if.c 21 Apr 2021 12:52:08 -
> @@ -238,7 +238,7 @@ int ifq_congestion;
>
> int netisr;
>
> -#define NET_TASKQ 1
> +#define NET_TASKQ 4
> struct taskq *nettqmp[NET_TASKQ];
>
> struct task if_input_task_locked = TASK_INITIALIZER(if_netisr, NULL);
> @@ -834,10 +834,10 @@ if_input_process(struct ifnet *ifp, stru
>* to PF globals, pipex globals, unicast and multicast addresses
>* lists and the socket layer.
>*/
> - NET_LOCK();
> + NET_RLOCK_IN_SOFTNET();
> while ((m = ml_dequeue(ml)) != NULL)
> (*ifp->if_input)(ifp, m);
> - NET_UNLOCK();
> + NET_RUNLOCK_IN_SOFTNET();
> }
>
> void
> @@ -895,6 +895,12 @@ if_netisr(void *unused)
> KERNEL_UNLOCK();
> }
> #endif
> + if (n & (1 << NETISR_IP))
> + ipintr();
> +#ifdef INET6
> + if (n & (1 << NETISR_IPV6))
> + ip6intr();
> +#endif
> #if NPPP > 0
> if (n & (1 << NETISR_PPP)) {
> KERNEL_LOCK();
> @@ -3316,12 +3322,15 @@ unhandled_af(int af)
> * globals aren't ready to be accessed by multiple threads in
> * parallel.
> */
> -int nettaskqs = NET_TASKQ;
> +int nettaskqs;
>
> struct taskq *
> net_tq(unsigned int ifindex)
> {
> struct taskq *t = NULL;
> +
> + if (nettaskqs == 0)
> + nettaskqs = min(NET_TASKQ, ncpus);
>
> t = nettqmp[ifindex % nettaskqs];
>
> Index: net/if_ethersubr.c
> ===
> RCS file: /data/mirror/openbsd/cvs/src/sys/net/if_ethersubr.c,v
> retrieving revision 1.274
> diff -u -p -r1.274 if_ethersubr.c
> --- net/if_ethersubr.c7 Mar 2021 06:02:32 - 1.274
> +++ net/if_ethersubr.c21 Apr 2021 22:16:05 -
> @@ -222,7 +222,10 @@ ether_resolve(struct ifnet *ifp, struct
>
> switch (af) {
> case AF_INET:
> + KERNEL_LOCK();
> + /* XXXSMP there is a MP race in arpresolve() */
> error = arpresolve(ifp, rt, m, dst, eh->ether_dhost);
> + KERNEL_UNLOCK();
> if (error)
> return (error);
> eh->ether_type = htons(ETHERTYPE_IP);
> @@ -245,7 +248,10 @@ ether_resolve(struct ifnet *ifp, struct
> break;
> #ifdef INET6
> case AF_INET6:
> + KERNEL_LOCK();
> + /* XXXSMP there is a MP race in nd6_resolve() */
> error = nd6_resolve(ifp, rt, m, dst, eh->ether_dhost);
> + KERNEL_UNLOCK();
> if (error)
> return (error);
> eh->ether_type = htons(ETHERTYPE_IPV6);
> Index: net/ifq.c
> ===
> RCS file: /data/mirror/openbsd/cvs/src/sys/net/ifq.c,v
> retrieving revision 1.43
> diff -u -p -r1.43 ifq.c
> --- net/ifq.c 20 Feb 2021 04:37:26 - 1.43
> +++ net/ifq.c 21 Apr 2021 13:12:44 -
> @@ -243,7 +243,7 @@ void
> ifq_init(struct ifqueue *ifq, struct ifnet *ifp, unsigned int idx)
> {
> ifq->ifq_if = ifp;
> - ifq->ifq_softnet = net_tq(ifp->if_index); /* + idx */
> + ifq->ifq_softnet = net_tq(ifp->if_index + idx);
> ifq->ifq_softc = NULL;
>
> mtx_init(>ifq_mtx, IPL_NET);
> @@ -617,7 +617,7 @@ void
> ifiq_init(struct ifiqueue *ifiq, struct ifnet *ifp, unsigned int idx)
> {
> ifiq->ifiq_if = ifp;
> - ifiq->ifiq_softnet = net_tq(ifp->if_index); /* + idx */
> + ifiq->ifiq_softnet = net_tq(ifp->if_index + idx);
> ifiq->ifiq_softc = NULL;
>
> mtx_init(>ifiq_mtx, IPL_NET);
> Index: net/netisr.h
> ===
> RCS file: /data/mirror/openbsd/cvs/src/sys/net/netisr.h,v
> retrieving revision 1.55
> diff -u -p -r1.55 netisr.h
> --- net/netisr.h 5 Jan 2021 20:43:36 - 1.55
> +++ net/netisr.h 21 Apr 2021 12:52:08 -
> @@ -41,8 +41,10 @@
> * interrupt used for scheduling the network code to calls
> * on the lowest level
Re: [External] : Re: running network stack forwarding in parallel
On 22.4.2021. 11:36, Hrvoje Popovski wrote:
> if you want i'll try to reproduce in on other boxes..
> maybe i can trigger it here easily because of 2 sockets ?
on second box with 6 x Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz,
3600.02 MHz..
r620-1# papnpaiancini:cc :p :op
opooolo_llc_ac_caccahhceh_ei_eti_tieetmme_mm__amgamigacigci__cc_hccehhcekcekc::
k :m bmubmfubfuppflp llc pc pcuup uf rfferree eel el iilsitss tm tom
omddoidfiiifeifeidde:d ::i ti etietmme m
a daddardd rd0 r0
xx0fxfddf88d08c0cc0c6c76afc9b3f04500400++01+61 610 6x0
fx0fxffdffdf88d08
00020720d72a8c0049703eb!ef!e==!0=x009x59x95995b9ebbaee3ae3ae344ef54f5a4bff7db07990a9
aa
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
514514 85981 0 0x14000 0x2003 softnet
*327946 47274 0 0x14000 0x2004 softnet
401838 53270 0 0x14000 0x2002 softnet
db_enter() at db_enter+0x10
panic(81df3a9d) at panic+0x12a
pool_cache_get(822fade8) at pool_cache_get+0x25b
pool_get(822fade8,2) at pool_get+0x5e
m_clget(0,2,802) at m_clget+0xdd
ixgbe_get_buf(800699d0,2a) at ixgbe_get_buf+0xa3
ixgbe_rxfill(800699d0) at ixgbe_rxfill+0xae
ixgbe_queue_intr(80024c80) at ixgbe_queue_intr+0x4f
intr_handler(80002486ae00,8006fa00) at intr_handler+0x6e
Xintr_ioapic_edge0_untramp() at Xintr_ioapic_edge0_untramp+0x18f
srp_leave(80002486aed0) at srp_leave+0x11
if_get(2) at if_get+0x6f
ip_output(fd80ccca9e00,0,80002486b130,1,0,0) at ip_output+0x393
ip_forward(fd80ccca9e00,80082048,fd83b39ada88,0) at
ip_forward+
0x261
end trace frame: 0x80002486b250, count: 0
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{4}> show panic
pool_cache_item_magic_check: mbufpl cpu free list modified: item addr
0xfd8
0ccca9300+16 0xfd80027dc47e!=0x959bea3e4ffab79a
ddb{4}> trace
db_enter() at db_enter+0x10
panic(81df3a9d) at panic+0x12a
pool_cache_get(822fade8) at pool_cache_get+0x25b
pool_get(822fade8,2) at pool_get+0x5e
m_clget(0,2,802) at m_clget+0xdd
ixgbe_get_buf(800699d0,2a) at ixgbe_get_buf+0xa3
ixgbe_rxfill(800699d0) at ixgbe_rxfill+0xae
ixgbe_queue_intr(80024c80) at ixgbe_queue_intr+0x4f
intr_handler(80002486ae00,8006fa00) at intr_handler+0x6e
Xintr_ioapic_edge0_untramp() at Xintr_ioapic_edge0_untramp+0x18f
srp_leave(80002486aed0) at srp_leave+0x11
if_get(2) at if_get+0x6f
ip_output(fd80ccca9e00,0,80002486b130,1,0,0) at ip_output+0x393
ip_forward(fd80ccca9e00,80082048,fd83b39ada88,0) at
ip_forward+0x261
ip_input_if(80002486b268,80002486b274,4,0,80082048) at
ip_input_if+0x608
ipv4_input(80082048,fd80ccca9e00) at ipv4_input+0x39
if_input_process(80082048,80002486b2e8) at if_input_process+0x6f
ifiq_process(80086200) at ifiq_process+0x69
taskq_thread(80030100) at taskq_thread+0x9f
end trace frame: 0x0, count: -19
ddb{4}> show locks
shared rwlock netlock r = 0 (0x82101bd8)
#0 witness_lock+0x339
#1 if_input_process+0x43
#2 ifiq_process+0x69
#3 taskq_thread+0x9f
#4 proc_trampoline+0x1c
shared rwlock softnet r = 0 (0x80030170)
#0 witness_lock+0x339
#1 taskq_thread+0x92
#2 proc_trampoline+0x1c
ddb{4}> show all locks
Process 53626 (ld) thread 0x800024972008 (479301)
exclusive rrwlock inode r = 0 (0xfd842e8d92c0)
#0 witness_lock+0x339
#1 rw_enter+0x286
#2 rrw_enter+0x56
#3 VOP_LOCK+0x33
#4 vn_lock+0x84
#5 uvn_get+0x15f
#6 uvm_fault_lower+0x2ae
#7 uvm_fault+0x19e
#8 upageflttrap+0x69
#9 usertrap+0x18b
#10 recall_trap+0x8
Process 47274 (softnet) thread 0x8000e2a0 (327946)
shared rwlock netlock r = 0 (0x82101bd8)
#0 witness_lock+0x339
#1 if_input_process+0x43
#2 ifiq_process+0x69
#3 taskq_thread+0x9f
#4 proc_trampoline+0x1c
shared rwlock softnet r = 0 (0x80030170)
#0 witness_lock+0x339
#1 taskq_thread+0x92
#2 proc_trampoline+0x1c
Process 53270 (softnet) thread 0x8000e540 (401838)
shared rwlock netlock r = 0 (0x82101bd8)
#0 witness_lock+0x339
#1 if_input_process+0x43
#2 ifiq_process+0x69
#3 taskq_thread+0x9f
#4 proc_trampoline+0x1c
shared rwlock softnet r = 0 (0x80030070)
#0 witness_lock+0x339
#1 taskq_thread+0x92
#2 proc_trampoline+0x1c
ddb{4}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
53626 479301 83760 0 3 0x2 biowait ld
83760 75359 86199 0 30x10008a sigsusp sh
86199 465340 39220 0 30x10008a sigsusp make
28046 189992 1 0 30x100083 ttyin ksh
39220 374653 1 0 30x10008b sigsusp ksh
4909 380794 1 0 30x100098 poll cron
38969 2
Re: [External] : Re: running network stack forwarding in parallel
On 22.4.2021. 11:02, Alexander Bluhm wrote:
> On Thu, Apr 22, 2021 at 09:03:22AM +0200, Hrvoje Popovski wrote:
>> something like this:
>>
>> x3550m4# pappnaiannc:iicc :p:o ppoolo_oolcla__ddcohoe__gg_eiettt::e m
>> _mmcbmualg2fkpilc2_:: chppeaag
>> gceke: ee mmmbppttuyfyp
>
> This was without my kernel lock around ARP bandage, right?
yes, yes ...
>
>> ddb{9}> mach ddbcpu 0xa
>> Stopped at x86_ipi_db+0x12:leave
>> x86_ipi_db(800021a2aff0) at x86_ipi_db+0x12
>> x86_ipi_handler() at x86_ipi_handler+0x80
>> Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
>> pool_get(8221e568,2) at pool_get+0x43
>> m_gethdr(2,1) at m_gethdr+0x3f
>> rtm_msg1(e,800026e3cf70) at rtm_msg1+0x4c
>> rtm_ifchg(805b3800) at rtm_ifchg+0x61
>> if_down(805b3800) at if_down+0xa0
>> if_downall() at if_downall+0x5b
>> boot(104) at boot+0x99
>> reboot(104) at reboot+0x5b
>> panic(81df855b) at panic+0x132
>> pool_do_get(8221ebc8,2,800026e3d294) at pool_do_get+0x309
>> pool_get(8221ebc8,2) at pool_get+0x95
>> end trace frame: 0x800026e3d340, count: 0
>>
>> ddb{10}> mach ddbcpu 0xb
>> Stopped at db_enter+0x10: popq%rbp
>> db_enter() at db_enter+0x10
>> panic(81df855b) at panic+0x12a
>> pool_do_get(8221e568,2,800026e43294) at pool_do_get+0x309
>> pool_get(8221e568,2) at pool_get+0x95
>> m_clget(0,2,802) at m_clget+0xdd
>> ixgbe_get_buf(8015c0e8,e) at ixgbe_get_buf+0xa3
>> ixgbe_rxfill(8015c0e8) at ixgbe_rxfill+0xae
>> ixgbe_queue_intr(8011ac40) at ixgbe_queue_intr+0x4f
>> intr_handler(800026e434b0,800cd700) at intr_handler+0x6e
>> Xintr_ioapic_edge4_untramp() at Xintr_ioapic_edge4_untramp+0x18f
>> acpicpu_idle() at acpicpu_idle+0x1ea
>> sched_idle(800021a33ff0) at sched_idle+0x27e
>> end trace frame: 0x0, count: 3
>
> Two processors 10 and 11 in pool get.
>
> CPU 10 does pool_get, panic, boot, pool_get again.
> CPU 11 was the one that originally stopped in ddb.
>
> Did you enter boot reboot before doing mach ddbcpu 0xa?
nope... is doing that ever useful?
> Or how did we get the boot sequence in this trace?
>
> Can it be that both CPU paniced simultaeously? The mangled massage
> indicates this. Then cpu 10 saw that cpu 11 already paniced to ddb
> and tried to reboot. There it paniced again and got stuck in a
> recursive call to pool_get().
>
> The if (db_panic) in the panic() function was not written with
> simultaneous panics on multiple CPUs in mind.
if you want i'll try to reproduce in on other boxes..
maybe i can trigger it here easily because of 2 sockets ?
Re: [External] : Re: running network stack forwarding in parallel
On 22.4.2021. 1:10, Hrvoje Popovski wrote:
> On 22.4.2021. 0:31, Alexandr Nedvedicky wrote:
>> Hello,
>>
>>
>>>> Hi,
>>>>
>>>> with this diff i'm getting panic when i'm pushing traffic over that box.
>>>> This is plain forwarding. To compile with witness ?
>>>
>>>
>>> with witness
>>>
>>
>> any chance to check other CPUs to see what code they are executing?
>> I hope to be lucky enough and see thread, which could corrupt the memory.
>
>
> no problem, will do that tomorrow
>
something like this:
x3550m4# pappnaiannc:iicc :p:o ppoolo_oolcla__ddcohoe__gg_eiettt::e m
_mmcbmualg2fkpilc2_:: chppeaag
gceke: ee mmmbppttuyfyp
l Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
110314 50947 0 0x14000 0x2002 softnet
370908 43261 0 0x14000 0x2003 softnet
271599 79025 0 0x14000 0x2000 softnet
121483 71835 0 0x14000 0x2001 softnet
db_enter() at db_enter+0x10
panic(81df855b) at panic+0x12a
pool_do_get(8221e568,2,800026e43294) at pool_do_get+0x309
pool_get(8221e568,2) at pool_get+0x95
m_clget(0,2,802) at m_clget+0xdd
ixgbe_get_buf(8015c0e8,e) at ixgbe_get_buf+0xa3
ixgbe_rxfill(8015c0e8) at ixgbe_rxfill+0xae
ixgbe_queue_intr(8011ac40) at ixgbe_queue_intr+0x4f
intr_handler(800026e434b0,800cd700) at intr_handler+0x6e
Xintr_ioapic_edge4_untramp() at Xintr_ioapic_edge4_untramp+0x18f
acpicpu_idle() at acpicpu_idle+0x1ea
sched_idle(800021a33ff0) at sched_idle+0x27e
end trace frame: 0x0, count: 3
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{11}>
ddb{11}> show panic
pool_cache_item_magic_check: mbufpl cpu free list modified: item addr
0xfd8
066b54500+16 0xfd80792f4bbe!=0xb49ab28a8a25d432
ddb{11}> trace
db_enter() at db_enter+0x10
panic(81df855b) at panic+0x12a
pool_do_get(8221e568,2,800026e43294) at pool_do_get+0x309
pool_get(8221e568,2) at pool_get+0x95
m_clget(0,2,802) at m_clget+0xdd
ixgbe_get_buf(8015c0e8,e) at ixgbe_get_buf+0xa3
ixgbe_rxfill(8015c0e8) at ixgbe_rxfill+0xae
ixgbe_queue_intr(8011ac40) at ixgbe_queue_intr+0x4f
intr_handler(800026e434b0,800cd700) at intr_handler+0x6e
Xintr_ioapic_edge4_untramp() at Xintr_ioapic_edge4_untramp+0x18f
acpicpu_idle() at acpicpu_idle+0x1ea
sched_idle(800021a33ff0) at sched_idle+0x27e
end trace frame: 0x0, count: -12
ddb{11}> show all locks
Process 59777 (ld) thread 0x800026ea97a8 (7337)
exclusive rrwlock inode r = 0 (0xfd887e57bc58)
#0 witness_lock+0x339
#1 rw_enter+0x286
#2 rrw_enter+0x56
#3 VOP_LOCK+0x33
#4 vn_lock+0x84
#5 uvn_get+0x15f
#6 uvm_fault_lower+0x2ae
#7 uvm_fault+0x19e
#8 upageflttrap+0x69
#9 usertrap+0x18b
#10 recall_trap+0x8
Process 50947 (softnet) thread 0x800026dc8a80 (110314)
shared rwlock netlock r = 0 (0x820e0ea8)
#0 witness_lock+0x339
#1 if_input_process+0x43
#2 ifiq_process+0x69
#3 taskq_thread+0x9f
#4 proc_trampoline+0x1c
shared rwlock softnet r = 0 (0x80030370)
#0 witness_lock+0x339
#1 taskq_thread+0x92
#2 proc_trampoline+0x1c
ddb{11}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
597777337 5166 0 3 0x2 biowait ld
5166 135990 73189 0 30x10008a sigsusp sh
73189 29786 50754 0 30x10008a sigsusp make
19125584 1 0 30x100083 ttyin ksh
50754 423311 1 0 30x10008b sigsusp ksh
41817 523177 1 0 30x100098 poll cron
55124 161629 41428720 30x90 kqreadlldpd
41428 491873 1 0 30x80 netio lldpd
1877 283481 6199 95 30x100092 kqreadsmtpd
65700 262335 6199103 30x100092 kqreadsmtpd
7456 289409 6199 95 30x100092 kqreadsmtpd
4455 333406 6199 95 30x100092 kqreadsmtpd
11921 364120 6199 95 30x100092 kqreadsmtpd
28535 449450 6199 95 30x100092 kqreadsmtpd
6199 140680 1 0 30x100080 kqreadsmtpd
69702 89036 1 0 30x80 selectsshd
98630 388346 1 0 30x100080 poll ntpd
65290 297654 68128 83 30x100092 poll ntpd
68128 253220 1 83 30x100092 poll ntpd
8458 234468 4179 73 30x100090 kqreadsyslogd
4179 69049 1 0 30x100082 netio syslogd
25880 194929 0 0 3 0x14200 bored smr
22006 336506 0 0 3 0x14200 pgzerozerothrea
Re: [External] : Re: running network stack forwarding in parallel
On 22.4.2021. 0:31, Alexandr Nedvedicky wrote: > Hello, > > >>> Hi, >>> >>> with this diff i'm getting panic when i'm pushing traffic over that box. >>> This is plain forwarding. To compile with witness ? >> >> >> with witness >> > > any chance to check other CPUs to see what code they are executing? > I hope to be lucky enough and see thread, which could corrupt the memory. no problem, will do that tomorrow
Re: running network stack forwarding in parallel
On 22.4.2021. 0:26, Alexander Bluhm wrote: > On Wed, Apr 21, 2021 at 11:28:17PM +0200, Hrvoje Popovski wrote: >> with this diff i'm getting panic when i'm pushing traffic over that box. > > Thanks for testing. > >> I'm sending traffic from host connected on ix0 from address 10.10.0.1 to >> host connected to ix1 to addresses 10.11.0.1 - 10.11.255.255 at cca 10Mpps > > I don't see the panic, but for you it is easily reproducable. I > use only 1 destination address, but you have 65000. Maybe is is a > routing or ARP issue. > >> x3550m4# panic: pool_cache_item_magic_check: mbufpl cpu free list >> modified: item addr 0xfd8066bbd6 > > This is a use after free bug with the mbuf. Either our pool is not > MP safe or mbuf handling anywhere in the driver or network stack > is buggy. > > As a wild guess, you could apply this diff on top. Something similar > has fixed IPv6 NDP problem I have seen. Maybe it is in the routing > table, that is used for ARP and NDP. > > bluhm > With this diff i can't reproduce panic and here are the numbers NAT_TASKQ 1 = 650Kpps NET_TASKQ 4 = 1Mpps NAT_TASKQ 12 = 720Kpps :) PID TID PRI NICE SIZE RES STATE WAIT TIMECPU COMMAND 94375 474911 6400K 1260K onproc/7 - 2:27 99.02% softnet 65830 241852 6400K 1260K onproc/1 - 2:26 99.02% softnet 74140 395507 6400K 1260K onproc/8 - 2:26 99.02% softnet 44640 313279 6400K 1260K onproc/2 - 2:26 99.02% softnet 42633 112756 6400K 1260K onproc/5 - 2:26 99.02% softnet 39742 473606 6400K 1260K onproc/11 - 2:26 99.02% softnet 77284 459909 6400K 1260K onproc/3 - 2:26 99.02% softnet 6070 206045 6400K 1260K onproc/10 - 2:26 99.02% softnet 31495 401200 6400K 1260K onproc/4 - 2:26 99.02% softnet 99034 582427 6400K 1260K onproc/9 - 2:26 99.02% softnet 61432 149664 1000K 1260K sleep/0 bored 0:17 14.26% softnet
Re: running network stack forwarding in parallel
On 21.4.2021. 23:28, Hrvoje Popovski wrote:
> On 21.4.2021. 21:36, Alexander Bluhm wrote:
>> Hi,
>>
>> For a while we are running network without kernel lock, but with a
>> network lock. The latter is an exclusive sleeping rwlock.
>>
>> It is possible to run the forwarding path in parallel on multiple
>> cores. I use ix(4) interfaces which provide one input queue for
>> each CPU. For that we have to start multiple softnet tasks and
>> replace the exclusive lock with a shared lock. This works for IP
>> and IPv6 input and forwarding, but not for higher protocols.
>>
>> So I implement a queue between IP and higher layers. We had that
>> before when we were using netlock for IP and kernel lock for TCP.
>> Now we have shared lock for IP and exclusive lock for TCP. By using
>> a queue, we can upgrade the lock once for multiple packets.
>>
>> As you can see here, forwardings performance doubles from 4.5x10^9
>> to 9x10^9 . Left column is current, right column is with my diff.
>> The other dots at 2x10^9 are with socket splicing which is not
>> affected.
>> http://bluhm.genua.de/perform/results/2021-04-21T10%3A50%3A37Z/gnuplot/forward.png
>>
>> Here are all numbers with various network tests.
>> http://bluhm.genua.de/perform/results/2021-04-21T10%3A50%3A37Z/perform.html
>> TCP performance gets less deterministic due to the addition queue.
>>
>> Kernel stack flame graph looks like this. Machine uses 4 CPU.
>> http://bluhm.genua.de/files/kstack-multiqueue-forward.svg
>>
>> Note the kernel lock around nd6_resolve(). I hat to put it there
>> as I have seen an MP related crash there. This can be fixed
>> independently of this diff.
>>
>> We need more MP preassure to find such bugs and races. I think now
>> is a good time to give this diff broader testing and commit it.
>> You need interfaces with multiple queues to see a difference.
>>
>> ok?
> Hi,
>
> with this diff i'm getting panic when i'm pushing traffic over that box.
> This is plain forwarding. To compile with witness ?
with witness
x3550m4# panic: pool_cache_item_magic_check: mbufpl cpu free list
modified: item addr 0xfd8066b5e5
00+16 0xfd8066b5e570!=0x1474deeb99bfdf06
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
*211939 58019 0 0x14000 0x2001 softnet
173790 68166 0 0x14000 0x2003 softnet
45539 46127 0 0x14000 0x2002 softnet
358228 28782 0 0x14000 0x2004 softnet
db_enter() at db_enter+0x10
panic(81df726e) at panic+0x12a
pool_cache_get(82203378) at pool_cache_get+0x25b
pool_get(82203378,2) at pool_get+0x5e
m_clget(0,2,802) at m_clget+0xdd
ixgbe_get_buf(8015c9f8,a) at ixgbe_get_buf+0xa3
ixgbe_rxfill(8015c9f8) at ixgbe_rxfill+0x93
ixgbe_queue_intr(8011aec0) at ixgbe_queue_intr+0x4f
intr_handler(800026df9740,800cc500) at intr_handler+0x6e
Xintr_ioapic_edge0_untramp() at Xintr_ioapic_edge0_untramp+0x18f
ip_forward(fd8066b58400,8015a048,fd878909fa80,0) at
ip_forward+0x1de
ip_input_if(800026df9a38,800026df9a44,4,0,8015a048) at
ip_input_if+0x608
ipv4_input(8015a048,fd8066b58400) at ipv4_input+0x39
if_input_process(8015a048,800026df9ab8) at if_input_process+0x6f
end trace frame: 0x800026df9b00, count: 0
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{1}> show panic
pool_cache_item_magic_check: mbufpl cpu free list modified: item addr
0xfd8
066b5e500+16 0xfd8066b5e570!=0x1474deeb99bfdf06
ddb{1}> trace
db_enter() at db_enter+0x10
panic(81df726e) at panic+0x12a
pool_cache_get(82203378) at pool_cache_get+0x25b
pool_get(82203378,2) at pool_get+0x5e
m_clget(0,2,802) at m_clget+0xdd
ixgbe_get_buf(8015c9f8,a) at ixgbe_get_buf+0xa3
ixgbe_rxfill(8015c9f8) at ixgbe_rxfill+0x93
ixgbe_queue_intr(8011aec0) at ixgbe_queue_intr+0x4f
intr_handler(800026df9740,800cc500) at intr_handler+0x6e
Xintr_ioapic_edge0_untramp() at Xintr_ioapic_edge0_untramp+0x18f
ip_forward(fd8066b58400,8015a048,fd878909fa80,0) at
ip_forward+0x1de
ip_input_if(800026df9a38,800026df9a44,4,0,8015a048) at
ip_input_if+0x608
ipv4_input(8015a048,fd8066b58400) at ipv4_input+0x39
if_input_process(8015a048,800026df9ab8) at if_input_process+0x6f
ifiq_process(8015ef00) at ifiq_process+0x69
taskq_thread(80030300) at taskq_thread+0x9f
end trace frame: 0x0, count: -16
ddb{1}> show locks
shared rwlock netlock r = 0 (0x82119770)
#0 witness_loc
Re: running network stack forwarding in parallel
On 21.4.2021. 21:36, Alexander Bluhm wrote:
> Hi,
>
> For a while we are running network without kernel lock, but with a
> network lock. The latter is an exclusive sleeping rwlock.
>
> It is possible to run the forwarding path in parallel on multiple
> cores. I use ix(4) interfaces which provide one input queue for
> each CPU. For that we have to start multiple softnet tasks and
> replace the exclusive lock with a shared lock. This works for IP
> and IPv6 input and forwarding, but not for higher protocols.
>
> So I implement a queue between IP and higher layers. We had that
> before when we were using netlock for IP and kernel lock for TCP.
> Now we have shared lock for IP and exclusive lock for TCP. By using
> a queue, we can upgrade the lock once for multiple packets.
>
> As you can see here, forwardings performance doubles from 4.5x10^9
> to 9x10^9 . Left column is current, right column is with my diff.
> The other dots at 2x10^9 are with socket splicing which is not
> affected.
> http://bluhm.genua.de/perform/results/2021-04-21T10%3A50%3A37Z/gnuplot/forward.png
>
> Here are all numbers with various network tests.
> http://bluhm.genua.de/perform/results/2021-04-21T10%3A50%3A37Z/perform.html
> TCP performance gets less deterministic due to the addition queue.
>
> Kernel stack flame graph looks like this. Machine uses 4 CPU.
> http://bluhm.genua.de/files/kstack-multiqueue-forward.svg
>
> Note the kernel lock around nd6_resolve(). I hat to put it there
> as I have seen an MP related crash there. This can be fixed
> independently of this diff.
>
> We need more MP preassure to find such bugs and races. I think now
> is a good time to give this diff broader testing and commit it.
> You need interfaces with multiple queues to see a difference.
>
> ok?
Hi,
with this diff i'm getting panic when i'm pushing traffic over that box.
This is plain forwarding. To compile with witness ?
I'm sending traffic from host connected on ix0 from address 10.10.0.1 to
host connected to ix1 to addresses 10.11.0.1 - 10.11.255.255 at cca 10Mpps
x3550m4# cat /etc/hostname.ix0
inet 192.168.10.1/24
!route add 10.10/16 192.168.10.11
up
x3550m4# cat /etc/hostname.ix1
192.168.11.1/24
!route add 10.11/16 192.168.11.11
up
10.10/16 192.168.10.11 UGS00 - 8 ix0
10.11/16 192.168.11.11 UGS00 - 8 ix1
x3550m4# panic: pool_cache_item_magic_check: mbufpl cpu free list
modified: item addr 0xfd8066bbd6
00+16 0xfd80669b017e!=0x2e1e87c6c9cef869
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
392766 26556 0 0x14000 0x2001 softnet
201422 58655 0 0x14000 0x2002 softnet
510972 89691 0 0x14000 0x2004 softnet
175943 76775 0 0x14000 0x2003 softnet
db_enter() at db_enter+0x10
panic(81ded24c) at panic+0x12a
pool_cache_get(8212a2c8) at pool_cache_get+0x25b
pool_get(8212a2c8,2) at pool_get+0x5e
m_clget(0,2,802) at m_clget+0xce
ixgbe_get_buf(8015b2b8,11) at ixgbe_get_buf+0xa3
ixgbe_rxfill(8015b2b8) at ixgbe_rxfill+0xae
ixgbe_queue_intr(8011acc0) at ixgbe_queue_intr+0x4f
intr_handler(800026e7d8f0,800bfd00) at intr_handler+0x6e
Xintr_ioapic_edge4_untramp() at Xintr_ioapic_edge4_untramp+0x18f
acpicpu_idle() at acpicpu_idle+0x1ea
sched_idle(800021a61ff0) at sched_idle+0x27e
end trace frame: 0x0, count: 3
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{9}>
ddb{9}> show panic
pool_cache_item_magic_check: mbufpl cpu free list modified: item addr
0xfd8
066bbd600+16 0xfd80669b017e!=0x2e1e87c6c9cef869
ddb{9}> trace
db_enter() at db_enter+0x10
panic(81ded24c) at panic+0x12a
pool_cache_get(8212a2c8) at pool_cache_get+0x25b
pool_get(8212a2c8,2) at pool_get+0x5e
m_clget(0,2,802) at m_clget+0xce
ixgbe_get_buf(8015b2b8,11) at ixgbe_get_buf+0xa3
ixgbe_rxfill(8015b2b8) at ixgbe_rxfill+0xae
ixgbe_queue_intr(8011acc0) at ixgbe_queue_intr+0x4f
intr_handler(800026e7d8f0,800bfd00) at intr_handler+0x6e
Xintr_ioapic_edge4_untramp() at Xintr_ioapic_edge4_untramp+0x18f
acpicpu_idle() at acpicpu_idle+0x1ea
sched_idle(800021a61ff0) at sched_idle+0x27e
end trace frame: 0x0, count: -12
ddb{9}>
x3550m4$ vmstat -iz
interrupt total rate
irq0/clock 197062 1152
irq0/ipi 6271 36
irq96/acpi0 00
irq97/ppb0 00
irq114/ixl0210
irq115/ixl0:0 50
irq116/ixl0:1 00
irq117/ixl0:2 00
irq118/ixl0:3 00
irq119/ixl0:4
Re: use 64bit ethernet addresses in carp(4)
On 5.3.2021. 6:11, David Gwynne wrote: > this passes the destination ethernet address from the network packet > as a uint64_t from ether_input into carp_input, so it can use it > to see if a carp interface should take the packet. > > it's been working on amd64 and sparc64. anyone else want to try? Hi, i'm hitting this diff on routers and doing ifconfig carp1 destroy && ifconfig carp0 destroy && sleep 3 && sh /etc/netstart && sleep 3 on both routers while sending traffic over carp interfaces .. both routers seems stable ..
pkg_add quirks log in snapshot
Hi all, i'm getting this log after update to latest snapshot pkg_add -ui quirks-3.580 signed on 2021-02-24T18:23:18Z |No change in quirks-3.580String found where operator expected at /usr/local/libdata/perl5/site_perl/OpenBSD/Quirks.pm line 2196, near ""Upstrem moved to unversioned tarballs, use the plan9port (same upstream) package instead"" (Missing semicolon on previous line?) Can't load quirk: syntax error at /usr/local/libdata/perl5/site_perl/OpenBSD/Quirks.pm line 2196, near ""Upstrem moved to unversioned tarballs, use the plan9port (same upstream) package instead"" Compilation failed in require at /usr/libdata/perl5/OpenBSD/AddDelete.pm line 350. pkg_info bash-5.1.4 GNU Bourne Again Shell gettext-runtime-0.21p1 GNU gettext runtime libraries and programs intel-firmware-20210216v0 microcode update binaries for Intel CPUs ipcalc-1.4p0small network calculator libevent-2.1.11 event notification library libiconv-1.16p0 character set conversion library libqrencode-4.1.1 library for encoding data in a QR Code symbol lldpd-1.0.8 LLDP (802.1ab)/CDP/EDP/SONMP/FDP daemon and SNMP subagent png-1.6.37 library for manipulating PNG images quirks-3.580exceptions to pkg_add rules reposync-20210113 script to update an OpenBSD CVS repository via rsync rsync-3.2.3 mirroring/synchronization over low bandwidth links vmm-firmware-1.11.0p3 firmware binary images for vmm(4) driver wireguard-tools-1.0.20200827v0 fast and secure VPN OpenBSD 6.9-beta (GENERIC.MP) #358: Wed Feb 24 17:11:53 MST 2021 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17055485952 (16265MB) avail mem = 16523235328 (15757MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xed9b0 (40 entries) bios0: vendor American Megatrends Inc. version "2.1" date 11/22/2019 bios0: Supermicro Super Server acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG UEFI DBG2 HPET WDDT SSDT SSDT SSDT PRAD DMAR HEST BERT ERST EINJ acpi0: wakeup devices IP2P(S4) EHC1(S4) EHC2(S4) RP01(S4) RP02(S4) RP03(S4) RP04(S4) RP05(S4) RP06(S4) RP07(S4) RP08(S4) BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4) BR2C(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.30 MHz, 06-56-03 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 100MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.02 MHz, 06-56-03 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.02 MHz, 06-56-03 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.02 MHz, 06-56-03 cpu3:
Re: switch(4): fix netlock assertion within ifpromisc()
On 19.2.2021. 21:50, Vitaliy Makkoveev wrote: > As it was reported [1] switch(4) triggers NET_ASSERT_LOCKED() while > we perform ifconfig(8) destroy. ifpromisc() requires netlock to be held. > This is true while switch_port_detach() and underlay ifpromisc() called > through switch_ioctl(). But while we destroy switch(4) interface we call > ifpromisc() without netlock. We can't add netlock just around > ifpromisc() because switch_port_detach() which calls it called by > switch_ioctl() with netlock held and by switch_clone_destroy() without > netlock held. So the solution is to add netlock to destroy path. Diff > below adds it around the whole foreach loop where we call > switch_port_detach(). > > 1. https://marc.info/?l=openbsd-bugs=161338077403538=2 i'm confirming that there are no more logs .. tnx ..
Re: i386 pmap diff
On 23.12.2020. 18:24, Mark Kettenis wrote:
> Diff below switches the i386 pmap to use the modern km_alloc(9)
> functions and uses IPL_VM for the pmap pool, following the example of
> amd64.
>
> Don't have easy access to an i386 machine right now, so this has only
> been compile tested.
>
> ok (if it works)?
Hi,
i've checkout tree few minutes ago and i'm getting this log when
applying diff
r620-2# cat arch/powerpc64/powerpc64/pmap.c.rej
@@ -995,7 +995,7 @@
{
int i;
- pool_init(_pmap_pool, sizeof(struct pmap), 0, IPL_NONE, 0,
+ pool_init(_pmap_pool, sizeof(struct pmap), 0, IPL_MPFLOOR, 0,
"pmap", _allocator_single);
pool_setlowat(_pmap_pool, 2);
pool_init(_vp_pool, sizeof(struct pmapvp1), 0, IPL_VM, 0,
but i've complied kernel with your diff and i'm getting this panic
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
panic: uvm_fault(0xd0e56c00, 0xf552f000, 0, 1) -> e
Stopped at db_enter+0x4: popl%ebp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
* 0 0 0 0x1 0x2000K swapper
db_enter(d0e2fab5,d110aa54,0,f552f000,d0e798c8) at db_enter+0x4
panic(d0c3302c,d0e56c00,f552f000,1,e) at panic+0xd3
kpageflttrap(d110aac4,f552f438,f552f438,,0) at kpageflttrap+0x14d
trap(d110aac4) at trap+0x26a
calltrap(8,10806,d0a46644,d110ab28,fee0) at calltrap+0xc
pmap_enter_special_pae(d0e1,fee0,3,10) at
pmap_enter_special_pae+0x21e
lapic_boot_init(fee0) at lapic_boot_init+0x43
acpimadt_attach(d6f08400,d6f34fc0,d110ac58) at acpimadt_attach+0x12c
config_attach(d6f08400,d0e202e0,d110ac58,d05abc50) at config_attach+0x18a
config_found_sm(d6f08400,d110ac58,d05abc50,d05ae190) at config_found_sm+0x29
acpi_attach_common(d6f08400,f0c40) at acpi_attach_common+0x556
acpi_attach(d6f120c0,d6f08400,d110adb8) at acpi_attach+0x2c
config_attach(d6f120c0,d0e204a0,d110adb8,d068fe40) at config_attach+0x18a
config_found_sm(d6f120c0,d110adb8,d068fe40,0) at config_found_sm+0x29
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
Re: Kernel panic with i386 on latest snapshot
On 15.12.2020. 18:57, Mark Kettenis wrote:
>> From: jungle Boogie
>> Date: Tue, 15 Dec 2020 08:07:04 -0800
>>
>> Hi All,
>>
>> On my i386 Toshiba netbook machine, I am getting a kernel panic with
>> the latest i386 snapshot.
>>
>> I hope this information helps someone with the issue.
>>
>>> show panic
>> kernel diagnostic assertion "_kernel_lock_held()" failed:
>> "/usr/src/sys/uvm/uvm_km.c", line 246
>>
>>> bt
>> db_enter(d0bc6fab,d0c2da31,d0c3a6bb,d0e36b7c,d0e36b7c) at db_enter+0x4
>> panic(d0bc6fab, d0c2da31, d0c3a6bb, d0c51f06, f6) at panic+0xd3
>> _assert(d0c2da31,d0c51f06,f6,d0c3a6bb,d0e71330) at _assert+0x19
>> uvm_km_pgremove(d0e578ec,2552c000,2553000) at uvm_km_pgremove+0x119
>> uvm_umap_kill_entry(d0e36b7c,d0e71330) at uvm_unmap_kill_entry+0x92
>> uvm_unmap(d0e36b7c,f552c000,f553) at uvm_unmap+0x53
>> uvm_km_free(d0e36b7c,f552c000,4000,4000) at uvm_km_free+0x25
>> cpu_ucode_setup(f092c000,f080,efff9000,8d565328,1107000) at
>> cpu_ucode_setup+0xeb
>> cpu_startup(8d565328,1107000,1116000,110a000,0) at cpu_startup+0x14a
>> main(0,0,0,0) at main+0x6b
>
> Does the diff below fix this?
>
Yes, it fixes panic.. tnx ..
OpenBSD 6.8-current (GENERIC.MP) #18: Tue Dec 15 19:36:39 CET 2020
hrv...@r620-2.srce.hr:/sys/arch/i386/compile/GENERIC.MP
real mem = 3441889280 (3282MB)
avail mem = 3362791424 (3207MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: date 12/06/19, BIOS32 rev. 0 @ 0xf14a0, SMBIOS rev.
2.7 @ 0xcf42c000 (99 entries)
bios0: vendor Dell Inc. version "2.9.0" date 12/06/2019
bios0: Dell Inc. PowerEdge R620
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WDAT SLIC ERST HEST
BERT EINJ TCPA PC__ SRAT SSDT
acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 4 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz ("GenuineIntel"
686-class) 3.61 GHz, 06-3e-04
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 100MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 6 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz ("GenuineIntel"
686-class) 3.51 GHz, 06-3e-04
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2 at mainbus0: apid 8 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz ("GenuineIntel"
686-class) 3.51 GHz, 06-3e-04
cpu2:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3 at mainbus0: apid 16 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz ("GenuineIntel"
686-class) 3.51 GHz, 06-3e-04
cpu3:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu4 at mainbus0: apid 18 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz ("GenuineIntel"
686-class) 3.51 GHz, 06-3e-04
cpu4:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu5 at mainbus0: apid 20 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz ("GenuineIntel"
686-class) 3.51 GHz, 06-3e-04
cpu5:
Re: Kernel panic with i386 on latest snapshot
On 15.12.2020. 17:07, jungle Boogie wrote:
> Hi All,
>
> On my i386 Toshiba netbook machine, I am getting a kernel panic with
> the latest i386 snapshot.
>
> I hope this information helps someone with the issue.
>
>> show panic
> kernel diagnostic assertion "_kernel_lock_held()" failed:
> "/usr/src/sys/uvm/uvm_km.c", line 246
>
>> bt
> db_enter(d0bc6fab,d0c2da31,d0c3a6bb,d0e36b7c,d0e36b7c) at db_enter+0x4
> panic(d0bc6fab, d0c2da31, d0c3a6bb, d0c51f06, f6) at panic+0xd3
> _assert(d0c2da31,d0c51f06,f6,d0c3a6bb,d0e71330) at _assert+0x19
> uvm_km_pgremove(d0e578ec,2552c000,2553000) at uvm_km_pgremove+0x119
> uvm_umap_kill_entry(d0e36b7c,d0e71330) at uvm_unmap_kill_entry+0x92
> uvm_unmap(d0e36b7c,f552c000,f553) at uvm_unmap+0x53
> uvm_km_free(d0e36b7c,f552c000,4000,4000) at uvm_km_free+0x25
> cpu_ucode_setup(f092c000,f080,efff9000,8d565328,1107000) at
> cpu_ucode_setup+0xeb
> cpu_startup(8d565328,1107000,1116000,110a000,0) at cpu_startup+0x14a
> main(0,0,0,0) at main+0x6b
>
> I've had no panics on amd64.
>
> Thanks!
>
Hi,
i can confirm this panic on i386 on Dell R620
OpenBSD 6.8-current (GENERIC.MP) #561: Tue Dec 15 03:03:30 MST 2020
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC.MP
real mem = 3441889280 (3282MB)
avail mem = 3362799616 (3207MB)
panic: kernel diagnostic assertion "_kernel_lock_held()" failed: file
"/usr/src$
sys/uvm/uvm_km.c", line 246
Stopped at db_enter+0x4: popl%ebp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
db_enter(d0e4901c,d1109e68,d0e48600,d0e2f5e0,d0e2f5e0) at db_enter+0x4
panic(d0bc86cd,d0c2d366,d0c3a01b,d0c508c6,f6) at panic+0xd3
__assert(d0c2d366,d0c508c6,f6,d0c3a01b,d0e48600) at __assert+0x19
uvm_km_pgremove(d0e2edbc,2552c000,2553) at uvm_km_pgremove+0x119
uvm_unmap_kill_entry(d0e2f5e0,d0e48600) at uvm_unmap_kill_entry+0x92
uvm_unmap_remove(d0e2f5e0,f552c000,f553,d1109f10,0,1) at
uvm_unmap_remove+0x1cb
uvm_unmap(d0e2f5e0,f552c000,f553) at uvm_unmap+0x53
uvm_km_free(d0e2f5e0,f552c000,4000,4000) at uvm_km_free+0x25
cpu_ucode_setup(f092c000,f080,efff9000,68cac0bc,1107000) at
cpu_ucode_setup+0xeb
cpu_startup(68cac0bc,1107000,1116000,110a000,0) at cpu_startup+0x14a
main(0,0,0,0,0) at main+0x6b
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
ddb{0}> show panic
kernel diagnostic assertion "_kernel_lock_held()" failed: file
"/usr/src/sys/uvm/uvm_km.c", line 246
ddb{0}> trace
db_enter(d0e4901c,d1109e68,d0e48600,d0e2f5e0,d0e2f5e0) at db_enter+0x4
panic(d0bc86cd,d0c2d366,d0c3a01b,d0c508c6,f6) at panic+0xd3
__assert(d0c2d366,d0c508c6,f6,d0c3a01b,d0e48600) at __assert+0x19
uvm_km_pgremove(d0e2edbc,2552c000,2553) at uvm_km_pgremove+0x119
uvm_unmap_kill_entry(d0e2f5e0,d0e48600) at uvm_unmap_kill_entry+0x92
uvm_unmap_remove(d0e2f5e0,f552c000,f553,d1109f10,0,1) at
uvm_unmap_remove+0x1cb
uvm_unmap(d0e2f5e0,f552c000,f553) at uvm_unmap+0x53
uvm_km_free(d0e2f5e0,f552c000,4000,4000) at uvm_km_free+0x25
cpu_ucode_setup(f092c000,f080,efff9000,68cac0bc,1107000) at
cpu_ucode_setup+0xeb
cpu_startup(68cac0bc,1107000,1116000,110a000,0) at cpu_startup+0x14a
main(0,0,0,0,0) at main+0x6b
ddb{0}>
ddb{0}> show reg
ds 0x10
es 0x10
fs 0x20
gs 0
edi 0xd0bc86cdacx100_txpower_maxim+0xc67c
esi0x100
ebp 0xd1109e40__kernel_bss_end+0x1c8e40
ebx 0xd1109e68__kernel_bss_end+0x1c8e68
edx0x2fd
ecx0
eax 0x1
eip 0xd05c05b4db_enter+0x4
cs 0x8
eflags 0x2
esp 0xd1109e40__kernel_bss_end+0x1c8e40
ss 0x10
db_enter+0x4: popl%ebp
ddb{0}>
Re: timekeep: fixing large skews on amd64 with RDTSCP
On 23.8.2020. 16:50, Claudio Jeker wrote: > On Sun, Aug 23, 2020 at 04:06:01PM +0200, Christian Weisgerber wrote: >> Scott Cheloha: >> >>> This "it might slow down the network stack" thing keeps coming up, and >>> yet nobody can point to (a) who expressed this concern or (b) what the >>> penalty is in practice. >> >> It was kettenis@ who simply raised the question and asked for >> comments from the network people. >> >> I think we should just go ahead and use rdtsc_lfence() in >> tsc_get_timecount(). It is *correct*. > > I agree. For the network stack there are bigger fishes to fry befor such a > micro optimisation would even matter. > Hi all, as you said, with this diff forwarding performance is little slower. 6 x Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.54 MHz, 06-3e-04 without diff 1.11 Mpps with diff 1.04 Mpps 12 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 2400.01 MHz, 06-3e-04 without diff 650 Kpps with diff 600 Kpps
Re: kstats for em(4)
On 7.7.2020. 10:51, David Gwynne wrote: > unfortunately em(4) covers a lot of chips of different vintages, so if > anyone has a super old one they can try this diff on with kstat enabled > in their kernel config, that would be appreciated. Hi, don't know if 82576 is old or super old but here are results ... i'm sending udp with random packet size from 64b to 1500b from host connected to em1 to host connected to em0.. em0:0:em-stats:0 rx crc errs: 0 packets rx align errs: 0 packets rx align errs: 0 packets rx errs: 0 packets rx missed: 0 packets tx single coll: 0 packets tx excess coll: 0 packets tx multi coll: 0 packets tx late coll: 0 packets tx coll: 0 tx defers: 0 tx no CRS: 0 packets seq errs: 0 carr ext errs: 0 packets rx len errs: 0 packets rx xon: 0 packets tx xon: 0 packets rx xoff: 0 packets tx xoff: 0 packets FC unsupported: 0 packets rx 64B: 33 packets rx 65-127B: 0 packets rx 128-255B: 1187 packets rx 256-511B: 0 packets rx 512-1023B: 0 packets rx 1024-maxB: 0 packets rx good: 1220 packets rx bcast: 0 packets rx mcast: 1187 packets tx good: 4739699733 packets rx good: 278683 bytes tx good: 6180591344197 bytes rx no buffers: 0 packets rx undersize: 0 packets rx fragments: 0 packets rx oversize: 0 packets rx jabbers: 0 packets rx mgmt: 0 packets rx mgmt drops: 0 packets tx mgmt: 0 packets rx total: 279844 bytes tx total: 4157661749214 bytes rx total: 1233 packets tx total: 4739699732 packets tx 64B: 67510552 packets tx 65-127B: 231664682 packets tx 128-255B: 470662473 packets tx 256-511B: 941174688 packets tx 512-1023B: 1881674253 packets tx 1024-maxB: 1768714760 packets tx mcast: 1188 packets tx bcast: 33 packets em0:0:rxq:0 packets: 1220 packets bytes: 273803 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets em0:0:txq:0 packets: 5361401666 packets bytes: 4136216461234 bytes qdrops: 1334588 packets errors: 0 packets qlen: 0 packets maxqlen: 511 packets oactive: false em1:0:em-stats:0 rx crc errs: 0 packets rx align errs: 0 packets rx align errs: 0 packets rx errs: 0 packets rx missed: 38992552 packets tx single coll: 0 packets tx excess coll: 0 packets tx multi coll: 0 packets tx late coll: 0 packets tx coll: 0 tx defers: 0 tx no CRS: 0 packets seq errs: 0 carr ext errs: 0 packets rx len errs: 0 packets rx xon: 0 packets tx xon: 0 packets rx xoff: 0 packets tx xoff: 0 packets FC unsupported: 0 packets rx 64B: 81172363 packets rx 65-127B: 231666120 packets rx 128-255B: 470687906 packets rx 256-511B: 941373440 packets rx 512-1023B: 1882746880 packets rx 1024-maxB: 1768751378 packets rx good: 4753594331 packets rx bcast: 0 packets rx mcast: 1186 packets tx good: 1188 packets rx good: 17400682469430 bytes tx good: 191268 bytes rx no buffers: 1151692 packets rx undersize: 0 packets rx fragments: 0 packets rx oversize: 0 packets rx jabbers: 0 packets rx mgmt: 0 packets rx mgmt drops: 0 packets tx mgmt: 0 packets rx total: 4161793823633 bytes tx total: 191268 bytes rx total: 4792586884 packets tx total: 1188 packets tx 64B: 0 packets tx 65-127B: 0 packets tx 128-255B: 1188 packets tx 256-511B: 0 packets tx 512-1023B: 0 packets tx 1024-maxB: 0 packets tx mcast: 1188 packets tx bcast: 0 packets em1:0:rxq:0 packets: 5376398072 packets bytes: 4137792685172 bytes qdrops: 13661822 packets errors: 0 packets qlen: 0 packets em1:0:txq:0 packets: 1189 packets bytes: 186558 bytes qdrops: 1 packets errors: 0 packets qlen: 0 packets maxqlen: 511 packets oactive: false
Re: fix races in if_clone_create()
On 29.6.2020. 10:59, Vitaliy Makkoveev wrote:
> I reworked tool for reproduce. Now I avoided fork()/exec() route and it
> takes couple of minutes to take panic on 4 cores. Also some screenshots
> attached.
>
> I hope anyone else will try it.
Hi,
i'm getting panic quite fast :)
i will leave box in ddb if more information is needed
r620-1# ./a.out bridge0
panic: kernel diagnostic assertion "TAILQ_EMPTY(>if_addrhooks)"
failed: file "/sys/net/if.c", line 1168
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
475311 7753 1000 0x3 00 ifconfig
*128110 3280 0 0x3 01K a.out
86419 3280 0 0x3 0x4004 a.out
352360 3280 0 0x3 0x4003 a.out
309715 3280 0 0x3 0x4005 a.out
268210 3280 0 0x3 0x4002 a.out
db_enter() at db_enter+0x10
panic(81df42d3) at panic+0x128
__assert(81e5d55e,81e5b1fa,490,81e408d9) at
__assert+0x2b
if_detach(81169000) at if_detach+0x45f
bridge_clone_destroy(81169000) at bridge_clone_destroy+0x17b
ifioctl(fd839209c828,80206979,8000248fa980,800024902618) at
ifioctl+0x1c2
soo_ioctl(fd83b04b34c8,80206979,8000248fa980,800024902618)
at soo_ioctl+0x171
sys_ioctl(800024902618,8000248faa90,8000248faaf0) at
sys_ioctl+0x2df
syscall(8000248fab60) at syscall+0x389
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7d3600, count: 5
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{1}>
Re: vlan and bridge panic with latest snapshot
On 22.6.2020. 11:11, Claudio Jeker wrote: > On Sun, Jun 21, 2020 at 08:51:53PM +0200, Hrvoje Popovski wrote: >> Hi all, >> >> with today's snapshot from 21-Jun-2020 09:34 >> OpenBSD 6.7-current (GENERIC.MP) #286: Sun Jun 21 08:51:29 MDT 2020 >> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP >> >> if i do "ifconfig vlan" i'm getting assert >> x3550m4# ifconfig vlan >> vlan100: flags=8splassert: vlan_ioctl: want 2 have 0 >> Starting stack trace... >> vlan_ioctl(80bb4800,c02069d3,800021f6f5d0) at vlan_ioctl+0x65 >> ifioctl(fd8785005668,c02069d3,800021f6f5d0,800021ffb130) at >> ifioctl+0x91c >> soo_ioctl(fd8784e6d630,c02069d3,800021f6f5d0,800021ffb130) >> at soo_ioctl+0x171 >> sys_ioctl(800021ffb130,800021f6f6e0,800021f6f740) at >> sys_ioctl+0x2df >> syscall(800021f6f7b0) at syscall+0x389 >> Xsyscall() at Xsyscall+0x128 >> end of kernel >> end trace frame: 0x7f7e53d0, count: 251 >> End of stack trace. >> >> >> with ifconfig bridge0 up everything seems fine but ifconfig bridge0 >> destroy and ifconfig after that get me panic .. >> >> x3550m4# ifconfig >> lo0: flags=8049 msplassert: vlan_ioctl: >> want 2 have 0 >> Starting stack trace... >> vlan_ioctl(80bb4800,c02069d3,800021f6f510) at vlan_ioctl+0x65 >> ifioctl(fd8785005668,c02069d3,800021f6f510,800021ffb130) at >> ifioctl+0x91c >> soo_ioctl(fd8784e6d630,c02069d3,800021f6f510,800021ffb130) >> at soo_ioctl+0x171 >> sys_ioctl(800021ffb130,800021f6f620,800021f6f680) at >> sys_ioctl+0x2df >> syscall(800021f6f6f0) at syscall+0x389 >> Xsyscall() at Xsyscall+0x128 >> end of kernel >> end trace frame: 0x7f7ddd20, count: 251 >> End of stack trace. >> tu 32768 >> indexpanic: netlock: lock not held >> Stopped at db_enter+0x10: popq%rbp >> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND >> *505095 3193 0 0x3 03K ifconfig >> db_enter() at db_enter+0x10 >> panic(81dbfaab) at panic+0x128 >> rw_exit_write(820e6138) at rw_exit_write+0xb5 >> bridge_ioctl(81754000,c02069d3,800021f6f510) at >> bridge_ioctl+0x42 >> ifioctl(fd8785005668,c02069d3,800021f6f510,800021ffb130) at >> ifioctl+0x91c >> soo_ioctl(fd8784e6d630,c02069d3,800021f6f510,800021ffb130) >> at soo_ioctl+0x171 >> sys_ioctl(800021ffb130,800021f6f620,800021f6f680) at >> sys_ioctl+0x2df >> syscall(800021f6f6f0) at syscall+0x389 >> Xsyscall() at Xsyscall+0x128 >> end of kernel >> end trace frame: 0x7f7ddd20, count: 6 >> https://www.openbsd.org/ddb.html describes the minimum info required in >> bugreports. Insufficient info makes it difficult to find and fix bugs. >> > > This crashes are because of wg(4) calling the interface ioctl handler > without holding the netlock() this is not allowed. > > As a quick fix this diff may work. > Hi, for some reason i couldn't reproduce panic if i compile kernel with WITNESS and after that with or without your "if.c if_wg.c" commit Thank you for fix ...
vlan and bridge panic with latest snapshot
Hi all,
with today's snapshot from 21-Jun-2020 09:34
OpenBSD 6.7-current (GENERIC.MP) #286: Sun Jun 21 08:51:29 MDT 2020
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
if i do "ifconfig vlan" i'm getting assert
x3550m4# ifconfig vlan
vlan100: flags=8splassert: vlan_ioctl: want 2 have 0
Starting stack trace...
vlan_ioctl(80bb4800,c02069d3,800021f6f5d0) at vlan_ioctl+0x65
ifioctl(fd8785005668,c02069d3,800021f6f5d0,800021ffb130) at
ifioctl+0x91c
soo_ioctl(fd8784e6d630,c02069d3,800021f6f5d0,800021ffb130)
at soo_ioctl+0x171
sys_ioctl(800021ffb130,800021f6f6e0,800021f6f740) at
sys_ioctl+0x2df
syscall(800021f6f7b0) at syscall+0x389
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7e53d0, count: 251
End of stack trace.
with ifconfig bridge0 up everything seems fine but ifconfig bridge0
destroy and ifconfig after that get me panic ..
x3550m4# ifconfig
lo0: flags=8049 msplassert: vlan_ioctl:
want 2 have 0
Starting stack trace...
vlan_ioctl(80bb4800,c02069d3,800021f6f510) at vlan_ioctl+0x65
ifioctl(fd8785005668,c02069d3,800021f6f510,800021ffb130) at
ifioctl+0x91c
soo_ioctl(fd8784e6d630,c02069d3,800021f6f510,800021ffb130)
at soo_ioctl+0x171
sys_ioctl(800021ffb130,800021f6f620,800021f6f680) at
sys_ioctl+0x2df
syscall(800021f6f6f0) at syscall+0x389
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ddd20, count: 251
End of stack trace.
tu 32768
indexpanic: netlock: lock not held
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
*505095 3193 0 0x3 03K ifconfig
db_enter() at db_enter+0x10
panic(81dbfaab) at panic+0x128
rw_exit_write(820e6138) at rw_exit_write+0xb5
bridge_ioctl(81754000,c02069d3,800021f6f510) at
bridge_ioctl+0x42
ifioctl(fd8785005668,c02069d3,800021f6f510,800021ffb130) at
ifioctl+0x91c
soo_ioctl(fd8784e6d630,c02069d3,800021f6f510,800021ffb130)
at soo_ioctl+0x171
sys_ioctl(800021ffb130,800021f6f620,800021f6f680) at
sys_ioctl+0x2df
syscall(800021f6f6f0) at syscall+0x389
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ddd20, count: 6
https://www.openbsd.org/ddb.html describes the minimum info required in
bugreports. Insufficient info makes it difficult to find and fix bugs.
ddb{3}> show panic
netlock: lock not held
ddb{3}> trace
db_enter() at db_enter+0x10
panic(81dbfaab) at panic+0x128
rw_exit_write(820e6138) at rw_exit_write+0xb5
bridge_ioctl(81754000,c02069d3,800021f6f510) at
bridge_ioctl+0x42
ifioctl(fd8785005668,c02069d3,800021f6f510,800021ffb130) at
ifioctl+0x91c
soo_ioctl(fd8784e6d630,c02069d3,800021f6f510,800021ffb130)
at soo_ioctl+0x171
sys_ioctl(800021ffb130,800021f6f620,800021f6f680) at
sys_ioctl+0x2df
syscall(800021f6f6f0) at syscall+0x389
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ddd20, count: -9
Re: multiple rings and cpus for ix(4)
On 17.6.2020. 13:13, Jonathan Matthew wrote: > On Wed, Jun 17, 2020 at 12:50:46PM +0200, Hrvoje Popovski wrote: >> On 17.6.2020. 12:45, Hrvoje Popovski wrote: >>> On 17.6.2020. 11:27, Hrvoje Popovski wrote: >>>> On 17.6.2020. 10:36, David Gwynne wrote: >>>>> this is an updated version of a diff from christiano haesbaert by way of >>>>> mpi@ to enable the use of multiple tx and rx rings with msi-x. >>>>> >>>>> the high level description is that that driver checks to see if msix is >>>>> available, and if so how many vectors it has. it then gets an intrmap >>>>> based on that information, and bumps the number of queues to the number >>>>> of cpus that intrmap says are available. >>>>> >>>>> once the queues are allocated, it then iterates over them and wires up >>>>> interrupts to the cpus provided by the intrmap. >>>>> >>>>> im happy for people to try this out, but i can't commit it until all the >>>>> architectures that ix(4) is enabled on support the APIs that it's using. >>>>> this basically means it'll work on amd64 (and a little bit on i386), but >>>>> not much else. please hold back your tears and cries of anguish. >>>>> >>>>> thanks to christiano and mpi for doing most of the work leading up to >>>>> this diff :) >>>> >>>> Hi, >>>> >>>> first, thank you all for mq work :) >>>> >>>> with this diff, if i'm sending traffic over ix and at the same time >>>> execute ifconfig ix down/up, forwarding stops until i stop generator, >>>> wait for few seconds and execute ifconfig ix down/up few times and than >>>> forwarding start normally >>> >> >> >> in vmstat i should see ix0:0-5 and ix1:0-5 ? > > vmstat -i only shows interrupts that have actually fired. Use -zi to show > all interrupts. > > This diff doesn't set up RSS, so received packets will only go to the first > vector, which is why only one of the ix1 interrupts has fired. Outgoing > packets are scattered across the tx queues, so all the ix0 interrupts have > fired. yes, thank you .. r620-1# vmstat -iz interrupt total rate irq0/clock4967987 599 irq0/ipi 14405128 1738 irq144/acpi040 irq114/ix0:0 20722297 2501 irq115/ix0:1 15585680 1881 irq116/ix0:2 14654922 1768 irq117/ix0:3861769301 104015 irq118/ix0:4 17121128 2066 irq119/ix0:5 17010524 2053 irq120/ix0 100 irq121/ix1:0 55895380 6746 irq122/ix1:100 irq123/ix1:200 irq124/ix1:300 irq125/ix1:400 irq126/ix1:500 irq127/ix1 160 irq96/ppb1 00 irq97/mfi0 380924 irq98/ppb3 00 irq128/ixl0 00 irq129/ixl0:0 00 irq130/ixl1 00 irq131/ixl1:0 00 irq132/ixl2 70 irq133/ixl2:0 5650 irq134/ixl3 70 irq135/ixl3:0 5590 irq99/ehci0 1390 irq136/em0 230872 irq137/em15590 irq100/ehci1 280 irq101/ahci010 irq145/com0 00 irq146/com1 44110 Total 1022199832 123379
Re: multiple rings and cpus for ix(4)
On 17.6.2020. 12:45, Hrvoje Popovski wrote: > On 17.6.2020. 11:27, Hrvoje Popovski wrote: >> On 17.6.2020. 10:36, David Gwynne wrote: >>> this is an updated version of a diff from christiano haesbaert by way of >>> mpi@ to enable the use of multiple tx and rx rings with msi-x. >>> >>> the high level description is that that driver checks to see if msix is >>> available, and if so how many vectors it has. it then gets an intrmap >>> based on that information, and bumps the number of queues to the number >>> of cpus that intrmap says are available. >>> >>> once the queues are allocated, it then iterates over them and wires up >>> interrupts to the cpus provided by the intrmap. >>> >>> im happy for people to try this out, but i can't commit it until all the >>> architectures that ix(4) is enabled on support the APIs that it's using. >>> this basically means it'll work on amd64 (and a little bit on i386), but >>> not much else. please hold back your tears and cries of anguish. >>> >>> thanks to christiano and mpi for doing most of the work leading up to >>> this diff :) >> >> Hi, >> >> first, thank you all for mq work :) >> >> with this diff, if i'm sending traffic over ix and at the same time >> execute ifconfig ix down/up, forwarding stops until i stop generator, >> wait for few seconds and execute ifconfig ix down/up few times and than >> forwarding start normally > in vmstat i should see ix0:0-5 and ix1:0-5 ? r620-1# vmstat -i interrupt total rate irq0/clock3985752 599 irq0/ipi 3462063 520 irq144/acpi040 irq114/ix0:0 8042709 1209 irq115/ix0:1 2906070 437 irq116/ix0:2 1975350 297 irq117/ix0:3849089681 127721 irq118/ix0:4 4441608 668 irq119/ix0:5 4330871 651 irq120/ix0 100 irq121/ix1:0 43209056 6499 irq127/ix1 160 irq97/mfi0 368465 irq132/ixl2 70 irq133/ixl2:0 4590 irq134/ixl3 70 irq135/ixl3:0 4510 irq99/ehci0 1390 irq136/em0 186372 irq137/em14510 irq100/ehci1 280 irq101/ahci010 irq146/com1 44110 Total 921504627 138613 > dmesg > > ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01, msix, 6 queues, > address ec:f4:bb:da:f7:f8 > ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01, msix, 6 queues, > address ec:f4:bb:da:f7:fa > > > > OpenBSD 6.7-current (GENERIC.MP) #71: Wed Jun 17 10:53:55 CEST 2020 > hrv...@r620-1.srce.hr:/sys/arch/amd64/compile/GENERIC.MP > real mem = 17115840512 (16322MB) > avail mem = 16582168576 (15813MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries) > bios0: vendor Dell Inc. version "2.9.0" date 12/06/2019 > bios0: Dell Inc. PowerEdge R620 > acpi0 at bios0: ACPI 3.0 > acpi0: sleep states S0 S4 S5 > acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WDAT SLIC ERST HEST > BERT EINJ TCPA PC__ SRAT SSDT > acpi0: wakeup devices PCI0(S5) > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 4 (boot processor) > cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.48 MHz, 06-3e-04 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 2, package 0 > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges > cpu0: apic clock running at 99MHz > cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE > cpu1 at mainbus0: apid 6 (application processor) > cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.00 MHz, 06-3e-04 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAI
Re: multiple rings and cpus for ix(4)
On 17.6.2020. 11:27, Hrvoje Popovski wrote: > On 17.6.2020. 10:36, David Gwynne wrote: >> this is an updated version of a diff from christiano haesbaert by way of >> mpi@ to enable the use of multiple tx and rx rings with msi-x. >> >> the high level description is that that driver checks to see if msix is >> available, and if so how many vectors it has. it then gets an intrmap >> based on that information, and bumps the number of queues to the number >> of cpus that intrmap says are available. >> >> once the queues are allocated, it then iterates over them and wires up >> interrupts to the cpus provided by the intrmap. >> >> im happy for people to try this out, but i can't commit it until all the >> architectures that ix(4) is enabled on support the APIs that it's using. >> this basically means it'll work on amd64 (and a little bit on i386), but >> not much else. please hold back your tears and cries of anguish. >> >> thanks to christiano and mpi for doing most of the work leading up to >> this diff :) > > Hi, > > first, thank you all for mq work :) > > with this diff, if i'm sending traffic over ix and at the same time > execute ifconfig ix down/up, forwarding stops until i stop generator, > wait for few seconds and execute ifconfig ix down/up few times and than > forwarding start normally dmesg ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01, msix, 6 queues, address ec:f4:bb:da:f7:f8 ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01, msix, 6 queues, address ec:f4:bb:da:f7:fa OpenBSD 6.7-current (GENERIC.MP) #71: Wed Jun 17 10:53:55 CEST 2020 hrv...@r620-1.srce.hr:/sys/arch/amd64/compile/GENERIC.MP real mem = 17115840512 (16322MB) avail mem = 16582168576 (15813MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries) bios0: vendor Dell Inc. version "2.9.0" date 12/06/2019 bios0: Dell Inc. PowerEdge R620 acpi0 at bios0: ACPI 3.0 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WDAT SLIC ERST HEST BERT EINJ TCPA PC__ SRAT SSDT acpi0: wakeup devices PCI0(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 4 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.48 MHz, 06-3e-04 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 2, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 6 (application processor) cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.00 MHz, 06-3e-04 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 3, package 0 cpu2 at mainbus0: apid 8 (application processor) cpu2: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.00 MHz, 06-3e-04 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 4, package 0 cpu3 at mainbus0: apid 16 (application processor) cpu3: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.00 MHz, 06-3e-04 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 8, package 0 cpu4 at mainbus0: apid 18 (application processor) cpu4: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04 cpu4: FPU,VME,DE,PSE,TSC,
Re: multiple rings and cpus for ix(4)
On 17.6.2020. 10:36, David Gwynne wrote: > this is an updated version of a diff from christiano haesbaert by way of > mpi@ to enable the use of multiple tx and rx rings with msi-x. > > the high level description is that that driver checks to see if msix is > available, and if so how many vectors it has. it then gets an intrmap > based on that information, and bumps the number of queues to the number > of cpus that intrmap says are available. > > once the queues are allocated, it then iterates over them and wires up > interrupts to the cpus provided by the intrmap. > > im happy for people to try this out, but i can't commit it until all the > architectures that ix(4) is enabled on support the APIs that it's using. > this basically means it'll work on amd64 (and a little bit on i386), but > not much else. please hold back your tears and cries of anguish. > > thanks to christiano and mpi for doing most of the work leading up to > this diff :) Hi, first, thank you all for mq work :) with this diff, if i'm sending traffic over ix and at the same time execute ifconfig ix down/up, forwarding stops until i stop generator, wait for few seconds and execute ifconfig ix down/up few times and than forwarding start normally
Re: em(4) hw setup vs queues
On 3.3.2020. 11:37, Martin Pieuchot wrote: > Currently em_hw_init() uses some hardcorded values to configure TX > rings. Diff below convert it to use the value of the first queue. > This is currently a no-op. It makes the code consistent with the > rest of the driver and reduce the size of upcoming diffs. > > Note that even if a single queue is currently used two of them are > setup. Document this has an historical behavior and keep it like it > is, there's not much need to introduce regression here :) no regression found on em0 at pci1 dev 0 function 0 "Intel 82571EB" rev 0x06: apic 0 int 16 em2 at pci0 dev 25 function 0 "Intel 82579LM" rev 0x04: msi em2 at pci2 dev 0 function 0 "Intel 82572EI" rev 0x06: apic 9 int 8 em3 at pci6 dev 0 function 0 "Intel I350" rev 0x01: msi em1 at pci1 dev 0 function 1 "Intel 82576" rev 0x01: msi
Re: em(4) towards multiqueues
On 14.2.2020. 18:28, Martin Pieuchot wrote: > I'm running this on: > > em0 at pci1 dev 0 function 0 "Intel I210" rev 0x03: msi > em0 at pci0 dev 20 function 0 "Intel I354 SGMII" rev 0x03: msi > > More tests are always welcome ;) em0 at pci0 dev 25 function 0 "Intel 82579LM" rev 0x04: msi em1 at pci1 dev 0 function 1 "Intel 82571EB" rev 0x06: apic 0 int 17 em4 at pci2 dev 0 function 1 "Intel 82576" rev 0x01: msi em5 at pci3 dev 0 function 0 "Intel 82572EI" rev 0x06: apic 0 int 16 em0 at pci9 dev 0 function 0 "Intel I350" rev 0x01: msi it just works .. :)
