Re: TCP FIN hangups in encrypted ESP tunnel
Hi Peter, it's not just you, I have similar problems since around July 1, but with a netcup server. Since then, downloading a bigger file from the netcup server using scp or rsync fails pretty consistently. Normal ssh sessions or other stuff like imap or xmpp remain stable, as far as I can tell. I run the scp/rsync over wg, but it doesn't matter, happens over pppoe too. Like you, I also spent the last evenings looking for mistakes on my side, besides having this working for years. So now I guess the problem is on their side or somewhere in between? I see the following when the file transfer fails: 192.168.100.1 is my router, where I run "scp 192.168.100.2:dump.gz ." 192.168.100.2 is the netcup server 237470 28.285237 192.168.100.1 -> 192.168.100.2 TCP 56 12534 -> 22 [ACK] Seq=55922 Ack=195360998 Win=120512 Len=0 TSval=2630531475 TSecr=89901171 237471 28.285242 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237472 28.285260 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237473 28.285288 192.168.100.1 -> 192.168.100.2 TCP 56 12534 -> 22 [ACK] Seq=55922 Ack=195363734 Win=117824 Len=0 TSval=2630531475 TSecr=89901171 237474 28.285293 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237475 28.285311 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237476 28.285339 192.168.100.1 -> 192.168.100.2 TCP 56 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=115072 Len=0 TSval=2630531475 TSecr=89901171 237477 28.285348 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: [TCP Previous segment not captured] , Encrypted packet (len=1368) 237478 28.285382 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Dup ACK 237476#1] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=115072 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195369206 237479 28.285498 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Window Update] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=123264 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195369206 237480 28.285863 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237481 28.285906 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Dup ACK 237476#2] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=123264 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195370574 237482 28.285914 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237483 28.285941 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Dup ACK 237476#3] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=123264 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195371942 237484 28.285946 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237485 28.285973 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Dup ACK 237476#4] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=123264 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195373310 237486 28.285979 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237487 28.286006 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Dup ACK 237476#5] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=123264 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195374678 237488 28.286016 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237489 28.286044 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Dup ACK 237476#6] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=123264 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195376046 237490 28.286054 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: Encrypted packet (len=1368) 237491 28.286081 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Dup ACK 237476#7] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=123264 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195377414 237492 28.286343 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Window Update] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=131456 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195377414 237493 28.286421 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Window Update] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=139648 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195377414 237494 28.287076 192.168.100.2 -> 192.168.100.1 TCP 56 22 -> 12534 [FIN, ACK] Seq=195377414 Ack=55922 Win=16384 Len=0 TSval=89901171 TSecr=2630531475 237495 28.287141 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Dup ACK 237476#8] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=139648 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195377414 237496 28.288062 192.168.100.1 -> 192.168.100.2 TCP 68 [TCP Window Update] 12534 -> 22 [ACK] Seq=55922 Ack=195366470 Win=147712 Len=0 TSval=2630531475 TSecr=89901171 SLE=195367838 SRE=195377414 237497 28.288586 192.168.100.1 -> 192.168.100.2 SSHv2 104 Client: Encrypted packet (len=36) 237498 28.295439 192.168.100.2 -> 192.168.100.1 SSHv2 1424 Server: [TCP Fast Retransmission] ,
Re: TCP FIN hangups in encrypted ESP tunnel
> On Jul 8, 2021, at 8:05 AM, Peter J. Philipp wrote: > > On Wed, Jul 07, 2021 at 11:57:50PM +0300, Ville Valkonen wrote: >> Hi, >> >> not sure if related but my Linux box (also in Hetzner) also started to have >> flaky connection lately. >> >> -- >> Regards, >> Ville > > I opened a ticket with Hetzner last week thinking it was an in-band DoS. They > assured me, they are not seeing this. > > My VPS is in Falkenstein for what it's worth. Because the problems started > occuring as I was upgrading my Telekom.de link I thought it was related to > that > until I did tcpdumps. I mentioned it to the telekom.de chat help line > despite. > > On your Linux box have you done any debugging as to why it became flaky? > > Some Linux equivalents that I know: ktrace/strace, tcpdump is the same. Are > you seeing these through an IPSEC tunnel or in plain Internetworking? > > Also are you using the Intel VPS's or the AMD Epyc VPS's? I think it may be > important to know if anything like spectre is able to write variables back to > the cloud instance. In that case we're f*cked and only Hetzner can help with > new hardware. > > Best Regards, > -peter > Are you changing the default TCPKeepAlive setting? It defaults to yes. It exists as options in sshd_ and ssh_config. Additionally, ClientAliveInterval and ServerAliveInterval might be handy. A sysctl also exists to turn TCP keep alive on for all connections by default. Not sure it’ll help. Does your download crawl to a halt, then after a period of time, you get the FIN? (Note: I don’t have any Hetzner hosts and I’m just guessing based on my experience with Azure) -Brian
Re: TCP FIN hangups in encrypted ESP tunnel
On Thu, Jul 08, 2021 at 12:18:09PM -0400, Brian Brombacher wrote: [..] > Are you changing the default TCPKeepAlive setting? It defaults to yes. It > exists as options in sshd_ and ssh_config. Additionally, ClientAliveInterval > and ServerAliveInterval might be handy. A sysctl also exists to turn TCP > keep alive on for all connections by default. > I didn't change that setting but I have this on pod's sshd_config: #TCPKeepAlive yes > Not sure it???ll help. Does your download crawl to a halt, then after a > period of time, you get the FIN? So what I'm doing is 'scp pod:Backup/*gz .' to the local directory on arda (local host). In the tcpdump the packets go by fairly rapidly in fact there is a lot of packets before the FP (fin packet) even makes it through to arda (because there is at least a dozen packets in flight). One note here: I applied the IPSEC to escape any in-band attacks, however when I did that last week, the very first time I tried to scp my backups the connection did indeed crawl to a halt. It was just sitting there. I worried back then that someone spoofed WIN 0's into the IPSEC'ed stream somehow but I applied anti-spoof rules on IPSEC so that can't happen and the fear was probably unfounded in retrospect. I still don't have an explanation for myself for that though. Perhaps it was the missing scrub to lower the MTU on enc0 which isn't adjustable? Anyhow last week then I did manage to download the 4 GB of backups, but doing so this week proved to not work. > (Note: I don???t have any Hetzner hosts and I???m just guessing based on my > experience with Azure) > > -Brian Thanks. I looked over my firewall rules on enc0 a little and did notice I have a scrub rule that changes the mss to 1240 I'm not sure tough if that will cause this behaviour. It wasn't used anyhow when I just did the plain scp without wrapping IPSEC around it, and then it still FIN'ed and subsequent RST'. Best Regards, -peter
Re: TCP FIN hangups in encrypted ESP tunnel
On Wed, Jul 07, 2021 at 11:57:50PM +0300, Ville Valkonen wrote: > Hi, > > not sure if related but my Linux box (also in Hetzner) also started to have > flaky connection lately. > > -- > Regards, > Ville I opened a ticket with Hetzner last week thinking it was an in-band DoS. They assured me, they are not seeing this. My VPS is in Falkenstein for what it's worth. Because the problems started occuring as I was upgrading my Telekom.de link I thought it was related to that until I did tcpdumps. I mentioned it to the telekom.de chat help line despite. On your Linux box have you done any debugging as to why it became flaky? Some Linux equivalents that I know: ktrace/strace, tcpdump is the same. Are you seeing these through an IPSEC tunnel or in plain Internetworking? Also are you using the Intel VPS's or the AMD Epyc VPS's? I think it may be important to know if anything like spectre is able to write variables back to the cloud instance. In that case we're f*cked and only Hetzner can help with new hardware. Best Regards, -peter
Re: TCP FIN hangups in encrypted ESP tunnel
Hi, not sure if related but my Linux box (also in Hetzner) also started to have flaky connection lately. -- Regards, Ville On Wed 7. Jul 2021 at 19.58, Peter J. Philipp wrote: > Hi, > > My VPS at Hetzner has very weird behaviour: > > last week it started hanging up scp'ing of large backups, so I worked hard > to > get these encrypted if it was a hangup attack. Well surprise to me too the > hangups are back. I have tcpdump'ed the enc0 from both sides and the FIN > does originate from the Hetzner VPS. It's inside the secure channel but I > did not activate it knowingly. Even a ktrace does not show much, no > signal, > no close(), no shutdown(). The connection just drops on FIN and resulting > RST's. Here is a catpure of the FIN: > > seen from pod: > > 18:02:59.040443 (authentic,confidential): SPI 0xf2d38877: > 2a01:4f8:c010:71dd::1 > 2003:a:60f:ce01::108: 2a01:4f8:c010:71dd::1.1022 > > 2003:a:60f:ce01::108.40358: FP [tcp sum ok] 45961186:45962414(1228) ack > 15902 win 268 [class 0x20] > [flowlabel 0x3fceb] (len 1260, hlim 64) [class 0x20] (len 1300, hlim 64) > > seen from arda: > > 18:02:59.064240 (authentic,confidential): SPI 0xf2d38877: > 2a01:4f8:c010:71dd::1 > 2003:a:60f:ce01::108: 2a01:4f8:c010:71dd::1.1022 > > 2003:a:60f:ce01::108.40358: FP [tcp sum ok] 45961186:45962414(1228) ack > 15902 win 268 [class 0x20] > [flowlabel 0x3fceb] (len 1260, hlim 64) (len 1300, hlim 55) > > The download downloads a few MB and then it hangs up. > > Has anyone seen this sort of behaviour? I don't think I changed much in my > pf rules because up until last month backups downloaded flawlessly. Here > is > my dmesg (after my signature): > > Best Regards, > -peter > > > OpenBSD 6.9 (GENERIC.MP) #3: Mon Jun 7 08:21:26 MDT 2021 > r...@syspatch-69-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/ > GENERIC.MP > real mem = 2080227328 (1983MB) > avail mem = 2001866752 (1909MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5b10 (9 entries) > bios0: vendor Hetzner version "2017" date 11/11/2017 > bios0: Hetzner vServer > acpi0 at bios0: ACPI 3.0 > acpi0: sleep states S5 > acpi0: tables DSDT FACP APIC HPET MCFG > acpi0: wakeup devices > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: AMD EPYC Processor (with IBPB), 2495.71 MHz, 17-01-02 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,SSBD,XSAVEOPT,XSAVEC,XGETBV1 > cpu0: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB > 64b/line 8-way L2 cache > cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped > cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges > cpu0: apic clock running at 1000MHz > cpu1 at mainbus0: apid 1 (application processor) > cpu1: AMD EPYC Processor (with IBPB), 2495.40 MHz, 17-01-02 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,SSBD,XSAVEOPT,XSAVEC,XGETBV1 > cpu1: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB > 64b/line 8-way L2 cache > cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped > cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped > cpu1: smt 0, core 0, package 1 > ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 24 pins > acpihpet0 at acpi0: 1 Hz > acpimcfg0 at acpi0 > acpimcfg0: addr 0xb000, bus 0-255 > acpiprt0 at acpi0: bus 0 (PCI0) > "ACPI0006" at acpi0 not configured > acpipci0 at acpi0 PCI0: 0x 0x0011 0x0001 > acpicmos0 at acpi0 > "APP0005" at acpi0 not configured > "PNP0A06" at acpi0 not configured > "PNP0A06" at acpi0 not configured > "QEMU0002" at acpi0 not configured > "ACPI0010" at acpi0 not configured > acpicpu0 at acpi0: C1(@1 halt!) > acpicpu1 at acpi0: C1(@1 halt!) > pvbus0 at mainbus0: KVM > pvclock0 at pvbus0 > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00 > vga1 at pci0 dev 1 function 0 "Qumranet Virtio 1.x GPU" rev 0x01 > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > ppb0 at pci0 dev 2 function 0 vendor "Red Hat", unknown product 0x000c rev > 0x00: apic 0 int 22 > pci1
TCP FIN hangups in encrypted ESP tunnel
Hi, My VPS at Hetzner has very weird behaviour: last week it started hanging up scp'ing of large backups, so I worked hard to get these encrypted if it was a hangup attack. Well surprise to me too the hangups are back. I have tcpdump'ed the enc0 from both sides and the FIN does originate from the Hetzner VPS. It's inside the secure channel but I did not activate it knowingly. Even a ktrace does not show much, no signal, no close(), no shutdown(). The connection just drops on FIN and resulting RST's. Here is a catpure of the FIN: seen from pod: 18:02:59.040443 (authentic,confidential): SPI 0xf2d38877: 2a01:4f8:c010:71dd::1 > 2003:a:60f:ce01::108: 2a01:4f8:c010:71dd::1.1022 > 2003:a:60f:ce01::108.40358: FP [tcp sum ok] 45961186:45962414(1228) ack 15902 win 268 [class 0x20] [flowlabel 0x3fceb] (len 1260, hlim 64) [class 0x20] (len 1300, hlim 64) seen from arda: 18:02:59.064240 (authentic,confidential): SPI 0xf2d38877: 2a01:4f8:c010:71dd::1 > 2003:a:60f:ce01::108: 2a01:4f8:c010:71dd::1.1022 > 2003:a:60f:ce01::108.40358: FP [tcp sum ok] 45961186:45962414(1228) ack 15902 win 268 [class 0x20] [flowlabel 0x3fceb] (len 1260, hlim 64) (len 1300, hlim 55) The download downloads a few MB and then it hangs up. Has anyone seen this sort of behaviour? I don't think I changed much in my pf rules because up until last month backups downloaded flawlessly. Here is my dmesg (after my signature): Best Regards, -peter OpenBSD 6.9 (GENERIC.MP) #3: Mon Jun 7 08:21:26 MDT 2021 r...@syspatch-69-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 2080227328 (1983MB) avail mem = 2001866752 (1909MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5b10 (9 entries) bios0: vendor Hetzner version "2017" date 11/11/2017 bios0: Hetzner vServer acpi0 at bios0: ACPI 3.0 acpi0: sleep states S5 acpi0: tables DSDT FACP APIC HPET MCFG acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD EPYC Processor (with IBPB), 2495.71 MHz, 17-01-02 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,SSBD,XSAVEOPT,XSAVEC,XGETBV1 cpu0: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 1000MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: AMD EPYC Processor (with IBPB), 2495.40 MHz, 17-01-02 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,SSBD,XSAVEOPT,XSAVEC,XGETBV1 cpu1: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu1: smt 0, core 0, package 1 ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 24 pins acpihpet0 at acpi0: 1 Hz acpimcfg0 at acpi0 acpimcfg0: addr 0xb000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) "ACPI0006" at acpi0 not configured acpipci0 at acpi0 PCI0: 0x 0x0011 0x0001 acpicmos0 at acpi0 "APP0005" at acpi0 not configured "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "QEMU0002" at acpi0 not configured "ACPI0010" at acpi0 not configured acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) pvbus0 at mainbus0: KVM pvclock0 at pvbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00 vga1 at pci0 dev 1 function 0 "Qumranet Virtio 1.x GPU" rev 0x01 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) ppb0 at pci0 dev 2 function 0 vendor "Red Hat", unknown product 0x000c rev 0x00: apic 0 int 22 pci1 at ppb0 bus 1 virtio0 at pci1 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01 vio0 at virtio0: address 96:00:00:a5:ca:09 virtio0: msix shared ppb1 at pci0 dev 2 function 1 vendor "Red Hat", unknown product 0x000c rev 0x00: apic 0 int 22 pci2 at ppb1 bus 2 xhci0 at pci2 dev 0 function 0 vendor "Red Hat", unknown product 0x000d rev 0x01: apic 0 int 22,
