>I see that RX checksumming is still off for you on virtio, this is >likely what's contribution to the problem. > >Here's how it looks for me: >ethtool -k eth1 > Offload parameters for eth1: > rx-checksumming: on > tx-checksumming: on > scatter-gather: on > tcp-segmentation-offload: on > udp-fragmentation-offload: on > generic-segmentation-offload: on > generic-receive-offload: on > large-receive-offload: off > When I select centos-6.3 as guest os, the rx-checksuming is on, too. After update the qemu from 1.4.0 to 2.0.0, the inter-vm throughput can achieve ~5Gbps via netper -t TCP_STREAM -m 1400. Here is ethtool -k eth1 on centos-6.3 guest, ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: on generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off
the only difference is gro, on for you, off for me, I run 'ethtool -K eth1 gro on' on my guest, below error reported, "Cannot set device GRO settings: Invalid argument" >you don't supply kernel versions for host or guest kernels, >so it's hard to judge what's going on exactly. > host: linux-3.10.27(directly download from kernel.org) qemu: qemu-2.0.0(directly download from wiki.qemu.org/Download) guest: centos-6.3(2.6.32-279.e16.x86_64), 2vcpu >Bridge configuration also plays a huge role. >Things like ebtables might affect performance as well, >sometimes even if they are only loaded, not even enabled. > I will check it. >Also, some old scheduler versions didn't put VMs on different >CPUs aggressively enough, this resulted in conflicts >when VMs compete for the same CPU. I will check it. >On numa systems, some older host kernels would split VM memory >across NUMA nodes, this might lead to bad performance. > local first. >On Sat, Jun 07, 2014 at 11:07:10AM +0800, Zhang Haoyu wrote: >> After updating the qemu from 1.4 to 2.0, the inter-vm throughput can achieve >> ~5Gbps via netper -t TCP_STREAM -m 1400, >> the performance gap(~2Gbps) between kvm and xen still exist. >> >> Thanks, >> Zhang Haoyu >> >> ------------------ >> Zhang Haoyu >> 2014-06-07 >> >> -----Original Message----- >> From: Zhang Haoyu >> Sent: 2014-06-07 09:27:16 >> To: Venkateswara Rao Nandigam; kvm; qemu-devel >> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi >> Subject: Re: [network performance question] only ~2Gbpsthroughputbetweentwo >> linux guests which are running on the same host vianetperf-tTCP_STREAM -m >> 1400, but xen can ac >> >> > Doesn't that answer your original question about performance gap! > Sorry, do you mean it's the offloadings cause the performance gap? >> But even OFF the checksum-offload, tso, gro, .etc, the performance gap still >> exist, >> if I understand correctly, kvm should have better performance than xen from >> the angle of implementation, because of shorter path, and fewer >> context-switches, >> especially inter-vm communication. >> >> And, why the performance gap is so big(~2G vs ~7G) when checksum-offload, >> tso, gro, .etc is on for both hypervisors? >> Why the packes' size can be so big(65160) and stable on xen, but most >> packets' size is 1448, only a few part is ~65000 on kvm, when netperf -t >> TCP_STREAM -m 1400 ? >> Does some TCP configurations have buissness with this? Or some virtio-net >> configurations? >> >> Thanks, >> Zhang Haoyu >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On Behalf >> Of Zhang Haoyu >> Sent: Friday, June 06, 2014 3:44 PM >> To: Venkateswara Rao Nandigam; kvm; qemu-devel >> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi >> Subject: Re: RE: [network performance question] only ~2Gbps >> throughputbetweentwo linux guests which are running on the same host via >> netperf-tTCP_STREAM -m 1400, but xen can ac >> >> > >> Try Rx/Tx checksum offload on the all the concerned guests of both >> > >> Hypervisors. >> > >> >> > >Already ON on both hypervisors, so some other offloadings(e.g. tso, gso) >> > >can be supported. >> > >> > Try Rx/Tx checksum offload "OFF" on the all the concerned guests of >> > both Hypervisors >> > >> Off Rx/Tx checksum offload on XEN guest, 1.6Gbps achived, tcpdump result on >> backend vif netdeivce shown that packets' size is 1448, stable. >> Off Rx/Tx checksum offload on KVM guest, only ~1Gbps ahchived, tcpdump >> result on backend tap netdevice shown that packets' size is 1448, stable. >> >> > And While launching the VM in KVM, in command line of virtio interface, >> > you can specify TSO, LRO, RxMergebuf. Try this instead of ethtool >> > interface. >> The cuurent qemu command shown as below, and I will change the virtio-net >> configuration later as your advise, /usr/bin/kvm -id 8572667846472 -chardev >> socket,id=qmp,path=/var/run/qemu-server/8572667846472.qmp,server,nowait -mon >> chardev=qmp,mode=control -vnc :0,websocket,to=200,x509,password -pidfile >> /var/run/qemu-server/8572667846472.pid -daemonize -name centos6-196.5.5.72 >> -smp sockets=1,cores=2 -cpu core2duo -nodefaults -vga cirrus -no-hpet -k >> en-us -boot menu=on,splash-time=8000 -m 4096 -usb -drive >> file=/sf/data/local/iso/vmtools/virtio_auto_install.iso,if=none,id=drive-ide0,media=cdrom,aio=threads,forecast=disable >> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200 >> -drive >> file=/sf/data/local/images/host-f8bc123b3e74/32f49b646d1e/centos6-196.5.5.72.vm/vm-disk-1.qcow2,if=none,id=drive-ide2,cache=directsync,aio=threads,forecast=disable >> -device ide-hd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100 >> -netdev type=tap,id=net0,ifname=857266784647200,sc r > ip >> t=/sf/etc/kvm/vtp-bridge,vhost=on,vhostforce=on -device >> virtio-net-pci,mac=FE:FC:FE:95:EC:A7,netdev=net0,bus=p >> ci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,clock=rt -global >> kvm-pit.lost_tick_policy=discard -global PIIX4_PM.disable_s3=1 -global >> PIIX4_PM.disable_s4=1 >> >> -----Original Message----- >> From: Zhang Haoyu [mailto:[email protected]] >> Sent: Friday, June 06, 2014 1:26 PM >> To: Venkateswara Rao Nandigam; kvm; qemu-devel >> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi >> Subject: RE: [network performance question] only ~2Gbps throughput >> betweentwo linux guests which are running on the same host via netperf >> -tTCP_STREAM -m 1400, but xen can ac >> >> Thanks for reply. >> > >>> And, vhost enabled, tx zero-copy enabled, virtio TSO enabled on kvm. >> > >> > Try lro "ON" on client side. This would require mergable Rx buffers to be >> > ON. >> > >> current setttings for gro and lro, >> generic-receive-offload: on >> large-receive-offload: off [fixed] >> >> > And Xen netfront to KVM virtio are not apples to apples because of their >> > implementation details. >> > >> You are right, I just want to make network performance comparison between >> the two virtualization platform from the view of user. >> >> > Try Rx/Tx checksum offload on the all the concerned guests of both >> > Hypervisors. >> > >> Already ON on both hypervisors, so some other offloadings(e.g. tso, gso) can >> be supported. >> >> kvm virtio-net nic: >> ethtool -k eth0 >> Features for eth0: >> rx-checksumming: off [fixed] >> tx-checksumming: on >> tx-checksum-ipv4: off [fixed]\ >> tx-checksum-ip-generic: on >> tx-checksum-ipv6: off [fixed] >> tx-checksum-fcoe-crc: off [fixed] >> tx-checksum-sctp: off [fixed] >> scatter-gather: on >> tx-scatter-gather: on >> scatter-gather-fraglist: on >> tcp-segmentation-offload: on >> tx-tcp-segmentation: on >> tx-tcp-ecn-segmentation: on >> tx-tcp6-segmentation: on >> udp-fragmentation-offload: on >> generic-segmentation-offload: on >> generic-receive-offload: on >> large-receive-offload: off [fixed] >> rx-vlan-offload: off [fixed] >> tx-vlan-offload: off [fixed] >> ntuple-filters: off [fixed] >> receive-hashing: off [fixed] >> highdma: on [fixed] >> rx-vlan-filter: on [fixed] >> vlan-challenged: off [fixed] >> tx-lockless: off [fixed] >> netns-local: off [fixed] >> tx-gso-robust: off [fixed] >> tx-fcoe-segmentation: off [fixed] >> tx-gre-segmentation: off [fixed] >> tx-udp_tnl-segmentation: off [fixed] >> fcoe-mtu: off [fixed] >> tx-nocache-copy: on >> loopback: off [fixed] >> rx-fcs: off [fixed] >> rx-all: off [fixed] >> tx-vlan-stag-hw-insert: off [fixed] >> rx-vlan-stag-hw-parse: off [fixed] >> rx-vlan-stag-filter: off [fixed] >> >> xen netfront nic: >> ethtool -k eth0 >> Offload features for eth0: >> rx-checksumming: on >> tx-checksumming: on >> scatter-gather: on >> tcp-segmentation-offload: on >> udp-fragmentation-offload: off >> generic-segmentation-offload: on >> generic-receive-offload: off >> large-receive-offload: off >> >> <piece of tcpdump result on xen backend vif netdevice > >> 15:46:41.279954 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193138968:1193204128, ack 1, win 115, options [nop,nop,TS val 102307210 ecr >> 102291188], length 65160 >> 15:46:41.279971 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193204128:1193269288, ack 1, win 115, options [nop,nop,TS val 102307210 ecr >> 102291188], length 65160 >> 15:46:41.279987 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193269288:1193334448, ack 1, win 115, options [nop,nop,TS val 102307210 ecr >> 102291188], length 65160 >> 15:46:41.280003 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193334448:1193399608, ack 1, win 115, options [nop,nop,TS val 102307210 ecr >> 102291188], length 65160 >> 15:46:41.280020 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193399608:1193464768, ack 1, win 115, options [nop,nop,TS val 102307210 ecr >> 102291188], length 65160 >> 15:46:41.280213 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193464768:1193529928, ack 1, win 115, options [nop,nop,TS val 102307211 ecr >> 102291189], length 65160 >> 15:46:41.280233 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193529928:1193595088, ack 1, win 115, options [nop,nop,TS val 102307211 ecr >> 102291189], length 65160 >> 15:46:41.280250 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193595088:1193660248, ack 1, win 115, options [nop,nop,TS val 102307211 ecr >> 102291189], length 65160 >> 15:46:41.280239 IP 196.6.6.71.53622 > 196.6.6.72.53507: Flags [.], ack >> 1193138968, win 22399, options [nop,nop,TS val 102291190 ecr 102307210], >> length 0 >> 15:46:41.280267 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193660248:1193725408, ack 1, win 115, options [nop,nop,TS val 102307211 ecr >> 102291189], length 65160 >> 15:46:41.280284 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq >> 1193725408:1193790568, ack 1, win 115, options [nop,nop,TS val 102307211 ecr >> 102291189], length 65160 >> >> Packets' size is very stable, 65160 Bytes. >> >> Thanks, >> Zhang Haoyu >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On Behalf >> Of Zhang Haoyu >> Sent: Friday, June 06, 2014 9:01 AM >> To: kvm; qemu-devel >> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi >> Subject: [network performance question] only ~2Gbps throughput between two >> linux guests which are running on the same host via netperf -t TCP_STREAM -m >> 1400, but xen can achieve ~7Gbps >> >> Hi, all >> >> I ran two linux guest on the same kvm host, then start the netserver on one >> vm, start netperf on the other one, netperf command and test result shown >> as below, netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 >> MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 >> () port 0 AF_INET : nodelay >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 87380 16384 1400 60.01 2355.45 >> >> but I ran two linux guest on the same xen hypervisor, ~7Gbps throughput >> achived, netperf command and test result shown as below, netperf -H >> 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED TCP STREAM TEST >> from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 87380 16384 1400 60.01 2349.82 >> >> many times test performed, the result is similar as above. >> >> When I tcpdump backend tap netdevice, found that most packets' size is >> 1448bytes on kvm, and few packets are ~60000Bytes, but I tcpdump backend >> vif netdevice, found that most packets' size is >60000bytes on xen. >> Test result of netperf -t TCP_STREAM -m 64 is similar, more larger packets >> on xen than kvm. >> >> And, vhost enabled, tx zero-copy enabled, virtio TSO enabled on kvm. >> >> Any ideas? >> >> Thanks, >> Zhang Haoyu >> >> <environment> >> guest: linux-3.10, or centos-6.3 >> host: linux-3.10(kvm), xenserver-6.2 >> >> <test tool> >> netperf-2.6 >> >> <test result details> >> 1. KVM >> netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 65000 -M 65000 MIGRATED TCP >> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 >> AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 87380 16384 65000 60.00 13743.44 >> >> netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED TCP >> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 >> AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 87380 16384 1400 60.01 2349.82 >> >> netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 64 -M 64 MIGRATED TCP STREAM >> TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 87380 16384 64 60.01 294.70 >> >> 2. XEN >> netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 65000 -M 65000 MIGRATED TCP >> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 >> AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 87380 16384 65000 60.01 6327.46 >> >> netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED TCP >> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 >> AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 87380 16384 1400 60.00 7250.63 >> >> netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 64 -M 64 MIGRATED TCP STREAM >> TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 87380 16384 64 60.00 668.25 >> >> <piece of tcpdump result on backend tap netdevice> >> 11:26:28.907130 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5421192:5422640, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907146 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5422640:5424088, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907161 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5419328, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907164 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5424088:5425536, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907183 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5426984:5428432, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907203 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5421192, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907217 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5428432:5429880, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907225 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5429880:5431328, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907238 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5431328:5432776, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907256 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5432776:5434224, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907268 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5426984, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907294 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq >> 5434224:5434692, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 468 >> 11:26:28.907311 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5429880, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907319 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5432776, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907345 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5434692:5436140, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907363 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5436140:5437588, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907373 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5434692, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907385 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5437588:5439036, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 > 11:26:28.907411 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq > 5439036:5440484, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], > length 1448 >> 11:26:28.907427 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq >> 5440484:5440692, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 208 >> 11:26:28.907427 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5437588, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907445 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5440692:5442140, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907453 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5442140:5443588, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907468 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5443588:5445036, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907494 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5445036:5446484, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907496 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5440484, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907506 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5446484:5447932, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907539 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5447932:5449380, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907551 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5445036, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907555 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5449380:5450828, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907566 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq >> 5450828:5451192, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 364 >> 11:26:28.907605 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5451192:5452640, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907623 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5452640:5454088, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907631 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5454088:5455536, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907646 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5455536:5456984, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907662 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5456984:5458432, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907670 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5458432:5459880, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907690 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5459880:5461328, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907699 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5461328:5462776, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907704 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5450828, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.907708 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5462776:5464224, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907721 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5464224:5465672, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907739 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5465672:5467120, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907764 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 5468568:5470016, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 1448 >> 11:26:28.907793 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq >> 5470016:5470692, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], >> length 676 >> 11:26:28.907796 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 5455536, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0 >> 11:26:28.937289 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7273240:7274688, ack 1, win 115, options [nop,nop,TS val 497416 ecr 497988], >> length 1448 >> 11:26:28.941798 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6259640, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941815 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6262536, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941820 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6265432, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941825 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6268328, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941829 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6271224, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941834 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6274120, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941839 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6277016, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941843 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6279912, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941848 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6282808, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941852 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6285704, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941856 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6288600, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941873 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6291496, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941888 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6294392, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941893 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6297288, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941904 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6300184, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941909 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6303080, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941919 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6305976, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941924 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6308872, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941934 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6311768, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941939 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6314664, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941950 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6317560, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941955 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6320456, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941961 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6323352, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941966 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6326248, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941967 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7274688:7276136, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], >> length 1448 >> 11:26:28.941970 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6329144, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941975 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6332040, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941980 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6334936, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941985 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6337832, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941991 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6342176, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.941995 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6345072, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.942001 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6362448, win 24441, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.942118 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7276136:7338400, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], >> length 62264 >> 11:26:28.942220 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6579648, win 22745, options [nop,nop,TS val 497993 ecr 497411], length 0 >> 11:26:28.942456 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7338400:7403560, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], >> length 65160 >> 11:26:28.942501 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7403560:7468720, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], >> length 65160 >> 11:26:28.942547 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7468720:7533880, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], >> length 65160 >> 11:26:28.942689 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6640464, win 22964, options [nop,nop,TS val 497994 ecr 497412], length 0 >> 11:26:28.942701 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6720104, win 23145, options [nop,nop,TS val 497994 ecr 497412], length 0 >> 11:26:28.942726 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6743272, win 23833, options [nop,nop,TS val 497994 ecr 497412], length 0 >> 11:26:28.942864 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7533880:7599040, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 65160 >> 11:26:28.942886 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6811328, win 24159, options [nop,nop,TS val 497994 ecr 497412], length 0 >> 11:26:28.942918 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7599040:7664200, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 65160 >> 11:26:28.942971 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7664200:7729360, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 65160 >> 11:26:28.943019 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7729360:7794520, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 65160 >> 11:26:28.943041 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6882280, win 24268, options [nop,nop,TS val 497994 ecr 497412], length 0 >> 11:26:28.943187 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7794520:7859680, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 65160 >> 11:26:28.943404 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6928616, win 24576, options [nop,nop,TS val 497994 ecr 497412], length 0 >> 11:26:28.943415 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6935856, win 24576, options [nop,nop,TS val 497994 ecr 497412], length 0 >> 11:26:28.943420 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6938752, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943427 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6941648, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943433 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6944544, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943438 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6947440, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943444 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6950336, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943452 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6953232, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943458 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6956128, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943463 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6959024, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943468 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6961920, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943473 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6964816, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943479 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6969160, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943487 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6972056, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943493 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6974952, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943500 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6977848, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943505 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6980744, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943511 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6983640, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943519 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6986536, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943525 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6989432, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943533 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6992328, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943542 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6995224, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943546 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 6998120, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943550 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7001016, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943554 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7003912, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943558 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7006808, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943562 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7011152, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943566 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7014048, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943570 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7859680:7924840, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 65160 >> 11:26:28.943607 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7077760, win 24079, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943673 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7924840:7990000, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 65160 >> 11:26:28.943711 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7096584, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943731 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7099480, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943750 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7102376, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943777 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 7990000:8055160, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 65160 >> 11:26:28.943822 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack >> 7177672, win 23988, options [nop,nop,TS val 497994 ecr 497413], length 0 >> 11:26:28.943833 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 8055160:8115976, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 60816 >> 11:26:28.943954 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq >> 8115976:8120320, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], >> length 4344
