100ms RTT >Which exact version of linux kernel are you using ? On the receiver side I could see the issue with any mainline kernel version >=4.19.86 which is the first kernel version that has patches [1] & [2] included. On the sender I am using kernel 5.4.0-rc6.
Links: [1] https://lore.kernel.org/patchwork/patch/1157936/ [2] https://lore.kernel.org/patchwork/patch/1157883/ Thank you. Hazem On 07/12/2020, 16:24, "Eric Dumazet" <[email protected]> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem <[email protected]> wrote: > > >Since I can not reproduce this problem with another NIC on x86, I > >really wonder if this is not an issue with ENA driver on PowerPC > >perhaps ? > > > I am able to reproduce it on x86 based EC2 instances using ENA or Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail. > > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side? 100ms RTT Which exact version of linux kernel are you using ? > > Thank you. > > Hazem > > On 07/12/2020, 15:26, "Eric Dumazet" <[email protected]> wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On Sat, Dec 5, 2020 at 1:03 PM Mohamed Abuelfotoh, Hazem > <[email protected]> wrote: > > > > Unfortunately few things are missing in this report. > > > > What is the RTT between hosts in your test ? > > >>>>>RTT in my test is 162 msec, but I am able to reproduce it with lower RTTs for example I could see the issue downloading from google endpoint with RTT of 16.7 msec, as mentioned in my previous e-mail the issue is reproducible whenever RTT exceeded 12msec given that the sender is using bbr. > > > > RTT between hosts where I run the iperf test. > > # ping 54.199.163.187 > > PING 54.199.163.187 (54.199.163.187) 56(84) bytes of data. > > 64 bytes from 54.199.163.187: icmp_seq=1 ttl=33 time=162 ms > > 64 bytes from 54.199.163.187: icmp_seq=2 ttl=33 time=162 ms > > 64 bytes from 54.199.163.187: icmp_seq=3 ttl=33 time=162 ms > > 64 bytes from 54.199.163.187: icmp_seq=4 ttl=33 time=162 ms > > > > RTT between my EC2 instances and google endpoint. > > # ping 172.217.4.240 > > PING 172.217.4.240 (172.217.4.240) 56(84) bytes of data. > > 64 bytes from 172.217.4.240: icmp_seq=1 ttl=101 time=16.7 ms > > 64 bytes from 172.217.4.240: icmp_seq=2 ttl=101 time=16.7 ms > > 64 bytes from 172.217.4.240: icmp_seq=3 ttl=101 time=16.7 ms > > 64 bytes from 172.217.4.240: icmp_seq=4 ttl=101 time=16.7 ms > > > > What driver is used at the receiving side ? > > >>>>>>I am using ENA driver version version: 2.2.10g on the receiver with scatter gathering enabled. > > > > # ethtool -k eth0 | grep scatter-gather > > scatter-gather: on > > tx-scatter-gather: on > > tx-scatter-gather-fraglist: off [fixed] > > This ethtool output refers to TX scatter gather, which is not relevant > for this bug. > > I see ENA driver might use 16 KB per incoming packet (if ENA_PAGE_SIZE is 16 KB) > > Since I can not reproduce this problem with another NIC on x86, I > really wonder if this is not an issue with ENA driver on PowerPC > perhaps ? > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 > > Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705 > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705
