Why my NIC shows NUMA -1?
Hi Sorry I am not sure if this is the right list to ask this question, please direct me to the correct one. thanks! I am using Intel 82599 on Dell PowerEdge R710 which should support NUMA node, the box is running ubuntu 14.01.1 LTS , I am wondering why I get NUMA socket -1. in BIOS, the memory "Node Interleaving" is disabled by default and I am using DPDK and DPDK sees my NIC NUMA -1, but i guess it is not related to DPDK since the sysfs shows NUMA -1, any clue? # uname -a Linux pktgen 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux EAL: Master lcore 0 is ready (tid=79b78900;cpuset=[0]) EAL: lcore 8 is ready (tid=f2deb700;cpuset=[8]) EAL: lcore 15 is ready (tid=ef5e4700;cpuset=[15]) EAL: lcore 5 is ready (tid=f45ee700;cpuset=[5]) EAL: lcore 2 is ready (tid=f5df1700;cpuset=[2]) EAL: lcore 10 is ready (tid=f1de9700;cpuset=[10]) EAL: lcore 6 is ready (tid=f3ded700;cpuset=[6]) EAL: lcore 7 is ready (tid=f35ec700;cpuset=[7]) EAL: lcore 11 is ready (tid=f15e8700;cpuset=[11]) EAL: lcore 1 is ready (tid=f65f2700;cpuset=[1]) EAL: lcore 13 is ready (tid=f05e6700;cpuset=[13]) EAL: lcore 4 is ready (tid=f4def700;cpuset=[4]) EAL: lcore 9 is ready (tid=f25ea700;cpuset=[9]) EAL: lcore 3 is ready (tid=f55f0700;cpuset=[3]) EAL: lcore 12 is ready (tid=f0de7700;cpuset=[12]) EAL: lcore 14 is ready (tid=efde5700;cpuset=[14]) EAL: PCI device :04:00.0 on NUMA socket -1 < EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device :04:00.1 on NUMA socket -1 <== EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7feb77c0 EAL: PCI memory mapped at 0x7feb77c8 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 14, SFP+: 6 PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x10fb Total number of attached devices: 1 Interface name: dpdk0 kernel sysfs output: #cat /sys/devices/pci:00/:00:05.0/:04:00.0/numa_node -1 #cat /sys/devices/pci:00/:00:05.0/:04:00.1/numa_node -1 # numactl --show policy: default preferred node: current physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cpubind: 0 1 nodebind: 0 1 membind: 0 1 # numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 2 4 6 8 10 12 14 node 0 size: 40317 MB node 0 free: 38699 MB node 1 cpus: 1 3 5 7 9 11 13 15 node 1 size: 32166 MB node 1 free: 30706 MB node distances: node 0 1 0: 10 20 1: 20 10 -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ip_no_pmtu_disc and UDP
ok, I observed if i increase the UDP client packet size > local interface MTU 1500, the client will fragment the packet first and then send it out, if the UDP client packet size < local interface MTU 1500, the DF bit will be set when ip_no_pmtu_disc set to 0, is this expected behavior ? On Mon, Oct 26, 2015 at 3:35 PM, Vincent Li <vincent.mc...@gmail.com> wrote: > I test again and i did see DF bit now, it is weird. I am going to do > more test, sorry for the noise. > > On Mon, Oct 26, 2015 at 3:12 PM, Hannes Frederic Sowa > <han...@stressinduktion.org> wrote: >> Hello, >> >> On Mon, Oct 26, 2015, at 23:00, Vincent Li wrote: >>> the UDP packet size is about 768, here is how packet path like: >>> >>> client >>> <router<-->server >>> (eth0 mtu 1500 ip 10.3.72.69) (eth0 mtu 1500 ip 10.3.72.1, >>> (eth0 mtu 1500 ip 10.2.72.99) >>> eth1.1102 mtu >>> 567 ip 10.2.72.139) >>> >>> >>> UDP client test script: >>> >>> [...] >>> >>> so I am hoping if I echo 0, 1, 2, 3 respectively to >>> /proc/sys/net/ipv4/ip_no_pmtu_disc, I am expected to see DF bit >>> set/unset from the client and should have shown me on the router eth0 >>> interface tcpdump, but instead, DF bit never set on the client. am I >>> misunderstanding something? >> >> This is strange... >> >> Can you please capture traffic on eth0 on the client? >> >> For outgoing packets only zero or non-zero matter. A '0' definitely >> generates a UDP packet with a DF bit on my side, anything else a frame >> with DF bit cleared. I just verified this on net-next with your script. >> It also does not cause any setsockopts but uses the default. >> >>> for example: >>> >>> two concurrent tcpdump on router eth0 (mtu 1500) and eth1.1102 (mtu >>> 576) interface: >>> >>> 1 #tcpdump -nn -i eth0 -v udp and host 10.3.72.69 & >>> >>> 14:51:11.946143 IP (tos 0x0, ttl 64, id 7193, offset 0, flags [none], >>> proto UDP (17), length 796) >>> 10.3.72.69.43748 > 10.2.72.99.: UDP, length 768 >>> >> >> As I said, I cannot reproduce that. :( Please test on eth0 directly so >> we can be sure the packet does not get mangled. >> >> Can you also show me the output of >> ip route get 10.2.72.139 >> on the client after you maybe already received a icmp pkt-too-big >> packet? >> >> Thanks, >> Hannes >> >> >> >> -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ip_no_pmtu_disc and UDP
the UDP packet size is about 768, here is how packet path like: client <router<-->server (eth0 mtu 1500 ip 10.3.72.69) (eth0 mtu 1500 ip 10.3.72.1, (eth0 mtu 1500 ip 10.2.72.99) eth1.1102 mtu 567 ip 10.2.72.139) UDP client test script: #!/usr/bin/perl use strict; use warnings; use IO::Socket::INET; my $socket = IO::Socket::INET->new( PeerPort => , PeerAddr => '10.2.72.99', Proto => 'udp', ) or die "Can't bind : $@\n"; $| = 1; my $data = "012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567012345670123456701234567"; $socket->send($data); sleep(10); $socket->close(); so I am hoping if I echo 0, 1, 2, 3 respectively to /proc/sys/net/ipv4/ip_no_pmtu_disc, I am expected to see DF bit set/unset from the client and should have shown me on the router eth0 interface tcpdump, but instead, DF bit never set on the client. am I misunderstanding something? for example: two concurrent tcpdump on router eth0 (mtu 1500) and eth1.1102 (mtu 576) interface: 1 #tcpdump -nn -i eth0 -v udp and host 10.3.72.69 & 14:51:11.946143 IP (tos 0x0, ttl 64, id 7193, offset 0, flags [none], proto UDP (17), length 796) 10.3.72.69.43748 > 10.2.72.99.: UDP, length 768 2# tcpdump -nn -i eth1.1102 -v udp and host 10.3.72.69 & 14:51:11.946164 IP (tos 0x0, ttl 63, id 7193, offset 0, flags [+], proto UDP (17), length 572) 14:51:11.946176 IP (tos 0x0, ttl 63, id 7193, offset 552, flags [none], proto UDP (17), length 244) 10.3.72.69.43748 > 10.2.72.99.: UDP, length 768 10.3.72.69 > 10.2.72.99: udp as you can see, the router was fragmenting the UDP packet and not sending icmp frag needed message, one reason I can think of is the DF bit is not set on the original UDP packet. client is on kernel 4.3.0-rc7+, router is on kernel 3.13.0-rc3 On Fri, Oct 23, 2015 at 3:34 PM, Hannes Frederic Sowa <han...@stressinduktion.org> wrote: > Hello, > > On Fri, Oct 23, 2015, at 18:45, Vincent Li wrote: >> It looks ip_no_pmtu_disc setting does not affect UDP IP packet DF bit >> setting, is that intended behavior? echo 0, 1, 2, 3 respectively to >> ip_no_pmtu_disc, UDP IP packet always have DF bit cleared, unless use >> IP_PMTUDISC_DO on IP_MTU_DISCOVER as ip man page says. > > Which size do the UDP packets have and what is your MTU? inet_create > also creates udp sockets and thus the setting does have effect. > >> >> in inet_create, seems to prove that. >> >>if (net->ipv4.sysctl_ip_no_pmtu_disc) >> inet->pmtudisc = IP_PMTUDISC_DONT; >> else >> inet->pmtudisc = IP_PMTUDISC_WANT; >> >> so I am wondering why UDP is excluded by ip_no_pmtu_disc, why in >> inet_create, not assign each individual ip_no_pmtu_disc setting to >> inet->pmtudisc but only check true and assign IP_PMTUDISC_DONT or >> IP_PMTUDISC_WANT only. > > ip_no_pmtu_disc sysctl != IP_MTU_DISCOVER setsockopt. Also we cannot > change this as it would disrupt communication easily relying on this > established behavior. > > See Documentation/ip-sysctl.txt: > > ip_no_pmtu_disc - INTEGER > Disable Path MTU Discovery. If enabled in mode 1 and a > fragmentation-required ICMP is received, the PMTU to this > destination will be set to min_pmtu (see below). You will need > to raise min_pmtu to the smallest interface MTU on your system > manually if you want to avoid locally generated fragments. > > In mode 2 incoming Path MTU Discovery messages will be > discarded. Outgoing frames are handled the same as in mode 1, > implicitly setting IP_PMTUDISC_DONT on every created socket. > > Mode 3 is a hardend pmtu discover mode. The kernel will only >
Re: ip_no_pmtu_disc and UDP
I test again and i did see DF bit now, it is weird. I am going to do more test, sorry for the noise. On Mon, Oct 26, 2015 at 3:12 PM, Hannes Frederic Sowa <han...@stressinduktion.org> wrote: > Hello, > > On Mon, Oct 26, 2015, at 23:00, Vincent Li wrote: >> the UDP packet size is about 768, here is how packet path like: >> >> client >> <router<-->server >> (eth0 mtu 1500 ip 10.3.72.69) (eth0 mtu 1500 ip 10.3.72.1, >> (eth0 mtu 1500 ip 10.2.72.99) >> eth1.1102 mtu >> 567 ip 10.2.72.139) >> >> >> UDP client test script: >> >> [...] >> >> so I am hoping if I echo 0, 1, 2, 3 respectively to >> /proc/sys/net/ipv4/ip_no_pmtu_disc, I am expected to see DF bit >> set/unset from the client and should have shown me on the router eth0 >> interface tcpdump, but instead, DF bit never set on the client. am I >> misunderstanding something? > > This is strange... > > Can you please capture traffic on eth0 on the client? > > For outgoing packets only zero or non-zero matter. A '0' definitely > generates a UDP packet with a DF bit on my side, anything else a frame > with DF bit cleared. I just verified this on net-next with your script. > It also does not cause any setsockopts but uses the default. > >> for example: >> >> two concurrent tcpdump on router eth0 (mtu 1500) and eth1.1102 (mtu >> 576) interface: >> >> 1 #tcpdump -nn -i eth0 -v udp and host 10.3.72.69 & >> >> 14:51:11.946143 IP (tos 0x0, ttl 64, id 7193, offset 0, flags [none], >> proto UDP (17), length 796) >> 10.3.72.69.43748 > 10.2.72.99.: UDP, length 768 >> > > As I said, I cannot reproduce that. :( Please test on eth0 directly so > we can be sure the packet does not get mangled. > > Can you also show me the output of > ip route get 10.2.72.139 > on the client after you maybe already received a icmp pkt-too-big > packet? > > Thanks, > Hannes > > > > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ip_no_pmtu_disc and UDP
I think the no_pmtu_disc could be renamed to pmtu_disc to be less confusion to users. pmtu_disc: IP_PMTUDISC_DONT, clear DF bit pmtu_disc: IP_PMTUDISC_WANT, set DF bit just my .2 cents On Fri, Oct 23, 2015 at 9:45 AM, Vincent Li <vincent.mc...@gmail.com> wrote: > Hi, > > It looks ip_no_pmtu_disc setting does not affect UDP IP packet DF bit > setting, is that intended behavior? echo 0, 1, 2, 3 respectively to > ip_no_pmtu_disc, UDP IP packet always have DF bit cleared, unless use > IP_PMTUDISC_DO on IP_MTU_DISCOVER as ip man page says. > > in inet_create, seems to prove that. > >if (net->ipv4.sysctl_ip_no_pmtu_disc) > inet->pmtudisc = IP_PMTUDISC_DONT; > else > inet->pmtudisc = IP_PMTUDISC_WANT; > > so I am wondering why UDP is excluded by ip_no_pmtu_disc, why in > inet_create, not assign each individual ip_no_pmtu_disc setting to > inet->pmtudisc but only check true and assign IP_PMTUDISC_DONT or > IP_PMTUDISC_WANT only. > > Thanks > > Vincent -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
ip_no_pmtu_disc and UDP
Hi, It looks ip_no_pmtu_disc setting does not affect UDP IP packet DF bit setting, is that intended behavior? echo 0, 1, 2, 3 respectively to ip_no_pmtu_disc, UDP IP packet always have DF bit cleared, unless use IP_PMTUDISC_DO on IP_MTU_DISCOVER as ip man page says. in inet_create, seems to prove that. if (net->ipv4.sysctl_ip_no_pmtu_disc) inet->pmtudisc = IP_PMTUDISC_DONT; else inet->pmtudisc = IP_PMTUDISC_WANT; so I am wondering why UDP is excluded by ip_no_pmtu_disc, why in inet_create, not assign each individual ip_no_pmtu_disc setting to inet->pmtudisc but only check true and assign IP_PMTUDISC_DONT or IP_PMTUDISC_WANT only. Thanks Vincent -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
