Re: FreeBSD 10-RC4: Got crash in igb driver
I work with Jack on FreeBSD network drivers, and we have a patch that we think might fix this problem. It re-implements the header pull-up code that was in the driver pre-2.4.0, but with IPv6 support. Alexandre, could you apply this patch to the igb version in HEAD and try it out on your network? We can't replicate your setup here. If it solves your problem, then the next step would be to port the patch to the ixgbe driver since, as Yonghyeon noted, it looks like that driver will encounter the same problem. We could then modify em to add IPv6 offload support as well. Thanks, - Eric Joyner Can I get in on network drivers, two in particular where I have the hardware involved? Drivers in my case are re, which fails on my motherboard, and athn, not currently present in FreeBSD. I would have NetBSD src tree to compare with, and NetBSD re driver works on my motherboard. Tom ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: FreeBSD 10-RC4: Got crash in igb driver
I have some compillation issues : ../../../dev/e1000/if_igb.c: In function 'igb_mq_start_locked': ../../../dev/e1000/if_igb.c:3936: warning: 'poff' may be used uninitialized in this function ../../../dev/e1000/if_igb.c:3936: note: 'poff' was declared here ../../../dev/e1000/if_igb.c:3936: warning: 'ip_hlen' may be used uninitialized in this function ../../../dev/e1000/if_igb.c:3936: note: 'ip_hlen' was declared here But this fix the crash. However, I have now a copy made of all ipv6 forwarded packet in igb_pullup_headers. It will be nice to m_free the copy of the packet made in ip6_forward before calling nd6_output function. It will put back the packet writable and avoid (in most cases) the copy. I have no time this week to check performance of this new driver version. I'll try to make that Monday. Thank you for this fix Le mardi 14 janvier 2014 14:22:43 Eric Joyner a écrit : All, I work with Jack on FreeBSD network drivers, and we have a patch that we think might fix this problem. It re-implements the header pull-up code that was in the driver pre-2.4.0, but with IPv6 support. Alexandre, could you apply this patch to the igb version in HEAD and try it out on your network? We can't replicate your setup here. If it solves your problem, then the next step would be to port the patch to the ixgbe driver since, as Yonghyeon noted, it looks like that driver will encounter the same problem. We could then modify em to add IPv6 offload support as well. Thanks, --- - Eric Joyner On Fri, Jan 10, 2014 at 02:35:29PM +0400, Gleb Smirnoff wrote: Yonghyeon, On Fri, Jan 10, 2014 at 10:21:14AM +0900, Yonghyeon PYUN wrote: Y I experience some troubles with the igb device driver on FreeBSD 10-RC4. Y Y The kernel make a pagefault in the igb_tx_ctx_setup function when accessing to Y a IPv6 header. Y Y The network configuration is the following: Y - box acting as an IPv6 router Y - one interface with an IPv6 (igb0) Y - another interface with a vlan, and IPv6 on it (vlan0 on igb1) Y Y Vlan Hardware tagging is set on both interfaces. Y Y The packet that cause the crash come from igb0 and go to vlan0. Y Y After investigation, i see that the mbuf is split in two. The first one carry Y the ethernet header, the second, the IPv6 header and data payload. Y Y The split is due to the m_copy done in ip6_forward, that make the mbuf not Y writable and the M_PREPEND in ether_output that insert the new mbuf before Y the original one. Y Y The kernel crashes only if the newly allocated mbuf is at the end of a memory Y page, and no page is available after this one. So, it's extremly rare. Y Y I inserted a KASSERT into the function (see attached patch) to check this Y behavior, and it raises on every IPv6 forwarded packet to the vlan. The Y problem disapear if i remove hardware tagging. Y Y In the commit 256200, i see that pullups has been removed. May it be related ? Y Y I think I introduced the header parsing code to meet controller Y requirement in em(4) and Jack borrowed that code in the past but it Y seems it was removed in r256200. It seems igb_tx_ctx_setup() Y assumes it can access ethernet/IP/TCP/UDP headers in the first mbuf Y of the chain. Y This looks wrong to me. Can you please restore the important code in head ASAP? Although crashes happen only when the mbuf is last in a page and page isn't mapped, we read thrash from next allocation on almost every packet. It seems other Intel ethernet drivers except em(4) have similar issues. I didn't check recent Intel controllers/drivers for long time so I don't know details on hardware requirements of offloading. Since Jack is very responsive and has hardwares to verify, he would be more appropriate person to handle these issues. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org -- Alexandre Martins NETASQ -- We secure IT smime.p7s Description: S/MIME cryptographic signature
Re: FreeBSD 10-RC4: Got crash in igb driver
All, I work with Jack on FreeBSD network drivers, and we have a patch that we think might fix this problem. It re-implements the header pull-up code that was in the driver pre-2.4.0, but with IPv6 support. Alexandre, could you apply this patch to the igb version in HEAD and try it out on your network? We can't replicate your setup here. If it solves your problem, then the next step would be to port the patch to the ixgbe driver since, as Yonghyeon noted, it looks like that driver will encounter the same problem. We could then modify em to add IPv6 offload support as well. Thanks, --- - Eric Joyner On Fri, Jan 10, 2014 at 02:35:29PM +0400, Gleb Smirnoff wrote: Yonghyeon, On Fri, Jan 10, 2014 at 10:21:14AM +0900, Yonghyeon PYUN wrote: Y I experience some troubles with the igb device driver on FreeBSD 10-RC4. Y Y The kernel make a pagefault in the igb_tx_ctx_setup function when accessing to Y a IPv6 header. Y Y The network configuration is the following: Y - box acting as an IPv6 router Y - one interface with an IPv6 (igb0) Y - another interface with a vlan, and IPv6 on it (vlan0 on igb1) Y Y Vlan Hardware tagging is set on both interfaces. Y Y The packet that cause the crash come from igb0 and go to vlan0. Y Y After investigation, i see that the mbuf is split in two. The first one carry Y the ethernet header, the second, the IPv6 header and data payload. Y Y The split is due to the m_copy done in ip6_forward, that make the mbuf not Y writable and the M_PREPEND in ether_output that insert the new mbuf before Y the original one. Y Y The kernel crashes only if the newly allocated mbuf is at the end of a memory Y page, and no page is available after this one. So, it's extremly rare. Y Y I inserted a KASSERT into the function (see attached patch) to check this Y behavior, and it raises on every IPv6 forwarded packet to the vlan. The Y problem disapear if i remove hardware tagging. Y Y In the commit 256200, i see that pullups has been removed. May it be related ? Y Y I think I introduced the header parsing code to meet controller Y requirement in em(4) and Jack borrowed that code in the past but it Y seems it was removed in r256200. It seems igb_tx_ctx_setup() Y assumes it can access ethernet/IP/TCP/UDP headers in the first mbuf Y of the chain. Y This looks wrong to me. Can you please restore the important code in head ASAP? Although crashes happen only when the mbuf is last in a page and page isn't mapped, we read thrash from next allocation on almost every packet. It seems other Intel ethernet drivers except em(4) have similar issues. I didn't check recent Intel controllers/drivers for long time so I don't know details on hardware requirements of offloading. Since Jack is very responsive and has hardwares to verify, he would be more appropriate person to handle these issues. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org fix.patch Description: Binary data ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: FreeBSD 10-RC4: Got crash in igb driver
On Fri, Jan 10, 2014 at 02:35:29PM +0400, Gleb Smirnoff wrote: Yonghyeon, On Fri, Jan 10, 2014 at 10:21:14AM +0900, Yonghyeon PYUN wrote: Y I experience some troubles with the igb device driver on FreeBSD 10-RC4. Y Y The kernel make a pagefault in the igb_tx_ctx_setup function when accessing to Y a IPv6 header. Y Y The network configuration is the following: Y - box acting as an IPv6 router Y - one interface with an IPv6 (igb0) Y - another interface with a vlan, and IPv6 on it (vlan0 on igb1) Y Y Vlan Hardware tagging is set on both interfaces. Y Y The packet that cause the crash come from igb0 and go to vlan0. Y Y After investigation, i see that the mbuf is split in two. The first one carry Y the ethernet header, the second, the IPv6 header and data payload. Y Y The split is due to the m_copy done in ip6_forward, that make the mbuf not Y writable and the M_PREPEND in ether_output that insert the new mbuf before Y the original one. Y Y The kernel crashes only if the newly allocated mbuf is at the end of a memory Y page, and no page is available after this one. So, it's extremly rare. Y Y I inserted a KASSERT into the function (see attached patch) to check this Y behavior, and it raises on every IPv6 forwarded packet to the vlan. The Y problem disapear if i remove hardware tagging. Y Y In the commit 256200, i see that pullups has been removed. May it be related ? Y Y I think I introduced the header parsing code to meet controller Y requirement in em(4) and Jack borrowed that code in the past but it Y seems it was removed in r256200. It seems igb_tx_ctx_setup() Y assumes it can access ethernet/IP/TCP/UDP headers in the first mbuf Y of the chain. Y This looks wrong to me. Can you please restore the important code in head ASAP? Although crashes happen only when the mbuf is last in a page and page isn't mapped, we read thrash from next allocation on almost every packet. It seems other Intel ethernet drivers except em(4) have similar issues. I didn't check recent Intel controllers/drivers for long time so I don't know details on hardware requirements of offloading. Since Jack is very responsive and has hardwares to verify, he would be more appropriate person to handle these issues. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: FreeBSD 10-RC4: Got crash in igb driver
Le 10 janv. 2014 à 02:21, Yonghyeon PYUN pyu...@gmail.com a écrit : On Thu, Jan 09, 2014 at 04:06:09PM +0100, Alexandre Martins wrote: Dear, I experience some troubles with the igb device driver on FreeBSD 10-RC4. The kernel make a pagefault in the igb_tx_ctx_setup function when accessing to a IPv6 header. The network configuration is the following: - box acting as an IPv6 router - one interface with an IPv6 (igb0) - another interface with a vlan, and IPv6 on it (vlan0 on igb1) Vlan Hardware tagging is set on both interfaces. The packet that cause the crash come from igb0 and go to vlan0. After investigation, i see that the mbuf is split in two. The first one carry the ethernet header, the second, the IPv6 header and data payload. The split is due to the m_copy done in ip6_forward, that make the mbuf not writable and the M_PREPEND in ether_output that insert the new mbuf before the original one. The kernel crashes only if the newly allocated mbuf is at the end of a memory page, and no page is available after this one. So, it's extremly rare. I inserted a KASSERT into the function (see attached patch) to check this behavior, and it raises on every IPv6 forwarded packet to the vlan. The problem disapear if i remove hardware tagging. In the commit 256200, i see that pullups has been removed. May it be related ? I think I introduced the header parsing code to meet controller requirement in em(4) and Jack borrowed that code in the past but it seems it was removed in r256200. It seems igb_tx_ctx_setup() assumes it can access ethernet/IP/TCP/UDP headers in the first mbuf of the chain. This looks wrong to me. Instead of patching each driver with pullup code we can add a generic pullup code ? - get the contiguous protocol requirement (L2, L3, L4) from underlying driver. - do the pullup Can you confirm the problem ? Probably Jack can tell more about change made in r256200. It's not easy for me to verify correctness of igb(4) at this moment. Best regards -- Alexandre Martins NETASQ -- We secure IT ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: FreeBSD 10-RC4: Got crash in igb driver
On Fri, Jan 10, 2014 at 09:37:33AM +0100, Fabien Thomas wrote: Le 10 janv. 2014 ? 02:21, Yonghyeon PYUN pyu...@gmail.com a ?crit : On Thu, Jan 09, 2014 at 04:06:09PM +0100, Alexandre Martins wrote: Dear, I experience some troubles with the igb device driver on FreeBSD 10-RC4. The kernel make a pagefault in the igb_tx_ctx_setup function when accessing to a IPv6 header. The network configuration is the following: - box acting as an IPv6 router - one interface with an IPv6 (igb0) - another interface with a vlan, and IPv6 on it (vlan0 on igb1) Vlan Hardware tagging is set on both interfaces. The packet that cause the crash come from igb0 and go to vlan0. After investigation, i see that the mbuf is split in two. The first one carry the ethernet header, the second, the IPv6 header and data payload. The split is due to the m_copy done in ip6_forward, that make the mbuf not writable and the M_PREPEND in ether_output that insert the new mbuf before the original one. The kernel crashes only if the newly allocated mbuf is at the end of a memory page, and no page is available after this one. So, it's extremly rare. I inserted a KASSERT into the function (see attached patch) to check this behavior, and it raises on every IPv6 forwarded packet to the vlan. The problem disapear if i remove hardware tagging. In the commit 256200, i see that pullups has been removed. May it be related ? I think I introduced the header parsing code to meet controller requirement in em(4) and Jack borrowed that code in the past but it seems it was removed in r256200. It seems igb_tx_ctx_setup() assumes it can access ethernet/IP/TCP/UDP headers in the first mbuf of the chain. This looks wrong to me. Instead of patching each driver with pullup code we can add a generic pullup code ? - get the contiguous protocol requirement (L2, L3, L4) from underlying driver. - do the pullup I believe Andre already planned that and he would be working on removing home-grown header parser implemented in drivers. Can you confirm the problem ? Probably Jack can tell more about change made in r256200. It's not easy for me to verify correctness of igb(4) at this moment. Best regards -- Alexandre Martins NETASQ -- We secure IT ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: FreeBSD 10-RC4: Got crash in igb driver
Yonghyeon, On Fri, Jan 10, 2014 at 10:21:14AM +0900, Yonghyeon PYUN wrote: Y I experience some troubles with the igb device driver on FreeBSD 10-RC4. Y Y The kernel make a pagefault in the igb_tx_ctx_setup function when accessing to Y a IPv6 header. Y Y The network configuration is the following: Y - box acting as an IPv6 router Y - one interface with an IPv6 (igb0) Y - another interface with a vlan, and IPv6 on it (vlan0 on igb1) Y Y Vlan Hardware tagging is set on both interfaces. Y Y The packet that cause the crash come from igb0 and go to vlan0. Y Y After investigation, i see that the mbuf is split in two. The first one carry Y the ethernet header, the second, the IPv6 header and data payload. Y Y The split is due to the m_copy done in ip6_forward, that make the mbuf not Y writable and the M_PREPEND in ether_output that insert the new mbuf before Y the original one. Y Y The kernel crashes only if the newly allocated mbuf is at the end of a memory Y page, and no page is available after this one. So, it's extremly rare. Y Y I inserted a KASSERT into the function (see attached patch) to check this Y behavior, and it raises on every IPv6 forwarded packet to the vlan. The Y problem disapear if i remove hardware tagging. Y Y In the commit 256200, i see that pullups has been removed. May it be related ? Y Y I think I introduced the header parsing code to meet controller Y requirement in em(4) and Jack borrowed that code in the past but it Y seems it was removed in r256200. It seems igb_tx_ctx_setup() Y assumes it can access ethernet/IP/TCP/UDP headers in the first mbuf Y of the chain. Y This looks wrong to me. Can you please restore the important code in head ASAP? Although crashes happen only when the mbuf is last in a page and page isn't mapped, we read thrash from next allocation on almost every packet. -- Totus tuus, Glebius. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: FreeBSD 10-RC4: Got crash in igb driver
The changes to igb were to add IPV6 support which previously was only in ixgbe, the transmit path code came from that code base, we did not see this issue in testing. Its not a simple matter of a few lines of code, I think we need to go forward not back... I'll look at the code. Jack On Fri, Jan 10, 2014 at 2:35 AM, Gleb Smirnoff gleb...@freebsd.org wrote: Yonghyeon, On Fri, Jan 10, 2014 at 10:21:14AM +0900, Yonghyeon PYUN wrote: Y I experience some troubles with the igb device driver on FreeBSD 10-RC4. Y Y The kernel make a pagefault in the igb_tx_ctx_setup function when accessing to Y a IPv6 header. Y Y The network configuration is the following: Y - box acting as an IPv6 router Y - one interface with an IPv6 (igb0) Y - another interface with a vlan, and IPv6 on it (vlan0 on igb1) Y Y Vlan Hardware tagging is set on both interfaces. Y Y The packet that cause the crash come from igb0 and go to vlan0. Y Y After investigation, i see that the mbuf is split in two. The first one carry Y the ethernet header, the second, the IPv6 header and data payload. Y Y The split is due to the m_copy done in ip6_forward, that make the mbuf not Y writable and the M_PREPEND in ether_output that insert the new mbuf before Y the original one. Y Y The kernel crashes only if the newly allocated mbuf is at the end of a memory Y page, and no page is available after this one. So, it's extremly rare. Y Y I inserted a KASSERT into the function (see attached patch) to check this Y behavior, and it raises on every IPv6 forwarded packet to the vlan. The Y problem disapear if i remove hardware tagging. Y Y In the commit 256200, i see that pullups has been removed. May it be related ? Y Y I think I introduced the header parsing code to meet controller Y requirement in em(4) and Jack borrowed that code in the past but it Y seems it was removed in r256200. It seems igb_tx_ctx_setup() Y assumes it can access ethernet/IP/TCP/UDP headers in the first mbuf Y of the chain. Y This looks wrong to me. Can you please restore the important code in head ASAP? Although crashes happen only when the mbuf is last in a page and page isn't mapped, we read thrash from next allocation on almost every packet. -- Totus tuus, Glebius. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: FreeBSD 10-RC4: Got crash in igb driver
On Thu, Jan 09, 2014 at 04:06:09PM +0100, Alexandre Martins wrote: Dear, I experience some troubles with the igb device driver on FreeBSD 10-RC4. The kernel make a pagefault in the igb_tx_ctx_setup function when accessing to a IPv6 header. The network configuration is the following: - box acting as an IPv6 router - one interface with an IPv6 (igb0) - another interface with a vlan, and IPv6 on it (vlan0 on igb1) Vlan Hardware tagging is set on both interfaces. The packet that cause the crash come from igb0 and go to vlan0. After investigation, i see that the mbuf is split in two. The first one carry the ethernet header, the second, the IPv6 header and data payload. The split is due to the m_copy done in ip6_forward, that make the mbuf not writable and the M_PREPEND in ether_output that insert the new mbuf before the original one. The kernel crashes only if the newly allocated mbuf is at the end of a memory page, and no page is available after this one. So, it's extremly rare. I inserted a KASSERT into the function (see attached patch) to check this behavior, and it raises on every IPv6 forwarded packet to the vlan. The problem disapear if i remove hardware tagging. In the commit 256200, i see that pullups has been removed. May it be related ? I think I introduced the header parsing code to meet controller requirement in em(4) and Jack borrowed that code in the past but it seems it was removed in r256200. It seems igb_tx_ctx_setup() assumes it can access ethernet/IP/TCP/UDP headers in the first mbuf of the chain. This looks wrong to me. Can you confirm the problem ? Probably Jack can tell more about change made in r256200. It's not easy for me to verify correctness of igb(4) at this moment. Best regards -- Alexandre Martins NETASQ -- We secure IT ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
