Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
Petr Štetiar [2020-12-02 21:25:12]:
Hi,
> > Can you check if commit
> > https://git.openwrt.org/?p=project/odhcp6c.git;a=commit;h=a7b2221f687264c020b7a18a4e690d79f312a667
> > fixes the issue in your setup ?
>
> yes, it seems so as it's currently almost 24h up without any issues.
I was testing on odhcp6c with the following commits:
odhcp6c: update to 2020-12-01 version
bcd86c748aeb script: handle possible issues with script termination
031c39471396 cmake: fix out of tree building with libubox library
a7b2221f6872 dhcpv6: avoid sending continuous renew/rebind messages
d7afa2b6d31a dhcpv6: add extra syslog info traces
f5728e40ff79 odhcp6c_find_entry: exclude priority from the list of fields
that must match
FYI I've just updated to latest master with your commit 13734075d256
("odhcp6c: update to git HEAD") and it seems like the issue is back.
http://ynezz.true.cz/openwrt/odhcp6c/syslog-2020-12-07.log.gz
-- ynezz
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
Hans Dedecker [2020-12-04 13:55:27]: > Could you run odhcpd with loglevel 7 as I would like to understand > what triggers the numerous transmission of the Reconfigure messages http://ynezz.true.cz/openwrt/odhcp6c/syslog-2020-12-05.log.gz ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
On Fri, Dec 4, 2020 at 11:45 AM Petr Štetiar wrote: > > Hans Dedecker [2020-12-03 21:18:11]: > > Hi, > > > Looking into the new pcap capture I must admit I've never seen such a DHCPv6 > > server with such weird behavior. > > it's OpenWrt so odhcpd. Could you run odhcpd with loglevel 7 as I would like to understand what triggers the numerous transmission of the Reconfigure messages Thx Hans > > -- ynezz ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
Hans Dedecker [2020-12-03 21:18:11]: Hi, > Looking into the new pcap capture I must admit I've never seen such a DHCPv6 > server with such weird behavior. it's OpenWrt so odhcpd. -- ynezz ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
On Wed, Dec 2, 2020 at 9:25 PM Petr Štetiar wrote: > > Hans Dedecker [2020-12-01 21:12:30]: > > Hi, > > > Can you check if commit > > https://git.openwrt.org/?p=project/odhcp6c.git;a=commit;h=a7b2221f687264c020b7a18a4e690d79f312a667 > > fixes the issue in your setup ? > > yes, it seems so as it's currently almost 24h up without any issues. Thank you for testing ! Looking into the new pcap capture I must admit I've never seen such a DHCPv6 server with such weird behavior. Hans > > > While doing the test can you put odhcp6c into verbose mode (-v) and > > take again a pcap trace on the wan ? > > http://ynezz.true.cz/openwrt/odhcp6c/capture-2020-12-02.pcap.gz > http://ynezz.true.cz/openwrt/odhcp6c/syslog-2020-12-02.log.gz > > Thanks! > > Cheers, > > Petr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
Hans Dedecker [2020-12-01 21:12:30]: Hi, > Can you check if commit > https://git.openwrt.org/?p=project/odhcp6c.git;a=commit;h=a7b2221f687264c020b7a18a4e690d79f312a667 > fixes the issue in your setup ? yes, it seems so as it's currently almost 24h up without any issues. > While doing the test can you put odhcp6c into verbose mode (-v) and > take again a pcap trace on the wan ? http://ynezz.true.cz/openwrt/odhcp6c/capture-2020-12-02.pcap.gz http://ynezz.true.cz/openwrt/odhcp6c/syslog-2020-12-02.log.gz Thanks! Cheers, Petr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
Hi Petr, On Sat, Nov 28, 2020 at 8:48 PM Hans Dedecker wrote: > > Hi, > > On Sat, Nov 28, 2020 at 5:04 PM Petr Štetiar wrote: > > > > Hans Dedecker [2020-11-26 21:19:30]: > > > > Hi, > > > > > Is it possible to have a pcap trace of the DHCPv6 messages on the wan ? > > > I hope to find some time next weekend to investigate the issue further > > > > http://ynezz.true.cz/openwrt/odhcp6c/capture.pcap.gz# > Thank you for the traces; looking into the DHCPv6 packets make the issue > clear. > Packet 22 is a DHCPv6 reply packet in response to a renew which in its > turn is triggered by a DHCPv6 reconfigure packet. > The packet contains an IA-PD option but no IA prefix option which is > strange. According to the DHCPv6 RFC renew messages must be kept > sending when T1 has expired for either an IA address or IA prefix when > the server does not reply the IA address or IA prefix which is > happening in this case. > I will do further investigation how we could protect odhpc6c against > such DHCPv6 server behavior Can you check if commit https://git.openwrt.org/?p=project/odhcp6c.git;a=commit;h=a7b2221f687264c020b7a18a4e690d79f312a667 fixes the issue in your setup ? While doing the test can you put odhcp6c into verbose mode (-v) and take again a pcap trace on the wan ? Hans > > Hans > > > > Cheers, > > > > Petr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
Hi, On Sat, Nov 28, 2020 at 5:04 PM Petr Štetiar wrote: > > Hans Dedecker [2020-11-26 21:19:30]: > > Hi, > > > Is it possible to have a pcap trace of the DHCPv6 messages on the wan ? > > I hope to find some time next weekend to investigate the issue further > > http://ynezz.true.cz/openwrt/odhcp6c/capture.pcap.gz# Thank you for the traces; looking into the DHCPv6 packets make the issue clear. Packet 22 is a DHCPv6 reply packet in response to a renew which in its turn is triggered by a DHCPv6 reconfigure packet. The packet contains an IA-PD option but no IA prefix option which is strange. According to the DHCPv6 RFC renew messages must be kept sending when T1 has expired for either an IA address or IA prefix when the server does not reply the IA address or IA prefix which is happening in this case. I will do further investigation how we could protect odhpc6c against such DHCPv6 server behavior Hans > > Cheers, > > Petr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
Hans Dedecker [2020-11-26 21:19:30]: Hi, > Is it possible to have a pcap trace of the DHCPv6 messages on the wan ? > I hope to find some time next weekend to investigate the issue further http://ynezz.true.cz/openwrt/odhcp6c/capture.pcap.gz Cheers, Petr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
Hi Petr On Thu, Nov 26, 2020 at 9:58 AM Petr Štetiar wrote: > > Hi Hans, > > I've tried to ping you on IRC few days ago as I've found some strange issue > with odhcp6c which leads to OOM/OOPS[1] (log with just the important parts). > It simply happens every time after about 10 hours for me when I boot the > system > in QEMU and have LAN/WAN networks connected there. > > It looks like that RENEW retry timeout is being decremented from 13060s down > to > 1s during those few hours, then it goes in the crazy loop and it seems to > trash > the machine completely, sometimes OOMs, sometimes OOPS and reboots due to the > squashfs issues. Thank you for the traces;it looks like the reply message in response to the renew message does not update the renew time which is strange. Is it possible to have a pcap trace of the DHCPv6 messages on the wan ? I hope to find some time next weekend to investigate the issue further Hans > > Nothing is happening on that machine, I just boot it and then let it idle. > > I've also uploaded complete unfiltered syslog[2] for you. I'm able to > reproduce it easily, so happy to help test the fix. > > This is on latest and greatest master, x86/64 in QEMU. Thanks! > > 1. http://sprunge.us/vv0Idx > 2. http://ynezz.true.cz/openwrt/odhcp6c/syslog.log.gz > > Cheers, > > Petr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
odhcp6c RENEW timeout issue leads to OOM/OOPS [Was: Re: QEMU x86/64 ubus issues ... ]
Hi Hans, I've tried to ping you on IRC few days ago as I've found some strange issue with odhcp6c which leads to OOM/OOPS[1] (log with just the important parts). It simply happens every time after about 10 hours for me when I boot the system in QEMU and have LAN/WAN networks connected there. It looks like that RENEW retry timeout is being decremented from 13060s down to 1s during those few hours, then it goes in the crazy loop and it seems to trash the machine completely, sometimes OOMs, sometimes OOPS and reboots due to the squashfs issues. Nothing is happening on that machine, I just boot it and then let it idle. I've also uploaded complete unfiltered syslog[2] for you. I'm able to reproduce it easily, so happy to help test the fix. This is on latest and greatest master, x86/64 in QEMU. Thanks! 1. http://sprunge.us/vv0Idx 2. http://ynezz.true.cz/openwrt/odhcp6c/syslog.log.gz Cheers, Petr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
