Hi Ivan, Will try to reproduce but given the types of crashes, it could be that the proxy app is not cleanly releasing the connections.
Regards, Florin > On Jul 22, 2020, at 8:29 AM, Ivan Shvedunov <[email protected]> wrote: > > Some preliminary observations concerning the crashes in the proxy example: > * !rb_tree_is_init(...) assertion failures are likely caused by multiple > active_open_connected_callback() invocations for the same connection > * f_update_ooo_deq() SIGSEGV crash is possibly caused for late callbacks for > connections that are already gone > > On Wed, Jul 22, 2020 at 2:18 PM Ivan Shvedunov via lists.fd.io > <http://lists.fd.io/> <[email protected] > <mailto:[email protected]>> wrote: > Hi, > sadly the patch apparently didn't work. It should have worked but for some > reason it didn't ... > > On the bright side, I've made a test case [1] using fresh upstream VPP code > with no UPF that reproduces the issues I mentioned, including both timer and > TCP retransmit one along with some other possible problems using http_static > plugin and the proxy example, along with nginx (with proxy) and wrk. > > It is docker-based, but the main scripts (start.sh and test.sh) can be used > without Docker, too. > I've used our own Dockerfiles to build the images, but I'm not sure if that > makes any difference. > I've added some log files resulting from the runs that crashed in different > places. For me, the tests crash on each run, but in different places. > > The TCP retransmit problem happens with http_static when using the debug > build. When using release build, some unrelated crash in timer_remove() > happens instead. > The SVM FIFO crash happens when using the proxy. It can happen with both > release and debug builds. > > Please see the repo [1] for details and crash logs. > > [1] https://github.com/ivan4th/vpp-tcp-test > <https://github.com/ivan4th/vpp-tcp-test> > > P.S. As the tests do expose some problems with VPP host stack and some of the > VPP plugins/examples, maybe we should consider adding them to the VPP CI, too? > > On Thu, Jul 16, 2020 at 8:33 PM Florin Coras <[email protected] > <mailto:[email protected]>> wrote: > Hi Ivan, > > Thanks for the detailed report! > > I assume this is a situation where most of the connections time out and the > rate limiting we apply on the pending timer queue delays handling for long > enough to be in a situation like the one you described. Here’s a draft patch > that starts tracking pending timers [1]. Let me know if it solves the first > problem. > > Regarding the second, it looks like the first chunk in the fifo is not > properly initialized/corrupted. It’s hard to tell what leads to that given > that I haven’t seen this sort of issues even with larger number of > connections. You could maybe try calling svm_fifo_is_sane() in the > enqueue/dequeue functions, or after the proxy allocates/shares the fifos to > catch the issue as early as possible. > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/27952 > <https://gerrit.fd.io/r/c/vpp/+/27952> > >> On Jul 16, 2020, at 2:03 AM, [email protected] <mailto:[email protected]> >> wrote: >> >> Hi, >> I'm working on the Travelping UPF project >> https://github.com/travelping/vpp <https://github.com/travelping/vpp>For >> variety of reasons, it's presently maintained as a fork of UPF that's >> rebased on top of upstream master from time to time, but really it's just a >> plugin. During 40K TCP connection test with netem, I found an issue with TCP >> timer race (timers firing after tcp_timer_reset() was called for them) which >> I tried to work around only to stumble into another crash, which I'm >> presently debugging (an SVM FIFO bug, possibly) but maybe some of you folks >> have some ideas what it could be. >> I've described my findings in this JIRA ticket: >> https://jira.fd.io/browse/VPP-1923 <https://jira.fd.io/browse/VPP-1923> >> Although the last upstream commit UPF is presently based on >> (afc233aa93c3f23b30b756cb4ae2967f968bbbb1) was some time ago, I believe the >> problems are still relevant as there were no changes in these parts of code >> in master since that commit. > > > > -- > Ivan Shvedunov <[email protected] <mailto:[email protected]>> > ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807 > > > > -- > Ivan Shvedunov <[email protected] <mailto:[email protected]>> > ;; My GPG fingerprint is: 2E61 0748 8E12 BB1A 5AB9 F7D0 613E C0F8 0BC5 2807
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17040): https://lists.fd.io/g/vpp-dev/message/17040 Mute This Topic: https://lists.fd.io/mt/75537746/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
