Glad to hear it. By the way, the Jira ticket you created was so well done that I was able to diagnose the problem in a few minutes.
Thanks for taking the time to create such a useful bug report. Dabe From: Yusuke Tatsumi <[email protected]> Sent: Friday, July 12, 2019 11:02 AM To: Luke, Chris <[email protected]>; Dave Barach (dbarach) <[email protected]>; [email protected] Subject: Re: vppctl makes VPP hang-up occasionally Hi Dave, Chris, Thank you for your kind support! With your patch, our VPPs works fine "without" hang-up so far in 24h. https://gerrit.fd.io/r/#/c/20573/ Regards, Tatsumi. ________________________________ 差出人: Luke, Chris <[email protected]<mailto:[email protected]>> 送信日時: 2019年7月11日 6:38 宛先: Dave Barach (dbarach); 立見 祐介; [email protected]<mailto:[email protected]> 件名: RE: vppctl makes VPP hang-up occasionally Sure, and in 99.99% of cases nobody would ever notice, too. For the curious, the most straightforward way to tickle this particular feature is something like “socat /run/vpp/cli.sock stdio” to get a CLI connection without any TELNET negotiation; there’s a ~1s pause while VPP waits for the negotiation to complete before it gives up and starts the session regardless. Chris. From: Dave Barach (dbarach) <[email protected]<mailto:[email protected]>> Sent: Wednesday, July 10, 2019 17:04 To: Luke, Chris <[email protected]<mailto:[email protected]>>; Yusuke Tatsumi <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: [EXTERNAL] RE: vppctl makes VPP hang-up occasionally +1, commenting out the timer_call(...) was a quick hack to make the pain go away... From: Luke, Chris <[email protected]<mailto:[email protected]>> Sent: Wednesday, July 10, 2019 4:55 PM To: Dave Barach (dbarach) <[email protected]<mailto:[email protected]>>; Yusuke Tatsumi <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: RE: vppctl makes VPP hang-up occasionally For the archives, https://gerrit.fd.io/r/#/c/20573/ was merged earlier to resolve this issue. We are hoping Tatsumi is able to confirm this, but since it is a better approach regardless. Chris. From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of Dave Barach via Lists.Fd.Io Sent: Tuesday, July 9, 2019 08:06 To: Yusuke Tatsumi <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally The root cause is obvious from the backtrace in VPP-1711: memory allocator takes a spin-lock, timer interrupt fires, back into the memory allocator, deadlock. The [sic] timer interrupt function in .../src/vppinfra/timers.c is a mistake. You managed to find the only place in all of vpp which uses it. As a quick 1-line workaround, comment out the “timer_call” shown below, around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an “if(0)”... /* In case the client doesn't negotiate terminal type, use * a timer to kick off the initial prompt. */ if (0) timer_call (unix_cli_file_welcome_timer, cf_index, 1); HTH... Dave From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of Yusuke Tatsumi Sent: Tuesday, July 9, 2019 1:11 AM To: [email protected]<mailto:[email protected]> Subject: [vpp-dev] vppctl makes VPP hang-up occasionally Hi all, We found that VPP itself freeze once a day under using "vppctl" command repeatedly. I think this is a kind of spin-lock problem but I can't understand the root cause. See here for details. https://jira.fd.io/browse/VPP-1711 If this goes on, It's hard to operate VPP as production service. So I need help to fix this issue. Could anyone help/suggest to this issue? Thanks, Tatsumi. - 立見 祐介 ヤフー株式会社 テクノロジーグループ システム統括本部 クラウドプラットフォーム本部 技術1部 コンピュート TEL: 03-6898-3081 mail: [email protected]<mailto:[email protected]> - Yusuke Tatsumi Compute team, Cloud Platform Division, System Management Group Yahoo Japan Corporation Direct: +81 (3) 6898 3081 mail: [email protected]<mailto:[email protected]>
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13504): https://lists.fd.io/g/vpp-dev/message/13504 Mute This Topic: https://lists.fd.io/mt/32402610/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
