Hi Shuva, Would you like to have an IRC chat in about 20 minutes?
Thanks, Koby ________________________________ From: Shuva Jyoti Kar <shuva.jyoti....@ericsson.com> Sent: Dec 13, 2016 9:21 AM To: Aizer, Koby; Kochba, Alon; Vishal Thapar; Sam Hague; Dayavanti Gopal Kamath Cc: Anil Vishnoi; openflowplugin-dev@lists.opendaylight.org Subject: RE: FW: [openflowplugin-dev] Error processing port status message Hi Koby, I am quite ignorant of the netvirt logic. trying to isolate the problem from the perspective of openflowplugin. If I install only ofplugin and dependent features(mdsal and restconf) without any of the services I donot encounter any errors during port-bringup. Need your help in understanding what you are trying to do that results in the issue Thanks Shuva From: Aizer, Koby [mailto:koby.ai...@hpe.com] Sent: Saturday, December 10, 2016 2:08 PM To: Shuva Jyoti Kar; Kochba, Alon; Vishal Thapar; Sam Hague; Dayavanti Gopal Kamath Cc: Anil Vishnoi; openflowplugin-dev@lists.opendaylight.org Subject: RE: FW: [openflowplugin-dev] Error processing port status message Hi Shuva, Sorry I haven’t seen this message on time so I didn’t join the call. We can talk beginning of next week and sync on the details. Please also see Dotan’s mail from Thursday where he reproduced the issue with DEBUG log level set for org.opendaylight.openflowplugin.impl Have you got a chance to reproduce the issue with the script Dotan has provided? Thanks, Koby From: Shuva Jyoti Kar [mailto:shuva.jyoti....@ericsson.com] Sent: Thursday, December 8, 2016 3:31 PM To: Aizer, Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>; Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>>; Vishal Thapar <vishal.tha...@ericsson.com<mailto:vishal.tha...@ericsson.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>> Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org> Subject: RE: FW: [openflowplugin-dev] Error processing port status message Hi Koby, Sorry for the late response. Could you join today’s call and lets discuss the issue ? -shuva From: Aizer, Koby [mailto:koby.ai...@hpe.com] Sent: Tuesday, December 06, 2016 8:14 PM To: Kochba, Alon; Shuva Jyoti Kar; Vishal Thapar; Sam Hague; Dayavanti Gopal Kamath Cc: Anil Vishnoi; openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org> Subject: RE: FW: [openflowplugin-dev] Error processing port status message Hi Shuva, In addition to the recreation efforts with higher debug level, please note that this issue occurred long after the DPN was connected (hard to see that in the log as it is trimmed). Looking in the log Dotan has attached to the ticket, between 2016-11-08 08:28:32 to 2016-11-08 09:22:18, you can see ~250 successful InterfaceStateChangeListeners events (which I believe are triggered from the port status messages). At this point something breaks, and you start seeing the “Error processing port status” WARNs. I also just noticed that these WARNs have started occurring after the following messages, could they give us some lead? 2016-11-08 09:22:19,682 | WARN | CommitFutures-4 | TransactionChainManager | 291 - org.opendaylight.openflowplugin.impl - 0.3.1.SNAPSHOT | txChain failed -> recreating due to {} 2016-11-08 09:22:19,684 | ERROR | CommitFutures-4 | ExecutionList | 38 - com.google.guava - 18.0.0 | RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@680785f9<mailto:com.google.common.util.concurrent.Futures$6@680785f9> with executor INSTANCE 2016-11-08 09:22:19,698 | ERROR | CommitFutures-4 | TransactionChainManager | 291 - org.opendaylight.openflowplugin.impl - 0.3.1.SNAPSHOT | Transaction commit failed. Thanks, Koby From: Kochba, Alon Sent: Tuesday, December 6, 2016 1:49 PM To: Shuva Jyoti Kar <shuva.jyoti....@ericsson.com<mailto:shuva.jyoti....@ericsson.com>>; Vishal Thapar <vishal.tha...@ericsson.com<mailto:vishal.tha...@ericsson.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>> Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>; openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org> Subject: RE: FW: [openflowplugin-dev] Error processing port status message Thanks Shuva, It happened to Dotan with only one compute. Can you try the script he provided to recreate? I asked him to try to recreate it with the debug logs and tcpdump in any case. --alon From: Shuva Jyoti Kar [mailto:shuva.jyoti....@ericsson.com] Sent: Tuesday, 6 December 2016 07:26 To: Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>>; Vishal Thapar <vishal.tha...@ericsson.com<mailto:vishal.tha...@ericsson.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>> Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>; openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org> Subject: RE: FW: [openflowplugin-dev] Error processing port status message Hi Alon, Sorry for the late response, got pulled into multiple things. The actual issue was reported as a part of bug 6595(https://bugs.opendaylight.org/show_bug.cgi?id=6595) wherein the OpenFlow plugin failed to process a port status message because the message came from a switch before it started the service as MASTER for the switch. Analysing the logs attached to this bug I was not able to come to any conclusion. To analyse the issue further we need the logs by setting the log level as log:set DEBUG org.opendaylight.openflowplugin.impl I assume that we see the issue when the number of switches(DPNs) are high, hence we need to check what is the typical delay between the MASTERSHIP grant and the ports coming up and adjust the delay accordingly. We can also get the relevant information in a wireshark capture that would demarcate when the ROLE_REQ is processed by the switch and when the PORT STATUS is sent. Thanks, shuva From: Kochba, Alon [mailto:alo...@hpe.com] Sent: Monday, December 05, 2016 9:27 PM To: Vishal Thapar; Sam Hague; Dayavanti Gopal Kamath; Shuva Jyoti Kar Cc: Anil Vishnoi; Aizer, Koby Subject: RE: FW: [openflowplugin-dev] Error processing port status message Hi, still waiting for *any* response. Please give some feedback as SR2 is very near and this is possibly blocking. Thanks, --alon From: Vishal Thapar [mailto:vishal.tha...@ericsson.com] Sent: Thursday, 1 December 2016 14:53 To: Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>; Shuva Jyoti Kar <shuva.jyoti....@ericsson.com<mailto:shuva.jyoti....@ericsson.com>> Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>> Subject: RE: FW: [openflowplugin-dev] Error processing port status message Sory Alon, totally forgot about it Adding Shuva. From: Kochba, Alon [mailto:alo...@hpe.com] Sent: 01 December 2016 18:20 To: Vishal Thapar <vishal.tha...@ericsson.com<mailto:vishal.tha...@ericsson.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>> Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>> Subject: RE: FW: [openflowplugin-dev] Error processing port status message Hi, Our QA stepped on this bug again today, could anyone please provide some input? Thanks, --alon From: Vishal Thapar [mailto:vishal.tha...@ericsson.com] Sent: Monday, 28 November 2016 19:26 To: Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>> Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>> Subject: RE: FW: [openflowplugin-dev] Error processing port status message Hi Sam, Tomorrow we’re having an offsite so no one will be able to take a look till Wednesday. If Anil is unable to take a quick look by them, I’ll bring it up with Shuva. Regarsd, Vishal. From: Sam Hague [mailto:sha...@redhat.com] Sent: 28 November 2016 22:53 To: Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>; Vishal Thapar <vishal.tha...@ericsson.com<mailto:vishal.tha...@ericsson.com>> Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>> Subject: Re: FW: [openflowplugin-dev] Error processing port status message Adding Daya and Vishal if they know someone who could take a look at this. On Sun, Nov 27, 2016 at 8:22 AM, Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>> wrote: Hey guys, We encountered a possibly critical bug in openflowplugin a few weeks back. Tried reaching out to openflowplugin-dev multiple times but no answer since the first one - Anil/Sam could you assist? It seems to be easily reproducible and causes a critical (if not blocker) state on the OVS where new ports cannot be created. I raised the bug severity to critical, was hoping to get confirmation from openflowplugin first but maybe this is the way. thanks, --alon From: openflowplugin-dev-boun...@lists.opendaylight.org<mailto:openflowplugin-dev-boun...@lists.opendaylight.org> [mailto:openflowplugin-dev-boun...@lists.opendaylight.org<mailto:openflowplugin-dev-boun...@lists.opendaylight.org>] On Behalf Of Kochba, Alon Sent: Tuesday, 22 November 2016 20:30 To: Aizer, Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>; shuva.jyoti....@ericsson.com<mailto:shuva.jyoti....@ericsson.com> Cc: openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org>; Sokolover, Dotan <dotan.sokolo...@hpe.com<mailto:dotan.sokolo...@hpe.com>> Subject: Re: [openflowplugin-dev] Error processing port status message Bumping this, Shuva or others could we get your inputs please? If confirmed this seems like a critical bug. Thanks! --alon From: openflowplugin-dev-boun...@lists.opendaylight.org<mailto:openflowplugin-dev-boun...@lists.opendaylight.org> [mailto:openflowplugin-dev-boun...@lists.opendaylight.org] On Behalf Of Aizer, Koby Sent: Wednesday, 16 November 2016 18:29 To: shuva.jyoti....@ericsson.com<mailto:shuva.jyoti....@ericsson.com> Cc: openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org>; Sokolover, Dotan <dotan.sokolo...@hpe.com<mailto:dotan.sokolo...@hpe.com>> Subject: [openflowplugin-dev] Error processing port status message Hi Shuva, (Sorry for creating a new thread, but I was somehow missing the original thread [1] in my mail client) We are hitting bug [2] when using netvirt on Boron SR2. This usually happens when creating/deleting a large number of VMs. Dotan has added to the bug description a small script that eventually reproduces it, and he did it while TransactionChainManager’s debug level is increased. We didn’t see any DEBUG messages from that class (there were several ERROR/WARNs though – added the relevant section of the log to the bug). Please let us know if we can do anything else in order to help getting this fixed, as it seem quite critical for us (once this issue occurs it persists and no more port updates are received from the switch). Thanks, Koby [1] https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-October/006155.html [2] https://bugs.opendaylight.org/show_bug.cgi?id=6908
_______________________________________________ openflowplugin-dev mailing list openflowplugin-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
