Re: [openflowplugin-dev] FW: Error processing port status message

[Aizer, Koby]

Hi Shuva,

In addition to the recreation efforts with higher debug level, please note that 
this issue occurred long after the DPN was connected (hard to see that in the 
log as it is trimmed).
Looking in the log Dotan has attached to the ticket, between 2016-11-08 
08:28:32 to 2016-11-08 09:22:18, you can see ~250 successful 
InterfaceStateChangeListeners events (which I believe are triggered from the 
port status messages). At this point something breaks, and you start seeing the 
“Error processing port status” WARNs.

I also just noticed that these WARNs have started occurring after the following 
messages, could they give us some lead?
2016-11-08 09:22:19,682 | WARN  | CommitFutures-4  | TransactionChainManager    
      | 291 - org.opendaylight.openflowplugin.impl - 0.3.1.SNAPSHOT | txChain 
failed -> recreating due to {}
2016-11-08 09:22:19,684 | ERROR | CommitFutures-4  | ExecutionList              
      | 38 - com.google.guava - 18.0.0 | RuntimeException while executing 
runnable com.google.common.util.concurrent.Futures$6@680785f9 with executor 
INSTANCE
2016-11-08 09:22:19,698 | ERROR | CommitFutures-4  | TransactionChainManager    
      | 291 - org.opendaylight.openflowplugin.impl - 0.3.1.SNAPSHOT | 
Transaction commit failed.

Thanks,
Koby

From: Kochba, Alon
Sent: Tuesday, December 6, 2016 1:49 PM
To: Shuva Jyoti Kar <shuva.jyoti....@ericsson.com>; Vishal Thapar 
<vishal.tha...@ericsson.com>; Sam Hague <sha...@redhat.com>; Dayavanti Gopal 
Kamath <dayavanti.gopal.kam...@ericsson.com>
Cc: Anil Vishnoi <vishnoia...@gmail.com>; Aizer, Koby <koby.ai...@hpe.com>; 
openflowplugin-dev@lists.opendaylight.org
Subject: RE: FW: [openflowplugin-dev] Error processing port status message

Thanks Shuva,

It happened to Dotan with only one compute.
Can you try the script he provided to recreate?

I asked him to try to recreate it with the debug logs and tcpdump in any case.

--alon

From: Shuva Jyoti Kar [mailto:shuva.jyoti....@ericsson.com]
Sent: Tuesday, 6 December 2016 07:26
To: Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>>; Vishal Thapar 
<vishal.tha...@ericsson.com<mailto:vishal.tha...@ericsson.com>>; Sam Hague 
<sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath 
<dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>
Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, 
Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>; 
openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org>
Subject: RE: FW: [openflowplugin-dev] Error processing port status message

Hi Alon,

Sorry for the late response, got pulled into multiple things.
The actual issue was reported as a part of bug 
6595(https://bugs.opendaylight.org/show_bug.cgi?id=6595) wherein the OpenFlow 
plugin failed to process a port status message because the message came from a 
switch before it started the service as MASTER for the switch.

Analysing the logs attached to this bug I was not able to come to any 
conclusion. To analyse the issue further we need the logs by setting the log 
level as
log:set DEBUG org.opendaylight.openflowplugin.impl

I assume that we  see the issue when the number of switches(DPNs) are high, 
hence we need to check what is the typical delay between the MASTERSHIP grant 
and the ports coming up and adjust the delay accordingly.

We can also get the relevant information in a wireshark capture that would 
demarcate when the ROLE_REQ is processed by the switch and when the PORT STATUS 
is sent.

Thanks,
shuva


From: Kochba, Alon [mailto:alo...@hpe.com]
Sent: Monday, December 05, 2016 9:27 PM
To: Vishal Thapar; Sam Hague; Dayavanti Gopal Kamath; Shuva Jyoti Kar
Cc: Anil Vishnoi; Aizer, Koby
Subject: RE: FW: [openflowplugin-dev] Error processing port status message

Hi, still waiting for *any* response.

Please give some feedback as SR2 is very near and this is possibly blocking.

Thanks,
--alon

From: Vishal Thapar [mailto:vishal.tha...@ericsson.com]
Sent: Thursday, 1 December 2016 14:53
To: Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>>; Sam Hague 
<sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath 
<dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>;
 Shuva Jyoti Kar 
<shuva.jyoti....@ericsson.com<mailto:shuva.jyoti....@ericsson.com>>
Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, 
Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>
Subject: RE: FW: [openflowplugin-dev] Error processing port status message

Sory Alon, totally forgot about it

Adding Shuva.

From: Kochba, Alon [mailto:alo...@hpe.com]
Sent: 01 December 2016 18:20
To: Vishal Thapar 
<vishal.tha...@ericsson.com<mailto:vishal.tha...@ericsson.com>>; Sam Hague 
<sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath 
<dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>
Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, 
Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>
Subject: RE: FW: [openflowplugin-dev] Error processing port status message

Hi,

Our QA stepped on this bug again today, could anyone please provide some input?

Thanks,
--alon

From: Vishal Thapar [mailto:vishal.tha...@ericsson.com]
Sent: Monday, 28 November 2016 19:26
To: Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Kochba, Alon 
<alo...@hpe.com<mailto:alo...@hpe.com>>; Dayavanti Gopal Kamath 
<dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>
Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, 
Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>
Subject: RE: FW: [openflowplugin-dev] Error processing port status message

Hi Sam,

Tomorrow we’re having an offsite so no one will be able to take a look till 
Wednesday. If Anil is unable to take a quick look by them, I’ll bring it up 
with Shuva.

Regarsd,
Vishal.

From: Sam Hague [mailto:sha...@redhat.com]
Sent: 28 November 2016 22:53
To: Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>>; Dayavanti Gopal 
Kamath 
<dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>;
 Vishal Thapar <vishal.tha...@ericsson.com<mailto:vishal.tha...@ericsson.com>>
Cc: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Aizer, 
Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>
Subject: Re: FW: [openflowplugin-dev] Error processing port status message

Adding Daya and Vishal if they know someone who could take a look at this.

On Sun, Nov 27, 2016 at 8:22 AM, Kochba, Alon 
<alo...@hpe.com<mailto:alo...@hpe.com>> wrote:
Hey guys,

We encountered a possibly critical bug in openflowplugin a few weeks back.
Tried reaching out to openflowplugin-dev multiple times but no answer since the 
first one - Anil/Sam could you assist?
It seems to be easily reproducible and causes a critical (if not blocker) state 
on the OVS where new ports cannot be created.

I raised the bug severity to critical, was hoping to get confirmation from 
openflowplugin first but maybe this is the way.

thanks,
--alon

From: 
openflowplugin-dev-boun...@lists.opendaylight.org<mailto:openflowplugin-dev-boun...@lists.opendaylight.org>
 
[mailto:openflowplugin-dev-boun...@lists.opendaylight.org<mailto:openflowplugin-dev-boun...@lists.opendaylight.org>]
 On Behalf Of Kochba, Alon
Sent: Tuesday, 22 November 2016 20:30
To: Aizer, Koby <koby.ai...@hpe.com<mailto:koby.ai...@hpe.com>>; 
shuva.jyoti....@ericsson.com<mailto:shuva.jyoti....@ericsson.com>
Cc: 
openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org>;
 Sokolover, Dotan <dotan.sokolo...@hpe.com<mailto:dotan.sokolo...@hpe.com>>
Subject: Re: [openflowplugin-dev] Error processing port status message

Bumping this, Shuva or others could we get your inputs please?
If confirmed this seems like a critical bug.

Thanks!
--alon

From: 
openflowplugin-dev-boun...@lists.opendaylight.org<mailto:openflowplugin-dev-boun...@lists.opendaylight.org>
 [mailto:openflowplugin-dev-boun...@lists.opendaylight.org] On Behalf Of Aizer, 
Koby
Sent: Wednesday, 16 November 2016 18:29
To: shuva.jyoti....@ericsson.com<mailto:shuva.jyoti....@ericsson.com>
Cc: 
openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org>;
 Sokolover, Dotan <dotan.sokolo...@hpe.com<mailto:dotan.sokolo...@hpe.com>>
Subject: [openflowplugin-dev] Error processing port status message

Hi Shuva,

(Sorry for creating a new thread, but I was somehow missing the original thread 
[1] in my mail client)

We are hitting bug [2] when using netvirt on Boron SR2. This usually happens 
when creating/deleting a large number of VMs.

Dotan has added to the bug description a small script that eventually 
reproduces it, and he did it while TransactionChainManager’s debug level is 
increased.
We didn’t see any DEBUG messages from that class (there were several 
ERROR/WARNs though – added the relevant section of the log to the bug).

Please let us know if we can do anything else in order to help getting this 
fixed, as it seem quite critical for us (once this issue occurs it persists and 
no more port updates are received from the switch).

Thanks,
Koby

[1] 
https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-October/006155.html
[2] https://bugs.opendaylight.org/show_bug.cgi?id=6908



_______________________________________________
openflowplugin-dev mailing list
openflowplugin-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev