Hi Faseela, We analyzed the logs and able to see that Table miss entry is missing for table 60. We are not able to see any success/failure future callback for the missing flow. I have added further logs in openflowplugin in patch https://git.opendaylight.org/gerrit/#/c/74717/. Can you please run the CSIT on the same.
Regards, Somashekhar From: Faseela K Sent: Tuesday, July 31, 2018 9:17 PM To: Vishal Thapar <vtha...@redhat.com>; SOMASHEKHAR MANOHARA JAVALAGI <somashekhar.manohara.javal...@ericsson.com> Cc: D Arunprakash <d.arunprak...@ericsson.com>; Anil Vishnoi <vishnoia...@gmail.com>; Gobinath . <gobin...@ericsson.com>; Sam Hague <sha...@redhat.com>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com>; Jamo Luhrsen <jluhr...@gmail.com>; openflowplugin-dev@lists.opendaylight.org Subject: RE: Netvirt Cluster CSIT : Table Miss Entry programming failing randomly Som/Arun, Have updated netvirt CSIT with information below as per Vishal’s inputs, and issue has been reproduced with DEBUG logs as per your suggestion. I have updated the JIRA with link to new logs. Thanks, Faseela From: Vishal Thapar [mailto:vtha...@redhat.com] Sent: Thursday, July 26, 2018 10:50 PM To: Faseela K <faseel...@ericsson.com<mailto:faseel...@ericsson.com>> Cc: D Arunprakash <d.arunprak...@ericsson.com<mailto:d.arunprak...@ericsson.com>>; Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>>; Gobinath . <gobin...@ericsson.com<mailto:gobin...@ericsson.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>; Jamo Luhrsen <jluhr...@gmail.com<mailto:jluhr...@gmail.com>>; openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org> Subject: Re: Netvirt Cluster CSIT : Table Miss Entry programming failing randomly Hi Faseela, Things to check: 1. Which odl is master for this switch and is it master for any other switch? 2. Is the flow present in config DS? 3. What was showsvc status for this ODL node? Regards, Vishal. On Thu, Jul 26, 2018 at 10:40 PM, Faseela K <faseel...@ericsson.com<mailto:faseel...@ericsson.com>> wrote: Taking a look at the logs, i can see the below log line in https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/260/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/18/odl_1/odl1_karaf.log.gz 10 mins before the test suite starts : 2018-07-25T23:00:41,688 | INFO | pool-241-thread-1 | WeightedCentralizedSwitchScheduler | 361 - org.opendaylight.netvirt.natservice-impl - 0.6.3.SNAPSHOT | addSwitch: Adding 167740445697660 dpnId to switchWeightsMap 2018-07-25T23:00:42,274 | DEBUG | epollEventLoopGroup-9-1 | SalFlowServiceImpl | 385 - org.opendaylight.openflowplugin.impl - 0.6.3.SNAPSHOT | Flow add with id=DHCPTableMissFlowForExternalTunnel finished without error But the flow is not present on the switch when the suite starts. We have 3 OVS in the test, but i see only this flow-id log for 2 switches. Anyways the compute in question, is 167740445697660, so let us assume that the flow was getting pushed for the same, but doesn't show up on switch. But there are different failures showing up in the logs including the below one. Let me know if this could impact: 2018-07-25T23:01:09,328 | ERROR | opendaylight-cluster-data-notification-dispatcher-97 | DefaultConfigPusher | 378 - org.opendaylight.openflowplugin.applications.of-switch-config-pusher - 0.6.3.SNAPSHOT | Future (eventually) failed: addFlow org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No implementation of RPC AbsoluteSchemaPath{path=[(urn:opendaylight:module:config?revision=2014-10-15)set-config]} available at org.opendaylight.controller.md.sal.dom.broker.impl.RoutedDOMRpcRoutingTableEntry.invokeRpc(RoutedDOMRpcRoutingTableEntry.java:85) [229:org.opendaylight.controller.sal-broker-impl:1.7.3.SNAPSHOT] at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:178) [229:org.opendaylight.controller.sal-broker-impl:1.7.3.SNAPSHOT] at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102) [229:org.opendaylight.controller.sal-broker-impl:1.7.3.SNAPSHOT] at Proxybb8ecbf1_eb9d_4480_a18e_4a2007f00a3c.invokeRpc(Unknown Source) [?:?] at Proxy40f45215_a726_4be9_ac6b_649862163662.invokeRpc(Unknown Source) [?:?] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:68) [226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:46) [226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:165) [226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:99) [226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at com.sun.proxy.$Proxy113.setConfig(Unknown Source) [388:org.opendaylight.openflowplugin.model.flow-service:0.6.3.SNAPSHOT] at org.opendaylight.openflowplugin.openflow.ofswitch.config.DefaultConfigPusher.onDataTreeChanged(DefaultConfigPusher.java:84) [378:org.opendaylight.openflowplugin.applications.of-switch-config-pusher:0.6.3.SNAPSHOT] at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41) [226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:67) [239:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:41) [239:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:38) [232:org.opendaylight.controller.sal-clustering-commons:1.7.3.SNAPSHOT] at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive(Actor.scala:517) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.ActorCell.invoke(ActorCell.scala:559) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [42:com.typesafe.akka.actor:2.5.11] From: Faseela K Sent: Thursday, July 26, 2018 7:50 AM To: D Arunprakash <d.arunprak...@ericsson.com<mailto:d.arunprak...@ericsson.com>>; 'Anil Vishnoi' <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>> Cc: Gobinath . <gobin...@ericsson.com<mailto:gobin...@ericsson.com>>; 'Sam Hague' <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>; 'Jamo Luhrsen' <jluhr...@gmail.com<mailto:jluhr...@gmail.com>>; 'Vishal Thapar' <vtha...@redhat.com<mailto:vtha...@redhat.com>>; 'openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org>' <openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org>> Subject: RE: Netvirt Cluster CSIT : Table Miss Entry programming failing randomly Arun, Here are the logs with DEBUG enabled. https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/260/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/18/ This time, table 18 entry is missing. Thanks, Faseela From: Faseela K Sent: Wednesday, July 25, 2018 4:34 PM To: D Arunprakash <d.arunprak...@ericsson.com<mailto:d.arunprak...@ericsson.com>>; Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>> Cc: Gobinath . <gobin...@ericsson.com<mailto:gobin...@ericsson.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>; Jamo Luhrsen <jluhr...@gmail.com<mailto:jluhr...@gmail.com>>; Vishal Thapar <vtha...@redhat.com<mailto:vtha...@redhat.com>>; openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org> Subject: RE: Netvirt Cluster CSIT : Table Miss Entry programming failing randomly Arun, I have enabled the requested TRACE logs and running the CSIT now. Ever since I enabled the TRACEs it has not failed ☺. Let us see if the issue gets reproduced in another 24 hours. Thanks, Faseela From: D Arunprakash Sent: Wednesday, July 25, 2018 7:53 AM To: Faseela K <faseel...@ericsson.com<mailto:faseel...@ericsson.com>>; Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>> Cc: Gobinath . <gobin...@ericsson.com<mailto:gobin...@ericsson.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>; Jamo Luhrsen <jluhr...@gmail.com<mailto:jluhr...@gmail.com>>; Vishal Thapar <vtha...@redhat.com<mailto:vtha...@redhat.com>>; openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org> Subject: RE: Netvirt Cluster CSIT : Table Miss Entry programming failing randomly Faseela, Could you please enable the following package logging and provide the logs? log:set DEBUG org.opendaylight.openflowplugin.impl Regards, Arun From: Faseela K Sent: Tuesday, July 24, 2018 11:44 PM To: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>> Cc: D Arunprakash <d.arunprak...@ericsson.com<mailto:d.arunprak...@ericsson.com>>; Gobinath . <gobin...@ericsson.com<mailto:gobin...@ericsson.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>; Jamo Luhrsen <jluhr...@gmail.com<mailto:jluhr...@gmail.com>>; Vishal Thapar <vtha...@redhat.com<mailto:vtha...@redhat.com>>; openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org> Subject: RE: Netvirt Cluster CSIT : Table Miss Entry programming failing randomly Hi Arun, We are still hitting this issue, and this can be seen on stable/oxygen as well. In some cases the flow will be programmed initially, but after some time it disappears. I do see a lot of “Switch Idle state occurred, node=/10.30.170.51:60858|auxId=0” messages in the logs. Is that anything to worry about? If there is any specific TRACE logs to be enabled while running the CSIT, could you please let me know? I can run the CSIT with the same and supply the logs. Thanks, Faseela From: Faseela K Sent: Tuesday, July 17, 2018 1:24 PM To: Anil Vishnoi <vishnoia...@gmail.com<mailto:vishnoia...@gmail.com>> Cc: D Arunprakash <d.arunprak...@ericsson.com<mailto:d.arunprak...@ericsson.com>>; Gobinath . <gobin...@ericsson.com<mailto:gobin...@ericsson.com>>; Sam Hague <sha...@redhat.com<mailto:sha...@redhat.com>>; Dayavanti Gopal Kamath <dayavanti.gopal.kam...@ericsson.com<mailto:dayavanti.gopal.kam...@ericsson.com>>; Jamo Luhrsen <jluhr...@gmail.com<mailto:jluhr...@gmail.com>>; Vishal Thapar <vtha...@redhat.com<mailto:vtha...@redhat.com>>; openflowplugin-dev@lists.opendaylight.org<mailto:openflowplugin-dev@lists.opendaylight.org> Subject: Netvirt Cluster CSIT : Table Miss Entry programming failing randomly Anil, I have raised the below JIRA, where I see Table Miss flow for Table 43 is sometimes not getting programmed on one of the computes in netvirt 3 node CSIT. https://jira.opendaylight.org/browse/OPNFLWPLUG-1028 I do see flow being pushed to config/inventory by netvirt, but it is missing on switch. This makes the basic DHCP itself fail for netvirt as this is one of the important flows needed for all broadcasts to work. Any pointers? Thanks, Faseela
_______________________________________________ openflowplugin-dev mailing list openflowplugin-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
