I added cluster logs to the bug.
> On Apr 20, 2017, at 12:00 AM, Jozef Bacigál <jozef.baci...@pantheon.tech> > wrote: > > Luis can you send me the karaf logs you made with DEBUG ? > > Jozef > Od: Luis Gomez <ece...@gmail.com <mailto:ece...@gmail.com>> > Odoslané: štvrtok, 20. apríla 2017 3:01:56 > Komu: Miroslav Macko; controller-dev > Kópia: Jozef Bacigál; openflowplugin-dev; > integration-...@lists.opendaylight.org > <mailto:integration-...@lists.opendaylight.org> > Predmet: Re: [integration-dev] CSIT test bugs > > CC-ing controller-dev anyway, > > These messages shown in one of the candidates (.101) after OF device owner > (.103) is killed. RPC "add-flow" to the candidate (.101) which is not the new > owner will fail until we see the message *Leader can perform its duties > again* below after few mins. > > This behavior is not observed in Boron where RPC works very quickly after OF > device owner is killed. > > Current bug: https://bugs.opendaylight.org/show_bug.cgi?id=8185 > <https://bugs.opendaylight.org/show_bug.cgi?id=8185> > > BR/Luis > > >> On Apr 19, 2017, at 5:28 PM, Luis Gomez <ece...@gmail.com >> <mailto:ece...@gmail.com>> wrote: >> >> I activated DEBUG for openflow and again I do not see anything conclusive, >> however I observed following cluster logging in the candidate instance I >> send the RPC and fails: >> >> 2017-04-19 23:45:48,003 | INFO | ult-dispatcher-4 | >> kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - >> com.typesafe.akka.slf4j - 2.4.17 | Cluster Node >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can >> currently not perform its duties, reachability status: >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> -> >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable >> [Unreachable] (1), akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> -> >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable >> [Unreachable] (4)], member status: >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> Up seen=true, >> akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> Up seen=true, >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550> Up seen=false] >> >> 2017-04-19 23:46:27,990 | INFO | ult-dispatcher-6 | >> kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - >> com.typesafe.akka.slf4j - 2.4.17 | Cluster Node >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can >> currently not perform its duties, reachability status: >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> -> >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable >> [Unreachable] (1), akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> -> >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable >> [Unreachable] (4)], member status: >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> Up seen=true, >> akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> Up seen=true, >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550> Up seen=false] >> >> 2017-04-19 23:47:28,000 | INFO | ult-dispatcher-3 | >> kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - >> com.typesafe.akka.slf4j - 2.4.17 | Cluster Node >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can >> currently not perform its duties, reachability status: >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> -> >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable >> [Unreachable] (1), akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> -> >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable >> [Unreachable] (4)], member status: >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> Up seen=true, >> akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> Up seen=true, >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550> Up seen=false] >> >> 2017-04-19 23:48:28,000 | INFO | ult-dispatcher-3 | >> kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - >> com.typesafe.akka.slf4j - 2.4.17 | Cluster Node >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can >> currently not perform its duties, reachability status: >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> -> >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable >> [Unreachable] (1), akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> -> >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable >> [Unreachable] (4)], member status: >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> Up seen=true, >> akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> Up seen=true, >> akka.tcp://opendaylight-cluster-data@192.168.0.103:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550> Up seen=false] >> >> 2017-04-19 23:48:31,999 | INFO | lt-dispatcher-17 | >> kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - >> com.typesafe.akka.slf4j - 2.4.17 | Cluster Node >> [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 >> <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can >> perform its duties again >> >> Also the RPC seems to work after the last line which is at least 3 mins >> after the owner goes down. Any idea what is going on or should we involve >> cluster people here? >> >> BR/Luis >> >> >> >>> On Apr 18, 2017, at 11:59 PM, Miroslav Macko <miroslav.ma...@pantheon.tech >>> <mailto:miroslav.ma...@pantheon.tech>> wrote: >>> >>> Luis it would be great, if you can try with DEBUG on >>> org.opendaylight.openflowplugin.impl. >>> >>> Thanks a lot, >>> Miro >>> Od: Luis Gomez <ece...@gmail.com <mailto:ece...@gmail.com>> >>> Odoslané: streda, 19. apríla 2017 5:57:35 >>> Komu: Jozef Bacigál >>> Kópia: openflowplugin-dev; Miroslav Macko; >>> integration-...@lists.opendaylight.org >>> <mailto:integration-...@lists.opendaylight.org> >>> Predmet: Re: [integration-dev] CSIT test bugs >>> >>> As far as I can tell the issue is not consistent, using local robot >>> automation I can reproduce after a while trying. The log of the failing RPC >>> instance does not show anything extraordinary, can you provide some DEBUG I >>> can try? >>> >>> BR/Luis >>> >>>> On Apr 18, 2017, at 9:55 AM, Luis Gomez <ece...@gmail.com >>>> <mailto:ece...@gmail.com>> wrote: >>>> >>>> OK, let me see if I find a way for easy reproduction. >>>> >>>>> On Apr 18, 2017, at 3:52 AM, Jozef Bacigál <jozef.baci...@pantheon.tech >>>>> <mailto:jozef.baci...@pantheon.tech>> wrote: >>>>> >>>>> Luis to the Cluster issue 2) >>>>> >>>>> it is failing always, because we were not able to reproduce it. Miroslav >>>>> started 3 node cluster with 3 switches connected to all controllers. If >>>>> send to any controler any RPC comming witout error. After killing master >>>>> a new master is elected and if sended an RPC to any controller is still >>>>> ok. RPC comming without error. Is there any other way to do it ? >>>>> >>>>> Tested on master actual master branch. >>>>> >>>>> Jozef >>>>> Od: Luis Gomez <ece...@gmail.com <mailto:ece...@gmail.com>> >>>>> Odoslané: piatok, 14. apríla 2017 21:30:41 >>>>> Komu: openflowplugin-dev >>>>> Kópia: integration-...@lists.opendaylight.org >>>>> <mailto:integration-...@lists.opendaylight.org> >>>>> Predmet: [integration-dev] CSIT test bugs >>>>> >>>>> It is been a while, lets review current CSIT test bugs after latest >>>>> patches: >>>>> >>>>> >>>>> Boron and Carbon bugs: >>>>> >>>>> 1) Switch scalability regression due to missing table miss flows. I think >>>>> table miss application does not scale and that impacts scalability test. >>>>> >>>>> Bug: https://bugs.opendaylight.org/show_bug.cgi?id=8103 >>>>> <https://bugs.opendaylight.org/show_bug.cgi?id=8103> >>>>> Tracked: >>>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/plot/Switch%20Scalability >>>>> >>>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/plot/Switch%20Scalability> >>>>> >>>>> >>>>> Carbon "cluster" bugs: >>>>> >>>>> 1) Switch is still seen after owner goes down in non OFHA scenario. >>>>> >>>>> Bug: https://bugs.opendaylight.org/show_bug.cgi?id=6459 >>>>> <https://bugs.opendaylight.org/show_bug.cgi?id=6459> >>>>> Tracked: >>>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon >>>>> >>>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon> >>>>> >>>>> 2) Send RPC to non-owner after original owner is killed is not stable in >>>>> Carbon. >>>>> >>>>> Bug: https://bugs.opendaylight.org/show_bug.cgi?id=8185 >>>>> <https://bugs.opendaylight.org/show_bug.cgi?id=8185> >>>>> Tracked: >>>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-carbon >>>>> >>>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-carbon> >>>>> >>>>> 3) Cluster perf regression in Carbon: >>>>> >>>>> Bug: https://bugs.opendaylight.org/show_bug.cgi?id=6755 >>>>> <https://bugs.opendaylight.org/show_bug.cgi?id=6755> >>>>> Tracked: >>>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon/ >>>>> >>>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon/> >>>>> >>>>> BR/Luis >>>>> _______________________________________________ >>>>> integration-dev mailing list >>>>> integration-...@lists.opendaylight.org >>>>> <mailto:integration-...@lists.opendaylight.org> >>>>> https://lists.opendaylight.org/mailman/listinfo/integration-dev >>>>> <https://lists.opendaylight.org/mailman/listinfo/integration-dev>
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
