CC-ing controller-dev anyway, These messages shown in one of the candidates (.101) after OF device owner (.103) is killed. RPC "add-flow" to the candidate (.101) which is not the new owner will fail until we see the message *Leader can perform its duties again* below after few mins.
This behavior is not observed in Boron where RPC works very quickly after OF device owner is killed. Current bug: https://bugs.opendaylight.org/show_bug.cgi?id=8185 BR/Luis > On Apr 19, 2017, at 5:28 PM, Luis Gomez <ece...@gmail.com> wrote: > > I activated DEBUG for openflow and again I do not see anything conclusive, > however I observed following cluster logging in the candidate instance I send > the RPC and fails: > > 2017-04-19 23:45:48,003 | INFO | ult-dispatcher-4 | > kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - > com.typesafe.akka.slf4j - 2.4.17 | Cluster Node > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can > currently not perform its duties, reachability status: > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> -> > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable > [Unreachable] (1), akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> -> > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable > [Unreachable] (4)], member status: > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> Up seen=true, > akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> Up seen=true, > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550> Up seen=false] > > 2017-04-19 23:46:27,990 | INFO | ult-dispatcher-6 | > kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - > com.typesafe.akka.slf4j - 2.4.17 | Cluster Node > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can > currently not perform its duties, reachability status: > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> -> > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable > [Unreachable] (1), akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> -> > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable > [Unreachable] (4)], member status: > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> Up seen=true, > akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> Up seen=true, > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550> Up seen=false] > > 2017-04-19 23:47:28,000 | INFO | ult-dispatcher-3 | > kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - > com.typesafe.akka.slf4j - 2.4.17 | Cluster Node > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can > currently not perform its duties, reachability status: > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> -> > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable > [Unreachable] (1), akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> -> > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable > [Unreachable] (4)], member status: > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> Up seen=true, > akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> Up seen=true, > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550> Up seen=false] > > 2017-04-19 23:48:28,000 | INFO | ult-dispatcher-3 | > kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - > com.typesafe.akka.slf4j - 2.4.17 | Cluster Node > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can > currently not perform its duties, reachability status: > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> -> > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable > [Unreachable] (1), akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> -> > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550: > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550:> Unreachable > [Unreachable] (4)], member status: > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550> Up seen=true, > akka.tcp://opendaylight-cluster-data@192.168.0.102:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.102:2550> Up seen=true, > akka.tcp://opendaylight-cluster-data@192.168.0.103:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.103:2550> Up seen=false] > > 2017-04-19 23:48:31,999 | INFO | lt-dispatcher-17 | > kka://opendaylight-cluster-data <kka://opendaylight-cluster-data>) | 179 - > com.typesafe.akka.slf4j - 2.4.17 | Cluster Node > [akka.tcp://opendaylight-cluster-data@192.168.0.101:2550 > <akka.tcp://opendaylight-cluster-data@192.168.0.101:2550>] - Leader can > perform its duties again > > Also the RPC seems to work after the last line which is at least 3 mins after > the owner goes down. Any idea what is going on or should we involve cluster > people here? > > BR/Luis > > > >> On Apr 18, 2017, at 11:59 PM, Miroslav Macko <miroslav.ma...@pantheon.tech >> <mailto:miroslav.ma...@pantheon.tech>> wrote: >> >> Luis it would be great, if you can try with DEBUG on >> org.opendaylight.openflowplugin.impl. >> >> Thanks a lot, >> Miro >> Od: Luis Gomez <ece...@gmail.com <mailto:ece...@gmail.com>> >> Odoslané: streda, 19. apríla 2017 5:57:35 >> Komu: Jozef Bacigál >> Kópia: openflowplugin-dev; Miroslav Macko; >> integration-...@lists.opendaylight.org >> <mailto:integration-...@lists.opendaylight.org> >> Predmet: Re: [integration-dev] CSIT test bugs >> >> As far as I can tell the issue is not consistent, using local robot >> automation I can reproduce after a while trying. The log of the failing RPC >> instance does not show anything extraordinary, can you provide some DEBUG I >> can try? >> >> BR/Luis >> >>> On Apr 18, 2017, at 9:55 AM, Luis Gomez <ece...@gmail.com >>> <mailto:ece...@gmail.com>> wrote: >>> >>> OK, let me see if I find a way for easy reproduction. >>> >>>> On Apr 18, 2017, at 3:52 AM, Jozef Bacigál <jozef.baci...@pantheon.tech >>>> <mailto:jozef.baci...@pantheon.tech>> wrote: >>>> >>>> Luis to the Cluster issue 2) >>>> >>>> it is failing always, because we were not able to reproduce it. Miroslav >>>> started 3 node cluster with 3 switches connected to all controllers. If >>>> send to any controler any RPC comming witout error. After killing master a >>>> new master is elected and if sended an RPC to any controller is still ok. >>>> RPC comming without error. Is there any other way to do it ? >>>> >>>> Tested on master actual master branch. >>>> >>>> Jozef >>>> Od: Luis Gomez <ece...@gmail.com <mailto:ece...@gmail.com>> >>>> Odoslané: piatok, 14. apríla 2017 21:30:41 >>>> Komu: openflowplugin-dev >>>> Kópia: integration-...@lists.opendaylight.org >>>> <mailto:integration-...@lists.opendaylight.org> >>>> Predmet: [integration-dev] CSIT test bugs >>>> >>>> It is been a while, lets review current CSIT test bugs after latest >>>> patches: >>>> >>>> >>>> Boron and Carbon bugs: >>>> >>>> 1) Switch scalability regression due to missing table miss flows. I think >>>> table miss application does not scale and that impacts scalability test. >>>> >>>> Bug: https://bugs.opendaylight.org/show_bug.cgi?id=8103 >>>> <https://bugs.opendaylight.org/show_bug.cgi?id=8103> >>>> Tracked: >>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/plot/Switch%20Scalability >>>> >>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/plot/Switch%20Scalability> >>>> >>>> >>>> Carbon "cluster" bugs: >>>> >>>> 1) Switch is still seen after owner goes down in non OFHA scenario. >>>> >>>> Bug: https://bugs.opendaylight.org/show_bug.cgi?id=6459 >>>> <https://bugs.opendaylight.org/show_bug.cgi?id=6459> >>>> Tracked: >>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon >>>> >>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon> >>>> >>>> 2) Send RPC to non-owner after original owner is killed is not stable in >>>> Carbon. >>>> >>>> Bug: https://bugs.opendaylight.org/show_bug.cgi?id=8185 >>>> <https://bugs.opendaylight.org/show_bug.cgi?id=8185> >>>> Tracked: >>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-carbon >>>> >>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-carbon> >>>> >>>> 3) Cluster perf regression in Carbon: >>>> >>>> Bug: https://bugs.opendaylight.org/show_bug.cgi?id=6755 >>>> <https://bugs.opendaylight.org/show_bug.cgi?id=6755> >>>> Tracked: >>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon/ >>>> >>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon/> >>>> >>>> BR/Luis >>>> _______________________________________________ >>>> integration-dev mailing list >>>> integration-...@lists.opendaylight.org >>>> <mailto:integration-...@lists.opendaylight.org> >>>> https://lists.opendaylight.org/mailman/listinfo/integration-dev >>>> <https://lists.opendaylight.org/mailman/listinfo/integration-dev>
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
