Issue is tracked under : https://jira.opendaylight.org/browse/OPNFLWPLUG-1069
Thanks, Faseela -----Original Message----- From: Faseela K Sent: Thursday, April 4, 2019 3:59 PM To: Jaya Priyadarshini <[email protected]>; SOMASHEKHAR MANOHARA JAVALAGI <[email protected]>; Jamo Luhrsen <[email protected]>; [email protected]; [email protected]; D Arunprakash <[email protected]> Cc: openflowplugin-dev <[email protected]> Subject: RE: [integration-dev] [openflowplugin-dev] Netvirt Apex jobs in jenkins - openflow stuck in starting state Thanks Jaya! @Som : Could you please check if respective logs from ofp are there? Looks like it is not. I think something is broken after the below patch got merged: https://git.opendaylight.org/gerrit/#/c/79793/ Thanks, Faseela -----Original Message----- From: Jaya Priyadarshini Sent: Thursday, April 4, 2019 2:16 PM To: Faseela K <[email protected]>; SOMASHEKHAR MANOHARA JAVALAGI <[email protected]>; Jamo Luhrsen <[email protected]>; [email protected]; [email protected]; D Arunprakash <[email protected]> Cc: openflowplugin-dev <[email protected]> Subject: RE: [integration-dev] [openflowplugin-dev] Netvirt Apex jobs in jenkins - openflow stuck in starting state Hi Faseela, Here are the logs: https://jenkins.opendaylight.org/sandbox/job/Jaya-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/2/ Regards Jaya -----Original Message----- From: Faseela K Sent: Thursday, April 4, 2019 10:19 AM To: SOMASHEKHAR MANOHARA JAVALAGI <[email protected]>; Jaya Priyadarshini <[email protected]>; Jamo Luhrsen <[email protected]>; [email protected]; [email protected]; D Arunprakash <[email protected]> Cc: openflowplugin-dev <[email protected]> Subject: RE: [integration-dev] [openflowplugin-dev] Netvirt Apex jobs in jenkins - openflow stuck in starting state Jaya, Could you please supply logs for Sodium apex jobs with TRACE enabled for org.opendaylight.openflowjava.protocol.impl.core.OpenflowDiagStatusProviderImpl ? Thanks, Faseela -----Original Message----- From: SOMASHEKHAR MANOHARA JAVALAGI Sent: Tuesday, April 2, 2019 9:05 PM To: Jaya Priyadarshini <[email protected]>; Jamo Luhrsen <[email protected]>; Faseela K <[email protected]>; [email protected]; [email protected]; D Arunprakash <[email protected]> Cc: openflowplugin-dev <[email protected]> Subject: RE: [integration-dev] [openflowplugin-dev] Netvirt Apex jobs in jenkins - openflow stuck in starting state Hi Jaya, We are sending status report of openflow service being operational to infrautils diagstatus service, as and when we receive systemBootReady notification. This happened in 2 mins from the start of controller. We also report the status to be ERROR in case of some issues. As I am not able to see any such logs, issue has to be checked at infrautils diagstatus service, whether it received status report properly. Regards, Somashekhar -----Original Message----- From: Jaya Priyadarshini Sent: Tuesday, April 2, 2019 4:46 PM To: Jamo Luhrsen <[email protected]>; SOMASHEKHAR MANOHARA JAVALAGI <[email protected]>; Faseela K <[email protected]>; [email protected]; [email protected]; D Arunprakash <[email protected]> Cc: openflowplugin-dev <[email protected]> Subject: RE: [integration-dev] [openflowplugin-dev] Netvirt Apex jobs in jenkins - openflow stuck in starting state Hi Som, As per info which u asked, https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/71/console-timestamp.log.gz ODL is started at 07:22:10 2019-03-31 07:22:10,517 INFO: Starting Opendaylight on node overcloud-controller-0.opnfvlf.org And it is checking if all bundles came up at around this time. 08:32:41 ============================================================================== 08:32:41 01_l2 :: Test suite to verify packet flows between vm instances. 08:32:41 ============================================================================== Regards Jaya -----Original Message----- From: Jamo Luhrsen <[email protected]> Sent: Monday, April 1, 2019 9:34 PM To: SOMASHEKHAR MANOHARA JAVALAGI <[email protected]>; Faseela K <[email protected]>; Jaya Priyadarshini <[email protected]>; [email protected]; [email protected]; D Arunprakash <[email protected]> Cc: openflowplugin-dev <[email protected]> Subject: Re: [integration-dev] [openflowplugin-dev] Netvirt Apex jobs in jenkins - openflow stuck in starting state something is broken. Looks like the failures are only there in sodium. here is a karaf log from job #71 failure: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/71/controller_sos/sosreport-controllerreport-20190331083711/opt/opendaylight/data/log/karaf.log.gz you can see some WARN messages in TracingBroker which seem to just repeat. the controller is up for at least 10m before the actual failure is finally noted. In a neon job I just checked, the same validation is completed in 3s. JamO On 4/1/19 4:44 AM, SOMASHEKHAR MANOHARA JAVALAGI wrote: > Hi Faseela, > > Karaf is started at Mar 31, 2019 7:23:59 AM. > > Once all the bundles are up, we are registering to SystemReadyMonitor > by 2019-03-31T07:25:31,172 > > After 16 seconds we are receiving onSystemBootready notification at > 2019-03-31T07:25:47,219 > > Immediately within a second we are marking openflow service status as > operational at 2019-03-31T07:25:48,709 > > As bundles coming up is taking more time, we are finding delay in operational > status change of openflow service. > > Regards, > > Somashekhar > > *From:* [email protected] > <[email protected]> *On Behalf Of > *Faseela K > *Sent:* Monday, April 1, 2019 12:43 PM > *To:* Jaya Priyadarshini <[email protected]>; > [email protected]; > [email protected]; D Arunprakash > <[email protected]> > *Cc:* openflowplugin-dev <[email protected]> > *Subject:* Re: [openflowplugin-dev] Netvirt Apex jobs in jenkins - > openflow stuck in starting state > > Arun, > > Any idea when this can happen? > > System Ready State says all bundles have come up, but OFP is still in > starting state. > > Thanks, > > Faseela > > *From:* [email protected] > <mailto:[email protected]> > <[email protected] > <mailto:[email protected]>> *On Behalf Of > *Jaya Priyadarshini > *Sent:* Monday, April 1, 2019 12:41 PM > *To:* [email protected] > <mailto:[email protected]>; > [email protected] > <mailto:[email protected]> > *Subject:* [netvirt-dev] Apex jobs in jenkins > > Here is the apex jobs status. > > Success > > > > 100% > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-fluorin > e/lastBuild> > > > > netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack- > fluorine > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-fluorin > e/> > > > > 8 hr 19 min - #142 > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-fluorin > e/lastSuccessfulBuild/> > > > > 18 days - #123 > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-fluorin > e/lastFailedBuild/> > > > > 1 hr 0 min > > > > 25 / 25 passed > https://jenkins.opendaylight.org/releng/plugin/robot/robot.png > > > > Success > > > > 100% > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-neon/la > stBuild> > > > > netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack- > neon > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-neon/> > > > > 15 hr - #158 > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-neon/la > stSuccessfulBuild/> > > > > 11 days - #148 > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-neon/la > stFailedBuild/> > > > > 52 min > > > > 25 / 25 passed > https://jenkins.opendaylight.org/releng/plugin/robot/robot.png > > > > In progress > > > > 0% > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/ > lastBuild> > > > > netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack- > sodium > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/ > > > > > > 23 hr - #71 > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/ > lastSuccessfulBuild/> > > > > 16 days - #40 > <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt > -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/ > lastFailedBuild/> > > > > 1 hr 36 min > > > > 0 / 25 passed > https://jenkins.opendaylight.org/releng/plugin/robot/robot.png > > > > 3^rd failure is due to following reason > > > > [?1l>[?2004lTimestamp: Sun Mar 31 08:33:15 UTC 2019 Node IP Address: > /127.0.0.1 System is operational: false System ready state: ACTIVE > > OPENFLOW : STARTING (INITIALIZING) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<It never > comes to operational. > > IFM : OPERATIONAL > > ITM : OPERATIONAL > > ELAN : OPERATIONAL (Service started) > > OVSDB : OPERATIONAL (OVSDB initialization complete) > > DATASTORE : OPERATIONAL [?1h=[90m~[0m > > So Suite setup itself has failed and hence all failures. > > Now how do we go ahead with apex, what should be next step? > > Regards > > Jaya > > > _______________________________________________ > integration-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/integration-dev > _______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
