On 2/16/18 11:38 AM, Tom Pantelis wrote: > > > On Fri, Feb 16, 2018 at 2:35 PM, Jamo Luhrsen <jluhr...@gmail.com > <mailto:jluhr...@gmail.com>> wrote: > > > > On 2/16/18 11:33 AM, Tom Pantelis wrote: > > > > > > On Fri, Feb 16, 2018 at 2:26 PM, Jamo Luhrsen <jluhr...@gmail.com > <mailto:jluhr...@gmail.com> <mailto:jluhr...@gmail.com > <mailto:jluhr...@gmail.com>>> wrote: > > > > I'm analyzing CSIT failures for our Carbon SR3 candidate. > > > > Something nasty went wrong in a netvirt CSIT job in the middle of > > the robot tests. Seems like all functionality is probably broken > > after that. > > > > in the karaf.log [0] I see a message about some akka circuit breaker > > Timed out, then a bunch of RuntimeExceptions: Transaction > > aborted due to shutdown. > > > > > > yeah that means akka persistence failed, ie it timed out waiting for > data to be written to the disk. That kills the > > shard actor with no recovery. This can happen if there's slow disk > access/contention in the env - seen this happen > > before with internal CSIT env before the disk issue was resolved. > > Thanks. I'll report to the infra guys that we are still likely seeing > some high disk IO latency. There was another job with similar issues. > > > The timeout(s) can be increased in the akka.conf (would have to look it up) > if it's really problematic although that's > really just a band-aid.
yeah, hopefully the disk IO stuff can be ironed out as the real fix. I'm sure it's affecting us in other ways we just don't know about yet. JamO > > JamO > > > Any ideas what's happening here? > > > > Thanks, > > JamO > > > > > [0]https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-carbon/200/odl_1/odl1_karaf.log.gz > > <https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-carbon/200/odl_1/odl1_karaf.log.gz> > > > <https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-carbon/200/odl_1/odl1_karaf.log.gz > > <https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-carbon/200/odl_1/odl1_karaf.log.gz>> > > _______________________________________________ > > controller-dev mailing list > > controller-dev@lists.opendaylight.org > <mailto:controller-dev@lists.opendaylight.org> > <mailto:controller-dev@lists.opendaylight.org > <mailto:controller-dev@lists.opendaylight.org>> > > https://lists.opendaylight.org/mailman/listinfo/controller-dev > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> > > <https://lists.opendaylight.org/mailman/listinfo/controller-dev > <https://lists.opendaylight.org/mailman/listinfo/controller-dev>> > > > > > > _______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
