On Fri, Feb 16, 2018 at 2:35 PM, Jamo Luhrsen <jluhr...@gmail.com> wrote:
> > > On 2/16/18 11:33 AM, Tom Pantelis wrote: > > > > > > On Fri, Feb 16, 2018 at 2:26 PM, Jamo Luhrsen <jluhr...@gmail.com > <mailto:jluhr...@gmail.com>> wrote: > > > > I'm analyzing CSIT failures for our Carbon SR3 candidate. > > > > Something nasty went wrong in a netvirt CSIT job in the middle of > > the robot tests. Seems like all functionality is probably broken > > after that. > > > > in the karaf.log [0] I see a message about some akka circuit breaker > > Timed out, then a bunch of RuntimeExceptions: Transaction > > aborted due to shutdown. > > > > > > yeah that means akka persistence failed, ie it timed out waiting for > data to be written to the disk. That kills the > > shard actor with no recovery. This can happen if there's slow disk > access/contention in the env - seen this happen > > before with internal CSIT env before the disk issue was resolved. > > Thanks. I'll report to the infra guys that we are still likely seeing > some high disk IO latency. There was another job with similar issues. > The timeout(s) can be increased in the akka.conf (would have to look it up) if it's really problematic although that's really just a band-aid. > > JamO > > > Any ideas what's happening here? > > > > Thanks, > > JamO > > > > [0]https://logs.opendaylight.org/releng/vex-yul-odl- > jenkins-1/netvirt-csit-1node-openstack-pike-upstream- > stateful-snat-conntrack-carbon/200/odl_1/odl1_karaf.log.gz > > <https://logs.opendaylight.org/releng/vex-yul-odl- > jenkins-1/netvirt-csit-1node-openstack-pike-upstream- > stateful-snat-conntrack-carbon/200/odl_1/odl1_karaf.log.gz> > > _______________________________________________ > > controller-dev mailing list > > controller-dev@lists.opendaylight.org <mailto:controller-dev@lists. > opendaylight.org> > > https://lists.opendaylight.org/mailman/listinfo/controller-dev > > <https://lists.opendaylight.org/mailman/listinfo/controller-dev> > > > > >
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev