On Fri, Feb 16, 2018 at 2:35 PM, Jamo Luhrsen <jluhr...@gmail.com> wrote:

>
>
> On 2/16/18 11:33 AM, Tom Pantelis wrote:
> >
> >
> > On Fri, Feb 16, 2018 at 2:26 PM, Jamo Luhrsen <jluhr...@gmail.com
> <mailto:jluhr...@gmail.com>> wrote:
> >
> >     I'm analyzing CSIT failures for our Carbon SR3 candidate.
> >
> >     Something nasty went wrong in a netvirt CSIT job in the middle of
> >     the robot tests. Seems like all functionality is probably broken
> >     after that.
> >
> >     in the karaf.log [0] I see a message about some akka circuit breaker
> >     Timed out, then a bunch of RuntimeExceptions: Transaction
> >     aborted due to shutdown.
> >
> >
> > yeah that means akka persistence failed, ie it timed out waiting for
> data to be written to the disk. That kills the
> > shard actor with no recovery.  This can happen if there's slow disk
> access/contention in the env - seen this happen
> > before with internal CSIT env before the disk issue was resolved.
>
> Thanks. I'll report to the infra guys that we are still likely seeing
> some high disk IO latency. There was another job with similar issues.
>

The timeout(s) can be increased in the akka.conf (would have to look it up)
if it's really problematic although that's really just a band-aid.


>
> JamO
>
> >     Any ideas what's happening here?
> >
> >     Thanks,
> >     JamO
> >
> >     [0]https://logs.opendaylight.org/releng/vex-yul-odl-
> jenkins-1/netvirt-csit-1node-openstack-pike-upstream-
> stateful-snat-conntrack-carbon/200/odl_1/odl1_karaf.log.gz
> >     <https://logs.opendaylight.org/releng/vex-yul-odl-
> jenkins-1/netvirt-csit-1node-openstack-pike-upstream-
> stateful-snat-conntrack-carbon/200/odl_1/odl1_karaf.log.gz>
> >     _______________________________________________
> >     controller-dev mailing list
> >     controller-dev@lists.opendaylight.org <mailto:controller-dev@lists.
> opendaylight.org>
> >     https://lists.opendaylight.org/mailman/listinfo/controller-dev
> >     <https://lists.opendaylight.org/mailman/listinfo/controller-dev>
> >
> >
>
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev

Reply via email to