On 08/22/2018 03:22 PM, Tom Pantelis wrote:
On Wed, Aug 22, 2018 at 6:17 PM, Tom Pantelis <tompante...@gmail.com
<mailto:tompante...@gmail.com>> wrote:
On Wed, Aug 22, 2018 at 4:57 PM, Jamo Luhrsen <jluhr...@gmail.com
<mailto:jluhr...@gmail.com>> wrote:
I am looking for some help/ideas. I am running three containers
on my laptop so I can test some cluster bugs locally. I am fine
with the netty based akka remoting (our default), but I've been
asked to reproduce a bug with the artery remoting.
Every time I start my controllers with artery, they all die with:
ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-56 |
ActorSystemImpl | 41 -
com.typesafe.akka.slf4j - 2.5.11 | Uncaught error from thread
[opendaylight-cluster-data-akka.remote.default-remote-dispatcher-7]: Di
rect buffer memory, shutting down JVM since
'akka.jvm-exit-on-fatal-error' is enabled for
ActorSystem[opendaylight-cluster-data]
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:695) ~[?:?]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
~[?:?]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
~[?:?]
at
akka.remote.artery.EnvelopeBufferPool.acquire(EnvelopeBufferPool.scala:34)
~[40:com.typesafe.akka.remote:2.5.11]
at akka.remote.artery.Encoder$$anon$2.onPush(Codecs.scala:93)
~[40:com.typesafe.akka.remote:2.5.11]
I can get past this error when I don't try to increase my docker containers
--shm-size, but I get a different problem about akka swallowing an exception
when trying to send a message. But, I think the suggestion below helped there.
I'm stuck in the mud on this one so far. I'm trying to tweak the
shared memory setting in the docker run command (to no avail).
The container stats are nowhere near any kind of limit.
It's not happening with netty based.
I've tried both udp and tcp protocols with the same OOM death.
appreciate any ideas or pointers. I'm literally just throwing mud
against
the wall at this point hoping something will stick.
TomP tells me it works on his laptop, but he's not using
docker/containers,
so that must be a clue.
The problem is this in configuration/factory/akka.conf:
artery {
advanced {
maximum-frame-size = 1 GiB
maximum-large-frame-size = 1 GiB
}
}
I had set these originally a while ago when we were first looking at artery
before I realized it was using direct
memory. I forgot to remove them. I'll submit a patch to do that - in the
mean time you can remove it locally.
I removed this config, which got me past some error about swallowing some
exception
when trying to send a message, so I think that helped.
but, now I have a different, but similar exception. snippet:
2018-08-22T23:00:05,336 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-3 | RepointableActorRef
| 41 - com.typesafe.akka.slf4j - 2
.5.11 | Error in stage [akka.remote.artery.InboundHandshake@1180a5b2]: IllegalStateException : Insufficient usable
storage for new log of length=50335744 in /dev
/shm (shm)
io.aeron.exceptions.RegistrationException: IllegalStateException : Insufficient usable storage for new log of
length=50335744 in /dev/shm (shm)
Any other configs I should look at?
I can try with the --shm-size tweak again I guess, now that maybe the
max-frame-size
1G is removed.
I think it was loading fine on my system fine b/c I have single node and the code that blows up gets executed when it
needs to do remoting.
single node? like, not in a cluster? or 3 java processes all running under
localhost
with different ports?
JamO
Thanks,
JamO
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev
