Hello,
we are observing an issue with akka-http client side since a few days. With
additional logging we tracked it down to the part where akka-http sends out
the request, i.e. the client starts the request but we never see it hitting
the target server.
The client pool is set up using the following way:
private val pool = scheme match {
case HttpScheme => Http(system).cachedHostConnectionPool[HttpRequest](host,
port, settings = settings.getOrElse(ConnectionPoolSettings(system)))
case HttpsScheme =>
Http(system).cachedHostConnectionPoolHttps[HttpRequest](host, port, settings =
settings.getOrElse(ConnectionPoolSettings(system)))
}
Actual call goes out like this:
// req: HttpRequest
val source = Source.single(req -> req)
val poolFlow = source.via(pool)
poolFlow .runWith(Sink.head).flatMap {
case (Success(response), _) => Future.successful(response)
case (Failure(ex), _) => Future.failed(ex)
}
With log statements we tracked the code until the "runWith" call. After
this there is no exception, no timeout or any other error in the log files.
The target server is never hit with the request.
No changes in configuration or setup that we can attribute to this, before
the issue appeared the service hasn't been updated for around 5 days and
was running in general without this issue since 2 months.
Scala 2.11, akka 2.4.7, JDK 1.8.0_121, running in a docker container on top
of Kubernetes.
The target server is SSL secured, so it's an HTTPS connection.
Other connections to SSL secured endpoints (different servers) running from
the same application are working even when this issue occurs, however, the
difference is that the connection pools of the working onces are hit
several times per minute, this endpoint maybe idle for hours and then
re-used.
After a restart the issue is fixed and it's working for a up to 12h, then
it's again in this state. Locally it wasn't reproducible.
What we tried so far:
- adding lot's of logging to get identify the line with the runWith
- switched to pseudo random number generator
(-Djava.security.egd=file:/dev/./urandom), had this issue a few times with
other secure connections on the database level that resulting in blocking
calls
- checked stack trace dumps, nothing except expected threads in waiting
and default akka and mongoDB pool threads, nothing that is different from
stack trace dump on a working system versus a non working system (dumps are
attached if helpful), this also showed us that our random number generator
issue was not the source
- right now we added a health check that constantly makes use of the
connection pool, to see if it has something to do with the connection not
being used
Is this an issue anyone has come across or has an idea how to further debug
the source?
Thanks,
Elmar
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode):
"default-akka.actor.default-dispatcher-13" #25 prio=5 os_prio=0
tid=0x00007f1448001000 nid=0x51 waiting on condition [0x00007f145b50e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-14" #26 prio=5 os_prio=0
tid=0x00007f1460026000 nid=0x50 waiting on condition [0x00007f145b60f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-12" #24 prio=5 os_prio=0
tid=0x00007f1454042000 nid=0x4f waiting on condition [0x00007f145b710000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-11" #23 prio=5 os_prio=0
tid=0x00007f147016c000 nid=0x4e waiting on condition [0x00007f145b811000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-10" #22 prio=5 os_prio=0
tid=0x00007f1478039000 nid=0x4d waiting on condition [0x00007f145bb89000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-9" #21 prio=5 os_prio=0
tid=0x00007f1460025000 nid=0x4c waiting on condition [0x00007f145bc8a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at
scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"DestroyJavaVM" #20 prio=5 os_prio=0 tid=0x00007f14d400d000 nid=0x34 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"default-akka.io.pinned-dispatcher-8" #19 prio=5 os_prio=0
tid=0x00007f146c03d800 nid=0x4b runnable [0x00007f145bd8b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000628b28260> (a sun.nio.ch.Util$3)
- locked <0x0000000628b28270> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000628a13068> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.tryRun(SelectionHandler.scala:115)
at
akka.io.SelectionHandler$ChannelRegistryImpl$Task.run(SelectionHandler.scala:219)
at
akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.run(SelePdiffctionHandler.scala:148)
at
akka.util.SerializedSuspendableExecutionContext.run$1(SerializedSuspendableExecutionContext.scala:67)
at
akka.util.SerializedSuspendableExecutionContext.run(SerializedSuspendableExecutionContext.scala:71)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"default-akka.actor.default-dispatcher-7" #18 prio=5 os_prio=0
tid=0x00007f146c00c000 nid=0x4a waiting on condition [0x00007f14a0117000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-6" #17 prio=5 os_prio=0
tid=0x00007f1478027800 nid=0x49 waiting on condition [0x00007f14a0218000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-5" #16 prio=5 os_prio=0
tid=0x00007f146c00b000 nid=0x48 waiting on condition [0x00007f14b4195000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"MongoCleaner1917082298" #15 daemon prio=5 os_prio=0 tid=0x00007f14d4a1c800
nid=0x47 waiting on condition [0x00007f14b5123000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.Mongo$CursorCleanerThread.run(Mongo.java:796)
"cluster-1-cpy-srv-app-mongo2:27017" #14 daemon prio=5 os_prio=0
tid=0x00007f14d4a0b800 nid=0x46 waiting on condition [0x00007f14b5224000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006286b8a28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
com.mongodb.ServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(ServerMonitor.java:177)
at
com.mongodb.ServerMonitor$ServerMonitorRunnable.waitForNext(ServerMonitor.java:158)
at com.mongodb.ServerMonitor$ServerMonitorRunnable.run(ServerMonitor.java:124)
at java.lang.Thread.run(Thread.java:745)
"default-akka.actor.default-dispatcher-4" #12 prio=5 os_prio=0
tid=0x00007f146c001000 nid=0x45 waiting on condition [0x00007f14b5b3f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-3" #11 prio=5 os_prio=0
tid=0x00007f14d48c6800 nid=0x44 waiting on condition [0x00007f14b5c40000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-2" #10 prio=5 os_prio=0
tid=0x00007f14d48bb800 nid=0x43 waiting on condition [0x00007f14b5d41000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-scheduler-1" #9 prio=5 os_prio=0 tid=0x00007f14d4821000 nid=0x42
waiting on condition [0x00007f14b6042000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:87)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:268)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f14d40c3000 nid=0x40
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f14d40c0000 nid=0x3f
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f14d40be800 nid=0x3e
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f14d40bb800 nid=0x3d
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f14d40ba000 nid=0x3c
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f14d4087000 nid=0x3b in
Object.wait() [0x00007f14b71f0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000062860b668> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f14d4082800 nid=0x3a
in Object.wait() [0x00007f14b72f1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x000000062860b820> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"VM Thread" os_prio=0 tid=0x00007f14d407b000 nid=0x39 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f14d4022000 nid=0x35
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f14d4024000 nid=0x36
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f14d4025800 nid=0x37
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f14d4027800 nid=0x38
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f14d40c7000 nid=0x41 waiting on
condition
JNI global references: 331
Heap
PSYoungGen total 66560K, used 20447K [0x0000000738200000,
0x000000073cc80000, 0x00000007c0000000)
eden space 56832K, 21% used
[0x0000000738200000,0x0000000738ded7d0,0x000000073b980000)
from space 9728K, 84% used
[0x000000073c300000,0x000000073cb0a5e8,0x000000073cc80000)
to space 9728K, 0% used
[0x000000073b980000,0x000000073b980000,0x000000073c300000)
ParOldGen total 290304K, used 99929K [0x0000000628600000,
0x000000063a180000, 0x0000000738200000)
object space 290304K, 34% used
[0x0000000628600000,0x000000062e796798,0x000000063a180000)
Metaspace used 51866K, capacity 52094K, committed 52352K, reserved
1095680K
class space used 6597K, capacity 6683K, committed 6784K, reserved 1048576K
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode):
"default-akka.actor.default-dispatcher-13" #25 prio=5 os_prio=0
tid=0x00007f1448001000 nid=0x51 waiting on condition [0x00007f145b50e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-14" #26 prio=5 os_prio=0
tid=0x00007f1460026000 nid=0x50 waiting on condition [0x00007f145b60f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-12" #24 prio=5 os_prio=0
tid=0x00007f1454042000 nid=0x4f waiting on condition [0x00007f145b710000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-11" #23 prio=5 os_prio=0
tid=0x00007f147016c000 nid=0x4e waiting on condition [0x00007f145b811000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-10" #22 prio=5 os_prio=0
tid=0x00007f1478039000 nid=0x4d waiting on condition [0x00007f145bb89000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at
scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-9" #21 prio=5 os_prio=0
tid=0x00007f1460025000 nid=0x4c waiting on condition [0x00007f145bc8a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"DestroyJavaVM" #20 prio=5 os_prio=0 tid=0x00007f14d400d000 nid=0x34 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"default-akka.io.pinned-dispatcher-8" #19 prio=5 os_prio=0
tid=0x00007f146c03d800 nid=0x4b runnable [0x00007f145bd8b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000628b28260> (a sun.nio.ch.Util$3)
- locked <0x0000000628b28270> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000628a13068> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.tryRun(SelectionHandler.scala:115)
at
akka.io.SelectionHandler$ChannelRegistryImpl$Task.run(SelectionHandler.scala:219)
at
akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.run(SelectionHandler.scala:148)
at
akka.util.SerializedSuspendableExecutionContext.run$1(SerializedSuspendableExecutionContext.scala:67)
at
akka.util.SerializedSuspendableExecutionContext.run(SerializedSuspendableExecutionContext.scala:71)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"default-akka.actor.default-dispatcher-7" #18 prio=5 os_prio=0
tid=0x00007f146c00c000 nid=0x4a waiting on condition [0x00007f14a0117000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-6" #17 prio=5 os_prio=0
tid=0x00007f1478027800 nid=0x49 waiting on condition [0x00007f14a0218000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-5" #16 prio=5 os_prio=0
tid=0x00007f146c00b000 nid=0x48 waiting on condition [0x00007f14b4195000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"MongoCleaner1917082298" #15 daemon prio=5 os_prio=0 tid=0x00007f14d4a1c800
nid=0x47 waiting on condition [0x00007f14b5123000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.Mongo$CursorCleanerThread.run(Mongo.java:796)
"cluster-1-cpy-srv-app-mongo2:27017" #14 daemon prio=5 os_prio=0
tid=0x00007f14d4a0b800 nid=0x46 waiting on condition [0x00007f14b5224000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006286b8a28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
com.mongodb.ServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(ServerMonitor.java:177)
at
com.mongodb.ServerMonitor$ServerMonitorRunnable.waitForNext(ServerMonitor.java:158)
at com.mongodb.ServerMonitor$ServerMonitorRunnable.run(ServerMonitor.java:124)
at java.lang.Thread.run(Thread.java:745)
"default-akka.actor.default-dispatcher-4" #12 prio=5 os_prio=0
tid=0x00007f146c001000 nid=0x45 waiting on condition [0x00007f14b5b3f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-3" #11 prio=5 os_prio=0
tid=0x00007f14d48c6800 nid=0x44 waiting on condition [0x00007f14b5c40000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-2" #10 prio=5 os_prio=0
tid=0x00007f14d48bb800 nid=0x43 waiting on condition [0x00007f14b5d41000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-scheduler-1" #9 prio=5 os_prio=0 tid=0x00007f14d4821000 nid=0x42
waiting on condition [0x00007f14b6042000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:87)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:268)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f14d40c3000 nid=0x40
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f14d40c0000 nid=0x3f
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f14d40be800 nid=0x3e
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f14d40bb800 nid=0x3d
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f14d40ba000 nid=0x3c
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f14d4087000 nid=0x3b in
Object.wait() [0x00007f14b71f0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000062860b668> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f14d4082800 nid=0x3a
in Object.wait() [0x00007f14b72f1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x000000062860b820> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"VM Thread" os_prio=0 tid=0x00007f14d407b000 nid=0x39 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f14d4022000 nid=0x35
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f14d4024000 nid=0x36
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f14d4025800 nid=0x37
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f14d4027800 nid=0x38
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f14d40c7000 nid=0x41 waiting on
condition
JNI global references: 325
Heap
PSYoungGen total 66560K, used 38631K [0x0000000738200000,
0x000000073cc80000, 0x00000007c0000000)
eden space 56832K, 52% used
[0x0000000738200000,0x0000000739f585a8,0x000000073b980000)
from space 9728K, 88% used
[0x000000073b980000,0x000000073c1e1680,0x000000073c300000)
to space 9728K, 0% used
[0x000000073c300000,0x000000073c300000,0x000000073cc80000)
ParOldGen total 290304K, used 99327K [0x0000000628600000,
0x000000063a180000, 0x0000000738200000)
object space 290304K, 34% used
[0x0000000628600000,0x000000062e6fffe0,0x000000063a180000)
Metaspace used 51866K, capacity 52094K, committed 52352K, reserved
1095680K
class space used 6597K, capacity 6683K, committed 6784K, reserved 1048576K
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode):
"default-akka.actor.default-dispatcher-13" #25 prio=5 os_prio=0
tid=0x00007f1448001000 nid=0x51 waiting on condition [0x00007f145b50e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-14" #26 prio=5 os_prio=0
tid=0x00007f1460026000 nid=0x50 waiting on condition [0x00007f145b60f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-12" #24 prio=5 os_prio=0
tid=0x00007f1454042000 nid=0x4f waiting on condition [0x00007f145b710000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-11" #23 prio=5 os_prio=0
tid=0x00007f147016c000 nid=0x4e runnable [0x00007f145b810000]
java.lang.Thread.State: RUNNABLE
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at
scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25)
at
scala.collection.TraversableLike$class.$plus$plus(TraversableLike.scala:147)
at
scala.collection.AbstractTraversable.$plus$plus(Traversable.scala:104)
at scala.collection.immutable.HashMap.$plus(HashMap.scala:63)
at scala.collection.immutable.Map$Map4.updated(Map.scala:179)
at spray.json.JsonParser.members$1(JsonParser.scala:82)
at spray.json.JsonParser.object(JsonParser.scala:86)
at spray.json.JsonParser.value(JsonParser.scala:60)
at spray.json.JsonParser.values$1(JsonParser.scala:102)
at spray.json.JsonParser.array(JsonParser.scala:106)
at spray.json.JsonParser.value(JsonParser.scala:61)
at spray.json.JsonParser.members$1(JsonParser.scala:81)
at spray.json.JsonParser.object(JsonParser.scala:86)
at spray.json.JsonParser.value(JsonParser.scala:60)
at spray.json.JsonParser.values$1(JsonParser.scala:102)
at spray.json.JsonParser.array(JsonParser.scala:106)
at spray.json.JsonParser.value(JsonParser.scala:61)
at spray.json.JsonParser.members$1(JsonParser.scala:81)
at spray.json.JsonParser.object(JsonParser.scala:86)
at spray.json.JsonParser.value(JsonParser.scala:60)
at spray.json.JsonParser.parseJsValue(JsonParser.scala:43)
at spray.json.JsonParser$.apply(JsonParser.scala:28)
at spray.json.PimpedString.parseJson(package.scala:45)
at
com.cupenya.zendesk.oauth.integration.common.HttpClient$$anonfun$doTypedRest$1$$anonfun$apply$5$$anonfun$6.apply(HttpClient.scala:119)
at scala.util.Try$.apply(Try.scala:192)
at
com.cupenya.zendesk.oauth.integration.common.HttpClient$$anonfun$doTypedRest$1$$anonfun$apply$5.apply(HttpClient.scala:119)
at
com.cupenya.zendesk.oauth.integration.common.HttpClient$$anonfun$doTypedRest$1$$anonfun$apply$5.apply(HttpClient.scala:118)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-10" #22 prio=5 os_prio=0
tid=0x00007f1478039000 nid=0x4d waiting on condition [0x00007f145bb89000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-9" #21 prio=5 os_prio=0
tid=0x00007f1460025000 nid=0x4c waiting on condition [0x00007f145bc8a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"DestroyJavaVM" #20 prio=5 os_prio=0 tid=0x00007f14d400d000 nid=0x34 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"default-akka.io.pinned-dispatcher-8" #19 prio=5 os_prio=0
tid=0x00007f146c03d800 nid=0x4b runnable [0x00007f145bd8b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000628b28260> (a sun.nio.ch.Util$3)
- locked <0x0000000628b28270> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000628a13068> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.tryRun(SelectionHandler.scala:115)
at
akka.io.SelectionHandler$ChannelRegistryImpl$Task.run(SelectionHandler.scala:219)
at
akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.run(SelectionHandler.scala:148)
at
akka.util.SerializedSuspendableExecutionContext.run$1(SerializedSuspendableExecutionContext.scala:67)
at
akka.util.SerializedSuspendableExecutionContext.run(SerializedSuspendableExecutionContext.scala:71)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"default-akka.actor.default-dispatcher-7" #18 prio=5 os_prio=0
tid=0x00007f146c00c000 nid=0x4a waiting on condition [0x00007f14a0117000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-6" #17 prio=5 os_prio=0
tid=0x00007f1478027800 nid=0x49 runnable [0x00007f14a0218000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-5" #16 prio=5 os_prio=0
tid=0x00007f146c00b000 nid=0x48 waiting on condition [0x00007f14b4195000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"MongoCleaner1917082298" #15 daemon prio=5 os_prio=0 tid=0x00007f14d4a1c800
nid=0x47 waiting on condition [0x00007f14b5123000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.Mongo$CursorCleanerThread.run(Mongo.java:796)
"cluster-1-cpy-srv-app-mongo2:27017" #14 daemon prio=5 os_prio=0
tid=0x00007f14d4a0b800 nid=0x46 waiting on condition [0x00007f14b5224000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006286b8a28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
com.mongodb.ServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(ServerMonitor.java:177)
at
com.mongodb.ServerMonitor$ServerMonitorRunnable.waitForNext(ServerMonitor.java:158)
at
com.mongodb.ServerMonitor$ServerMonitorRunnable.run(ServerMonitor.java:124)
at java.lang.Thread.run(Thread.java:745)
"default-akka.actor.default-dispatcher-4" #12 prio=5 os_prio=0
tid=0x00007f146c001000 nid=0x45 waiting on condition [0x00007f14b5b3f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-3" #11 prio=5 os_prio=0
tid=0x00007f14d48c6800 nid=0x44 waiting on condition [0x00007f14b5c40000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-akka.actor.default-dispatcher-2" #10 prio=5 os_prio=0
tid=0x00007f14d48bb800 nid=0x43 waiting on condition [0x00007f14b5d41000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000628602ef0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"default-scheduler-1" #9 prio=5 os_prio=0 tid=0x00007f14d4821000 nid=0x42
waiting on condition [0x00007f14b6042000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:87)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:268)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f14d40c3000 nid=0x40
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f14d40c0000 nid=0x3f
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f14d40be800 nid=0x3e
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f14d40bb800 nid=0x3d
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f14d40ba000 nid=0x3c
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f14d4087000 nid=0x3b in
Object.wait() [0x00007f14b71f0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000062860b668> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f14d4082800 nid=0x3a
in Object.wait() [0x00007f14b72f1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x000000062860b820> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"VM Thread" os_prio=0 tid=0x00007f14d407b000 nid=0x39 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f14d4022000 nid=0x35
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f14d4024000 nid=0x36
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f14d4025800 nid=0x37
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f14d4027800 nid=0x38
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f14d40c7000 nid=0x41 waiting on
condition
JNI global references: 323
Heap
PSYoungGen total 66048K, used 20981K [0x0000000738200000,
0x000000073cd00000, 0x00000007c0000000)
eden space 56832K, 21% used
[0x0000000738200000,0x0000000738de8290,0x000000073b980000)
from space 9216K, 95% used
[0x000000073c300000,0x000000073cb95470,0x000000073cc00000)
to space 9728K, 0% used
[0x000000073b980000,0x000000073b980000,0x000000073c300000)
ParOldGen total 290304K, used 99157K [0x0000000628600000,
0x000000063a180000, 0x0000000738200000)
object space 290304K, 34% used
[0x0000000628600000,0x000000062e6d57d8,0x000000063a180000)
Metaspace used 51865K, capacity 52094K, committed 52352K, reserved
1095680K
class space used 6596K, capacity 6683K, committed 6784K, reserved 1048576K
