[
https://issues.apache.org/jira/browse/NIFI-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865975#comment-15865975
]
Andre commented on NIFI-3313:
-----------------------------
[~alopresto] apologies for the delay on this, here it goes jstacks's output
{code}
2017-02-15 02:35:39
Full thread dump OpenJDK 64-Bit Server VM (25.121-b13 mixed mode):
"Attach Listener" #37 daemon prio=9 os_prio=0 tid=0x00007f65ec001000 nid=0x9ed
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"H2 Log Writer NIFI-USER-KEYS" #36 daemon prio=5 os_prio=0
tid=0x00007f6667d65800 nid=0x916 in Object.wait() [0x00007f65b97f9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.h2.store.WriterThread.run(WriterThread.java:103)
- locked <0x00000000fb9bb740> (a org.h2.store.WriterThread)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"H2 File Lock Watchdog
/home/afucs/nifi/nifi-assembly/target/nifi-1.2.0-SNAPSHOT-bin/nifi-1.2.0-SNAPSHOT/database_repository/nifi-user-keys.lock.db"
#35 daemon prio=9 os_prio=0 tid=0x00007f65b5d2c000 nid=0x915 waiting on
condition [0x00007f65b9afa000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.h2.store.FileLock.run(FileLock.java:517)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Generate Seed" #34 daemon prio=5 os_prio=0 tid=0x00007f65b5d6e000 nid=0x903
runnable [0x00007f65b9bfb000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at
sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:539)
at
sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:144)
at
sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:139)
at java.security.SecureRandom.generateSeed(SecureRandom.java:533)
at org.h2.util.MathUtils$1.run(MathUtils.java:84)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"org.eclipse.jetty.server.session.HashSessionManager@1b4872bcTimer" #33 daemon
prio=5 os_prio=0 tid=0x00007f65b59a5800 nid=0x8f1 waiting on condition
[0x00007f65ba6fc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ecb80eb8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"org.eclipse.jetty.server.session.HashSessionManager@61191222Timer" #32 daemon
prio=5 os_prio=0 tid=0x00007f65b58f9000 nid=0x8f0 waiting on condition
[0x00007f65ba9fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ecc869b0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"org.eclipse.jetty.server.session.HashSessionManager@713a35c5Timer" #31 daemon
prio=5 os_prio=0 tid=0x00007f65b58d0800 nid=0x8ef waiting on condition
[0x00007f65baafe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ecc86c10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"org.eclipse.jetty.server.session.HashSessionManager@72443081Timer" #30 daemon
prio=5 os_prio=0 tid=0x00007f6666f8c000 nid=0x8ee waiting on condition
[0x00007f65bad00000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e072dfe8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"org.eclipse.jetty.server.session.HashSessionManager@1f9d4b0eTimer" #29 daemon
prio=5 os_prio=0 tid=0x00007f6666f76000 nid=0x8ea waiting on condition
[0x00007f65bae01000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e120> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"org.eclipse.jetty.server.session.HashSessionManager@6b649efaTimer" #28 daemon
prio=5 os_prio=0 tid=0x00007f6664d42000 nid=0x8db waiting on condition
[0x00007f65baf1a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e380> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-27" #27 prio=5 os_prio=0 tid=0x00007f659c001800 nid=0x8da
waiting on condition [0x00007f65bb11c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-26" #26 prio=5 os_prio=0 tid=0x00007f6594001000 nid=0x8d9
waiting on condition [0x00007f65bb01b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-24" #24 prio=5 os_prio=0 tid=0x00007f66679d6800 nid=0x8d7
waiting on condition [0x00007f65bb4a5000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-23" #23 prio=5 os_prio=0 tid=0x00007f66679d5000 nid=0x8d6
waiting on condition [0x00007f65bb5a6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-22" #22 prio=5 os_prio=0 tid=0x00007f66679d3000 nid=0x8d5
waiting on condition [0x00007f65bb6a7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-21" #21 prio=5 os_prio=0 tid=0x00007f666447f000 nid=0x8d4
waiting on condition [0x00007f65bb7a8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-20" #20 prio=5 os_prio=0 tid=0x00007f666447d000 nid=0x8d3
waiting on condition [0x00007f65bb8a9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-19" #19 prio=5 os_prio=0 tid=0x00007f666447b800 nid=0x8d2
waiting on condition [0x00007f65bb9aa000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-18" #18 prio=5 os_prio=0 tid=0x00007f666447a000 nid=0x8d1
waiting on condition [0x00007f65bbaab000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"NiFi Web Server-17" #17 prio=5 os_prio=0 tid=0x00007f6664ef0000 nid=0x8d0
waiting on condition [0x00007f65bbbac000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068e5e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"process reaper" #16 daemon prio=10 os_prio=0 tid=0x00007f6666cd7000 nid=0x8ce
waiting on condition [0x00007f65c6b1c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e068f2a0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Timer-0" #15 daemon prio=5 os_prio=0 tid=0x00007f6664610800 nid=0x8aa in
Object.wait() [0x00007f6601517000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e05bb840> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000e05bb840> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Detect Timing Issues" #14 daemon prio=5 os_prio=0 tid=0x00007f666460d000
nid=0x8a9 waiting on condition [0x00007f6601618000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e05bbf10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Listen to Bootstrap" #13 daemon prio=5 os_prio=0 tid=0x00007f6664604800
nid=0x8a7 runnable [0x00007f6601719000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
org.apache.nifi.BootstrapListener$Listener.run(BootstrapListener.java:158)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x00007f66642af000 nid=0x8a5
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007f66642aa800
nid=0x8a4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f66642a8800
nid=0x8a3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f66642a4000
nid=0x8a2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f66642a2000
nid=0x8a1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f666429f800 nid=0x8a0
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007f666429d800 nid=0x89f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f6664276000 nid=0x89e in
Object.wait() [0x00007f6602c26000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e0275c78> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000e0275c78> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f6664271800
nid=0x89d in Object.wait() [0x00007f6602d27000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e0275e30> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000e0275e30> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"main" #1 prio=5 os_prio=0 tid=0x00007f666400b800 nid=0x886 runnable
[0x00007f666bd06000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at
sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:539)
at
sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:144)
at
sun.security.provider.SecureRandom$SeederHolder.<clinit>(SecureRandom.java:203)
at
sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:221)
- locked <0x00000000f0eae3d8> (a sun.security.provider.SecureRandom)
at java.security.SecureRandom.nextBytes(SecureRandom.java:468)
at
org.jasypt.salt.RandomSaltGenerator.generateSalt(RandomSaltGenerator.java:92)
- locked <0x00000000f0eae6f8> (a java.security.SecureRandom)
at
org.jasypt.encryption.pbe.StandardPBEByteEncryptor.encrypt(StandardPBEByteEncryptor.java:891)
at
org.jasypt.encryption.pbe.StandardPBEStringEncryptor.encrypt(StandardPBEStringEncryptor.java:642)
at
org.apache.nifi.encrypt.StringEncryptor.encrypt(StringEncryptor.java:130)
at
org.apache.nifi.encrypt.StringEncryptor.createEncryptor(StringEncryptor.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at
org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
- locked <0x00000000eca00758> (a java.util.concurrent.ConcurrentHashMap)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at
org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
at
org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1481)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
- locked <0x00000000eca00758> (a java.util.concurrent.ConcurrentHashMap)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060)
at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:53)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x00000000e130f790> (a java.lang.Object)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x00000000e072ec68> (a java.lang.Object)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x00000000e072ed18> (a java.lang.Object)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:411)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:378)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
- locked <0x00000000e072ede8> (a java.lang.Object)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:675)
at org.apache.nifi.NiFi.<init>(NiFi.java:157)
at org.apache.nifi.NiFi.main(NiFi.java:264)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007f666426a000 nid=0x89c runnable
"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f6664021000
nid=0x888 runnable
"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f6664022800
nid=0x889 runnable
"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f6664024800
nid=0x88a runnable
"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f6664026000
nid=0x88b runnable
"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f6664028000
nid=0x88c runnable
"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007f6664029800
nid=0x88d runnable
"Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007f666402b800
nid=0x88e runnable
"Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007f666402d000
nid=0x88f runnable
"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007f6664049000 nid=0x899
runnable
"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f666404a800
nid=0x89a runnable
"Gang worker#1 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f666404c800
nid=0x89b runnable
"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007f666403e000 nid=0x898
runnable
"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007f666403c000 nid=0x897
runnable
"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007f666403a800 nid=0x896
runnable
"G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007f6664038800 nid=0x895
runnable
"G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007f6664036800 nid=0x894
runnable
"G1 Concurrent Refinement Thread#5" os_prio=0 tid=0x00007f6664035000 nid=0x893
runnable
"G1 Concurrent Refinement Thread#6" os_prio=0 tid=0x00007f6664033000 nid=0x892
runnable
"G1 Concurrent Refinement Thread#7" os_prio=0 tid=0x00007f6664031800 nid=0x891
runnable
"G1 Concurrent Refinement Thread#8" os_prio=0 tid=0x00007f666402f800 nid=0x890
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f66642b2000 nid=0x8a6 waiting on
condition
JNI global references: 402
{code}
> First deployment of NiFi can hang on VMs without sufficient entropy if using
> /dev/random
> ----------------------------------------------------------------------------------------
>
> Key: NIFI-3313
> URL: https://issues.apache.org/jira/browse/NIFI-3313
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.1.1
> Reporter: Andy LoPresto
> Assignee: Andy LoPresto
> Priority: Critical
> Labels: entropy, security, virtual-machine
>
> h1. Analysis of Issue
> h2. Statement of Problem:
> NiFi deployed on headless VM (little user interaction by way of keyboard and
> mouse I/O) can take 5-10 minutes (reported) to start up. User reports this
> occurs on a "secure" cluster. Further examination is required to determine
> which specific process requires the large amount of random input (no steps to
> reproduce, configuration files, logs, or VM environment information
> provided).
> h2. Context
> The likely cause of this issue is that a process is attempting to read from
> _/dev/random_, a \*nix "device" providing a pseudo-random number generator
> (PRNG). Also available is _/dev/urandom_, a related PRNG. Despite common
> misperceptions, _/dev/urandom_ is not "less-secure" than _/dev/random_ for
> all general use cases. _/dev/random_ blocks if the entropy *estimate* (a
> "guess" of the existing entropy introduced into the pool) is lower than the
> amount of random data requested by the caller. In contrast, _/dev/urandom_
> does not block, but provides the output of the same cryptographically-secure
> PRNG (CSPRNG) that _/dev/random_ reads from \[myths\]. After as little as 256
> bytes of initial seeding, accessing _/dev/random_ and _/dev/urandom_ are
> functionally equivalent, as the long period of random data generated will not
> require re-seeding before sufficient entropy can be provided again.
> As mentioned earlier, further examination is required to determine if the
> process requiring random input occurs at application boot or only at
> "machine" (hardware or VM) boot. On the first deployment of the system with
> certificates, the certificate generation process will require substantial
> random input. However, on application launch and connection to a cluster,
> even the TLS/SSL protocol requires some amount of random input.
> h2. Proposed Solutions
> h3. rngd
> A software toolset for accessing dedicated hardware PRNG (*true* RNG, or
> TRNG) called _rng-tools_ \[rngtools\] exists for Linux. Specialized hardware,
> as well as Intel chips from IvyBridge and on (2012), can provide
> hardware-generated random input to the kernel. Using the daemon _rngd_ to
> seed the _/dev/random_ and _/dev/urandom_ entropy pool is the simplest
> solution.
> *Note: Do not use _/dev/urandom_ to seed _/dev/random_ using _rngd_. This is
> like running a garden hose from a car's exhaust back into its gas tank and
> trying to drive.*
> h3. Instruct Java to use /dev/urandom
> The Java Runtime Environment (JRE) can be instructed to use _/dev/urandom_
> for all invocations of {{SecureRandom}}, either on a per-Java process basis
> \[jdk-urandom\] or in the JVM configuration \[oracle-urandom\], which means
> it will not block on server startup. The NiFi {{bootstrap.conf}} file can be
> modified to contain an additional Java argument directing the JVM to use
> _/dev/urandom_.
> h2. Other Solutions
> h3. Entropy Gathering Tools
> Tools to gather entropy from non-standard sources (audio card noise, video
> capture from webcams, etc.) have been developed such as audio-entropyd
> \[wagner\], but these tools are not verified or well-examined -- usually when
> tested, they are only tested for the strength of their PRNG, not the ability
> of the tool to capture entropy and generate sufficiently random data
> unavailable to an attacker who may be able to determine the internal state.
> h3. haveged
> A solution has been proposed to use {{havaged}} \[haveged\], a user-space
> daemon relying on the HAVEGE (HArdware Volatile Entropy Gathering and
> Expansion) construct to continually increase the entropy on the system,
> allowing _/dev/random_ to run without blocking.
> However, on further investigation, multiple sources indicate this solution
> may be insecure \[dice\]\[leek-havege\].
> Michael Kerrisk:
> bq. Having read a number of papers about HAVEGE, Peter \[Anvin\] said he had
> been unable to work out whether this was a "real thing". Most of the papers
> that he has read run along the lines, "we took the output from HAVEGE, and
> ran some tests on it and all of the tests passed". The problem with this sort
> of reasoning is the point that Peter made earlier: there are no tests for
> randomness, only for non-randomness.
> bq. One of Peter's colleagues replaced the random input source employed by
> HAVEGE with a constant stream of ones. All of the same tests passed. In other
> words, all that the test results are guaranteeing is that the HAVEGE
> developers have built a very good PRNG. It is possible that HAVEGE does
> generate some amount of randomness, Peter said. But the problem is that the
> proposed source of randomness is simply too complex to analyze; thus it is
> not possible to make a definitive statement about whether it is truly
> producing randomness. (By contrast, the HWRNGs that Peter described earlier
> have been analyzed to produce a quantum theoretical justification that they
> are producing true randomness.) "So, while I can't really recommend it, I
> can't not recommend it either." If you are going to run HAVEGE, Peter
> strongly recommended running it together with rngd, rather than as a
> replacement for it.
> Tom Leek:
> bq. Of course, the whole premise of HAVEGE is questionable. For any practical
> security, you need a few "real random" bits, no more than 200, which you use
> as seed in a cryptographically secure PRNG. The PRNG will produce gigabytes
> of pseudo-\[data\] indistinguishable from true randomness, and that's good
> enough for all practical purposes.
> bq. Insisting on going back to the hardware for every bit looks like yet
> another outbreak of that flawed idea which sees entropy as a kind of
> gasoline, which you burn up when you look at it.
> h2. Next Steps
> As described above, further investigation is necessary, but moving forward,
> barring new information, I would propose directing the JVM to use
> _/dev/urandom_ and making _rngd_ available to systems that support a TRNG.
> [myths] http://www.2uo.de/myths-about-urandom/
> [rngtools]
> https://git.kernel.org/cgit/utils/kernel/rng-tools/rng-tools.git/about/
> [jdk-urandom] http://stackoverflow.com/a/2325109/70465
> [oracle-urandom]
> https://docs.oracle.com/cd/E13209_01/wlcp/wlss30/configwlss/jvmrand.html
> [wagner] https://people.eecs.berkeley.edu/~daw/rnd/
> [haveged] http://www.issihosts.com/haveged/
> [dice] https://lwn.net/Articles/525459/
> [leek-havege] http://security.stackexchange.com/a/34552/16485
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
