I am reposting my previous question with additional logs to see if someone
may help.
I have a simple flow with 2 processors connected.
The first one is GenerateFlowFile generating 64KB at 10 batch. and second
one is PutAzureBlobStorage to send flowfiles generated from the first
processor to Azure Blob.
When I increase the ConcurrentTask to 20 and leaves the flow, eventually, I
see SocketTimeoutException in UI.
And when I capture the thread dump with n='nifi.sh dump', I see 31 blocked
threads. For example,:
hreadId:1428 - state:BLOCKED
stackTrace:
at
org.apache.nifi.provenance.store.RecordWriterLease.tryClaim(RecordWriterLease.java:49)
at
org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:178)
at
org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:103)
at
org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:180)
at
org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:766)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:360)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:343)
- waiting on
org.apache.nifi.controller.repository.StandardProcessSession@64231af5
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@1246c12d
I see these running threads are blocking others.
1) case 1
"NiFi Web
Server-53-acceptor-3@1ef6f7eb-ServerConnector@2d5d15df{HTTP/1.1,[http/1.1]}{cj-dev-seyan-nifi000000:8080}"
- Thread t@53
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <3c156d01> (a java.lang.Object)
at
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
2) case 2
"Timer-Driven Process Thread-11" - Thread t@189
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
- locked <1f21f193> (a java.io.BufferedOutputStream)
at
org.apache.nifi.stream.io.ByteCountingOutputStream.flush(ByteCountingOutputStream.java:59)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at
org.apache.nifi.provenance.serialization.CompressableRecordWriter.flush(CompressableRecordWriter.java:189)
- locked <56a80a6e> (a
org.apache.nifi.provenance.EventIdFirstSchemaRecordWriter)
at
org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:313)
at
org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:194)
at
org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:103)
at
org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:180)
at
org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:766)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:360)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:343)
- locked <3673531> (a
org.apache.nifi.controller.repository.StandardProcessSession)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- locked <6fed0c3e> (a
java.util.concurrent.ThreadPoolExecutor$Worker
3) case 3
"Replicate Request Thread-1" - Thread t@169
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at okio.Okio$2.read(Okio.java:140)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355)
at
okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227)
at
okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
at
okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
at
okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at
okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at
okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at
okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
at okhttp3.RealCall.execute(RealCall.java:77)
at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:138)
at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:132)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I am attaching the two set of thread dumps while this happened.
threaddump-9-11-2019.txt
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t765/threaddump-9-11-2019.txt>
threaddump-9-11-2019-2.txt
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t765/threaddump-9-11-2019-2.txt>
And, this happens when we increase the timer-thread count to 100 under
controller setting and go away when we decrease the count to 50 below. What
is the guideline for thread count?
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
