Jeff Storck created NIFI-3317:
---------------------------------
Summary: Under load, ThreadPoolRequestReplicator is experiencing
SocketTimeoutExceptions
Key: NIFI-3317
URL: https://issues.apache.org/jira/browse/NIFI-3317
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.1.0
Reporter: Jeff Storck
In a cluster under load (4 nodes, ~80 active threads per node) requests for
starting and stopping processors (and, for example, the timed UI thread
requesting status) can fail due to socket timeouts during request replication.
{code}2017-01-10 17:46:21,878 INFO [StandardProcessScheduler Thread-8]
o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling
GenerateFlowFile[id=87de49bc-0158-1000-6329-33d0dd20bd3a] to run
2017-01-10 17:46:21,921 WARN [Replicate Request Thread-9]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request PUT
/nifi-api/processors/87dd1f69-0158-1000-0e85-aaf63af20c89 to m10wn03.local
domain:9091 due to {}
com.sun.jersey.api.client.ClientHandlerException:
java.net.SocketTimeoutException: Read timed out
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:529)
~[jersey-client-1.19.jar:1.19]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:590)
~[nifi-framework-cluster-1.1.0.jar:1.1.0]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
~[nifi-framework-cluster-1.1.0.jar:1.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_111]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
~[na:1.8.0_111]
at java.net.SocketInputStream.read(SocketInputStream.java:170)
~[na:1.8.0_111]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[na:1.8.0_111]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
~[na:1.8.0_111]
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
~[na:1.8.0_111]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
~[na:1.8.0_111]
at
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
~[na:1.8.0_111]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
~[na:1.8.0_111]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
~[na:1.8.0_111]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
~[na:1.8.0_111]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
~[na:1.8.0_111]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
~[na:1.8.0_111]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
~[na:1.8.0_111]
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
~[na:1.8.0_111]
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
~[na:1.8.0_111]
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
~[na:1.8.0_111]
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
~[na:1.8.0_111]
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
~[jersey-client-1.19.jar:1.19]
... 12 common frames omitted{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
