[
https://issues.apache.org/jira/browse/NIFI-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Storck updated NIFI-3316:
------------------------------
Description:
Under heavy load, PostHTTP is encountering issues with chunking. This is
occurring while PostHTTP is set with a concurrent task setting of 20, posting
to a HandleHttpRequest with a concurrent task setting of 10 and a container
queue size of 2000.
The following exceptions are being thrown:
{code}2017-01-10 15:40:06,538 WARN [qtp1416553214-346304]
org.eclipse.jetty.server.HttpChannel //localhost:9090/generatedData
java.lang.IllegalStateException: cannot reset buffer on committed response
at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1222)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:371)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:266)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2017-01-10 15:40:06,538 ERROR [Timer-Driven Process Thread-108]
o.a.n.p.standard.HandleHttpResponse
HandleHttpResponse[id=e01117b3-0158-1000-b20e-5b35e91052cc] Failed to complete
HTTP Transaction for StandardFlowFileRecord[uuid=f0ea4d16-64c5-
4d33-86be-db93ed40a9ec,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1484080806524-39463, container=default,
section=551], offset=60840,
length=500],offset=0,name=5545915104168051,size=500] due to
java.lang.IllegalStateEx
ception: s=DISPATCHED i=false a=NOT_ASYNC: java.lang.IllegalStateException:
s=DISPATCHED i=false a=NOT_ASYNC
2017-01-10 15:40:06,551 ERROR [Timer-Driven Process Thread-72]
o.a.nifi.processors.standard.PostHTTP
PostHTTP[id=87e329e8-0158-1000-3b69-5972c3033611] Failed to Post
[StandardFlowFileRecord[uuid=586f8de0-01c3-4200-97f6-4ac96d05f47a,claim=Stan
dardContentClaim [resourceClaim=StandardResourceClaim[id=1484080803983-39150,
container=default, section=238], offset=13764,
length=500],offset=0,name=5545912553595898,size=500]] due to
org.apache.http.MalformedChunkCodingException: CRLF expe
cted at end of chunk; transferring to failure:
org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
2017-01-10 15:40:06,559 ERROR [Timer-Driven Process Thread-108]
o.a.n.p.standard.HandleHttpResponse
java.lang.IllegalStateException: s=DISPATCHED i=false a=NOT_ASYNC
at
org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:584)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at
org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:99)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at
org.apache.nifi.http.StandardHttpContextMap.complete(StandardHttpContextMap.java:131)
~[na:na]
at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
at
org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177)
~[na:na]
at com.sun.proxy.$Proxy91.complete(Unknown Source) ~[na:na]
at
org.apache.nifi.processors.standard.HandleHttpResponse.onTrigger(HandleHttpResponse.java:175)
~[nifi-standard-processors-1.1.0.jar:1.1.0]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_111]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2017-01-10 15:40:06,560 ERROR [Timer-Driven Process Thread-72]
o.a.nifi.processors.standard.PostHTTP
org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
at
org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
~[httpcore-4.4.1.jar:4.4.1]{code}
was:
Under heavy load, PostHTTP is encountering issues with chunking. The following
exceptions are being thrown:
{code}2017-01-10 15:40:06,538 WARN [qtp1416553214-346304]
org.eclipse.jetty.server.HttpChannel //localhost:9090/generatedData
java.lang.IllegalStateException: cannot reset buffer on committed response
at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1222)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:371)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:266)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2017-01-10 15:40:06,538 ERROR [Timer-Driven Process Thread-108]
o.a.n.p.standard.HandleHttpResponse
HandleHttpResponse[id=e01117b3-0158-1000-b20e-5b35e91052cc] Failed to complete
HTTP Transaction for StandardFlowFileRecord[uuid=f0ea4d16-64c5-
4d33-86be-db93ed40a9ec,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1484080806524-39463, container=default,
section=551], offset=60840,
length=500],offset=0,name=5545915104168051,size=500] due to
java.lang.IllegalStateEx
ception: s=DISPATCHED i=false a=NOT_ASYNC: java.lang.IllegalStateException:
s=DISPATCHED i=false a=NOT_ASYNC
2017-01-10 15:40:06,551 ERROR [Timer-Driven Process Thread-72]
o.a.nifi.processors.standard.PostHTTP
PostHTTP[id=87e329e8-0158-1000-3b69-5972c3033611] Failed to Post
[StandardFlowFileRecord[uuid=586f8de0-01c3-4200-97f6-4ac96d05f47a,claim=Stan
dardContentClaim [resourceClaim=StandardResourceClaim[id=1484080803983-39150,
container=default, section=238], offset=13764,
length=500],offset=0,name=5545912553595898,size=500]] due to
org.apache.http.MalformedChunkCodingException: CRLF expe
cted at end of chunk; transferring to failure:
org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
2017-01-10 15:40:06,559 ERROR [Timer-Driven Process Thread-108]
o.a.n.p.standard.HandleHttpResponse
java.lang.IllegalStateException: s=DISPATCHED i=false a=NOT_ASYNC
at
org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:584)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at
org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:99)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at
org.apache.nifi.http.StandardHttpContextMap.complete(StandardHttpContextMap.java:131)
~[na:na]
at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
at
org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177)
~[na:na]
at com.sun.proxy.$Proxy91.complete(Unknown Source) ~[na:na]
at
org.apache.nifi.processors.standard.HandleHttpResponse.onTrigger(HandleHttpResponse.java:175)
~[nifi-standard-processors-1.1.0.jar:1.1.0]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_111]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2017-01-10 15:40:06,560 ERROR [Timer-Driven Process Thread-72]
o.a.nifi.processors.standard.PostHTTP
org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
at
org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
~[httpcore-4.4.1.jar:4.4.1]{code}
> PostHTTP chunking issue under load
> ----------------------------------
>
> Key: NIFI-3316
> URL: https://issues.apache.org/jira/browse/NIFI-3316
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.1.0
> Reporter: Jeff Storck
>
> Under heavy load, PostHTTP is encountering issues with chunking. This is
> occurring while PostHTTP is set with a concurrent task setting of 20, posting
> to a HandleHttpRequest with a concurrent task setting of 10 and a container
> queue size of 2000.
> The following exceptions are being thrown:
> {code}2017-01-10 15:40:06,538 WARN [qtp1416553214-346304]
> org.eclipse.jetty.server.HttpChannel //localhost:9090/generatedData
> java.lang.IllegalStateException: cannot reset buffer on committed response
> at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1222)
> ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:371)
> [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
> at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:266)
> [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> 2017-01-10 15:40:06,538 ERROR [Timer-Driven Process Thread-108]
> o.a.n.p.standard.HandleHttpResponse
> HandleHttpResponse[id=e01117b3-0158-1000-b20e-5b35e91052cc] Failed to
> complete HTTP Transaction for StandardFlowFileRecord[uuid=f0ea4d16-64c5-
> 4d33-86be-db93ed40a9ec,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484080806524-39463,
> container=default, section=551], offset=60840,
> length=500],offset=0,name=5545915104168051,size=500] due to
> java.lang.IllegalStateEx
> ception: s=DISPATCHED i=false a=NOT_ASYNC: java.lang.IllegalStateException:
> s=DISPATCHED i=false a=NOT_ASYNC
> 2017-01-10 15:40:06,551 ERROR [Timer-Driven Process Thread-72]
> o.a.nifi.processors.standard.PostHTTP
> PostHTTP[id=87e329e8-0158-1000-3b69-5972c3033611] Failed to Post
> [StandardFlowFileRecord[uuid=586f8de0-01c3-4200-97f6-4ac96d05f47a,claim=Stan
> dardContentClaim [resourceClaim=StandardResourceClaim[id=1484080803983-39150,
> container=default, section=238], offset=13764,
> length=500],offset=0,name=5545912553595898,size=500]] due to
> org.apache.http.MalformedChunkCodingException: CRLF expe
> cted at end of chunk; transferring to failure:
> org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
> 2017-01-10 15:40:06,559 ERROR [Timer-Driven Process Thread-108]
> o.a.n.p.standard.HandleHttpResponse
> java.lang.IllegalStateException: s=DISPATCHED i=false a=NOT_ASYNC
> at
> org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:584)
> ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
> at
> org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:99)
> ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
> at
> org.apache.nifi.http.StandardHttpContextMap.complete(StandardHttpContextMap.java:131)
> ~[na:na]
> at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
> ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_111]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
> at
> org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177)
> ~[na:na]
> at com.sun.proxy.$Proxy91.complete(Unknown Source) ~[na:na]
> at
> org.apache.nifi.processors.standard.HandleHttpResponse.onTrigger(HandleHttpResponse.java:175)
> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_111]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_111]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_111]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> 2017-01-10 15:40:06,560 ERROR [Timer-Driven Process Thread-72]
> o.a.nifi.processors.standard.PostHTTP
> org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
> at
> org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
> ~[httpcore-4.4.1.jar:4.4.1]{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
