Koji Kawamura created NIFI-5065:
-----------------------------------
Summary: HTTP Site-to-Site fails sending data more than 2GB at once
Key: NIFI-5065
URL: https://issues.apache.org/jira/browse/NIFI-5065
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.0.0
Reporter: Koji Kawamura
Assignee: Koji Kawamura
NiFi Site-to-Site (S2S) is designed to support sending large files. While RAW
transport protocol works fine with large files (I tested up to 8GB, but it can
be more), HTTP transport protocol starts failing if more than 2GB data is sent
at a time, meaning by sending a 2GB FlowFile, or batching multiple FlowFiles to
make more than 2GB into one transaction.
When it fails, following log messages are written:
{code}
2018-04-10 16:05:45,006 ERROR [I/O dispatcher 25]
o.a.n.r.util.SiteToSiteRestApiClient Failed to send data to
http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee642179
c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files due to
java.lang.RuntimeException: Sending data to
http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee
642179c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files has
reached to its end, but produced : read : wrote byte sizes (659704502 :
659704502 : 9249639094) were not equal. Something went wr
ong.
java.lang.RuntimeException: Sending data to
http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee642179c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files
h
as reached to its end, but produced : read : wrote byte sizes (659704502 :
659704502 : 9249639094) were not equal. Something went wrong.
at
org.apache.nifi.remote.util.SiteToSiteRestApiClient$4.produceContent(SiteToSiteRestApiClient.java:848)
at
org.apache.http.impl.nio.client.MainClientExec.produceContent(MainClientExec.java:262)
at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.produceContent(DefaultClientExchangeHandlerImpl.java:140)
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.outputReady(HttpAsyncRequestExecutor.java:241)
at
org.apache.http.impl.nio.DefaultNHttpClientConnection.produceOutput(DefaultNHttpClientConnection.java:290)
at
org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:86)
at
org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:39)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:145)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:188)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:341)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:745)
2018-04-10 16:06:25,009 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.remote.StandardRemoteGroupPort
RemoteGroupPort[name=input,targets=http://localhost:8080/nifi] failed to
communicate with remote NiFi instance due to java.io.IOException: Failed to
confirm transaction with Peer[url=http://localhost:8080/nifi-api] due to
java.io.IOException: Awaiting transferDataLatch has been timeout.
2018-04-10 16:06:25,009 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.remote.StandardRemoteGroupPort
java.io.IOException: Failed to confirm transaction with
Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Awaiting
transferDataLatch has been timeout.
at
org.apache.nifi.remote.AbstractTransaction.confirm(AbstractTransaction.java:264)
at
org.apache.nifi.remote.StandardRemoteGroupPort.transferFlowFiles(StandardRemoteGroupPort.java:369)
at
org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:285)
at
org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:250)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Awaiting transferDataLatch has been timeout.
at
org.apache.nifi.remote.util.SiteToSiteRestApiClient.finishTransferFlowFiles(SiteToSiteRestApiClient.java:938)
at
org.apache.nifi.remote.protocol.http.HttpClientTransaction.readTransactionResponse(HttpClientTransaction.java:93)
at
org.apache.nifi.remote.AbstractTransaction.confirm(AbstractTransaction.java:239)
... 12 common frames omitted
{code}
The cause of this issue is that SiteToSiteRestApiClient.openConnectionForSend
method uses int variables to track bytes it sends. When the total size exceeds
Integer.MAX_VALUE, the int variables get overflown. Then the check condition
goes wrong.
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/util/SiteToSiteRestApiClient.java#L777
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
