[ 
https://issues.apache.org/jira/browse/NIFI-5065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432942#comment-16432942
 ] 

ASF GitHub Bot commented on NIFI-5065:
--------------------------------------

Github user MikeThomsen commented on the issue:

    https://github.com/apache/nifi/pull/2622
  
    @ijokarumawak Merged, but I accidentally put the Jira # in the close line 
and not the PR.


> HTTP Site-to-Site fails sending data more than 2GB at once
> ----------------------------------------------------------
>
>                 Key: NIFI-5065
>                 URL: https://issues.apache.org/jira/browse/NIFI-5065
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.0.0
>            Reporter: Koji Kawamura
>            Assignee: Koji Kawamura
>            Priority: Major
>
> NiFi Site-to-Site (S2S) is designed to support sending large files. While RAW 
> transport protocol works fine with large files (I tested up to 8GB, but it 
> can be more), HTTP transport protocol starts failing if more than 2GB data is 
> sent at a time, meaning by sending a 2GB FlowFile, or batching multiple 
> FlowFiles to make more than 2GB into one transaction.
>  
> When it fails, following log messages are written:
> {code}
> 2018-04-10 16:05:45,006 ERROR [I/O dispatcher 25] 
> o.a.n.r.util.SiteToSiteRestApiClient Failed to send data to 
> http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee642179
> c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files due to 
> java.lang.RuntimeException: Sending data to 
> http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee
> 642179c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files has 
> reached to its end, but produced : read : wrote byte sizes (659704502 : 
> 659704502 : 9249639094) were not equal. Something went wr
> ong.
> java.lang.RuntimeException: Sending data to 
> http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee642179c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files
>  h
> as reached to its end, but produced : read : wrote byte sizes (659704502 : 
> 659704502 : 9249639094) were not equal. Something went wrong.
>         at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient$4.produceContent(SiteToSiteRestApiClient.java:848)
>         at 
> org.apache.http.impl.nio.client.MainClientExec.produceContent(MainClientExec.java:262)
>         at 
> org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.produceContent(DefaultClientExchangeHandlerImpl.java:140)
>         at 
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.outputReady(HttpAsyncRequestExecutor.java:241)
>         at 
> org.apache.http.impl.nio.DefaultNHttpClientConnection.produceOutput(DefaultNHttpClientConnection.java:290)
>         at 
> org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:86)
>         at 
> org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:39)
>         at 
> org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:145)
>         at 
> org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:188)
>         at 
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:341)
>         at 
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
>         at 
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
>         at 
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
>         at 
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
>         at java.lang.Thread.run(Thread.java:745)
> 2018-04-10 16:06:25,009 ERROR [Timer-Driven Process Thread-3] 
> o.a.nifi.remote.StandardRemoteGroupPort 
> RemoteGroupPort[name=input,targets=http://localhost:8080/nifi] failed to 
> communicate with remote NiFi instance due to java.io.IOException: Failed to 
> confirm transaction with Peer[url=http://localhost:8080/nifi-api] due to 
> java.io.IOException: Awaiting transferDataLatch has been timeout.
> 2018-04-10 16:06:25,009 ERROR [Timer-Driven Process Thread-3] 
> o.a.nifi.remote.StandardRemoteGroupPort
> java.io.IOException: Failed to confirm transaction with 
> Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Awaiting 
> transferDataLatch has been timeout.
>         at 
> org.apache.nifi.remote.AbstractTransaction.confirm(AbstractTransaction.java:264)
>         at 
> org.apache.nifi.remote.StandardRemoteGroupPort.transferFlowFiles(StandardRemoteGroupPort.java:369)
>         at 
> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:285)
>         at 
> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:250)
>         at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
>         at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Awaiting transferDataLatch has been timeout.
>         at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.finishTransferFlowFiles(SiteToSiteRestApiClient.java:938)
>         at 
> org.apache.nifi.remote.protocol.http.HttpClientTransaction.readTransactionResponse(HttpClientTransaction.java:93)
>         at 
> org.apache.nifi.remote.AbstractTransaction.confirm(AbstractTransaction.java:239)
>         ... 12 common frames omitted
> {code}
> The cause of this issue is that SiteToSiteRestApiClient.openConnectionForSend 
> method uses int variables to track bytes it sends. When the total size 
> exceeds Integer.MAX_VALUE (2,147,483,647 = 2GB), the int variables get 
> overflown. Then the check condition goes wrong.
> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/util/SiteToSiteRestApiClient.java#L777



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to