[
https://issues.apache.org/jira/browse/NIFI-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015099#comment-16015099
]
ASF GitHub Bot commented on NIFI-3894:
--------------------------------------
Github user ijokarumawak commented on the issue:
https://github.com/apache/nifi/pull/1820
For future reference...
There are 4 places that uses Inflater/Deflater at S2S client and server:
1. S2S Client receives data from remote output-port: **NOT** calling
inflator.end explicitly (added by previous PR but removed by new PR)
2. S2S Client sends data to remote input-port: Calls deflator.end
3. S2S Server receives data through input-port: Calls inflator.end (fixed
consume stream and inflator.end by new PR)
4. S2S Server sends data through output-port: Calls deflator.end
Ideally we should call Inflater/Deflater.end as often as possible, however
DataPacket which is holding the inputStream is exposed to S2S clients and we
can't safely close the inflater since we don't know when the stream is
consumed, we can't call inflator.end for the case.1 in above list. But even if
we don't call inflater/deflater.end explicitly, when JVM finalizes it, it will
be cleared.
> NiFi Crashes with OOM if compression is enabled on an RPG with high number of
> FlowFiles
> ---------------------------------------------------------------------------------------
>
> Key: NIFI-3894
> URL: https://issues.apache.org/jira/browse/NIFI-3894
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 0.7.0
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Fix For: 1.3.0
>
>
> RemoteProcessGroup can be configured to use compression per port.
> OutOfMemoryError occurred with a large amount of FlowFiles queued in an
> output port and a NiFi S2S client pulled those via RPG with compression
> enabled. Without compression, the same amount of FlowFiles can be transferred
> without issue. Here is the stack trace:
> {code}
> 2017-04-06 18:21:59,063 WARN [Timer-Driven Process Thread-5]
> o.a.n.c.t.ContinuallyRunConnectableTask
> StandardRootGroupPort[id=cc3ad50e-0159-1000-0000-00001c8cf0ef]
> Administratively Pausing for 10 seconds due to processing failure:
> java.lang.OutOfMemoryError
> java.lang.OutOfMemoryError: null
> at java.util.zip.Deflater.init(Native Method) ~[na:1.8.0_77]
> at java.util.zip.Deflater.<init>(Deflater.java:171) ~[na:1.8.0_77]
> at java.util.zip.Deflater.<init>(Deflater.java:180) ~[na:1.8.0_77]
> at
> org.apache.nifi.remote.io.CompressionOutputStream.<init>(CompressionOutputStream.java:54)
> ~[nifi-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
> {code}
> It happens because CompressionOutputStream.close does not call end() method
> of a Deflater It calls resets() but not end().
> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/remote/io/CompressionOutputStream.java#L141
> The issue is the same with this:
> JDK-4797189 : Finalizers not called promptly enough
> http://bugs.java.com/view_bug.do?bug_id=4797189
> {quote}
> WORK AROUND
> Invoke "end()" on any Inflator or Deflator once it is no longer in use. This
> will clean up the process space (non Java/GC) memory.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
