[
https://issues.apache.org/jira/browse/NIFI-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015153#comment-16015153
]
ASF subversion and git services commented on NIFI-3894:
-------------------------------------------------------
Commit 36e7bd6164f6d7294ec17fe6487a74b042ba2f25 in nifi's branch
refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=36e7bd6 ]
NIFI-3894: This closes #1820. Fixed close and consume order with compression.
Before this fix, 'NullPointerException: Inflater has been closed' can be thrown
as the Inflater is closed before input stream is consumed.
Also, calling close from AbstractTransaction.receive is removed, because the
DataPacket is exposed as its return value and this class will not be able to
know when to close the stream.
Signed-off-by: joewitt <[email protected]>
> NiFi Crashes with OOM if compression is enabled on an RPG with high number of
> FlowFiles
> ---------------------------------------------------------------------------------------
>
> Key: NIFI-3894
> URL: https://issues.apache.org/jira/browse/NIFI-3894
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Fix For: 1.3.0
>
>
> RemoteProcessGroup can be configured to use compression per port.
> OutOfMemoryError occurred with a large amount of FlowFiles queued in an
> output port and a NiFi S2S client pulled those via RPG with compression
> enabled. Without compression, the same amount of FlowFiles can be transferred
> without issue. Here is the stack trace:
> {code}
> 2017-04-06 18:21:59,063 WARN [Timer-Driven Process Thread-5]
> o.a.n.c.t.ContinuallyRunConnectableTask
> StandardRootGroupPort[id=cc3ad50e-0159-1000-0000-00001c8cf0ef]
> Administratively Pausing for 10 seconds due to processing failure:
> java.lang.OutOfMemoryError
> java.lang.OutOfMemoryError: null
> at java.util.zip.Deflater.init(Native Method) ~[na:1.8.0_77]
> at java.util.zip.Deflater.<init>(Deflater.java:171) ~[na:1.8.0_77]
> at java.util.zip.Deflater.<init>(Deflater.java:180) ~[na:1.8.0_77]
> at
> org.apache.nifi.remote.io.CompressionOutputStream.<init>(CompressionOutputStream.java:54)
> ~[nifi-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
> {code}
> It happens because CompressionOutputStream.close does not call end() method
> of a Deflater It calls resets() but not end().
> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/remote/io/CompressionOutputStream.java#L141
> The issue is the same with this:
> JDK-4797189 : Finalizers not called promptly enough
> http://bugs.java.com/view_bug.do?bug_id=4797189
> {quote}
> WORK AROUND
> Invoke "end()" on any Inflator or Deflator once it is no longer in use. This
> will clean up the process space (non Java/GC) memory.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
