Michael Hogue created NIFI-10273:
------------------------------------
Summary: MergeContent tar compression fails on FlowFiles > 8.5GB
Key: NIFI-10273
URL: https://issues.apache.org/jira/browse/NIFI-10273
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.16.3
Reporter: Michael Hogue
When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the
following error:
{noformat}
java.lang.RuntimeException: entry size '<FlowFileSize>' is too big (>
8589934591){noformat}
This is because apache commons-compress library has a hard maximum :
[https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]
077777777777 octal bytes is 8589934591 decimal bytes (~8.5GB).
This can be addressed by configuring a BigNumberMode on the
{{TarArchiveOutputStream:
[https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
}}
{{Here's an example of another project doing this:}}
{{https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
