[
https://issues.apache.org/jira/browse/NIFI-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough reassigned NIFI-6999:
----------------------------------
Assignee: Nathan Gough (was: Andy LoPresto)
> Encrypt Config Toolkit fails on very large flow.xml.gz files
> ------------------------------------------------------------
>
> Key: NIFI-6999
> URL: https://issues.apache.org/jira/browse/NIFI-6999
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
> Affects Versions: 1.2.0, 1.10.0
> Reporter: Andy LoPresto
> Assignee: Nathan Gough
> Priority: Critical
> Labels: documentation, encryption, heap, security, streaming,
> toolkit
>
> A user reported failure when using the encrypt config toolkit to process
> (encrypt) a large {{flow.xml.gz}}. The compressed file was 49 MB, but was 687
> MB uncompressed. It contained 545 encrypted values, and approximately 90
> templates. This caused the toolkit to fail during {{loadFlowXml()}} unless
> the toolkit invocation set the heap to 8 GB via {{-Xms2g -Xmx8g}}. Even with
> the expanded heap, the serialization of the newly-encrypted flow XML to the
> file system fails with the following exception:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.lang.StringCoding.encode(StringCoding.java:350)
> at java.lang.String.getBytes(String.java:941)
> at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
> at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at
> org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
> at groovy.lang.Closure.call(Closure.java:426)
> at groovy.lang.Closure.call(Closure.java:442)
> at
> org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
> at
> org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
> at
> org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5.doCall(ConfigEncryptionTool.groovy:691)
> {code}
> The immediate fix was to remove the duplicated template definitions in the
> flow definition, returning the file to a reasonable size. However, if run as
> an inline replacement, this can cause the {{flow.xml.gz}} to be overwritten
> with an empty file, potentially leading to data loss. The following steps
> should be taken:
> # Guard against loading/operating on/serializing large files (log statements,
> simple conditional checks)
> # Handle large files internally (change from direct {{String}} access to
> {{BufferedInputStream}}, etc.)
> # Document the internal memory usage of the toolkit in the toolkit guide
> # Document best practices and steps to resolve issue in the toolkit guide
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
