In addition to what Steve suggested, if you want to increase the memory
available for the Daffodil CLI tool, you can change/set the following
environment variables:
DAFFODIL_JAVA_OPTS
JAVA_OPTS
The Daffodil CLI tool will check the DAFFODIL_JAVA_OPTS variable first but if
it is not set will use the JAVA_OPTS environment variable. If that isn't set
either, it will default to the following options:
"-Xms1024m"
"-Xmx1024m"
"-XX:ReservedCodeCacheSize=128m"
"-Dlog4j.configurationFile=$CONFDIR/log4j2.xml"
The "-Xmx1024m" is the most important here as this sets the upper limit for
memory usage. So a quick way to double the amount of memory for the CLI tool
would be to do the following:
export DAFFODIL_JAVA_OPTS="-Xms1024m -Xmx2048m -XX:ReservedCodeCacheSize=128m"
Josh
________________________________
From: Steve Lawrence <slawre...@apache.org>
Sent: Thursday, August 4, 2022 8:15 AM
To: users@daffodil.apache.org <users@daffodil.apache.org>
Subject: Re: Daffodil 3.2.1 - An Unexpected exception occurred...
Thanks for reporting. How much memory have you given to your JVM? And
what version of Daffodil are you using?
It looks like the deliberate junk value you reference is about 1.5 GB.
So Daffodil will try to create a 1.5GB array to store the hex binary,
and if you don't have enough memory it will result in the OOM exception.
There are a couple of solutions here:
1) Put an assert on the length field to ensure it is a reasonable size.
For example:
<xs:element name="MDO_MovieDataSize" type="xs:int" ...>
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/";>
<dfdl:assert>{ . le 1000000 }</dfdl:assert>
</xs:appinfo>
</xs:annotation>
</xs:element>
2) Similar to above, put an xs:restriction on the length field to ensure
it's a reasonable size, add an assert to check that restriction:
<xs:element name="MDO_MovieDataSize" ...>
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/";>
<dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert>
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:int">
<xs:maxInclusive value="1000000" />
</xs:restriction>
</xs:simpleType>
</xs:element>
3) Set the "maxHexBinaryLengthInBytes" tunable. This will create a
processing error if the length of a hexBinary field is larger than that
tunable. This value defaults to 2GB, but could be set to a lower value
if you know your hex binary will never be that large.
- Steve
On 8/3/22 6:40 PM, Thompson, Mark M [US] (DS) wrote:
> All,
>
> I am reporting the occurrence of an exception as requested. The attached file
> contains a trace of the Exception with
>
> what I believe the relevant info.
>
> * Command executed
>
> daffodil -t -vv parse -s ..\..\..\MPRemote.dfdl.xsd -V limited -o
> image_file_name-8.invalid.dfdl.xml -r MP_REMOTE_Fields
> image_file_name-8.invalid
>
> * It appears that Daffodil does not like large values (deliberate junk in
> this
> case) when used as a size for an xs:hexBinary element. See
>
> <MDO_MovieDataSize> and <MDO_MovieData> respectively in the attached trace. In
> this case, there is nowhere near that much data available
>
> In the test binary input. Normally, in the case of insufficient data, Daffodil
> errors out gracefully and indicates that there was insufficient dat.
>
> Overview:
>
> * I am not at liberty to provide the actual schema files.
> * I may be able to provide test messages if necessary. I’m hoping that the
> attached trace provides more than enough info.
> * The Input test files to Daffodil are binary.
> * The command used in this case is listed above.
> * Command line options: -t -vv parse -V limited
> * Daffodil version: 3.2.1
> * Offending element: <MDO_MovieData>
>
> Thank you for your time,
>
> Mark M. Thompson
>
> Northrop Grumman Defense Systems
>
> Software Engineer
>
> (818) 712-7439
>
