Steve,
Thanks so much! This is a HUGE help! I will save off the parser and
load it into the boot image and store it on the device's file system. Then
load the saved parser via the JAVA API.
Thanks!
Rob
-----Original Message-----
From: Steve Lawrence <slawre...@apache.org>
Sent: Thursday, November 21, 2019 5:11 PM
To: dev@daffodil.apache.org
Subject: Re: JAVA API - Out of heap space
Daffodil is pretty memory hungry at the moment, especially when it comes to
compiling very large schemas like VMF. I can almost guarantee you won't be able
to compile that schema on 2GB of memory. For reference, I usually give my JVM
8GB when compling the VMF schema, and I think Daffodil uses a good chunk of
that. This is something we plan to address in furture releases, with 2.5.0 have
some initial steps towards improving this.
In the meantime, I've got a few suggestions that might help.
First, if you're using the master branch of Daffodil, don't do that.
It's currently broken and requires exponential amounts of memory, and VMF being
the size it is basically means it will never finish compiling.
So for now stick to the 2.4.0 release if you aren't already. This will be
addressed by the time 2.5.0 comes out.
Second, I would definitely reccommend compiling schemas off box with a bunch of
memory, saving the parser, and then reloading just the compiled parser on box.
You can save a parser with the Java API, but it might be easier to just use the
daffodil CLI, something like this:
daffodil save-parser --schema path/to/schema.dfdl.xsdf parser.bin
That will compile your schema and save it to parser.bin, which can be reloaded
on box and should require signicantly less memory. I suspect 2GB should be
plenty for VMF, but I haven't compiled the vmf schema in a while to be sure.
With the CLI, you can use the -P option to use a saved parser, e.g.
daffodil parse -P parser.bin file_to_parse
Or with the API you can use the comilers reload() method, e.g.:
Compiler c = Daffodil.compiler();
DataProcessor dp = c.reload(savedParserFile);
Last suggestion, the VMF schema on DI2E is pretty large because it supports all
the different message types (something like 100+). If you only need to support
a small subset of those, I woud recommend removing any message elements you do
not need. That will make schema compilation quicker, require less memory, and
make the saved parser much smaller as well.
With just handful of messages supported and compiling off box, you should be
able to easily get VMF to run with less than 2GB of memory.
- Steve
On 11/21/19 3:56 PM, Rob Rose wrote:
> Hello all,
>
> I am trying to compile the DataProcessor using the Apache DFDL VMF
> schema from the DI2E site. I am trying to run DFDL on a small embedded
> device with an ARM processor containing 2G of memory. I tried setting the
> max and min heap space values for the JVM to several different values 128,
> 256, 512 and each time the device panicked and ran out of memory. The very
> first time I tried to execute DFDL I did not set the min and max JVM heap
> values and received the Out of heap space exception.
>
> GIven the magnitude of the schema, is it even possible to run DFDL on
> such a limited device ?
>
> Can I precompile the parser and load it onto the embedded device prior
> to executing DFDL?
>
> Any suggestions are greatly appreciated!
>
> Here is some log info:
>
>
> /usr/lib/jvm/image-aarch64/bin/java -jar MessageTestApp.jar vmb
> vmf.dfdl.xsd T imestamp prior to entering COMPILE SCHEMA:
> 1574365652553ms
>
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at org.apache.daffodil.dsom.TermFactory$.apply(ModelGroup.scala:11[
> 1554.981162] audit: type=1400 audit(1574365826.688:1601): avc: denied
> { getsched } for pid=1910 comm="java"
> scontext=system_u:system_r:busybox_t
> tcontext=system_u:system_r:busybox_t tclass=process permissive=1
> 6)
> at org.apache.daffodil.dsom[ 1555.003192] audit: type=1400
> audit(1574365826.708:1602): avc: denied { remove_name } for
> pid=1910 comm="java" name="1910" dev="rootfs" ino=21993
> scontext=system_u:system_r:busybox_t tcontext=system_u:object_r:tmp_t
> tclass=dir permissive=1 .GroupDefLike.$a[ 1555.025259] audit:
> type=1400 audit(1574365826.708:1603): avc: denied { unlink } for
> pid=1910 comm="java" name="1910" dev="rootfs" ino=21993
> scontext=system_u:system_r:busybox_t tcontext=system_u:object_r:tmp_t
> tclass=file permissive=1
> nonfun$groupMembers$2(GroupDef.scala:77)
> at
> org.apache.daffodil.dsom.GroupDefLike$$Lambda$471/0x0000007f5eb0f040.apply(Unknown
> Source)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$Lambda$38/0x0000007f5f7fa040.apply(Unknown
> Source)
> at scala.collection.immutable.List.foreach(List.scala:389)
> at scala.collection.TraversableLike.map(TraversableLike.scala:234)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
> at scala.collection.immutable.List.map(List.scala:295)
> at
> org.apache.daffodil.dsom.GroupDefLike.$anonfun$groupMembers$1(GroupDef.scala:75)
> at
> org.apache.daffodil.dsom.GroupDefLike$$Lambda$467/0x0000007f5eb06040.apply(Unknown
> Source)
> at
> org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.liftedTree1$1(OOLAG.scala:549)
> at
> org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.valueAsAny$lzycompute(OOLAG.scala:547)
> at
> org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.valueAsAny(OOLAG.scala:544)
> at
> org.apache.daffodil.oolag.OOLAG$OOLAGValue.value$lzycompute(OOLAG.scala:594)
> at org.apache.daffodil.oolag.OOLAG$OOLAGValue.value(OOLAG.scala:594)
> at org.apache.daffodil.dsom.GroupDefLike.groupMembers(GroupDef.scala:72)
> at
> org.apache.daffodil.dsom.GroupDefLike.groupMembers$(GroupDef.scala:72)
> at
> org.apache.daffodil.dsom.Sequence.groupMembers$lzycompute(SequenceGroup.scala:320)
> at
> org.apache.daffodil.dsom.Sequence.groupMembers(SequenceGroup.scala:320)
> at
> org.apache.daffodil.dsom.ModelGroup.$anonfun$elementChildren$1(ModelGroup.scala:186)
> at
> org.apache.daffodil.dsom.ModelGroup$$Lambda$466/0x0000007f5eb05040.apply(Unknown
> Source)
> at
> org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.liftedTree1$1(OOLAG.scala:549)
> at
> org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.valueAsAny$lzycompute(OOLAG.scala:547)
> at
> org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.valueAsAny(OOLAG.scala:544)
> at
> org.apache.daffodil.oolag.OOLAG$OOLAGValue.value$lzycompute(OOLAG.scala:594)
> at org.apache.daffodil.oolag.OOLAG$OOLAGValue.value(OOLAG.scala:594)
> at
> org.apache.daffodil.dsom.ModelGroup.elementChildren$lzycompute(ModelGroup.scala:185)
> at
> org.apache.daffodil.dsom.ModelGroup.elementChildren(ModelGroup.scala:185)
> at
> org.apache.daffodil.dsom.ElementBase.$anonfun$elementChildren$1(ElementBase.scala:612)
> at
> org.apache.daffodil.dsom.ElementBase$$Lambda$457/0x0000007f5ea4e840.apply(Unknown
> Source)
> at
> org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.liftedTree1$1(OOLAG.sca
> la:549)
> #
>
>
> /usr/lib/jvm/image-aarch64/bin/java -Xmx256m -jar MessageTestApp.jar
> vmb vmf.dfdl.xsd Timestamp prior to entering COMPILE SCHEMA: 1574366380842ms
> PERF_SYS System load and free memory [System=0.7% User=15.1% IRQ/IO=0.0%
> Idle=84.2%; Uptime 20:00:13 up 35 min, load average: 1.05, 0.62, 0.40; Mem
> tot=778960,used=546256,free=232704]
> TACDS 0101 CFG_CHKR INFO PERF_SYS Flash filesystem usage [18%]
> [ 2166.210992] dmesg invoked oom-killer:
> gfp_mask=0x16080c0(GFP_KERNEL|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null),
> order=0, oom_score_adj=0
> [ 2166.223381] CPU: 1 PID: 2221 Comm: dmesg Tainted: G O
> 4.14.0-okl4-hyp #11
> [ 2166.231269] Hardware name: ZynqMP Ultrazed Split1 Hybrid Linux Cell
> (OKL4 Guest) (DT) [ 2166.239060] Call trace:
> [ 2166.241509] [<ffffff8008088888>] dump_backtrace+0x0/0x370 [
> 2166.246881] [<ffffff8008088c0c>] show_stack+0x14/0x20 [ 2166.251912]
> [<ffffff800859fea8>] dump_stack+0x98/0xb8 [ 2166.256941]
> [<ffffff8008134a2c>] dump_header.isra.6+0x7c/0x1a8 [ 2166.262746]
> [<ffffff80081340bc>] oom_kill_process+0x2cc/0x518 [ 2166.268464]
> [<ffffff8008134634>] out_of_memory+0xbc/0x3d0 [ 2166.273840]
> [<ffffff80081394dc>] __alloc_pages_nodemask+0x9fc/0xa78
> [ 2166.280078] [<ffffff800815bd88>] __pte_alloc+0x30/0x148 [
> 2166.285279] [<ffffff8008167f14>] move_page_tables+0x5ac/0x678 [
> 2166.290999] [<ffffff80081973c0>] shift_arg_pages+0x88/0x198 [
> 2166.296544] [<ffffff8008197694>] setup_arg_pages+0x1c4/0x1e0 [
> 2166.302181] [<ffffff80081e4b6c>] load_elf_binary+0x3cc/0x1070 [
> 2166.307896] [<ffffff8008198b6c>] search_binary_handler+0xc4/0x270 [
> 2166.313962] [<ffffff800819951c>]
> do_execveat_common.isra.13+0x4b4/0x5e0
> [ 2166.320546] [<ffffff8008199818>] SyS_execve+0x38/0x48 [
> 2166.325573] Exception stack(0xffffff800b6a3ec0 to 0xffffff800b6a4000)
> [ 2166.331985] 3ec0: 000000000040a068 0000007fea2d7f30
> 0000007fea2da568 0000000000000000 [ 2166.339777] 3ee0:
> 0000000000000001 000000000040aab5 0000007fea2d9f1c 0000000000000000 [
> 2166.347571] 3f00: 00000000000000dd 0000007fea2d9f30 0101010101010101
> 0000000000000018 [ 2166.355364] 3f20: 0000000000002090
> 0000000000000000 6d68732f7665642f 000000000000007f [ 2166.363158]
> 3f40: 0000000000002090 000000000041c2e0 0000000000040e00
> 0000007fea2d7f40 [ 2166.370951] 3f60: 0000000000000003
> 0000007fea2d7f30 0000000000000400 0000007fea2d7f30 [ 2166.378744]
> 3f80: 000000000040a068 0000000000000000 0000000000000000
> 0000000000000012 [ 2166.386538] 3fa0: 0000000010c6fc48
> 0000007fea2d9f60 0000007f96ed1de0 0000007fea2d7ed0 [ 2166.394331] 3fc0:
> 0000007f96ed1a08 00000000a0000000 000000000040a068 00000000000000dd [
> 2166.402123] 3fe0: 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000 [ 2166.409920] [<ffffff8008082fb0>] el0_svc_naked+0x24/0x28
> [ 2166.415296] Mem-Info:
> [ 2166.417587] active_anon:134711 inactive_anon:14 isolated_anon:0 [
> 2166.417587] active_file:0 inactive_file:0 isolated_file:0 [
> 2166.417587] unevictable:0 dirty:0 writeback:0 unstable:0 [
> 2166.417587] slab_reclaimable:672 slab_unreclaimable:2655 [
> 2166.417587] mapped:5875 shmem:43049 pagetables:684 bounce:0 [
> 2166.417587] free:54146 free_pcp:178 free_cma:48514 [ 2166.450137]
> Node 0 active_anon:538844kB inactive_anon:56kB active_file:0kB
> inactive_file:0kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB mapped:23500kB dirty:0kB writeback:0kB
> shmem:172196kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB
> writeback_tmp:0kB unstable:0kB all_unreclaimable? yes [ 2166.476905]
> DMA free:216584kB min:22528kB low:28160kB high:33792kB
> active_anon:538908kB inactive_anon:56kB active_file:0kB
> inactive_file:0kB unevictable:0kB writepending:0kB present:819200kB
> managed:778960kB mlocked:0kB kernel_stack:2736kB pagetables:2736kB
> bounce:0kB free_pcp:712kB local_pcp:360kB free_cma:194056kB [
> 2166.504783] lowmem_reserve[]: 0 0 0 [ 2166.508283] DMA: 50*4kB (EHC)
> 40*8kB (UEH) 14*16kB (UEH) 11*32kB (UEHC) 5*64kB (UE) 5*128kB (EHC)
> 4*256kB (EHC) 3*512kB (UE) 5*1024kB (UEC) 3*2048kB (UE) 49*4096kB (EC)
> = 216584kB [ 2166.524258] Node 0 hugepages_total=0 hugepages_free=0
> hugepages_surp=0 hugepages_size=2048kB [ 2166.532675] 43049 total pagecache
> pages [ 2166.536510] 0 pages in swap cache [ 2166.539824] Swap cache stats:
> add 0, delete 0, find 0/0 [ 2166.545050] Free swap = 0kB [ 2166.547911]
> Total swap = 0kB [ 2166.550800] 204800 pages RAM [ 2166.553679] 0 pages
> HighMem/MovableOnly [ 2166.557509] 10060 pages reserved [ 2166.560717] 65536
> pages cma reserved
> [ 2166.564289] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents
> oom_score_adj name
> [ 2166.572814] [ 1231] 0 1231 821 407 5 3 0
> 0 sh
> [ 2166.581158] [ 1232] 102 1232 33662 1970 16 3 0
> 0 SDCMDCSvc
> [ 2166.590102] [ 1233] 101 1233 3271 666 6 3 0
> 0 SDCConfigChecke
> [ 2166.599568] [ 1234] 113 1234 1193 338 7 3 0
> 0 SDCSanitizer
> [ 2166.608781] [ 1237] 100 1237 21236 1792 11 4 0
> 0 SDCLogSvc
> [ 2166.617731] [ 1240] 1052 1240 25500 1766 12 3 0
> 0 SDCFilter2Svc_1
> [ 2166.627198] [ 1241] 1042 1241 25582 1803 14 3 0
> 0 SDCFilterSvc_10
> [ 2166.636665] [ 1242] 1041 1242 25581 1830 13 3 0
> 0 SDCFilter3Svc_1
> [ 2166.646129] [ 1243] 1044 1243 25582 1821 12 3 0
> 0 SDCFilter3Svc_1
> [ 2166.655591] [ 1244] 1046 1244 25502 1847 13 3 0
> 0 SDCFilter2Svc_1
> [ 2166.665059] [ 1245] 1035 1245 25451 1794 12 3 0
> 0 SDCFilterSvc_10
> [ 2166.674534] [ 1246] 1039 1246 25581 1811 13 3 0
> 0 SDCFilterSvc_10
> [ 2166.683996] [ 1247] 1055 1247 25499 1802 13 3 0
> 0 SDCFilter2Svc_1
> [ 2166.693458] [ 1248] 1063 1248 25514 1829 13 3 0
> 0 SDCFilterSvc_10
> [ 2166.702918] [ 1249] 1038 1249 25565 1808 12 3 0
> 0 SDCFilter3Svc_1
> [ 2166.712370] [ 1250] 1065 1250 25514 1785 13 4 0
> 0 SDCFilter3Svc_1
> [ 2166.721820] [ 1251] 1036 1251 25565 1808 13 3 0
> 0 SDCFilterSvc_10
> [ 2166.731286] [ 1252] 1059 1252 27089 3056 16 4 0
> 0 SDCFilter3Svc_1
> [ 2166.740749] [ 1253] 1057 1253 27089 3052 16 3 0
> 0 SDCFilterSvc_10
> [ 2166.750212] [ 1254] 1034 1254 23884 371 9 3 0
> 0 SDCInterDomainR
> [ 2166.759687] [ 1255] 103 1255 68688 3918 23 3 0
> 0 SDCEndpointSvc
> [ 2166.769077] [ 1256] 1048 1256 25453 1803 12 3 0
> 0 SDCFilterSvc_10
> [ 2166.778537] [ 1257] 1061 1257 25622 1832 13 3 0
> 0 SDCFilter2Svc_1
> [ 2166.788007] [ 1258] 1043 1258 25582 1794 12 3 0
> 0 SDCFilter2Svc_1
> [ 2166.797476] [ 1259] 1053 1259 25500 1849 12 3 0
> 0 SDCFilter3Svc_1
> [ 2166.806942] [ 1260] 1051 1260 25500 1799 13 3 0
> 0 SDCFilterSvc_10
> TACDS 0105 SDCMD[ 2166.816396] [ 1261] 1045 1261 25502 1840 13
> 3 0 0 SDCFilterSvc_10
> C ERROR SYST[ 2166.827229] [ 1262] 1047 1262 25502 1883 13
> 3 0 0 SDCFilter3Svc_1
> EM Applicat[ 2166.838043] [ 1263] 1040 1263 25581 1795 13
> 3 0 0 SDCFilter2Svc_1
> ion signature ve[ 2166.848880] [ 1264] 1050 1264 25453 1786 13
> 3 0 0 SDCFilter3Svc_1
> rify failed. [ 2166.859702] [ 1265] 1056 1265 25499 1794 12
> 3 0 0 SDCFilter3Svc_1
> [ 2166.870250] [ 1266] 1054 1266 25499 1821 13 3 0
> 0 SDCFilterSvc_10
> [ 2166.879709] [ 1267] 1064 1267 25514 1794 12 4 0
> 0 SDCFilter2Svc_1
>
> [ 2166.889156] [ 1268] 1037 1268 25565 1843 13 3 0
> 0 SDCFilter2Svc_1
> [ 2166.898764] [ 1269] 1058 1269 27089 3427 16 3 0
> 0 SDCFilter2Svc_1
> [ 2166.908230] [ 1270] 1033 1270 23883 698 10 3 0
> 0 SDCInterDomainT
> [ 2166.917702] [ 1271] 1049 1271 25453 1789 12 3 0
> 0 SDCFilter2Svc_1
> [ 2166.927169] [ 1272] 1060 1272 25622 1823 13 3 0
> 0 SDCFilterSvc_10
> [ 2166.936660] [ 1273] 1062 1273 25622 1863 13 3 0
> 0 SDCFilter3Svc_1
> [ 2166.946135] [ 1278] 105 1278 1185 310 6 3 0
> 0 SDCFileIntegrit
> [ 2166.955601] [ 2189] 0 2189 289638 79897 184 4 0
> 0 java
> [ 2166.964123] [ 2221] 101 2221 1 1 1 1 0
> 0 dmesg
> [ 2166.972727] Out of memory: Kill process 2189 (java) score 398 or
> sacrifice child [ 2166.980178] Killed process 2189 (java)
> total-vm:1158552kB, anon-rss:313340kB, file-rss:8kB, shmem-rss:6240kB
>
>
>
>
