Hello all,
I am trying to compile the DataProcessor using the Apache DFDL VMF
schema from the DI2E site. I am trying to run DFDL on a small embedded device
with an ARM processor containing 2G of memory. I tried setting the max and min
heap space values for the JVM to several different values 128, 256, 512 and
each time the device panicked and ran out of memory. The very first time I
tried to execute DFDL I did not set the min and max JVM heap values and
received the Out of heap space exception.
GIven the magnitude of the schema, is it even possible to run DFDL on
such a limited device ?
Can I precompile the parser and load it onto the embedded device prior to
executing DFDL?
Any suggestions are greatly appreciated!
Here is some log info:
/usr/lib/jvm/image-aarch64/bin/java -jar MessageTestApp.jar vmb vmf.dfdl.xsd
T
imestamp prior to entering COMPILE SCHEMA: 1574365652553ms
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.apache.daffodil.dsom.TermFactory$.apply(ModelGroup.scala:11[
1554.981162] audit: type=1400 audit(1574365826.688:1601): avc: denied {
getsched } for pid=1910 comm="java" scontext=system_u:system_r:busybox_t
tcontext=system_u:system_r:busybox_t tclass=process permissive=1
6)
at org.apache.daffodil.dsom[ 1555.003192] audit: type=1400
audit(1574365826.708:1602): avc: denied { remove_name } for pid=1910
comm="java" name="1910" dev="rootfs" ino=21993
scontext=system_u:system_r:busybox_t tcontext=system_u:object_r:tmp_t
tclass=dir permissive=1
.GroupDefLike.$a[ 1555.025259] audit: type=1400 audit(1574365826.708:1603):
avc: denied { unlink } for pid=1910 comm="java" name="1910" dev="rootfs"
ino=21993 scontext=system_u:system_r:busybox_t tcontext=system_u:object_r:tmp_t
tclass=file permissive=1
nonfun$groupMembers$2(GroupDef.scala:77)
at
org.apache.daffodil.dsom.GroupDefLike$$Lambda$471/0x0000007f5eb0f040.apply(Unknown
Source)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$Lambda$38/0x0000007f5f7fa040.apply(Unknown
Source)
at scala.collection.immutable.List.foreach(List.scala:389)
at scala.collection.TraversableLike.map(TraversableLike.scala:234)
at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
at scala.collection.immutable.List.map(List.scala:295)
at
org.apache.daffodil.dsom.GroupDefLike.$anonfun$groupMembers$1(GroupDef.scala:75)
at
org.apache.daffodil.dsom.GroupDefLike$$Lambda$467/0x0000007f5eb06040.apply(Unknown
Source)
at
org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.liftedTree1$1(OOLAG.scala:549)
at
org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.valueAsAny$lzycompute(OOLAG.scala:547)
at
org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.valueAsAny(OOLAG.scala:544)
at
org.apache.daffodil.oolag.OOLAG$OOLAGValue.value$lzycompute(OOLAG.scala:594)
at org.apache.daffodil.oolag.OOLAG$OOLAGValue.value(OOLAG.scala:594)
at org.apache.daffodil.dsom.GroupDefLike.groupMembers(GroupDef.scala:72)
at
org.apache.daffodil.dsom.GroupDefLike.groupMembers$(GroupDef.scala:72)
at
org.apache.daffodil.dsom.Sequence.groupMembers$lzycompute(SequenceGroup.scala:320)
at
org.apache.daffodil.dsom.Sequence.groupMembers(SequenceGroup.scala:320)
at
org.apache.daffodil.dsom.ModelGroup.$anonfun$elementChildren$1(ModelGroup.scala:186)
at
org.apache.daffodil.dsom.ModelGroup$$Lambda$466/0x0000007f5eb05040.apply(Unknown
Source)
at
org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.liftedTree1$1(OOLAG.scala:549)
at
org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.valueAsAny$lzycompute(OOLAG.scala:547)
at
org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.valueAsAny(OOLAG.scala:544)
at
org.apache.daffodil.oolag.OOLAG$OOLAGValue.value$lzycompute(OOLAG.scala:594)
at org.apache.daffodil.oolag.OOLAG$OOLAGValue.value(OOLAG.scala:594)
at
org.apache.daffodil.dsom.ModelGroup.elementChildren$lzycompute(ModelGroup.scala:185)
at
org.apache.daffodil.dsom.ModelGroup.elementChildren(ModelGroup.scala:185)
at
org.apache.daffodil.dsom.ElementBase.$anonfun$elementChildren$1(ElementBase.scala:612)
at
org.apache.daffodil.dsom.ElementBase$$Lambda$457/0x0000007f5ea4e840.apply(Unknown
Source)
at
org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.liftedTree1$1(OOLAG.scala:549)
#
/usr/lib/jvm/image-aarch64/bin/java -Xmx256m -jar MessageTestApp.jar vmb
vmf.dfdl.xsd
Timestamp prior to entering COMPILE SCHEMA: 1574366380842ms
PERF_SYS System load and free memory [System=0.7% User=15.1% IRQ/IO=0.0%
Idle=84.2%; Uptime 20:00:13 up 35 min, load average: 1.05, 0.62, 0.40; Mem
tot=778960,used=546256,free=232704]
TACDS 0101 CFG_CHKR INFO PERF_SYS Flash filesystem usage [18%]
[ 2166.210992] dmesg invoked oom-killer:
gfp_mask=0x16080c0(GFP_KERNEL|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null),
order=0, oom_score_adj=0
[ 2166.223381] CPU: 1 PID: 2221 Comm: dmesg Tainted: G O
4.14.0-okl4-hyp #11
[ 2166.231269] Hardware name: ZynqMP Ultrazed Split1 Hybrid Linux Cell (OKL4
Guest) (DT)
[ 2166.239060] Call trace:
[ 2166.241509] [<ffffff8008088888>] dump_backtrace+0x0/0x370
[ 2166.246881] [<ffffff8008088c0c>] show_stack+0x14/0x20
[ 2166.251912] [<ffffff800859fea8>] dump_stack+0x98/0xb8
[ 2166.256941] [<ffffff8008134a2c>] dump_header.isra.6+0x7c/0x1a8
[ 2166.262746] [<ffffff80081340bc>] oom_kill_process+0x2cc/0x518
[ 2166.268464] [<ffffff8008134634>] out_of_memory+0xbc/0x3d0
[ 2166.273840] [<ffffff80081394dc>] __alloc_pages_nodemask+0x9fc/0xa78
[ 2166.280078] [<ffffff800815bd88>] __pte_alloc+0x30/0x148
[ 2166.285279] [<ffffff8008167f14>] move_page_tables+0x5ac/0x678
[ 2166.290999] [<ffffff80081973c0>] shift_arg_pages+0x88/0x198
[ 2166.296544] [<ffffff8008197694>] setup_arg_pages+0x1c4/0x1e0
[ 2166.302181] [<ffffff80081e4b6c>] load_elf_binary+0x3cc/0x1070
[ 2166.307896] [<ffffff8008198b6c>] search_binary_handler+0xc4/0x270
[ 2166.313962] [<ffffff800819951c>] do_execveat_common.isra.13+0x4b4/0x5e0
[ 2166.320546] [<ffffff8008199818>] SyS_execve+0x38/0x48
[ 2166.325573] Exception stack(0xffffff800b6a3ec0 to 0xffffff800b6a4000)
[ 2166.331985] 3ec0: 000000000040a068 0000007fea2d7f30 0000007fea2da568
0000000000000000
[ 2166.339777] 3ee0: 0000000000000001 000000000040aab5 0000007fea2d9f1c
0000000000000000
[ 2166.347571] 3f00: 00000000000000dd 0000007fea2d9f30 0101010101010101
0000000000000018
[ 2166.355364] 3f20: 0000000000002090 0000000000000000 6d68732f7665642f
000000000000007f
[ 2166.363158] 3f40: 0000000000002090 000000000041c2e0 0000000000040e00
0000007fea2d7f40
[ 2166.370951] 3f60: 0000000000000003 0000007fea2d7f30 0000000000000400
0000007fea2d7f30
[ 2166.378744] 3f80: 000000000040a068 0000000000000000 0000000000000000
0000000000000012
[ 2166.386538] 3fa0: 0000000010c6fc48 0000007fea2d9f60 0000007f96ed1de0
0000007fea2d7ed0
[ 2166.394331] 3fc0: 0000007f96ed1a08 00000000a0000000 000000000040a068
00000000000000dd
[ 2166.402123] 3fe0: 0000000000000000 0000000000000000 0000000000000000
0000000000000000
[ 2166.409920] [<ffffff8008082fb0>] el0_svc_naked+0x24/0x28
[ 2166.415296] Mem-Info:
[ 2166.417587] active_anon:134711 inactive_anon:14 isolated_anon:0
[ 2166.417587] active_file:0 inactive_file:0 isolated_file:0
[ 2166.417587] unevictable:0 dirty:0 writeback:0 unstable:0
[ 2166.417587] slab_reclaimable:672 slab_unreclaimable:2655
[ 2166.417587] mapped:5875 shmem:43049 pagetables:684 bounce:0
[ 2166.417587] free:54146 free_pcp:178 free_cma:48514
[ 2166.450137] Node 0 active_anon:538844kB inactive_anon:56kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:23500kB dirty:0kB writeback:0kB shmem:172196kB shmem_thp: 0kB
shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB
all_unreclaimable? yes
[ 2166.476905] DMA free:216584kB min:22528kB low:28160kB high:33792kB
active_anon:538908kB inactive_anon:56kB active_file:0kB inactive_file:0kB
unevictable:0kB writepending:0kB present:819200kB managed:778960kB mlocked:0kB
kernel_stack:2736kB pagetables:2736kB bounce:0kB free_pcp:712kB local_pcp:360kB
free_cma:194056kB
[ 2166.504783] lowmem_reserve[]: 0 0 0
[ 2166.508283] DMA: 50*4kB (EHC) 40*8kB (UEH) 14*16kB (UEH) 11*32kB (UEHC)
5*64kB (UE) 5*128kB (EHC) 4*256kB (EHC) 3*512kB (UE) 5*1024kB (UEC) 3*2048kB
(UE) 49*4096kB (EC) = 216584kB
[ 2166.524258] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=2048kB
[ 2166.532675] 43049 total pagecache pages
[ 2166.536510] 0 pages in swap cache
[ 2166.539824] Swap cache stats: add 0, delete 0, find 0/0
[ 2166.545050] Free swap = 0kB
[ 2166.547911] Total swap = 0kB
[ 2166.550800] 204800 pages RAM
[ 2166.553679] 0 pages HighMem/MovableOnly
[ 2166.557509] 10060 pages reserved
[ 2166.560717] 65536 pages cma reserved
[ 2166.564289] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents
oom_score_adj name
[ 2166.572814] [ 1231] 0 1231 821 407 5 3 0
0 sh
[ 2166.581158] [ 1232] 102 1232 33662 1970 16 3 0
0 SDCMDCSvc
[ 2166.590102] [ 1233] 101 1233 3271 666 6 3 0
0 SDCConfigChecke
[ 2166.599568] [ 1234] 113 1234 1193 338 7 3 0
0 SDCSanitizer
[ 2166.608781] [ 1237] 100 1237 21236 1792 11 4 0
0 SDCLogSvc
[ 2166.617731] [ 1240] 1052 1240 25500 1766 12 3 0
0 SDCFilter2Svc_1
[ 2166.627198] [ 1241] 1042 1241 25582 1803 14 3 0
0 SDCFilterSvc_10
[ 2166.636665] [ 1242] 1041 1242 25581 1830 13 3 0
0 SDCFilter3Svc_1
[ 2166.646129] [ 1243] 1044 1243 25582 1821 12 3 0
0 SDCFilter3Svc_1
[ 2166.655591] [ 1244] 1046 1244 25502 1847 13 3 0
0 SDCFilter2Svc_1
[ 2166.665059] [ 1245] 1035 1245 25451 1794 12 3 0
0 SDCFilterSvc_10
[ 2166.674534] [ 1246] 1039 1246 25581 1811 13 3 0
0 SDCFilterSvc_10
[ 2166.683996] [ 1247] 1055 1247 25499 1802 13 3 0
0 SDCFilter2Svc_1
[ 2166.693458] [ 1248] 1063 1248 25514 1829 13 3 0
0 SDCFilterSvc_10
[ 2166.702918] [ 1249] 1038 1249 25565 1808 12 3 0
0 SDCFilter3Svc_1
[ 2166.712370] [ 1250] 1065 1250 25514 1785 13 4 0
0 SDCFilter3Svc_1
[ 2166.721820] [ 1251] 1036 1251 25565 1808 13 3 0
0 SDCFilterSvc_10
[ 2166.731286] [ 1252] 1059 1252 27089 3056 16 4 0
0 SDCFilter3Svc_1
[ 2166.740749] [ 1253] 1057 1253 27089 3052 16 3 0
0 SDCFilterSvc_10
[ 2166.750212] [ 1254] 1034 1254 23884 371 9 3 0
0 SDCInterDomainR
[ 2166.759687] [ 1255] 103 1255 68688 3918 23 3 0
0 SDCEndpointSvc
[ 2166.769077] [ 1256] 1048 1256 25453 1803 12 3 0
0 SDCFilterSvc_10
[ 2166.778537] [ 1257] 1061 1257 25622 1832 13 3 0
0 SDCFilter2Svc_1
[ 2166.788007] [ 1258] 1043 1258 25582 1794 12 3 0
0 SDCFilter2Svc_1
[ 2166.797476] [ 1259] 1053 1259 25500 1849 12 3 0
0 SDCFilter3Svc_1
[ 2166.806942] [ 1260] 1051 1260 25500 1799 13 3 0
0 SDCFilterSvc_10
TACDS 0105 SDCMD[ 2166.816396] [ 1261] 1045 1261 25502 1840 13
3 0 0 SDCFilterSvc_10
C ERROR SYST[ 2166.827229] [ 1262] 1047 1262 25502 1883 13
3 0 0 SDCFilter3Svc_1
EM Applicat[ 2166.838043] [ 1263] 1040 1263 25581 1795 13
3 0 0 SDCFilter2Svc_1
ion signature ve[ 2166.848880] [ 1264] 1050 1264 25453 1786 13
3 0 0 SDCFilter3Svc_1
rify failed. [ 2166.859702] [ 1265] 1056 1265 25499 1794 12
3 0 0 SDCFilter3Svc_1
[ 2166.870250] [ 1266] 1054 1266 25499 1821 13 3 0
0 SDCFilterSvc_10
[ 2166.879709] [ 1267] 1064 1267 25514 1794 12 4 0
0 SDCFilter2Svc_1
[ 2166.889156] [ 1268] 1037 1268 25565 1843 13 3 0
0 SDCFilter2Svc_1
[ 2166.898764] [ 1269] 1058 1269 27089 3427 16 3 0
0 SDCFilter2Svc_1
[ 2166.908230] [ 1270] 1033 1270 23883 698 10 3 0
0 SDCInterDomainT
[ 2166.917702] [ 1271] 1049 1271 25453 1789 12 3 0
0 SDCFilter2Svc_1
[ 2166.927169] [ 1272] 1060 1272 25622 1823 13 3 0
0 SDCFilterSvc_10
[ 2166.936660] [ 1273] 1062 1273 25622 1863 13 3 0
0 SDCFilter3Svc_1
[ 2166.946135] [ 1278] 105 1278 1185 310 6 3 0
0 SDCFileIntegrit
[ 2166.955601] [ 2189] 0 2189 289638 79897 184 4 0
0 java
[ 2166.964123] [ 2221] 101 2221 1 1 1 1 0
0 dmesg
[ 2166.972727] Out of memory: Kill process 2189 (java) score 398 or sacrifice
child
[ 2166.980178] Killed process 2189 (java) total-vm:1158552kB,
anon-rss:313340kB, file-rss:8kB, shmem-rss:6240kB
