Hi Jena list.
I've been trying to load wikidata .nt file using tdbloader,
but I'm getting an out of memory error. Last year I was able
to do it with an older version of Jena (not sure if 3.14 or 3.17)
and a bit smaller wikidata file.
Now I'm using jena 4.1 and don't know if this is a bug or
if I did something incorrectly on the way.
I built jena into a docker image and ran
the following command:
>
> docker run -m 120g --cpus=12 -e JVM_ARGS="-Xms8G -Xmx100G" \
> --mount 'type=bind,src=/home/cmiranda/docker-mnts,dst=/external' \
> --name jena-indexing-wikidata cmirandat/jena \
> tdbloader --loc /external/jena-indexed-data
> /external/cache-preprocessing/latest-truthy-3apr2021.nt
Then after some time I had this error on the console:
> 16:39:07 INFO loader :: Elapsed: 63,991.91 seconds [2021/06/20
> 16:39:07 UTC]
> 16:39:10 INFO loader :: Add: 4,227,050,000 triples (Batch:
> 21,561 / Avg: 66,053)
> OpenJDK 64-Bit Server VM warning: INFO:
> os::commit_memory(0x00007f4962e40000, 262144, 0) failed; error='Not enough
> space' (errno=12)
> #
>
> # There is insufficient memory for the Java Runtime Environment to
> continue.
> # Native memory allocation (mmap) failed to map 262144 bytes for
> committing reserved memory.
>
> # An error report file with more information is saved as:
>
> # /jenadir/hs_err_pid8.log
For me it's intriguing that this happens near the 2^32 triples, maybe there
is an overflow somewhere? but this might just be a coincidence.
Some parts of the log file:
>
> # There is insufficient memory for the Java Runtime Environment to
> continue.
> # Native memory allocation (mmap) failed to map 262144 bytes for
> committing reserved memory.
> # Possible reasons:
> # The system is out of physical RAM or swap space
> # Possible solutions:
> # Reduce memory load on the system
> # Increase physical memory or swap space
> # Check if swap backing store is full
> # Decrease Java heap size (-Xmx/-Xms)
> # Decrease number of Java threads
> # Decrease Java thread stack sizes (-Xss)
> # Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> # Out of Memory Error (os_linux.cpp:2985), pid=8, tid=9
> #
> # JRE version: OpenJDK Runtime Environment (11.0.11+9) (build
> 11.0.11+9-post-Debian-1deb10u1)
> # Java VM: OpenJDK 64-Bit Server VM (11.0.11+9-post-Debian-1deb10u1, mixed
> mode, tiered, g1 gc, linux-amd64)
> # No core dump will be written. Core dumps have been disabled. To enable
> core dumping, try "ulimit -c unlimited" before starting Java again
> #
> ...
> VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
> [0x00007f986c015790] MetaspaceExpand_lock - owner thread:
> 0x00007f986c019800
> OutOfMemory and StackOverflow Exception counts:
> OutOfMemoryError java_heap_errors=2
> LinkageErrors=58
> Heap:
> garbage-first heap total 8388608K, used 198434K [0x00007f7f50000000,
> 0x00007f9850000000)
> region size 16384K, 1 young (16384K), 0 survivors (0K)
> Metaspace used 30487K, capacity 30976K, committed 30976K, reserved
> 32768K
> Heap Regions: E=young(eden), S=young(survivor), O=old,
> HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free,
> A=archive, TAMS=top-at-mark-start (previous, next)
> | 0|0x00007f7f50000000, 0x00007f7f50ffffd8, 0x00007f7f51000000| 99%| O|
> |TAMS 0x00007f7f50000000, 0x00007f7f50000000| Untracked
> | 1|0x00007f7f51000000, 0x00007f7f51b0e3a0, 0x00007f7f52000000| 69%| O|
> |TAMS 0x00007f7f51000000, 0x00007f7f51000000| Untracked
> ...
>
> Internal exceptions (20 events):
> Event: 1.847 Thread 0x00007f986c019800 Implicit null exception at
> 0x00007f985c39c0e8 to 0x00007f985c39c16a
> Event: 1.847 Thread 0x00007f986c019800 Implicit null exception at
> 0x00007f985c35ca2f to 0x00007f985c35ca99
> Event: 1.868 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f814eef2688}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> int)'> (0x00007f814eef2688) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 1.869 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f814ef03920}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> int, int)'> (0x00007f814ef03920) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 1.926 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f814d260688}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, int)'>
> (0x00007f814d260688) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 1.928 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f814d282b90}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, int, int)'>
> (0x00007f814d282b90) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 1.929 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f814d2a6608}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, int, int,
> java.lang.Object)'> (0x00007f814d2a6608) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 1.930 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f814d2cbca8}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, int, int,
> java.lang.Object, java.lang.Object)'> (0x00007f814d2cbca8) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 3.984 Thread 0x00007f986c674800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f814f821eb0}: 'void
> java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object)'> (0x00007f814f821eb0) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 3.986 Thread 0x00007f986c674800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f814f83ab20}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> int)'> (0x00007f814f83ab20) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 3.987 Thread 0x00007f986c674800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f814f840168}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
> java.lang.Object, int)'> (0x00007f814f840168) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 8.845 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f81ad9aec20}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object,
> java.lang.Object, int, int, int, java.lang.Object, java.lang.Object,
> java.lang.Object)'> (0x00007f81ad9aec20) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 8.845 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f81ad9b32d8}: 'java.lang.Object
> java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(java.lang.Object,
> int, int, int, java.lang.Object, java.lang.Object, java.lang.Object)'>
> (0x00007f81ad9b32d8) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 62.494 Thread 0x00007f986c019800 Implicit null exception at
> 0x00007f985c65c6d1 to 0x00007f985c65cd0c
> Event: 3249.491 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f829fd2ac48}: 'java.lang.Object
> java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object,
> java.lang.Object, int, int, int, java.lang.Object, int, java.lang.Object,
> java.lang.Object, java.lang.Object)'> (0x00007f829fd2ac48) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 3249.491 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f829fd2f748}: 'java.lang.Object
> java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(java.lang.Object,
> int, int, int, java.lang.Object, int, java.lang.Object, java.lang.Object,
> java.lang.Object)'> (0x00007f829fd2f748) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 3249.495 Thread 0x00007f986c019800 Exception <a
> 'java/lang/NoSuchMethodError'{0x00007f829fdafd30}: 'java.lang.Object
> java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, int,
> java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'>
> (0x00007f829fdafd30) thrown at
> [src/hotspot/share/interpreter/linkResolver.cpp, line 773]
> Event: 63996.243 Thread 0x00007f986c019800 Exception <a
> 'java/lang/OutOfMemoryError'{0x00007f83f336f048}: Map failed>
> (0x00007f83f336f048) thrown at [src/hotspot/share/prims/jni.cpp, line 636]
> Event: 64008.399 Thread 0x00007f986c019800 Exception <a
> 'java/lang/OutOfMemoryError'{0x00007f814f000ad8}: Map failed>
> (0x00007f814f000ad8) thrown at [src/hotspot/share/prims/jni.cpp, line 636]
> Event: 64008.415 Thread 0x00007f986c674800 Implicit null exception at
> 0x00007f985c7bb444 to 0x00007f985c7bbd30
> ...
>
> VM Arguments:
> jvm_args: -Xms8G -Xmx100G
> -Dlog4j.configurationFile=file:/RDFJenaServerBase/jenadir/jena/apache-jena/log4j2.properties
> java_command: tdb.tdbloader --loc /external/jena-indexed-data
> /external/cache-preprocessing/latest-truthy-3apr2021.nt
> java_class_path (initial):
> /RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-bundle-4.1.0-javadoc.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jetty-server-9.4.40.v20210413.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-tdb-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-dboe-base-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-shaded-guava-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-text-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-shacl-4.1.0-sources.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-tdb2-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-shacl-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/libthrift-0.14.1.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/lucene-sandbox-8.8.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-core-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/original-jena-fuseki-geosparql-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/log4j-core-2.14.1.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jetty-util-ajax-9.4.40.v20210413.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-fuseki-access-4.1.0-test-sources.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-mem-4.1.0-test-javadoc.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-fuseki-geosparql-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/simpleclient_common-0.9.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-integration-tests-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-arq-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-tdb-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jetty-util-9.4.40.v20210413.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-remote-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-integration-
> ...
>
>
> --------------- S Y S T E M ---------------
> OS:PRETTY_NAME="Debian GNU/Linux 10 (buster)"
> NAME="Debian GNU/Linux"
> VERSION_ID="10"
> VERSION="10 (buster)"
> VERSION_CODENAME=buster
> ID=debian
> HOME_URL="https://www.debian.org/";
> SUPPORT_URL="https://www.debian.org/support";
> BUG_REPORT_URL="https://bugs.debian.org/";
> uname:Linux 5.10.0-0.bpo.5-amd64 #1 SMP Debian 5.10.24-1~bpo10+1
> (2021-03-29) x86_64
> OS uptime: 51 days 16:39 hours
> libc:glibc 2.28 NPTL 2.28
> rlimit (soft/hard): STACK 8192k/infinity , CORE 0k/infinity , NPROC
> infinity/infinity , NOFILE 1048576/1048576 , AS infinity/infinity , CPU
> infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity ,
> MEMLOCK 64k/64k
> load average:4.41 3.31 3.10
> /proc/meminfo:
> MemTotal: 771614128 kB
> MemFree: 4588316 kB
> MemAvailable: 748598400 kB
> Buffers: 1148388 kB
> Cached: 733238364 kB
> SwapCached: 41300 kB
> Active: 332274448 kB
> Inactive: 415663672 kB
> Active(anon): 11943628 kB
> Inactive(anon): 1593996 kB
> Active(file): 320330820 kB
> Inactive(file): 414069676 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 67108860 kB
> SwapFree: 66484988 kB
> Dirty: 74908016 kB
> Writeback: 56 kB
> AnonPages: 13525108 kB
> Mapped: 256622612 kB
> Shmem: 2040 kB
> KReclaimable: 16044104 kB
> Slab: 17857072 kB
> SReclaimable: 16044104 kB
> SUnreclaim: 1812968 kB
> KernelStack: 8832 kB
> PageTables: 1042544 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 452915924 kB
> Committed_AS: 23765740 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 34012 kB
> VmallocChunk: 0 kB
> Percpu: 10624 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 13185024 kB
> ShmemHugePages: 0 kB
> ShmemPmdMapped: 0 kB
> FileHugePages: 0 kB
> FilePmdMapped: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> Hugetlb: 0 kB
> DirectMap4k: 620540 kB
> DirectMap2M: 102180864 kB
> DirectMap1G: 682622976 kB
> /sys/kernel/mm/transparent_hugepage/enabled:
> [always] madvise never
> /sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts
> parameter):
> always defer defer+madvise [madvise] never
> Process Memory:
> Virtual Size: 646288896K (peak: 646288896K)
> Resident Set Size: 269789276K (peak: 269789616K) (anon: 13301076K, file:
> 256488200K, shmem: 0K)
> Swapped out: 579680K
> C-Heap outstanding allocations: 47803K (may have wrapped)
> /proc/sys/kernel/threads-max (system-wide limit on the number of threads):
> 6028021
> /proc/sys/vm/max_map_count (maximum number of memory map areas a process
> may have):
> 65530
> /proc/sys/kernel/pid_max (system-wide limit on number of process
> identifiers):
> 32768
> container (cgroup) information:
> container_type: cgroupv1
> cpu_cpuset_cpus: 0-31
> cpu_memory_nodes: 0
> active_processor_count: 12
> cpu_quota: 1200000
> cpu_period: 100000
> cpu_shares: -1
> memory_limit_in_bytes: 128849018880
> memory_and_swap_limit_in_bytes: 257698037760
> memory_soft_limit_in_bytes: -1
> memory_usage_in_bytes: 128849002496
> memory_max_usage_in_bytes: 128849018880
> Xen hardware-assisted virtualization detected
> Steal ticks since vm start: 88915
> Steal ticks percentage since vm start: 0.000
> CPU:total 32 (initial active 12) (16 cores per cpu, 2 threads per core)
> family 6 model 85 stepping 4 microcode 0x2006a0a, cmov, cx8, fxsr, mmx,
> sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul,
> erms, rtm, 3dnowpref, lzcnt, ht, tsc, bmi1, bmi2, adx, fma
> CPU Model and flags from /proc/cpuinfo:
> model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
> pse36 clflush acpi mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm
> constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 fma
> cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave
> avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault
> invpcid_single pti intel_ppin ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1
> hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap
> clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves
> pku ospke md_clear flush_l1d
> Online cpus:
> 0-31
> Offline cpus:
> BIOS frequency limitation:
> <Not Available>
> Frequency switch latency (ns):
> <Not Available>
> Available cpu frequencies:
> <Not Available>
> Current governor:
> <Not Available>
> Core performance/turbo boost:
> <Not Available>
> Memory: 4k page, physical 125829120k(4k free), swap 67108860k(66484988k
> free)
> vm_info: OpenJDK 64-Bit Server VM (11.0.11+9-post-Debian-1deb10u1) for
> linux-amd64 JRE (11.0.11+9-post-Debian-1deb10u1), built on Apr 21 2021
> 15:55:49 by "unknown" with gcc 8.3.0
> END.
Sorry for the big logs, I don't know what is the best way to send those
here.
Any help or suggestions would be appreciated.
Cristobal M.
