Re: tdbloader out of memory error

Mon, 21 Jun 2021 12:09:55 -0700 

Hi Cristóbal,

https://jena.apache.org/documentation/tdb2/tdb2_cmds.html
This was added in the last few days to the documentation - it applies to TDB1 as well. 

"""
Loading very large datasets (like Wikidata) with tdb2.tdbloader may sometimes on linux configurations fail with errors like: 

Native memory allocation (mmap) failed to map NNNNNN bytes for
committing reserved memory.
This can be avoided by adding a larger value to the vm.max_map_count option. The command sudo sysctl -w vm.max_map_count=262144 updates the value for your current session, or you can persist the change by editing the value in /etc/sysctl.conf or in /etc/sysctl.d/* override files if available. 
"""

Looks like you have
 /proc/sys/vm/max_map_count = 65530
from the stats below so it could be this.
errno=12 is an error from the OS and may result from other reasons like not enough phsical RAM for the docker container. 


--
Unrelated - you don't need -Xmx100G -- in fact, 8G should be enough. This leaves more space for the memory mapped files. 

    Andy

On 21/06/2021 17:22, Cristóbal Miranda wrote:

Hi Jena list.

I've been trying to load wikidata .nt file using tdbloader,
but I'm getting an out of memory error. Last year I was able
to do it with an older version of Jena (not sure if 3.14 or 3.17)
and a bit smaller wikidata file.

Now I'm using jena 4.1 and don't know if this is a bug or
if I did something incorrectly on the way.

I built jena into a docker image and ran
the following command:



docker run -m 120g --cpus=12 -e JVM_ARGS="-Xms8G -Xmx100G" \
  --mount 'type=bind,src=/home/cmiranda/docker-mnts,dst=/external' \
  --name jena-indexing-wikidata cmirandat/jena \
tdbloader --loc /external/jena-indexed-data
/external/cache-preprocessing/latest-truthy-3apr2021.nt



Then after some time I had this error on the console:



16:39:07 INFO  loader          ::   Elapsed: 63,991.91 seconds [2021/06/20
16:39:07 UTC]
16:39:10 INFO  loader          :: Add: 4,227,050,000 triples (Batch:
21,561 / Avg: 66,053)
OpenJDK 64-Bit Server VM warning: INFO:
os::commit_memory(0x00007f4962e40000, 262144, 0) failed; error='Not enough
space' (errno=12)
#

# There is insufficient memory for the Java Runtime Environment to
continue.
# Native memory allocation (mmap) failed to map 262144 bytes for
committing reserved memory.

# An error report file with more information is saved as:

# /jenadir/hs_err_pid8.log



For me it's intriguing that this happens near the 2^32 triples, maybe there
is an overflow somewhere? but this might just be a coincidence.

Some parts of the log file:



# There is insufficient memory for the Java Runtime Environment to
continue.
# Native memory allocation (mmap) failed to map 262144 bytes for
committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2985), pid=8, tid=9
#
# JRE version: OpenJDK Runtime Environment (11.0.11+9) (build
11.0.11+9-post-Debian-1deb10u1)
# Java VM: OpenJDK 64-Bit Server VM (11.0.11+9-post-Debian-1deb10u1, mixed
mode, tiered, g1 gc, linux-amd64)
# No core dump will be written. Core dumps have been disabled. To enable
core dumping, try "ulimit -c unlimited" before starting Java again
#
...
VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007f986c015790] MetaspaceExpand_lock - owner thread:
0x00007f986c019800
OutOfMemory and StackOverflow Exception counts:
OutOfMemoryError java_heap_errors=2
LinkageErrors=58
Heap:
  garbage-first heap   total 8388608K, used 198434K [0x00007f7f50000000,
0x00007f9850000000)
   region size 16384K, 1 young (16384K), 0 survivors (0K)
  Metaspace       used 30487K, capacity 30976K, committed 30976K, reserved
32768K
Heap Regions: E=young(eden), S=young(survivor), O=old,
HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free,
A=archive, TAMS=top-at-mark-start (previous, next)
|   0|0x00007f7f50000000, 0x00007f7f50ffffd8, 0x00007f7f51000000| 99%| O|
  |TAMS 0x00007f7f50000000, 0x00007f7f50000000| Untracked
|   1|0x00007f7f51000000, 0x00007f7f51b0e3a0, 0x00007f7f52000000| 69%| O|
  |TAMS 0x00007f7f51000000, 0x00007f7f51000000| Untracked
...

Internal exceptions (20 events):
Event: 1.847 Thread 0x00007f986c019800 Implicit null exception at
0x00007f985c39c0e8 to 0x00007f985c39c16a
Event: 1.847 Thread 0x00007f986c019800 Implicit null exception at
0x00007f985c35ca2f to 0x00007f985c35ca99
Event: 1.868 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f814eef2688}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
int)'> (0x00007f814eef2688) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 1.869 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f814ef03920}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
int, int)'> (0x00007f814ef03920) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 1.926 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f814d260688}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, int)'>
(0x00007f814d260688) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 1.928 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f814d282b90}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, int, int)'>
(0x00007f814d282b90) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 1.929 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f814d2a6608}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, int, int,
java.lang.Object)'> (0x00007f814d2a6608) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 1.930 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f814d2cbca8}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, int, int,
java.lang.Object, java.lang.Object)'> (0x00007f814d2cbca8) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 3.984 Thread 0x00007f986c674800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f814f821eb0}: 'void
java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object)'> (0x00007f814f821eb0) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 3.986 Thread 0x00007f986c674800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f814f83ab20}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
int)'> (0x00007f814f83ab20) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 3.987 Thread 0x00007f986c674800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f814f840168}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object,
java.lang.Object, int)'> (0x00007f814f840168) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 8.845 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f81ad9aec20}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object,
java.lang.Object, int, int, int, java.lang.Object, java.lang.Object,
java.lang.Object)'> (0x00007f81ad9aec20) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 8.845 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f81ad9b32d8}: 'java.lang.Object
java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(java.lang.Object,
int, int, int, java.lang.Object, java.lang.Object, java.lang.Object)'>
(0x00007f81ad9b32d8) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 62.494 Thread 0x00007f986c019800 Implicit null exception at
0x00007f985c65c6d1 to 0x00007f985c65cd0c
Event: 3249.491 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f829fd2ac48}: 'java.lang.Object
java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object,
java.lang.Object, int, int, int, java.lang.Object, int, java.lang.Object,
java.lang.Object, java.lang.Object)'> (0x00007f829fd2ac48) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 3249.491 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f829fd2f748}: 'java.lang.Object
java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(java.lang.Object,
int, int, int, java.lang.Object, int, java.lang.Object, java.lang.Object,
java.lang.Object)'> (0x00007f829fd2f748) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 3249.495 Thread 0x00007f986c019800 Exception <a
'java/lang/NoSuchMethodError'{0x00007f829fdafd30}: 'java.lang.Object
java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, int,
java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'>
(0x00007f829fdafd30) thrown at
[src/hotspot/share/interpreter/linkResolver.cpp, line 773]
Event: 63996.243 Thread 0x00007f986c019800 Exception <a
'java/lang/OutOfMemoryError'{0x00007f83f336f048}: Map failed>
(0x00007f83f336f048) thrown at [src/hotspot/share/prims/jni.cpp, line 636]
Event: 64008.399 Thread 0x00007f986c019800 Exception <a
'java/lang/OutOfMemoryError'{0x00007f814f000ad8}: Map failed>
(0x00007f814f000ad8) thrown at [src/hotspot/share/prims/jni.cpp, line 636]
Event: 64008.415 Thread 0x00007f986c674800 Implicit null exception at
0x00007f985c7bb444 to 0x00007f985c7bbd30
  ...

VM Arguments:
jvm_args: -Xms8G -Xmx100G
-Dlog4j.configurationFile=file:/RDFJenaServerBase/jenadir/jena/apache-jena/log4j2.properties
java_command: tdb.tdbloader --loc /external/jena-indexed-data
/external/cache-preprocessing/latest-truthy-3apr2021.nt
java_class_path (initial):
/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-bundle-4.1.0-javadoc.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jetty-server-9.4.40.v20210413.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-tdb-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-dboe-base-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-shaded-guava-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-text-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-shacl-4.1.0-sources.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-tdb2-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-shacl-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/libthrift-0.14.1.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/lucene-sandbox-8.8.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-core-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/original-jena-fuseki-geosparql-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/log4j-core-2.14.1.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jetty-util-ajax-9.4.40.v20210413.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-fuseki-access-4.1.0-test-sources.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-mem-4.1.0-test-javadoc.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-fuseki-geosparql-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/simpleclient_common-0.9.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-integration-tests-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-arq-4.1.0-tests.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-tdb-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jetty-util-9.4.40.v20210413.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-jdbc-driver-remote-4.1.0.jar:/RDFJenaServerBase/jenadir/jena/apache-jena/lib/jena-integration-
...


---------------  S Y S T E M  ---------------
OS:PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/";
SUPPORT_URL="https://www.debian.org/support";
BUG_REPORT_URL="https://bugs.debian.org/";
uname:Linux 5.10.0-0.bpo.5-amd64 #1 SMP Debian 5.10.24-1~bpo10+1
(2021-03-29) x86_64
OS uptime: 51 days 16:39 hours
libc:glibc 2.28 NPTL 2.28
rlimit (soft/hard): STACK 8192k/infinity , CORE 0k/infinity , NPROC
infinity/infinity , NOFILE 1048576/1048576 , AS infinity/infinity , CPU
infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity ,
MEMLOCK 64k/64k
load average:4.41 3.31 3.10
/proc/meminfo:
MemTotal:       771614128 kB
MemFree:         4588316 kB
MemAvailable:   748598400 kB
Buffers:         1148388 kB
Cached:         733238364 kB
SwapCached:        41300 kB
Active:         332274448 kB
Inactive:       415663672 kB
Active(anon):   11943628 kB
Inactive(anon):  1593996 kB
Active(file):   320330820 kB
Inactive(file): 414069676 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      67108860 kB
SwapFree:       66484988 kB
Dirty:          74908016 kB
Writeback:            56 kB
AnonPages:      13525108 kB
Mapped:         256622612 kB
Shmem:              2040 kB
KReclaimable:   16044104 kB
Slab:           17857072 kB
SReclaimable:   16044104 kB
SUnreclaim:      1812968 kB
KernelStack:        8832 kB
PageTables:      1042544 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    452915924 kB
Committed_AS:   23765740 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       34012 kB
VmallocChunk:          0 kB
Percpu:            10624 kB
HardwareCorrupted:     0 kB
AnonHugePages:  13185024 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      620540 kB
DirectMap2M:    102180864 kB
DirectMap1G:    682622976 kB
/sys/kernel/mm/transparent_hugepage/enabled:
[always] madvise never
/sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts
parameter):
always defer defer+madvise [madvise] never
Process Memory:
Virtual Size: 646288896K (peak: 646288896K)
Resident Set Size: 269789276K (peak: 269789616K) (anon: 13301076K, file:
256488200K, shmem: 0K)
Swapped out: 579680K
C-Heap outstanding allocations: 47803K (may have wrapped)
/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
6028021
/proc/sys/vm/max_map_count (maximum number of memory map areas a process
may have):
65530
/proc/sys/kernel/pid_max (system-wide limit on number of process
identifiers):
32768
container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-31
cpu_memory_nodes: 0
active_processor_count: 12
cpu_quota: 1200000
cpu_period: 100000
cpu_shares: -1
memory_limit_in_bytes: 128849018880
memory_and_swap_limit_in_bytes: 257698037760
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 128849002496
memory_max_usage_in_bytes: 128849018880
Xen hardware-assisted virtualization detected
Steal ticks since vm start: 88915
Steal ticks percentage since vm start:  0.000
CPU:total 32 (initial active 12) (16 cores per cpu, 2 threads per core)
family 6 model 85 stepping 4 microcode 0x2006a0a, cmov, cx8, fxsr, mmx,
sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul,
erms, rtm, 3dnowpref, lzcnt, ht, tsc, bmi1, bmi2, adx, fma
CPU Model and flags from /proc/cpuinfo:
model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush acpi mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm
constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 fma
cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave
avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault
invpcid_single pti intel_ppin ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1
hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap
clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves
pku ospke md_clear flush_l1d
Online cpus:
0-31
Offline cpus:
BIOS frequency limitation:
<Not Available>
Frequency switch latency (ns):
<Not Available>
Available cpu frequencies:
<Not Available>
Current governor:
<Not Available>
Core performance/turbo boost:
<Not Available>
Memory: 4k page, physical 125829120k(4k free), swap 67108860k(66484988k
free)
vm_info: OpenJDK 64-Bit Server VM (11.0.11+9-post-Debian-1deb10u1) for
linux-amd64 JRE (11.0.11+9-post-Debian-1deb10u1), built on Apr 21 2021
15:55:49 by "unknown" with gcc 8.3.0
END.



Sorry for the big logs, I don't know what is the best way to send those
here.


Any help or suggestions would be appreciated.

Cristobal M.

Reply via email to