moon-fall opened a new issue #3695:
URL: https://github.com/apache/iceberg/issues/3695
flink version:1.12.1
iceberg version: 0.12.0
When flinkSink job runs on yarn serveral hours , the container is killed
because physical memory use beyond physical memory limits and report errors
like this:
> 2021-12-06 00:16:36,280 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Worker container_e10_1634010428940_125776_01_000003 is terminated. Diagnostics:
Container [pid=122806,containerID=container_e10_1634010428940_125776_01_000003]
is running beyond physical memory limits. Current usage: 4.0 GB of 4 GB
physical memory used; 6.2 GB of 8.4 GB virtual memory used. Killing container.
Dump of the process-tree for container_e10_1634010428940_125776_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 122920 122806 122806 122806 (java) 1926809 110163 6567804928 1048262
/usr/local/jdk8/bin/java -Xmx2383706830 -Xms2383706830
-XX:MaxDirectMemorySize=493921243 -XX:MaxMetaspaceSize=268435456
I try to avoid some related flink problems by running flink job like this:
./bin/flink run -m yarn-cluster -p 1 -yjm 2048m -ytm 8000m -yD
taskmanager.memory.jvm-overhead.min=2g -yD
taskmanager.memory.jvm-overhead.max=2g -yD
containerized.taskmanager.env.MALLOC_ARENA_MAX=1 -ys 1 -c icebergTest
icebergTest.jar
parameter containerized.taskmanager.env.MALLOC_ARENA_MAX=1 is to solve the
Arena "leak" in glibc like this
https://publib.boulder.ibm.com/httpserv/cookbook/Operating_Systems-Linux.html?lang=en
parameter taskmanager.memory.jvm-overhead.min=2g and
taskmanager.memory.jvm-overhead.max=2g is to increase the overhead memory and
this makes taskmanager run longger
and use FsStateBackend as flink State Backends
but it still report errors every serveral hours
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
