Hi Guys,
I need help with Spark memory errors when executing ML pipelines.
The error that I see is:
16/02/02 20:34:17 INFO Executor: Executor is trying to kill task
32.0 in stage 32.0 (TID 3298)
16/02/02 20:34:17 INFO Executor: Executor is trying to kill task
12.0 in stage 32.0 (TID 3278)
16/02/02 20:34:39 INFO MemoryStore: ensureFreeSpace(2004728720)
called with curMem=296303415, maxMem=8890959790
16/02/02 20:34:39 INFO MemoryStore: Block taskresult_3298 stored
as bytes in memory (estimated size 1911.9 MB, free 6.1 GB)
16/02/02 20:34:39 ERROR CoarseGrainedExecutorBackend: RECEIVED
SIGNAL 15: SIGTERM
16/02/02 20:34:39 ERROR Executor: Exception in task 12.0 in
stage 32.0 (TID 3278)
java.lang.OutOfMemoryError:
Java heap space
at
java.util.Arrays.copyOf(Arrays.java:2271)
at
java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191)
at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:86)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:256)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
java.lang.Thread.run(Thread.java:745)
16/02/02 20:34:39 INFO DiskBlockManager: Shutdown hook called
16/02/02 20:34:39 INFO Executor: Finished task 32.0 in stage
32.0 (TID 3298). 2004728720 bytes result sent via BlockManager)
16/02/02 20:34:39 ERROR SparkUncaughtExceptionHandler: Uncaught
exception in thread Thread[Executor task launch worker-8,5,main]
java.lang.OutOfMemoryError:
Java heap space
at
java.util.Arrays.copyOf(Arrays.java:2271)
at
java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191)
at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:86)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:256)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
java.lang.Thread.run(Thread.java:745)
16/02/02 20:34:39 INFO ShutdownHookManager: Shutdown hook called
16/02/02 20:34:39 INFO MetricsSystemImpl: Stopping
azure-file-system metrics system...
16/02/02 20:34:39 INFO MetricsSinkAdapter: azurefs2 thread
interrupted.
16/02/02 20:34:39 INFO MetricsSystemImpl: azure-file-system
metrics system stopped.
16/02/02 20:34:39 INFO MetricsSystemImpl: azure-file-system
metrics system shutdown complete.
And …..
16/02/02 20:09:03 INFO impl.ContainerManagementProtocolProxy:
Opening proxy : 10.0.0.5:30050
16/02/02 20:33:51 INFO yarn.YarnAllocator: Completed container
container_1454421662639_0011_01_000005 (state: COMPLETE, exit status: -104)
16/02/02 20:33:51 WARN yarn.YarnAllocator: Container killed by YARN for
exceeding memory limits. 16.8 GB of 16.5 GB physical memory used. Consider
boosting spark.yarn.executor.memoryOverhead.
16/02/02 20:33:56 INFO yarn.YarnAllocator: Will request 1
executor containers, each with 2 cores and 16768 MB memory including 384 MB
overhead
16/02/02 20:33:56 INFO yarn.YarnAllocator: Container request
(host: Any, capability: <memory:16768, vCores:2>)
16/02/02 20:33:57 INFO yarn.YarnAllocator: Launching container
container_1454421662639_0011_01_000037 for on host 10.0.0.8
16/02/02 20:33:57 INFO yarn.YarnAllocator: Launching
ExecutorRunnable. driverUrl:
akka.tcp://[email protected]:47446/user/CoarseGrainedScheduler,
executorHostname: 10.0.0.8
16/02/02 20:33:57 INFO yarn.YarnAllocator: Received 1 containers
from YARN, launching executors on 1 of them.
I'll really appreciate any help here.
Thank you,
Stefan Panayotov, PhD
Home: 610-355-0919
Cell: 610-517-5586
email: [email protected]
[email protected]
[email protected]
