Just a thought; this has worked for me before on standalone client with a similar OOM error in a driver thread. Try setting: export SPARK_DAEMON_MEMORY=4G #or whatever size you can afford on your machine in your environment/spark-env.sh before running spark-submit. Mike
On 9/2/15, ankit tyagi <ankittyagi.mn...@gmail.com> wrote: > Hi All, > > I am using spark-sql 1.3.1 with hadoop 2.4.0 version. I am running sql > query against parquet files and wanted to save result on s3 but looks like > https://issues.apache.org/jira/browse/SPARK-2984 problem still coming while > saving data to s3. > > Hence Now i am saving result on hdfs and with the help > of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil > in onApplicationEnd method. But my job is getting failed with OOM in spark > driver. > > *5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking each > executor to shut down* > *15/09/02 04:17:59 INFO > scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: > OutputCommitCoordinator stopped!* > *Exception in thread "Reporter" * > *Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "Reporter"* > *Exception in thread "SparkListenerBus" * > *Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "SparkListenerBus"* > *Exception in thread "Driver" * > *Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "Driver"* > > > Strage part is, result is getting saved on HDFS but while copying file job > is getting failed. size of file is under 1MB. > > Any help or leads would be appreciated. > -- Thanks, Mike --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
