[
https://issues.apache.org/jira/browse/SPARK-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226822#comment-15226822
]
Steve Johnston commented on SPARK-14389:
----------------------------------------
Here are the configuration settings we modify when bringing up the cluster:
||Classification||Key||Value||
|hadoop-env.export|ADC_RUNNING_ON_EMR_CLUSTER|1|
|hadoop-env.export|PYTHONPATH|/home/hadoop/aqa_root/src:/home/hadoop/aqa_root/src/algebraixlib:/usr/lib/spark/python|
|spark|maximizeResourceAllocation|true|
|spark-defaults|spark.driver.maxResultSize|0|
|spark-defaults|spark.executorEnv.PYTHONPATH|/home/hadoop/aqa_root/src:/home/hadoop/aqa_root/src/algebraixlib:/usr/lib/spark/python|
|spark-defaults|spark.driver.extraClassPath|/home/hadoop/aqa_root/working/java/*:/etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*|
|spark-defaults|spark.executorEnv.PYTHONHASHSEED|0|
|spark-defaults|spark.executorEnv.ADC_RUNNING_ON_EMR_CLUSTER|1|
|spark-defaults|spark.task.maxFailures|1|
|spark-defaults|spark.yarn.max.executor.failures|3|
|spark-defaults|spark.logConf|true|
|spark-env.export|PYTHONHASHSEED|0|
|spark-env.export|SPARK_YARN_USER_ENV|"PYTHONHASHSEED=0"|
|spark-env.export|ADC_RUNNING_ON_EMR_CLUSTER|1|
|spark-env.export|PYTHONPATH|/home/hadoop/aqa_root/src:/home/hadoop/aqa_root/src/algebraixlib:/usr/lib/spark/python|
|spark-env.export|PYSPARK_PYTHON|/usr/bin/python3.4|
> OOM during BroadcastNestedLoopJoin
> ----------------------------------
>
> Key: SPARK-14389
> URL: https://issues.apache.org/jira/browse/SPARK-14389
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.0
> Environment: OS: Amazon Linux AMI 2015.09
> EMR: 4.3.0
> Hadoop: Amazon 2.7.1
> Spark 1.6.0
> Ganglia 3.7.2
> Master: m3.xlarge
> Core: m3.xlarge
> m3.xlarge: 4 CPU, 15GB mem, 2x40GB SSD
> Reporter: Steve Johnston
> Attachments: lineitem.tbl, sample_script.py, stdout.txt
>
>
> When executing attached sample_script.py in client mode with a single
> executor an exception occurs, "java.lang.OutOfMemoryError: Java heap space",
> during the self join of a small table, TPC-H lineitem generated for a 1M
> dataset. Also see execution log stdout.txt attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
