[
https://issues.apache.org/jira/browse/SPARK-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kai zhao updated SPARK-27854:
-----------------------------
Environment:
Spark Version:1.6.2
HDP Version:2.5
JDK Version:1.8
OS Version:Redhat 7.3
Runtime Information
Java Home=/opt/jdk1.8.0_131/jre
Java Version=1.8.0_131 (Oracle Corporation)
Scala Version=version 2.10.5
Spark Properties
spark.app.id=application_1558686555626_0024
spark.app.name=org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
spark.driver.appUIAddress=http://172.17.3.2:4040
spark.driver.extraClassPath=/yinhai_platform/resources/spark_dep_jar/*
spark.driver.extraLibraryPath=/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
spark.driver.host=172.17.3.2
spark.driver.maxResultSize=16g
spark.driver.port=44591
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.initialExecutors=0
spark.dynamicAllocation.maxExecutors=200
spark.dynamicAllocation.minExecutors=0
spark.eventLog.dir=hdfs:///spark-history
spark.eventLog.enabled=true
spark.executor.cores=5
spark.executor.extraClassPath=/yinhai_platform/resources/spark_dep_jar/*
spark.executor.extraJavaOptions=-XX:MaxPermSize=10240m
spark.executor.extraLibraryPath=/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
spark.executor.id=driver
spark.executor.memory=16g
spark.externalBlockStore.folderName=spark-058bff7c-f76c-4a0e-86a3-b390f2f06d1a
spark.hadoop.cacheConf=false
spark.history.fs.logDirectory=hdfs:///spark-history
spark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider
spark.kryo.referenceTracking=false
spark.kryoserializer.buffer.max=1024m
spark.local.dir=/data/disk1/spark-local-dir
spark.master=yarn-client
spark.network.timeout=600s
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS=ambari-node-2
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES=http://ambari-node-2:8088/proxy/application_1558686555626_0024
spark.scheduler.allocation.file
/usr/hdp/current/spark-thriftserver/conf/spark-thrift-fairscheduler.xml
spark.scheduler.mode=FAIR
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.shuffle.managr=SORT
spark.shuffle.service.enabled=true
spark.shuffle.service.port=9339
spark.submit.deployMode=client
spark.ui.filters=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
spark.yarn.am.cores=5
spark.yarn.am.memory=16g
spark.yarn.queue=default
> [Spark-SQL] OOM when using unequal join sql
> --------------------------------------------
>
> Key: SPARK-27854
> URL: https://issues.apache.org/jira/browse/SPARK-27854
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.2
> Environment: Spark Version:1.6.2
> HDP Version:2.5
> JDK Version:1.8
> OS Version:Redhat 7.3
>
> Runtime Information
> Java Home=/opt/jdk1.8.0_131/jre
> Java Version=1.8.0_131 (Oracle Corporation)
> Scala Version=version 2.10.5
> Spark Properties
> spark.app.id=application_1558686555626_0024
> spark.app.name=org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
> spark.driver.appUIAddress=http://172.17.3.2:4040
> spark.driver.extraClassPath=/yinhai_platform/resources/spark_dep_jar/*
> spark.driver.extraLibraryPath=/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
> spark.driver.host=172.17.3.2
> spark.driver.maxResultSize=16g
> spark.driver.port=44591
> spark.dynamicAllocation.enabled=true
> spark.dynamicAllocation.initialExecutors=0
> spark.dynamicAllocation.maxExecutors=200
> spark.dynamicAllocation.minExecutors=0
> spark.eventLog.dir=hdfs:///spark-history
> spark.eventLog.enabled=true
> spark.executor.cores=5
> spark.executor.extraClassPath=/yinhai_platform/resources/spark_dep_jar/*
> spark.executor.extraJavaOptions=-XX:MaxPermSize=10240m
> spark.executor.extraLibraryPath=/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
> spark.executor.id=driver
> spark.executor.memory=16g
> spark.externalBlockStore.folderName=spark-058bff7c-f76c-4a0e-86a3-b390f2f06d1a
> spark.hadoop.cacheConf=false
> spark.history.fs.logDirectory=hdfs:///spark-history
> spark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider
> spark.kryo.referenceTracking=false
> spark.kryoserializer.buffer.max=1024m
> spark.local.dir=/data/disk1/spark-local-dir
> spark.master=yarn-client
> spark.network.timeout=600s
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS=ambari-node-2
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES=http://ambari-node-2:8088/proxy/application_1558686555626_0024
> spark.scheduler.allocation.file
> /usr/hdp/current/spark-thriftserver/conf/spark-thrift-fairscheduler.xml
> spark.scheduler.mode=FAIR
> spark.serializer=org.apache.spark.serializer.KryoSerializer
> spark.shuffle.managr=SORT
> spark.shuffle.service.enabled=true
> spark.shuffle.service.port=9339
> spark.submit.deployMode=client
> spark.ui.filters=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> spark.yarn.am.cores=5
> spark.yarn.am.memory=16g
> spark.yarn.queue=default
> Reporter: kai zhao
> Priority: Major
>
> I am using Spark 1.6.2 which is from hdp package.
> Reproduce Steps:
> 1.Start Spark thrift server Or write DataFrame Code
> 2.Execute SQL like:select * from table_a left join table_b on table_a.fieldA
> <= table_b.fieldB
> 3.Wait until the job is finished
>
> Actual Result:
> SQL won't execute failed With multipule task error:
> a)ExecutorLostFailure (executor 119 exited caused by one of the running
> tasks) Reason: Container marked as failed:
> b)java.lang.OutOfMemoryError: Java heap space
> Expect:
> SQL runs Successfully .
> I have tried every method on the Internet . But it still won't work
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
