[
https://issues.apache.org/jira/browse/SPARK-29202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-29202.
-----------------------------------
Fix Version/s: 3.0.0
Assignee: Sandeep Katta
Resolution: Fixed
This is resolved via https://github.com/apache/spark/pull/25889
> --driver-java-options are not passed to driver process in yarn client mode
> --------------------------------------------------------------------------
>
> Key: SPARK-29202
> URL: https://issues.apache.org/jira/browse/SPARK-29202
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 3.0.0
> Reporter: Sandeep Katta
> Assignee: Sandeep Katta
> Priority: Major
> Fix For: 3.0.0
>
>
> Run the below command
> ./bin/spark-sql --master yarn
> --driver-java-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5555"
>
> In Spark 2.3.3
> /opt/softwares/Java/jdk1.8.0_211/bin/java -cp
> /opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/conf/:/opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/
> -Xmx1g -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555
> org.apache.spark.deploy.SparkSubmit --master yarn --conf
> spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555
> --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
> spark-internal
>
> In Spark 3.0
> /opt/softwares/Java/jdk1.8.0_211/bin/java -cp
> /opt/apache/git/sparkSourceCode/spark/conf/:/opt/apache/git/sparkSourceCode/spark/assembly/target/scala-2.12/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/
> org.apache.spark.deploy.SparkSubmit --master yarn --conf
> spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5556
> --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
> spark-internal
> We can see that java options are not passed to driver process in spark3
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
