[
https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740683#comment-16740683
]
Ravindra edited comment on SPARK-26606 at 1/11/19 7:17 PM:
-----------------------------------------------------------
So, if you look at the below image, you can see the spark launch command for my
spark job. So, you can see the launch command which has the expected values.
But the individual parts like "-Dapp.env=prod" "-Dapp.country=US"
"-Dapp.banner=WMT" are missing. I am suspecting this could be the reason why my
source code throwing an exception that the jvm params are null. !Screen Shot
2019-01-11 at 1.12.33 PM.png!
was (Author: rrb441):
So, if you look at the below image, you can see the spark launch command for my
spark job. So, you can see the launch command which has the expected values.
But the parts "-Dapp.env=prod" "-Dapp.country=US" "-Dapp.banner=WMT" are
missing. I am suspecting this could be the reason why my source code throwing
an exception that the jvm params are null. !Screen Shot 2019-01-11 at 1.12.33
PM.png!
> extraJavaOptions does not work
> -------------------------------
>
> Key: SPARK-26606
> URL: https://issues.apache.org/jira/browse/SPARK-26606
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.3.1
> Reporter: Ravindra
> Priority: Major
> Labels: java, spark
> Attachments: Screen Shot 2019-01-09 at 4.31.01 PM.png, Screen Shot
> 2019-01-11 at 1.12.33 PM.png
>
>
> driver.extraJavaOptions and executor.extraJavaOptions does not work in spark
> submit command. Even though I see the parameters are being passed fine in the
> spark launch command I do not see these parameters are being picked up for
> some unknown reason.
>
> This is my spark submit command:
> output=`spark-submit \
> --class com.demo.myApp.App \
> --conf 'spark.executor.extraJavaOptions=-Dapp.env=dev -Dapp.country=US
> -Dapp.banner=ABC -Doracle.net.tns_admin=/work/artifacts/oracle/current
> -Djava.security.egd=[file:/dev/./urandom|file:///dev/urandom]' \
> --conf 'spark.driver.extraJavaOptions=-Dapp.env=dev -Dapp.country=US
> -Dapp.banner=ABC -Doracle.net.tns_admin=/work/artifacts/oracle/current
> -Djava.security.egd=[file:/dev/./urandom|file:///dev/urandom]' \
> --executor-memory "$EXECUTOR_MEMORY" \
> --executor-cores "$EXECUTOR_CORES" \
> --total-executor-cores "$TOTAL_CORES" \
> --driver-memory "$DRIVER_MEMORY" \
> --deploy-mode cluster \
> /home/spark/asm//current/myapp-*.jar 2>&1 &`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
