[
https://issues.apache.org/jira/browse/SPARK-25995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683876#comment-16683876
]
Thomas Graves commented on SPARK-25995:
---------------------------------------
I haven't looked at the details but I would say whatever is easier. I'm not
sure what users do with the args in this case anyway. Can they be used?
Another option would be to make the port file an actual spark internal arg or
configuration. It looks like it has a SparkConf object at the time its read
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/RBackend.scala#L135
> sparkR should ensure user args are after the argument used for the port
> -----------------------------------------------------------------------
>
> Key: SPARK-25995
> URL: https://issues.apache.org/jira/browse/SPARK-25995
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Affects Versions: 2.3.2
> Reporter: Thomas Graves
> Priority: Minor
>
> Currently if you run sparkR and accidentally specify an argument, it fails
> with a useless error message. For example:
> $SPARK_HOME/bin/sparkR --master yarn --deploy-mode client fooarg
> This gets turned into:
> Launching java with spark-submit command spark-submit "--master" "yarn"
> "--deploy-mode" "client" "sparkr-shell" "fooarg"
> /tmp/Rtmp6XBGz2/backend_port162806ea36bca
> Notice that "fooarg" got put before /tmp file which is how R and jvm know
> which port to connect to. SparkR eventually fails with timeout exception
> after 10 seconds.
>
> SparkR should either not allow args or make sure the order is correct so the
> backend_port is always first. see
> https://github.com/apache/spark/blob/master/R/pkg/R/sparkR.R#L129
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
