[
https://issues.apache.org/jira/browse/PIG-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021488#comment-15021488
]
Srikanth Sundarrajan commented on PIG-4741:
-------------------------------------------
SPARK_DIST_CLASSPATH is used by AM launcher via Node
Manager::ContainerExecutor. If the environment variable holds un-substituted
PWD, the Container Launcher script would substitute this with container home
directory which would be something along the lines of
{{<yarn.local.dir>/usercache/<user-id>/appcache/<app-id>/<container-id>}}. PWD
will be substituted with this home directory location on the AM. On the other
hand if it is left directly as {{${PWD}}}, it would get replaced with the local
directory from where the pig script is launched and you are most likely to see
Container exit with class not found exception.
> the value of $SPARK_DIST_CLASSPATH in pig file is invalid
> ---------------------------------------------------------
>
> Key: PIG-4741
> URL: https://issues.apache.org/jira/browse/PIG-4741
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4741.patch
>
>
> the value of
> [$SPARK_DIST_CLASSPATH|https://github.com/apache/pig/blob/spark/bin/pig#L380]
> in bin/pig is invalid
> {code}
> SPARK_DIST_CLASSPATH=${SPARK_DIST_CLASSPATH}:\${PWD}/`basename $f`
> {code}
> there is no need to escape the {{PWD}}. If we add "\", the value of
> SPARK_DIST_CLASSPATH will like:
> {code}
>
> ${PWD}/akka-actor_2.10-2.3.4-spark.jar:${PWD}/akka-remote_2.10-2.3.4-spark.jar
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
