[
https://issues.apache.org/jira/browse/PIG-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171977#comment-14171977
]
Rohini Palaniswamy edited comment on PIG-4233 at 10/15/14 4:51 AM:
-------------------------------------------------------------------
[~praveenr019],
The pig-withouthadoop.jar should not contain spark specific jars and should
only be pig dependencies. The actual fat jar containing hadoop dependencies has
been totally removed in 0.14. If you need to create a fat jar with spark
dependencies create a separate build target.
was (Author: rohini):
[~praveenr019],
The pig-withouthadoop.jar should not contain spark specific jars and should
only be pig dependencies. The actual fat jar containing hadoop dependencies has
been totally removed in 0.14.
> Package pig along with dependencies into a fat jar while job submission to
> Spark cluster
> ----------------------------------------------------------------------------------------
>
> Key: PIG-4233
> URL: https://issues.apache.org/jira/browse/PIG-4233
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Praveen Rachabattuni
> Assignee: Praveen Rachabattuni
> Attachments: PIG-4233.patch
>
>
> Currently we have a fat jar created in legacy directory which contains pig
> along with dependencies.
> Would need to modify build.xml to add spark dependency jars to include in
> legacy fat jar.
> Running job on Spark cluster:
> 1. export SPARK_HOME=/path/to/spark
> 2. export
> SPARK_PIG_JAR=$PIG_HOME/legacy/pig-0.14.0-SNAPSHOT-withouthadoop-h1.jar
> 3. export SPARK_MASTER=spark://localhost:7077
> 4 export HADOOP_HOME=/path/to/hadoop
> 5. Launch the job using ./bin/pig -x spark
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
