[
https://issues.apache.org/jira/browse/PIG-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated PIG-4946:
----------------------------------
Attachment: PIG-4946.patch
[~kexianda]:
After PIG-4903 check in, the process of starting a pig script changes(before we
add all jars under $PIG_HOME/lib and $PIG_HOME/lib/spark/* to the classpath and
export them to SPARK_YARN_DIST_FILES and SPARK_DIST_CLASSPATH) now we
dynamically load libs(PIG-4893)
I found that some redudant code in PIG-4903's check in. So checkout latest code
and patch -p1<PIG-4946.patch to verify this jira.
Can you verify the process as following both in spark local and spark
yarn-client mode?
*export SPARK_MASTER="yarn-client”*
1. not export SPARK_HOME to run:
{code}
error message like "Error: SPARK_HOME is not set!" is thrown out
{code}
2. export SPARK_HOME but not export SPARK_JAR to run:
{code}
error message like "Error: SPARK_JAR is not set, SPARK_JAR stands for the
hdfs location of spark-assembly*.jar. This allows YARN to cache
spark-assembly*.jar on nodes so that it doesn't need to be distributed each
time an application runs."
{code}
3. export SPARK_HOME and SPARK_JAR to run:
{code}
export SPARK_SOURCE=/home/zly/prj/oss/spark_source/
export
SPARK_JAR=hdfs://zly1.sh.intel.com:8020/user/root/spark-assembly-1.6.0-hadoop2.6.0.jar
./pig -x spark xx.pig
{code}
*export SPARK_MASTER="local"
1. we need not export SPARK_HOME, SPARK_JAR in local mode, so even we don't
export these two variables, the script should pass.
> Remove redudant code of bin/pig in spark mode after PIG-4903 check in
> ---------------------------------------------------------------------
>
> Key: PIG-4946
> URL: https://issues.apache.org/jira/browse/PIG-4946
> Project: Pig
> Issue Type: Bug
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Attachments: PIG-4946.patch
>
>
> After PIG-4903 checkin, some redudant code of bin/pig in spark branch is
> generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
