[
https://issues.apache.org/jira/browse/PIG-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kexianda updated PIG-4605:
--------------------------
Description:
The function SparkLauncher.addJarToSparkJobWorkingDirectory() try to copy the
user's Jar files to the working directory.
when SPARK_MATER == LOCAL, and if the src & dest Jar files have the same
absolute path, an exception will be thrown.
Reproduce steps:
1. build pig tutorial
2. cd PIG_HOME/tutorial/pigtmp
pig -x spark
REGISTER ./tutorial.jar
raw = LOAD '/user/excite-small.log' USING PigStorage('\t') AS (user, time,
query);
clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);
store clean1 into '/user/result1';
was:
The function SparkLauncher.addJarToSparkJobWorkingDirectory() try to copy the
user's Jar files to the working directory.
when SPARK_MATER == LOCAL, and if the src & dest Jar files have the same
absolute path, an exception will be thrown.
Reproduce steps:
1. build pig tutorial
2. cd PIG_HOME/tutorial/pigtmp
pig -x spark
REGISTER ./tutorial.jar
raw = LOAD '/user/excite-small.log' USING PigStorage('\t') AS (user, time,
query);
clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);
store clean1 into '/user/result1'
> fix a bug when coping Jar to SparkJob working directory
> -------------------------------------------------------
>
> Key: PIG-4605
> URL: https://issues.apache.org/jira/browse/PIG-4605
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: kexianda
> Assignee: kexianda
> Fix For: spark-branch
>
>
> The function SparkLauncher.addJarToSparkJobWorkingDirectory() try to copy the
> user's Jar files to the working directory.
> when SPARK_MATER == LOCAL, and if the src & dest Jar files have the same
> absolute path, an exception will be thrown.
> Reproduce steps:
> 1. build pig tutorial
> 2. cd PIG_HOME/tutorial/pigtmp
> pig -x spark
> REGISTER ./tutorial.jar
> raw = LOAD '/user/excite-small.log' USING PigStorage('\t') AS (user,
> time, query);
> clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);
> store clean1 into '/user/result1';
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
