Pallavi Rao created PIG-4746:
--------------------------------
Summary: Enable spork in Oozie
Key: PIG-4746
URL: https://issues.apache.org/jira/browse/PIG-4746
Project: Pig
Issue Type: Sub-task
Components: spark
Reporter: Pallavi Rao
Fix For: spark-branch
I was able get PIG on SPARK going with Oozie. But, only in "local" mode. Here
is what I did:
1. Used workflow schema version uri:oozie:workflow:0.2 and passed exectype as
an argument.
2. Copied Spark jars + kyro jar into workflow app lib.
To get spork going in yarn-client/yarn-master mode, couple of enhancements will
need to be made:
1. Right now, spark launcher reads SPARK_MASTER as env. variable. Need to make
this a PIG property.
2. The spark libraries need to be in classpath of the driver in case of
yarn-client/yarn-cluster mode. This will need a fix similar to PIG-4667 and
PIG-4681.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
