[
https://issues.apache.org/jira/browse/PIG-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146272#comment-14146272
]
Praveen Rachabattuni commented on PIG-4174:
-------------------------------------------
Currently, I see a conflict with hadoop when having it in the shell
environment, the e2e test doesn't run with Spark. But works fine when hadoop
isn't available in the environment.
Would like to know if there any other issues.
> e2e tests for Spark
> -------------------
>
> Key: PIG-4174
> URL: https://issues.apache.org/jira/browse/PIG-4174
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Praveen Rachabattuni
> Assignee: Praveen Rachabattuni
> Attachments: PIG-4174-1.patch
>
>
> Setup e2e tests for pig on spark like that pig on map-reduce and pig on tez.
> Steps to setup e2e tests:
> 1. Initialize Variables
> export OLD_PIG_HOME=/usr/local/Cellar/pig/0.12.0 # Should rather be 14
> export HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/1.0.4/conf
> export HADOOP_BIN=/usr/local/Cellar/hadoop/1.0.4/bin/hadoop
> 2. Generate Data
> ant -Dharness.old.pig=$OLD_PIG_HOME -Dharness.cluster.conf=$HADOOP_CONF_DIR
> -Dharness.cluster.bin=$HADOOP_BIN test-e2e-deploy-local
> (You might want to install necessary cpan modules incase of any dependency
> errors
> https://cwiki.apache.org/confluence/display/PIG/HowToTest#HowToTest-End-to-endTesting)
> Copy data to hdfs to use with Spark
> hadoop fs -put test/e2e/pig/testdist/data ./
> 3. Run particular test
> ant -Dharness.old.pig=$OLD_PIG_HOME -Dharness.cluster.conf=$HADOOP_CONF_DIR
> -Dharness.cluster.bin=$HADOOP_BIN -Dtests.to.run="-t Checkin_1" test-e2e-spark
> 4. Run all tests
> ant -Dharness.old.pig=$OLD_PIG_HOME -Dharness.cluster.conf=$HADOOP_CONF_DIR
> -Dharness.cluster.bin=$HADOOP_BIN test-e2e-spark
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
