[
https://issues.apache.org/jira/browse/PIG-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151026#comment-14151026
]
liyunzhang_intel commented on PIG-4206:
---------------------------------------
Need add -Dharness.hadoop.home in command(the reason see PIG-4205), I can run
e2e tests for spark successfully with patch in the following way:
Code:
1. Use branch spark 9cb14bc
9cb14bc - PIG-4174: e2e tests for Spark setup
2. patch
patch -p1<PIG-4206.patch
Env:
hadoop 1 + Spark 0.9
How to build:
ant jar
How to run:
1. Export variables:
export PIG_CLASSPATH=$PIG_HOME/build/ivy/lib/Pig/*
export HADOOP_USER_CLASSPATH_FIRST="true"
export HADOOP_HOME=xxx
export HADOOP_CONF_DIR=xxx
export HADOOP_BIN=$HADOOP_HOME/bin/hadoop
2. Generate Data
ant -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN
-Dharness.hadoop.home=$HADOOP_HOME test-e2e-deploy
3. Run particular test
ant -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN
-Dharness.hadoop.home=$HADOOP_HOME -Dtests.to.run="-t Checkin_1" test-e2e-spark
4. Run all tests
Ant -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN
-Dharness.hadoop.home=$HADOOP_HOME test-e2e-spark
attached test_harnesss_1411881848 is the result of Step3 "Run particular test"
> e2e tests for Spark can not work in hadoop env
> ----------------------------------------------
>
> Key: PIG-4206
> URL: https://issues.apache.org/jira/browse/PIG-4206
> Project: Pig
> Issue Type: Bug
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Attachments: PIG-4206.patch, test_harnesss_1411881848
>
>
> Code:
> Use branch spark 9cb14bc
> 9cb14bc - PIG-4174: e2e tests for Spark setup
> Env:
> hadoop 1 + Spark 0.9
> How to build:
> ant jar
> Steps to reproduce:
> 1. Initialize Variables
> export OLD_PIG_HOME=/usr/local/Cellar/pig/0.12.0 # Should rather be 14
> export HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/1.0.4/conf
> export HADOOP_BIN=/usr/local/Cellar/hadoop/1.0.4/bin/hadoop
> 2. Generate Data
> ant -Dharness.old.pig=$OLD_PIG_HOME -Dharness.cluster.conf=$HADOOP_CONF_DIR
> -Dharness.cluster.bin=$HADOOP_BIN test-e2e-deploy-local
> (You might want to install necessary cpan modules incase of any dependency
> errors
> https://cwiki.apache.org/confluence/display/PIG/HowToTest#HowToTest-End-to-endTesting)
> Copy data to hdfs to use with Spark
> hadoop fs -put test/e2e/pig/testdist/data ./
> 3. Run particular test
> ant -Dharness.old.pig=$OLD_PIG_HOME -Dharness.cluster.conf=$HADOOP_CONF_DIR
> -Dharness.cluster.bin=$HADOOP_BIN -Dtests.to.run="-t Checkin_1" test-e2e-spark
> following error message found in Step 3 "Run particular test"
> [exec] ERROR: driver->run() returned the following error message
> [./test_harness.pl at 156: Cannot create HDFS directory
> /user/pig/out/root-1411634072-nightly.conf/: 256 - Illegal seek
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
