[ https://issues.apache.org/jira/browse/PIG-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163176#comment-14163176 ]
liyunzhang_intel commented on PIG-4206: --------------------------------------- Hi [~praveenr019]: I agree with you that 'local' mode should be benchmark exec type because it is quicker than mapred. so i modified the patch and uploaded PIG-4206_1.patch following ways is how to test it. 0. env which hadoop /home/zly/prj/oss/hadoop-1.2.1/bin/hadoop 1. patch patch -p1<PIG-4206_1.patch 2. build ant jar 3. export variables export HADOOP_HOME=xxx export HADOOP_CONF_DIR=xxx export HADOOP_BIN=$HADOOP_HOME/bin/hadoop export PIG_CLASSPATH=$PIG_HOME/build/ivy/lib/Pig/*;$HADOOP_CONF_DIR export HADOOP_USER_CLASSPATH_FIRST="true" 4.generate data ant -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN test-e2e-deploy-local 5.run particular test ant -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN -Dharness.hadoop.home=$HADOOP_HOME -Dtests.to.run="-t Checkin_1" test-e2e-spark 6.run all tests ant -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN -Dharness.hadoop.home=$HADOOP_HOME test-e2e-spark > e2e tests for Spark can not work in hadoop env > ---------------------------------------------- > > Key: PIG-4206 > URL: https://issues.apache.org/jira/browse/PIG-4206 > Project: Pig > Issue Type: Bug > Components: spark > Reporter: liyunzhang_intel > Assignee: liyunzhang_intel > Attachments: PIG-4206.patch, test_harnesss_1411881848 > > > Code: > Use branch spark 9cb14bc > 9cb14bc - PIG-4174: e2e tests for Spark setup > Env: > hadoop 1 + Spark 0.9 > How to build: > ant jar > Steps to reproduce: > 1. Initialize Variables > export OLD_PIG_HOME=/usr/local/Cellar/pig/0.12.0 # Should rather be 14 > export HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/1.0.4/conf > export HADOOP_BIN=/usr/local/Cellar/hadoop/1.0.4/bin/hadoop > 2. Generate Data > ant -Dharness.old.pig=$OLD_PIG_HOME -Dharness.cluster.conf=$HADOOP_CONF_DIR > -Dharness.cluster.bin=$HADOOP_BIN test-e2e-deploy-local > (You might want to install necessary cpan modules incase of any dependency > errors > https://cwiki.apache.org/confluence/display/PIG/HowToTest#HowToTest-End-to-endTesting) > Copy data to hdfs to use with Spark > hadoop fs -put test/e2e/pig/testdist/data ./ > 3. Run particular test > ant -Dharness.old.pig=$OLD_PIG_HOME -Dharness.cluster.conf=$HADOOP_CONF_DIR > -Dharness.cluster.bin=$HADOOP_BIN -Dtests.to.run="-t Checkin_1" test-e2e-spark > following error message found in Step 3 "Run particular test" > [exec] ERROR: driver->run() returned the following error message > [./test_harness.pl at 156: Cannot create HDFS directory > /user/pig/out/root-1411634072-nightly.conf/: 256 - Illegal seek -- This message was sent by Atlassian JIRA (v6.3.4#6332)