All:
I've seen a thread with a similar issue and it was left unresolved.
So, here it goes again -
(a) I'm trying to get PIG to connect to a HADOOP cluster and execute a
script.
(a.1) The hadoop-site.xml file is in /home/hadoop and the script is
/home/hadoop/tmp/pig
(b) PIG finds the data file in DFS but does not run any mapreduce jobs on
the cluster (of 6 nodes). Instead it runs all the mapreduce jobs using a
local job runner.
(c) What am I missing? How do I get PIG to schedule its mapred jobs on the
cluster?
Thanks,
Prashanth
>> verbose debug
java -cp /home/hadoop:pig.jar org.apache.pig.Main -v /home/hadoop/tmp.pig
2008-05-23 16:43:44,636 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: local
2008-05-23 16:43:44,656 [main] DEBUG org.apache.hadoop.conf.Configuration -
java.io.IOException: config()
at
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:156)
at
org.apache.pig.backend.hadoop.datastorage.ConfigurationUtil.toConfiguration(ConfigurationUtil.java:14)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:45)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:36)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:139)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:106)
at org.apache.pig.impl.PigContext.connect(PigContext.java:177)
at org.apache.pig.PigServer.<init>(PigServer.java:149)
at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:43)
at org.apache.pig.Main.main(Main.java:295)
...
>> simple debug
java -cp /home/hadoop:pig.jar org.apache.pig.Main /home/hadoop/tmp.pig
2008-05-23 16:51:13,076 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: local
2008-05-23 16:51:13,386 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId=
2008-05-23 16:51:14,038 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - ----- MapReduce
Job -----
2008-05-23 16:51:14,038 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - Input:
[/user/hadoop/prashanth/log1:PigStorage(','),
/user/hadoop/prashanth/log1:PigStorage(','),
/user/hadoop/prashanth/log1:PigStorage(',')]
2008-05-23 16:51:14,039 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map:
[[*]->GENERATE {[FLOOR(GENERATE {[PROJECT $1]})],[PROJECT $3],[PROJECT
$38],[PROJECT $37],[PROJECT $32]}->GENERATE
{[org.apache.pig.impl.builtin.MULTIPLY(GENERATE {[FLOOR(GENERATE
{[org.apache.pig.impl.builtin.DIVIDE(GENERATE {[PROJECT
$0],['2']})]})],['2']})],[PROJECT $0],[PROJECT $1],[PROJECT $3],[PROJECT
$2],[PROJECT $4]}, [*]->GENERATE {[FLOOR(GENERATE {[PROJECT $1]})],[PROJECT
$3],[PROJECT $38],[PROJECT $37],[PROJECT $32]}->GENERATE
{[org.apache.pig.impl.builtin.MULTIPLY(GENERATE
{[org.apache.pig.impl.builtin.ADD(GENERATE {[FLOOR(GENERATE
{[org.apache.pig.impl.builtin.DIVIDE(GENERATE {[PROJECT
$0],['2']})]})],['1']})],['2']})],[PROJECT $0],[PROJECT $1],[PROJECT
$3],[PROJECT $2],[PROJECT $4]}, [*]->GENERATE {[FLOOR(GENERATE {[PROJECT
$1]})],[PROJECT $3],[PROJECT $38],[PROJECT $37],[PROJECT $32]}->GENERATE
{[org.apache.pig.impl.builtin.MULTIPLY(GENERATE
{[org.apache.pig.impl.builtin.ADD(GENERATE {[FLOOR(GENERATE
{[org.apache.pig.impl.builtin.DIVIDE(GENERATE {[PROJECT
$0],['2']})]})],['2']})],['2']})],[PROJECT $0],[PROJECT $1],[PROJECT
$3],[PROJECT $2],[PROJECT $4]}]
2008-05-23 16:51:14,039 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - Group: null
2008-05-23 16:51:14,039 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - Combine: null
2008-05-23 16:51:14,039 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce: null
2008-05-23 16:51:14,039 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - Output:
/tmp/temp-711499347/tmp128716201:org.apache.pig.builtin.BinStorage
2008-05-23 16:51:14,040 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - Split: null
2008-05-23 16:51:14,040 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map parallelism:
-1
2008-05-23 16:51:14,040 [main] INFO
org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce
parallelism: -1
2008-05-23 16:51:15,640 [Thread-14] INFO org.apache.hadoop.mapred.MapTask -
numReduceTasks: 1
2008-05-23 16:51:16,531 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher
- Pig progress = 0%
2008-05-23 16:51:17,344 [Thread-14] WARN
org.apache.hadoop.mapred.LocalJobRunner - job_local_1
