Hi,
I am using pig 0.7.0 in hadoop mapreduce mode.
The problem I have is that I simply can't use
STORE INTO alias USING PigStorage();
I can load dataset in, write UDFs to manipulate the dataset, but I can't
store it. The output is a directory in HDFS with 0 bytes.
As an example, I've been testing with a simple script:
W = load 'wordbag' using PigStorage(' ') as (f1:int, f2:int, name:chararray,
type:chararray);
store W into 'wordtesting' using PigStorage(' ');
I run the code in grunt, and the output of hadoop fs -ls is:
drwxr-xr-x - awang supergroup 0 2010-09-21 13:45
/user/awang/wordtesting
The grunt messages are:
grunt> store filteredW into 'wordtesting' using PigStorage(' ');
2010-09-21 13:45:35,210 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns
- No column pruned for W
2010-09-21 13:45:35,210 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns
- No map keys pruned for W
2010-09-21 13:45:35,440 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine
- (Name: Store(hdfs://pineal:9000/user/awang/wordtesting:PigStorage(' ')) -
1-46 Operator Key: 1-46)
2010-09-21 13:45:35,498 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2010-09-21 13:45:35,498 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2010-09-21 13:45:35,549 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-09-21 13:45:38,100 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-09-21 13:45:38,166 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2010-09-21 13:45:38,173 [Thread-15] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-09-21 13:45:38,307 [Thread-15] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat
- Total input paths to process : 1
2010-09-21 13:45:38,307 [Thread-15] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
- Total input paths to process : 1
2010-09-21 13:45:38,670 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201009211320_0002
2010-09-21 13:45:38,670 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://pineal:50030/jobdetails.jsp?jobid=job_201009211320_0002
2010-09-21 13:45:38,673 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2010-09-21 13:45:48,755 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 50% complete
2010-09-21 13:45:53,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2010-09-21 13:45:53,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Successfully stored result in: "hdfs://pineal:9000/user/awang/wordtesting"
2010-09-21 13:45:53,846 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Records written : 1
2010-09-21 13:45:53,846 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Bytes written : 20
2010-09-21 13:45:53,846 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Spillable Memory Manager spill count : 0
2010-09-21 13:45:53,847 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Proactive spill count : 0
2010-09-21 13:45:53,847 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Success!
I've been struggling with this for a long time…. It works if I have a one
bytearray in my tuple, but once I defined my schema, it no longer works.
Anyone has any idea? Please help!! Thanks!
Best regards,
Alex
