Hi all,
I'm defining a udf to store Library description information.
Here is the pig script:
register fasta.jar;
register /usr/lib/zookeeper/zookeeper-3.3.5-cdh3u4.jar;
register /usr/lib/hbase/hbase-0.90.6-cdh3u4.jar;
register /usr/lib/hbase/lib/guava-r06.jar;
register Library_Udf.jar;
library = LOAD 'discovery_rnaseq_library' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('A:organism' ,'-loadKey true')
as (id:int, name:chararray);
grpd = group library all;
data = foreach grpd generate ((MAX(library.id))+1) as id;
library346_tag = LOAD '/user/zhaji/Sequence/library/346_lib_description.txt'
USING PigStorage(',') AS (tag:chararray, value:chararray);
tmp_Lib346 = foreach library346_tag generate STRSPLIT(tag, ',') as t,
STRSPLIT(value, ',') as v;
Lib_tuple = CROSS data, tmp_Lib346;
dump Lib_tuple;
store Lib_tuple into 'hbase://discovery_rnaseq_library' using LibraryStorage();
And here is the udf java code:
public void putNext(Tuple input) throws IOException
{
// (library_id, tag, value)
List<Object> tuple = input.getAll();
int lib_id;
String tag, val;
lib_id = Integer.parseInt(tuple.get(0).toString());
tag = ((String) tuple.get(1)).toString(); // this is line 60
where error coresponds to here!!!!
val = ((String) tuple.get(2)).toString();
Put put = new Put(Bytes.toBytes(lib_id));
put.add(Bytes.toBytes("A"), Bytes.toBytes(tag),
Bytes.toBytes(val));
getTable().put(put);
}
It dumps the tuples fine like here:
(268,(REPLICATE_NUMBER),(1))
(268,(QC_RESULT),)
(268,(LIMS_PROJECT_ID),(FAKE500))
(268,(ECTOPIC_TYPE),)
(268,(COMMON_NAME),(corn))
(268,(SEQUENCER),(Solid))
(268,(DEV_STAGE),(V8-V9))
(268,(DATE_SENT),)
(268,(REPLICATE_TYPE),)
(268,(EVENT_ID),)
(268,(TISSUE),(tassel))
(268,(SEQUENCING_RUN),(84))
(268,(DATE_RETURN),(2/21/2011))
(268,(LIB_NAME),(LIB0346))
(268,(SCIENTIFIC_NAME),(Zea mays))
(268,(ECTOPIC_ID),)
(268,(ADAPTOR_5),)
(268,(ADAPTOR_3),(CGCCTTGGCCGTACAGCAG))
(268,(TREATMENT),("Microspore mother cell stage-premeiosis (V8-V9)))
(268,(EXPERIMENT_DESC),(tassel-specific siRNA discovery))
(268,(QC_NOTE),)
(268,(GENETIC_BACKGROUND),(LH244))
But when it wants to store it into hbase table gives me this error:
2012-09-25 11:11:45,242 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201209201232_0127 has failed! Stop running all dependent jobs
2012-09-25 11:11:45,245 [main] WARN org.apache.pig.tools.pigstats.JobStats -
unable to get input counter for
/user/zhaji/Sequence/library/346_lib_description.txt
2012-09-25 11:11:45,245 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2012-09-25 11:11:45,256 [main] ERROR org.apache.pig.tools.pigstats.PigStats -
ERROR: org.apache.pig.data.BinSedesTuple cannot be cast to java.lang.String
2012-09-25 11:11:45,256 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil
- 1 map reduce job(s) failed!
2012-09-25 11:11:45,256 [main] INFO org.apache.pig.tools.pigstats.PigStats -
Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u3 0.8.1-cdh3u3 zhaji 2012-09-25 11:09:55 2012-09-25
11:11:45 GROUP_BY,CROSS
Some jobs have failed! Stop running all dependent jobs
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime
MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_201209201232_0126 1 1 3 3 3 13 13
13 Lib_tuple,data,grpd,library GROUP_BY
Failed Jobs:
JobId Alias Feature Message Outputs
job_201209201232_0127 Lib_tuple,library346_tag,tmp_Lib346
Message: Job failed! Error - NA hbase://discovery_rnaseq_library,
Input(s):
Successfully read 267 records (352 bytes) from: "discovery_rnaseq_library"
Failed to read data from "/user/zhaji/Sequence/library/346_lib_description.txt"
Output(s):
Failed to produce result in "hbase://discovery_rnaseq_library"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_201209201232_0126 -> job_201209201232_0127,
job_201209201232_0127
2012-09-25 11:11:45,265 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Some jobs have failed! Stop running all dependent jobs
2012-09-25 11:11:45,267 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
0: java.io.IOException: No FileSystem for scheme: hbase
Details at logfile: /home/ZHAJI/pig_1348589313588.log
The pig log file is also here:
Backend error message
---------------------
java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast
to java.lang.String
at LibraryStorage.putNext(LibraryStorage.java:60)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:514)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:396)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:374)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:251)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.a
Pig Stack Trace
---------------
ERROR 0: java.io.IOException: No FileSystem for scheme: hbase
org.apache.pig.backend.executionengine.ExecException: ERROR 0:
java.io.IOException: No FileSystem for scheme: hbase
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1223)
at org.apache.pig.PigServer.execute(PigServer.java:1201)
at org.apache.pig.PigServer.access$100(PigServer.java:129)
at org.apache.pig.PigServer$Graph.execute(PigServer.java:1528)
at org.apache.pig.PigServer.executeBatchEx(PigServer.java:373)
at org.apache.pig.PigServer.executeBatch(PigServer.java:340)
at
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:115)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:172)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: No FileSystem for scheme: hbase
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1560)
at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
at
org.apache.pig.StoreFunc.cleanupOnFailureImpl(StoreFunc.java:171)
at org.apache.pig.StoreFunc.cleanupOnFailure(StoreFunc.java:158)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1220)
... 11 more
Any help is highly appreciated.
-Zahra
This e-mail message may contain privileged and/or confidential information, and
is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please
notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of
this e-mail by you is strictly prohibited.
All e-mails and attachments sent and received are subject to monitoring,
reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking
for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage
caused by any such code transmitted by or accompanying
this e-mail or any attachment.
The information contained in this email may be subject to the export control
laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and
sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this
information you are obligated to comply with all
applicable U.S. export laws and regulations.
