[resending as previous attempt did not make it through]
Hi Praveenesh,
Thanks for your suggestion but I checked and the HBase/ZK jars and dirs are
correct in our CLASSPATH and in PIG_CLASSPATH.
A few more details might help someone to point out where we are going wrong:
We have the following script called 'hbase_sample.pig'...
raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
listing_id: chararray,
fname: chararray,
lname: chararray );
STORE raw_data INTO 'hbase://hello_world' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage (
'info:fname info:lname');
We have created the 'hello_world' table in HBase with the following command:
create 'hello_world','info'
There is an input file called 'sample_data.csv' containing the following:
1, John, Smith
2, Jane, Doe
3, George, Washington
4, Ben, Franklin
We have stored 'sample_data.csv' in the same directory as the
'hbase_sample.pig' script and we have also stored a copy of the CSV file on
HDFS under the home directory for the current user
"hdfs://user/hadoop1/sample_data.csv".
We ran the script in local mode SUCCESSFULLY with the following command:
pig -x local -f hbase_sample.pig
We then ran the script using the mapreduce mode (default for pig):
pig -x mapreduce -f hbase_sample.pig
The jobtracker shows the following error(repeatedly) in the log output for
both map and reduce tasks:
Error: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.filter.WritableByteArrayComparable
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperat
ors.POStore.getStoreFunc(POStore.java:232)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.getCommitters(PigOutputCommitter.java:87)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.<init>(PigOutputCommitter.java:69)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat
.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
In addition, we see the following errors from pig (on the stdout):
Input(s):
Failed to read data from
"hdfs://10.0.0.235:8020/user/hadoop1/sample_data.csv"
Output(s):
Failed to produce result in "hbase://hello_world"
So we have established that we can run a pig script successfully in local
mode to store data in HBase (and we have verified that this has actually
worked), however we cannot seem to get the same script to work with HBase in
MR mode.
Here is the complete output from pig:
http://pastebin.com/vVvsCHXv
Thanks,
Royston
-----Original Message-----
From: praveenesh kumar [mailto:[email protected]]
Sent: 03 February 2012 14:38
To: [email protected]
Subject: Re: Cannot STORE from Pig to HBase 0.9x
Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath Also you can try
editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just Add your
Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig script.
Don't know whether these methods are the best method, but they work for me
:-)
Thanks,
Praveenesh
On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman <
[email protected]> wrote:
> Thanks for your reply. WritableByteArrayComparable is in the same
> place in HBase 0.93. I also registered my HBase jar from within Pig
> i.e register /opt/hbase/hbase-0.93.jar and that command returned
> without error. I also edited the line in the pig startup script that
specifies the path to HBase.
> Is there some other config file that controls how Pig finds HBase jar?
> Is there a default location Pig searches for jars that I could copy my
> jar into?
>
> Cheers,
> Royston
>
>
>
> On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <[email protected]> wrote:
>
> > "Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.hbase.filter.WritableByteArrayComparable"
> > indicates
> that
> > you don't have HBase on your classpath, or that the version of HBase
> > you are testing against moved this class someplace else. We've
> > tested against the 0.90 series, but not 0.92+... did they refactor
something?
> >
> > D
> >
> > On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
> > [email protected]> wrote:
> >
> >> Hi,
> >>
> >>
> >>
> >> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from
> >> HBase
> >> trunk) and following the tutorial.
> >>
> >>
> >>
> >> This line loads the sample file from HDFS successfully:
> >>
> >> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user,
> time,
> >> query);
> >>
> >>
> >>
> >> This line seems to work:
> >>
> >> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time),
> >> query;
> >>
> >>
> >>
> >> Because when I do:
> >>
> >> DUMP T;
> >>
> >> I get the response I expected.
> >>
> >>
> >>
> >> But if I then do:
> >>
> >> STORE T INTO 'excite' USING
> >> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
> >>
> >> I get (in the log):
> >>
> >> Pig Stack Trace
> >>
> >> ---------------
> >>
> >> ERROR 2998: Unhandled internal error.
> >> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>
> >>
> >>
> >> java.lang.NoClassDefFoundError:
> >> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>
> >> at java.lang.Class.forName0(Native Method)
> >>
> >> at java.lang.Class.forName(Class.java:247)
> >>
> >> at
> >> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428
> >> )
> >>
> >> at
> >>
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java
> :458)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanB
> uilder
> >> .java:723)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuil
> der.ja
> >> va:712)
> >>
> >> at
> >>
> >>
>
org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
> >> java:4340)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGen
> erator
> >> .java:5956)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenera
> tor.ja
> >> va:1122)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPl
> anGene
> >> rator.java:683)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenera
> tor.ja
> >> va:483)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.
> java:3
> >> 69)
> >>
> >> at
> >>
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:1
> 71)
> >>
> >> at
> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
> >>
> >> at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
> >>
> >> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
> >>
> >> at
> >> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:
> >> 942)
> >>
> >> at
> >>
> >>
>
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
> >> java:386)
> >>
> >> at
> >>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja
> va:188
> >> )
> >>
> >> at
> >>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja
> va:164
> >> )
> >>
> >> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> >>
> >> at org.apache.pig.Main.run(Main.java:495)
> >>
> >> at org.apache.pig.Main.main(Main.java:111)
> >>
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> Method)
> >>
> >> at
> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:39
> >> )
> >>
> >> at
> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl
> >> .java:25)
> >>
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >>
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
> >>
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>
> >> at java.security.AccessController.doPrivileged(Native Method)
> >>
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>
> >> at
> >> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>
> >> ... 28 more
> >>
> >>
> >>
> >> Grateful for any help with this.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Royston
> >>
> >>
>
