Hi Praveenesh,
Thanks for your suggestion but I checked and the HBase/ZK jars and dirs are
correct in our CLASSPATH and in PIG_CLASSPATH.
A few more details might help someone to point out where we are going wrong:
We have the following script called 'hbase_sample.pig'...
raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
listing_id: chararray,
fname: chararray,
lname: chararray );
STORE raw_data INTO 'hbase://hello_world' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage (
'info:fname info:lname');
We have created the 'hello_world' table in HBase with the following command:
create 'hello_world','info'
There is an input file called 'sample_data.csv' containing the following:
1, John, Smith
2, Jane, Doe
3, George, Washington
4, Ben, Franklin
We have stored 'sample_data.csv' in the same directory as the
'hbase_sample.pig' script and we have also stored a copy of the CSV file on
HDFS under the home directory for the current user
"hdfs://user/hadoop1/sample_data.csv".
We ran the script in local mode SUCCESSFULLY with the following command:
pig -x local -f hbase_sample.pig
We then ran the script using the mapreduce mode (default for pig):
pig -x mapreduce -f hbase_sample.pig
The jobtracker shows the following error(repeatedly) in the log output for both
map and reduce tasks:
Error: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.filter.WritableByteArrayComparable
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:87)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:69)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
In addition, we see the following errors from pig (on the stdout):
Input(s):
Failed to read data from "hdfs://10.0.0.235:8020/user/hadoop1/sample_data.csv"
Output(s):
Failed to produce result in "hbase://hello_world"
So we have established that we can run a pig script successfully in local mode
to store data in HBase (and we have verified that this has actually worked),
however we cannot seem to get the same script to work with HBase in MR mode.
Here is the complete output from pig:
http://pastebin.com/vVvsCHXv
Thanks,
Royston
On 3 Feb 2012, at 14:38, praveenesh kumar wrote:
> Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath
> Also you can try editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just
> Add your Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig
> script.
> Don't know whether these methods are the best method, but they work for me
> :-)
>
> Thanks,
> Praveenesh
>
> On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman <
> [email protected]> wrote:
>
>> Thanks for your reply. WritableByteArrayComparable is in the same place in
>> HBase 0.93. I also registered my HBase jar from within Pig i.e register
>> /opt/hbase/hbase-0.93.jar and that command returned without error. I also
>> edited the line in the pig startup script that specifies the path to HBase.
>> Is there some other config file that controls how Pig finds HBase jar? Is
>> there a default location Pig searches for jars that I could copy my jar
>> into?
>>
>> Cheers,
>> Royston
>>
>>
>>
>> On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <[email protected]> wrote:
>>
>>> "Caused by: java.lang.ClassNotFoundException:
>>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable" indicates
>> that
>>> you don't have HBase on your classpath, or that the version of HBase you
>>> are testing against moved this class someplace else. We've tested against
>>> the 0.90 series, but not 0.92+... did they refactor something?
>>>
>>> D
>>>
>>> On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
>>> [email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
>>>> trunk) and following the tutorial.
>>>>
>>>>
>>>>
>>>> This line loads the sample file from HDFS successfully:
>>>>
>>>> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user,
>> time,
>>>> query);
>>>>
>>>>
>>>>
>>>> This line seems to work:
>>>>
>>>> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
>>>>
>>>>
>>>>
>>>> Because when I do:
>>>>
>>>> DUMP T;
>>>>
>>>> I get the response I expected.
>>>>
>>>>
>>>>
>>>> But if I then do:
>>>>
>>>> STORE T INTO 'excite' USING
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
>>>>
>>>> I get (in the log):
>>>>
>>>> Pig Stack Trace
>>>>
>>>> ---------------
>>>>
>>>> ERROR 2998: Unhandled internal error.
>>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>>>>
>>>>
>>>>
>>>> java.lang.NoClassDefFoundError:
>>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>>>>
>>>> at java.lang.Class.forName0(Native Method)
>>>>
>>>> at java.lang.Class.forName(Class.java:247)
>>>>
>>>> at
>>>> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
>>>>
>>>> at
>>>>
>> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
>>>> .java:723)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
>>>> va:712)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
>>>> java:4340)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
>>>> .java:5956)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
>>>> va:1122)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
>>>> rator.java:683)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
>>>> va:483)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
>>>> 69)
>>>>
>>>> at
>>>>
>> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
>>>>
>>>> at
>> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
>>>>
>>>> at
>> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>>>>
>>>> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>>>>
>>>> at
>>>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
>>>> java:386)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
>>>> )
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
>>>> )
>>>>
>>>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>>>>
>>>> at org.apache.pig.Main.run(Main.java:495)
>>>>
>>>> at org.apache.pig.Main.main(Main.java:111)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>>
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>>>> )
>>>>
>>>> at
>>>>
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>>>> .java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
>>>>
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>
>>>> ... 28 more
>>>>
>>>>
>>>>
>>>> Grateful for any help with this.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Royston
>>>>
>>>>
>>
