Hi Dmitriy, Well thanks, that fixed it.
A few days ago I WAS registering the HBase jar and getting the same error. It turns out that you also need the jar in the PIG_CLASSPATH and I didn't. So I was never at a point where both things were simultaneously true. Many thanks for the reminder. I think it would be helpful if the sample Pig code in the javadoc for HBaseStorage included something like "register etc/hbase/hbase-0.92.jar" before the LOAD and STORE examples as this is one of the first places one naturally looks for help on HBaseStorage. Many of the other examples that turn up in a Google search are also missing the "register" statement. So is the HBase book (not your problem). I only stumbled across it by luck. Of course, none of the above means I'm not an idiot. Regards, Royston -----Original Message----- From: Dmitriy Ryaboy [mailto:[email protected]] Sent: 06 February 2012 21:22 To: [email protected] Subject: Re: Cannot STORE from Pig to HBase 0.9x Did you explicitly "register" the hbase jar? D On Mon, Feb 6, 2012 at 3:37 AM, Royston Sellman < [email protected]> wrote: > Hi Praveenesh, > > Thanks for your suggestion but I checked and the HBase/ZK jars and > dirs are correct in our CLASSPATH and in PIG_CLASSPATH. > > A few more details might help someone to point out where we are going > wrong: > > We have the following script called 'hbase_sample.pig'... > > raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS ( > listing_id: chararray, > fname: chararray, > lname: chararray ); > > STORE raw_data INTO 'hbase://hello_world' USING > org.apache.pig.backend.hadoop.hbase.HBaseStorage ( > 'info:fname info:lname'); > > We have created the 'hello_world' table in HBase with the following > command: > > create 'hello_world','info' > > There is an input file called 'sample_data.csv' containing the following: > > 1, John, Smith > 2, Jane, Doe > 3, George, Washington > 4, Ben, Franklin > > We have stored 'sample_data.csv' in the same directory as the > 'hbase_sample.pig' script and we have also stored a copy of the CSV > file on HDFS under the home directory for the current user > "hdfs://user/hadoop1/sample_data.csv". > > We ran the script in local mode SUCCESSFULLY with the following command: > > pig -x local -f hbase_sample.pig > > We then ran the script using the mapreduce mode (default for pig): > > pig -x mapreduce -f hbase_sample.pig > > The jobtracker shows the following error(repeatedly) in the log output > for both map and reduce tasks: > > Error: java.lang.ClassNotFoundException: > org.apache.hadoop.hbase.filter.WritableByteArrayComparable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428) > at > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperat ors.POStore.getStoreFunc(POStore.java:232) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit ter.getCommitters(PigOutputCommitter.java:87) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit ter.<init>(PigOutputCommitter.java:69) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat .getOutputCommitter(PigOutputFormat.java:279) > at org.apache.hadoop.mapred.Task.initialize(Task.java:515) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja va:1083) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > > > In addition, we see the following errors from pig (on the stdout): > > Input(s): > Failed to read data from "hdfs:// > 10.0.0.235:8020/user/hadoop1/sample_data.csv" > > Output(s): > Failed to produce result in "hbase://hello_world" > > > So we have established that we can run a pig script successfully in > local mode to store data in HBase (and we have verified that this has > actually worked), however we cannot seem to get the same script to > work with HBase in MR mode. > > Here is the complete output from pig: > > http://pastebin.com/vVvsCHXv > > Thanks, > Royston > > On 3 Feb 2012, at 14:38, praveenesh kumar wrote: > > > Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath Also you > > can try editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just Add > > your Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig > > script. > > Don't know whether these methods are the best method, but they work > > for > me > > :-) > > > > Thanks, > > Praveenesh > > > > On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman < > > [email protected]> wrote: > > > >> Thanks for your reply. WritableByteArrayComparable is in the same > >> place > in > >> HBase 0.93. I also registered my HBase jar from within Pig i.e > >> register /opt/hbase/hbase-0.93.jar and that command returned > >> without error. I > also > >> edited the line in the pig startup script that specifies the path > >> to > HBase. > >> Is there some other config file that controls how Pig finds HBase jar? > Is > >> there a default location Pig searches for jars that I could copy my > >> jar into? > >> > >> Cheers, > >> Royston > >> > >> > >> > >> On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <[email protected]> wrote: > >> > >>> "Caused by: java.lang.ClassNotFoundException: > >>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable" > >>> indicates > >> that > >>> you don't have HBase on your classpath, or that the version of > >>> HBase > you > >>> are testing against moved this class someplace else. We've tested > against > >>> the 0.90 series, but not 0.92+... did they refactor something? > >>> > >>> D > >>> > >>> On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman < > >>> [email protected]> wrote: > >>> > >>>> Hi, > >>>> > >>>> > >>>> > >>>> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from > HBase > >>>> trunk) and following the tutorial. > >>>> > >>>> > >>>> > >>>> This line loads the sample file from HDFS successfully: > >>>> > >>>> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS > >>>> (user, > >> time, > >>>> query); > >>>> > >>>> > >>>> > >>>> This line seems to work: > >>>> > >>>> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), > >>>> query; > >>>> > >>>> > >>>> > >>>> Because when I do: > >>>> > >>>> DUMP T; > >>>> > >>>> I get the response I expected. > >>>> > >>>> > >>>> > >>>> But if I then do: > >>>> > >>>> STORE T INTO 'excite' USING > >>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query') > >>>> ; > >>>> > >>>> I get (in the log): > >>>> > >>>> Pig Stack Trace > >>>> > >>>> --------------- > >>>> > >>>> ERROR 2998: Unhandled internal error. > >>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable > >>>> > >>>> > >>>> > >>>> java.lang.NoClassDefFoundError: > >>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable > >>>> > >>>> at java.lang.Class.forName0(Native Method) > >>>> > >>>> at java.lang.Class.forName(Class.java:247) > >>>> > >>>> at > >>>> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:4 > >>>> 28) > >>>> > >>>> at > >>>> > >> > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java > :458) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanB > uilder > >>>> .java:723) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuil > der.ja > >>>> va:712) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator. > >>>> java:4340) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGen > erator > >>>> .java:5956) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenera > tor.ja > >>>> va:1122) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPl > anGene > >>>> rator.java:683) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenera > tor.ja > >>>> va:483) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator. > java:3 > >>>> 69) > >>>> > >>>> at > >>>> > >> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:1 > 71) > >>>> > >>>> at > >> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609) > >>>> > >>>> at > >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582) > >>>> > >>>> at > >>>> org.apache.pig.PigServer.registerQuery(PigServer.java:584) > >>>> > >>>> at > >>>> > org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942 > ) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser. > >>>> java:386) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja > va:188 > >>>> ) > >>>> > >>>> at > >>>> > >>>> > >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja > va:164 > >>>> ) > >>>> > >>>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) > >>>> > >>>> at org.apache.pig.Main.run(Main.java:495) > >>>> > >>>> at org.apache.pig.Main.main(Main.java:111) > >>>> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>> Method) > >>>> > >>>> at > >>>> > >>>> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > ava:39 > >>>> ) > >>>> > >>>> at > >>>> > >>>> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > orImpl > >>>> .java:25) > >>>> > >>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > >>>> > >>>> Caused by: java.lang.ClassNotFoundException: > >>>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable > >>>> > >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > >>>> > >>>> at java.security.AccessController.doPrivileged(Native > >>>> Method) > >>>> > >>>> at > >>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190) > >>>> > >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > >>>> > >>>> at > >>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >>>> > >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > >>>> > >>>> ... 28 more > >>>> > >>>> > >>>> > >>>> Grateful for any help with this. > >>>> > >>>> > >>>> > >>>> Thanks, > >>>> > >>>> Royston > >>>> > >>>> > >> > >
