Matt and I got the patch to apply and work but another bug showed up. I'll see about fixing it later tonight.
D On Tue, Feb 15, 2011 at 3:00 PM, Matt Davies <[email protected]> wrote: > Dmitriy, > > Appreciate the help. We have tried against trunk and here's the output. > > > > patching file > src/org/apache/pig/backend/hadoop/hbase/HBaseTableInputFormat.java > patching file src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java > Hunk #6 FAILED at 80. > Hunk #7 succeeded at 115 (offset 1 line). > Hunk #9 succeeded at 167 (offset 1 line). > Hunk #11 succeeded at 253 (offset 1 line). > Hunk #13 succeeded at 288 (offset 1 line). > Hunk #15 succeeded at 324 (offset 1 line). > Hunk #17 succeeded at 377 (offset 1 line). > Hunk #19 succeeded at 417 (offset 1 line). > Hunk #21 succeeded at 451 (offset 1 line). > 1 out of 21 hunks FAILED -- saving rejects to file > src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java.rej > patching file test/hbase-site.xml > patching file test/org/apache/pig/test/TestHBaseStorage.java > Hunk #3 FAILED at 42. > 1 out of 3 hunks FAILED -- saving rejects to file > test/org/apache/pig/test/TestHBaseStorage.java.rej > patching file build.xml > Hunk #1 succeeded at 51 (offset 1 line). > Hunk #2 FAILED at 199. > Hunk #3 succeeded at 587 (offset -83 lines). > 1 out of 3 hunks FAILED -- saving rejects to file build.xml.rej > patching file ivy.xml > Hunk #1 succeeded at 84 (offset -5 lines). > patching file ivy/libraries.properties > Hunk #1 succeeded at 28 with fuzz 2 (offset -1 lines). > patching file > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java > Hunk #1 succeeded at 111 (offset 1 line). > Hunk #3 succeeded at 139 (offset 1 line). > Hunk #4 succeeded at 175 with fuzz 1 (offset -3 lines). > Hunk #5 succeeded at 201 (offset 1 line). > Hunk #6 succeeded at 229 (offset -3 lines). > Hunk #7 succeeded at 270 (offset 1 line). > Hunk #8 succeeded at 276 (offset -3 lines). > Hunk #9 succeeded at 298 (offset 1 line). > Hunk #10 succeeded at 320 (offset -3 lines). > Hunk #11 FAILED at 385. > Hunk #12 succeeded at 394 with fuzz 1 (offset -2 lines). > Hunk #13 succeeded at 402 (offset -4 lines). > Hunk #14 succeeded at 418 (offset -2 lines). > Hunk #15 succeeded at 430 (offset -4 lines). > Hunk #16 succeeded at 457 (offset -2 lines). > Hunk #17 FAILED at 484. > Hunk #18 succeeded at 484 (offset -13 lines). > Hunk #19 succeeded at 514 (offset -2 lines). > Hunk #20 succeeded at 539 (offset -19 lines). > Hunk #21 succeeded at 564 (offset -2 lines). > Hunk #22 succeeded at 566 with fuzz 2 (offset -22 lines). > Hunk #23 succeeded at 605 (offset -2 lines). > Hunk #24 succeeded at 595 (offset -22 lines). > Hunk #25 succeeded at 635 (offset -2 lines). > Hunk #26 FAILED at 651. > Hunk #27 FAILED at 680. > Hunk #28 succeeded at 673 (offset -24 lines). > Hunk #29 succeeded at 705 (offset -2 lines). > Hunk #30 succeeded at 694 with fuzz 1 (offset -24 lines). > Hunk #31 succeeded at 746 (offset -2 lines). > Hunk #32 succeeded at 740 (offset -24 lines). > Hunk #33 succeeded at 775 (offset -2 lines). > Hunk #34 succeeded at 771 (offset -24 lines). > Hunk #35 succeeded at 806 (offset -2 lines). > Hunk #36 succeeded at 806 (offset -24 lines). > Hunk #37 succeeded at 892 (offset -2 lines). > Hunk #38 succeeded at 914 (offset -24 lines). > Hunk #39 succeeded at 948 (offset -2 lines). > Hunk #40 succeeded at 936 (offset -24 lines). > Hunk #41 succeeded at 974 (offset -2 lines). > Hunk #42 FAILED at 1072. > Hunk #43 succeeded at 1093 (offset -31 lines). > Hunk #44 succeeded at 1158 (offset -2 lines). > Hunk #45 succeeded at 1141 (offset -31 lines). > Hunk #46 succeeded at 1221 (offset -2 lines). > Hunk #47 succeeded at 1212 (offset -31 lines). > Hunk #48 succeeded at 1264 (offset -2 lines). > Hunk #49 FAILED at 1290. > Hunk #50 FAILED at 1302. > Hunk #51 FAILED at 1322. > > > > > 8 out of 51 hunks FAILED -- saving rejects to file > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java.rej > > > > On Tue, Feb 15, 2011 at 3:56 PM, Dmitriy Ryaboy <[email protected]> > wrote: > > > Thought it was against the trunk.. I did make it ignore whitespace > changes, > > maybe there's a problem there. I'll email you another patch off-list, > let's > > see if we can get it to work. > > > > D > > > > On Tue, Feb 15, 2011 at 2:45 PM, Matt Davies <[email protected]> > wrote: > > > > > Dmitriy, > > > > > > I saw the patch posted to PIG-1680 from last night. Thanks! We have > been > > > trying to apply it, and find that the patch can't match some of the > > hunks. > > > Are you patching against trunk or a certain version? > > > > > > > > > Thanks! > > > -Matt > > > > > > On Mon, Feb 14, 2011 at 4:14 PM, Matt Davies <[email protected]> > > wrote: > > > > > > > You are welcome! We did a filter before the store, and I confirmed > that > > > > there weren't any nulls. Just something undocumented feature ;) > > > > > > > > > > > > On Mon, Feb 14, 2011 at 3:03 PM, jacob <[email protected]> > > > wrote: > > > > > > > >> Thanks for the mention. Off the top of my head I got these sorts of > > > >> errors when trying to store either null records or records with null > > > >> fields. What happens if you FILTER out any null values you might > have. > > > >> Does the problem persist? > > > >> > > > >> --jacob > > > >> @thedatachef > > > >> > > > >> On Mon, 2011-02-14 at 14:57 -0700, Matt Davies wrote: > > > >> > Hey All, > > > >> > > > > >> > Running into a problem storing data from a pig script storing > > results > > > >> into > > > >> > HBase. > > > >> > > > > >> > We are getting the following error: > > > >> > > > > >> > java.lang.NullPointerException > > > >> > at > > > >> > > > > > > org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126) > > > >> > at > > > >> > > > > > > org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81) > > > >> > at > > > >> > > > > > > org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364) > > > >> > at > > > >> > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138) > > > >> > at > > > >> > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97) > > > >> > at > > > >> > > > > > > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523) > > > >> > at > > > >> > > > > > > org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) > > > >> > at > > > >> > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48) > > > >> > at > > > >> > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238) > > > >> > at > > > >> > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231) > > > >> > at > > > >> > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53) > > > >> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > > > >> > at > > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639) > > > >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315) > > > >> > at org.apache.hadoop.mapred.Child$4.run(Child.java:217) > > > >> > at java.security.AccessController.doPrivileged(Native > Method) > > > >> > at javax.security.auth.Subject.doAs(Subject.java:396) > > > >> > at > > > >> > > > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063) > > > >> > at org.apache.hadoop.mapred.Child.main(Child.java:211) > > > >> > > > > >> > > > > >> > We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've > > > >> > followed the instructions to get pig 0.8.0 to work with CDH3 from > > > >> > thedatachef(Thanks!) > > > >> > > > > >> > > > > > > http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html > > > >> . > > > >> > > > > >> > The relevant line from the pig script is below. We've applied the > > > >> > patch to get the "-noWAL" working: > > > >> > > > > >> > STORE links INTO 'p' USING > > > >> > org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2 > > > >> > a:g','-noWAL'); > > > >> > > > > >> > > > > >> > > > > >> > Anyone know what could be causing this problem? > > > >> > > > > >> > > > > >> > Thanks in advance, > > > >> > > > > >> > > > > >> > Matt > > > >> > > > >> > > > >> > > > > > > > > > >
