[
https://issues.apache.org/jira/browse/HCATALOG-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229578#comment-13229578
]
[email protected] commented on HCATALOG-302:
--------------------------------------------------------
bq. On 2012-03-14 03:05:47, Francis Liu wrote:
bq. >
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java,
line 163
bq. > <https://reviews.apache.org/r/4322/diff/2/?file=91963#file91963line163>
bq. >
bq. > Does this work without the foolproof code? Let's try to contain the
bug with pig in HCatStorer. It'll just add more complexity.
bq.
bq. Rohini Palaniswamy wrote:
bq. Nothing complex. Just added to check to ensure we don't create
transaction again if same jobConf is passed again with a different
OutputJobInfo. In Pig's case we fixed to pass same OutputJobInfo but it is with
a different JobConf . Added a test case assert for the two cases.
You are adding logic to support misuse. Granted we have to support it because
of a bug in Pig. Let's try and localize it in HCatStorer. Adding logic here
even little would make it confusing as a developer would have to understand how
pig works. I'm fine with this going in for now but it should be removed once
HCATALOG-314 is out.
- Francis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4322/#review5917
-----------------------------------------------------------
On 2012-03-14 06:48:57, Rohini Palaniswamy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4322/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-14 06:48:57)
bq.
bq.
bq. Review request for hcatalog, Alan Gates, Francis Liu, and Vandana
Ayyalasomayajula.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Fixes to get Hbase storagehandler working with Pig.
bq.
bq.
bq. 1)Includes changes to HCatStorer used by Pig. Modified HCatStorer to pass
the same OutputJobInfo in the multiple calls made in Pig to
HCatOutputFormat.setOutput. Else we ended up creating multiple orphaned
transactions which were never committed in zookeeper.
bq. 2) Moved stuff from getInputSplits and checkoutputSpecs to configure
methods in storagehandler.
bq. 3) Fixed a logic error in hbase record reader.
bq.
bq.
bq. This addresses bug HCATALOG-302.
bq. https://issues.apache.org/jira/browse/HCATALOG-302
bq.
bq.
bq. Diffs
bq. -----
bq.
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/pig/HCatStorer.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseBulkOutputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseInputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HbaseSnapshotRecordReader.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseInputFormat.java
1299921
bq.
bq. Diff: https://reviews.apache.org/r/4322/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Unit tests and Integration testing done.
bq.
bq.
bq. Thanks,
bq.
bq. Rohini
bq.
bq.
> unable to write to hbase channel. HBaseHCatStorageHandler class not found
> -------------------------------------------------------------------------
>
> Key: HCATALOG-302
> URL: https://issues.apache.org/jira/browse/HCATALOG-302
> Project: HCatalog
> Issue Type: Bug
> Components: hbase
> Affects Versions: 0.4
> Reporter: David Capwell
> Assignee: Rohini Palaniswamy
> Fix For: 0.4
>
> Attachments: HCATALOG-302-1.patch, HCATALOG-302-2.patch
>
>
> This is the pig script:
> PigServer pigServer = PigServerBuilder.create(this.client.getConf());
> pigServer.registerQuery("A = LOAD '"+input+"' USING PigStorage()
> AS(key:chararray, value:chararray);");
> pigServer.registerQuery("STORE A INTO '"+this.getDb().getName() + "." +
> this.getTable().getName()+"' USING org.apache.hcatalog.pig.HCatStorer();");
> Error:
> 2012-03-09 03:04:12,105 WARN org.apache.hadoop.mapred.Child: Error running
> child
> java.io.IOException: Error in loading storage
> handler.org.apache.hcatalog.hbase.HBaseHCatStorageHandler
> at
> org.apache.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:518)
> at
> org.apache.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:474)
> at
> org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getOutputFormat(HCatBaseOutputFormat.java:77)
> at
> org.apache.hcatalog.mapreduce.HCatOutputFormat.getOutputCommitter(HCatOutputFormat.java:250)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:89)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:67)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:278)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1082)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hcatalog.hbase.HBaseHCatStorageHandler
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:512)
> ... 13 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
