[
https://issues.apache.org/jira/browse/HCATALOG-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229029#comment-13229029
]
[email protected] commented on HCATALOG-302:
--------------------------------------------------------
bq. On 2012-03-14 03:05:47, Francis Liu wrote:
bq. >
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/pig/HCatStorer.java,
line 80
bq. > <https://reviews.apache.org/r/4322/diff/1/?file=91790#file91790line80>
bq. >
bq. > nitpick: use getProperty isntead so you don't have to cast it to
string.
Done.
bq. On 2012-03-14 03:05:47, Francis Liu wrote:
bq. >
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java,
line 89
bq. > <https://reviews.apache.org/r/4322/diff/2/?file=91963#file91963line89>
bq. >
bq. > nitpick: since you're changing this can you fix the formatting? :-)
sure.
bq. On 2012-03-14 03:05:47, Francis Liu wrote:
bq. >
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java,
line 112
bq. > <https://reviews.apache.org/r/4322/diff/2/?file=91963#file91963line112>
bq. >
bq. > I'd rather add it directly to JobConf. That way it's less of a hack
and less dependent on an unplanned behavior.
Done.
bq. On 2012-03-14 03:05:47, Francis Liu wrote:
bq. >
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java,
line 128
bq. > <https://reviews.apache.org/r/4322/diff/2/?file=91963#file91963line128>
bq. >
bq. > add a TODO to remove this once HCAT-308 is fixed.
Done.
bq. On 2012-03-14 03:05:47, Francis Liu wrote:
bq. >
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java,
line 195
bq. > <https://reviews.apache.org/r/4322/diff/2/?file=91963#file91963line195>
bq. >
bq. > shouldn't you add it to copy? for the output you don't need the hack
of changing conf do you?
Yes. Will remove it when we remove for the input after HCAT-308. Just put in
there so it is similar to input.
bq. On 2012-03-14 03:05:47, Francis Liu wrote:
bq. >
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java,
line 163
bq. > <https://reviews.apache.org/r/4322/diff/2/?file=91963#file91963line163>
bq. >
bq. > Does this work without the foolproof code? Let's try to contain the
bug with pig in HCatStorer. It'll just add more complexity.
Nothing complex. Just added to check to ensure we don't create transaction
again if same jobConf is passed again with a different OutputJobInfo. In Pig's
case we fixed to pass same OutputJobInfo but it is with a different JobConf .
Added a test case assert for the two cases.
- Rohini
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4322/#review5917
-----------------------------------------------------------
On 2012-03-14 01:07:44, Rohini Palaniswamy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4322/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-14 01:07:44)
bq.
bq.
bq. Review request for hcatalog, Alan Gates, Francis Liu, and Vandana
Ayyalasomayajula.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Fixes to get Hbase storagehandler working with Pig.
bq.
bq.
bq. 1)Includes changes to HCatStorer used by Pig. Modified HCatStorer to pass
the same OutputJobInfo in the multiple calls made in Pig to
HCatOutputFormat.setOutput. Else we ended up creating multiple orphaned
transactions which were never committed in zookeeper.
bq. 2) Moved stuff from getInputSplits and checkoutputSpecs to configure
methods in storagehandler.
bq. 3) Fixed a logic error in hbase record reader.
bq.
bq.
bq. This addresses bug HCATALOG-302.
bq. https://issues.apache.org/jira/browse/HCATALOG-302
bq.
bq.
bq. Diffs
bq. -----
bq.
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/pig/HCatStorer.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseBulkOutputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseInputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HbaseSnapshotRecordReader.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputFormat.java
1299921
bq.
http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseInputFormat.java
1299921
bq.
bq. Diff: https://reviews.apache.org/r/4322/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Unit tests and Integration testing done.
bq.
bq.
bq. Thanks,
bq.
bq. Rohini
bq.
bq.
> unable to write to hbase channel. HBaseHCatStorageHandler class not found
> -------------------------------------------------------------------------
>
> Key: HCATALOG-302
> URL: https://issues.apache.org/jira/browse/HCATALOG-302
> Project: HCatalog
> Issue Type: Bug
> Components: hbase
> Affects Versions: 0.4
> Reporter: David Capwell
> Assignee: Rohini Palaniswamy
> Fix For: 0.4
>
> Attachments: HCATALOG-302-1.patch
>
>
> This is the pig script:
> PigServer pigServer = PigServerBuilder.create(this.client.getConf());
> pigServer.registerQuery("A = LOAD '"+input+"' USING PigStorage()
> AS(key:chararray, value:chararray);");
> pigServer.registerQuery("STORE A INTO '"+this.getDb().getName() + "." +
> this.getTable().getName()+"' USING org.apache.hcatalog.pig.HCatStorer();");
> Error:
> 2012-03-09 03:04:12,105 WARN org.apache.hadoop.mapred.Child: Error running
> child
> java.io.IOException: Error in loading storage
> handler.org.apache.hcatalog.hbase.HBaseHCatStorageHandler
> at
> org.apache.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:518)
> at
> org.apache.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:474)
> at
> org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getOutputFormat(HCatBaseOutputFormat.java:77)
> at
> org.apache.hcatalog.mapreduce.HCatOutputFormat.getOutputCommitter(HCatOutputFormat.java:250)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:89)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:67)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:278)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1082)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hcatalog.hbase.HBaseHCatStorageHandler
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:512)
> ... 13 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
