[
https://issues.apache.org/jira/browse/HCATALOG-73?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122677#comment-13122677
]
[email protected] commented on HCATALOG-73:
-------------------------------------------------------
bq. On 2011-10-05 23:03:10, David Capwell wrote:
bq. >
storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputFormat.java,
line 40
bq. > <https://reviews.apache.org/r/2060/diff/3/?file=48147#file48147line40>
bq. >
bq. > Are you not extending TableOutputformat because of future changes
like disabling WAL and cacheing?
bq. >
bq. > http://hbase.apache.org/book/perf.writing.html#def.log.flush
bq. >
bq. > Deferred WAL writes seems to be table only and stuff like cacheing
can be controlled via the config object itself.
Disabling WAL is what I'd like to try but deferred WAL sounds good too. In
general having our own OutputFormat keeps us flexible. Deferred is a table
config tho we need it per session since we will most likely be support some
sort of point write mechanism.
bq. On 2011-10-05 23:03:10, David Capwell wrote:
bq. >
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/ManyMiniCluster.java,
line 59
bq. > <https://reviews.apache.org/r/2060/diff/3/?file=48149#file48149line59>
bq. >
bq. > Almost every place that uses this variable turns it into a File
object. Can we switch this to a File?
sure
bq. On 2011-10-05 23:03:10, David Capwell wrote:
bq. >
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/ManyMiniCluster.java,
line 121
bq. > <https://reviews.apache.org/r/2060/diff/3/?file=48149#file48149line121>
bq. >
bq. > Might also need to close the FileSystem. If you look at the HBase
test code itself, it closes it after it stops the DFS cluster.
bq. >
bq. > FileSystem fs = this.dfsCluster.getFileSystem();
bq. > if (fs != null) {
bq. > fs.close();
bq. > }
bq. > FileSystem.closeAll();
sounds good
bq. On 2011-10-05 23:03:10, David Capwell wrote:
bq. >
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/ManyMiniCluster.java,
line 205
bq. > <https://reviews.apache.org/r/2060/diff/3/?file=48149#file48149line205>
bq. >
bq. > Why not have HBase use HDFS directly?
Right now there is no need to. We don't have tests which require DFS, it would
just add overhead on resources and running time. I think running
LocalFileSystem disables block caching in hbase but we're not really testing
that.One thing I'd like to have running as well is a metastore daemon since the
fixture is running local mode right now.
bq. On 2011-10-05 23:03:10, David Capwell wrote:
bq. >
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/ManyMiniCluster.java,
line 171
bq. > <https://reviews.apache.org/r/2060/diff/3/?file=48149#file48149line171>
bq. >
bq. > Can we have this configurable via Configuration or via constructor?
Ok changed the instantiation pattern to be a builder. now you can set
numTaskTrackers, the different Conf objects and enabled/disable daemons. This
way you run only what you need or you can make the tests run against an actual
cluster. Doubt our tests will pass against a real cluster tho :D.
- Francis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2060/#review2371
-----------------------------------------------------------
On 2011-10-05 16:55:28, Francis Liu wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2060/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-10-05 16:55:28)
bq.
bq.
bq. Review request for hcatalog and Vandana Ayyalasomayajula.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hbase direct output storage driver as describe here:
bq.
bq.
https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Output+Storage+Driver+-+Design
bq.
bq.
bq. This addresses bug hcatalog-73.
bq. https://issues.apache.org/jira/browse/hcatalog-73
bq.
bq.
bq. Diffs
bq. -----
bq.
bq.
storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputFormat.java
PRE-CREATION
bq.
storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java
PRE-CREATION
bq.
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/ManyMiniCluster.java
PRE-CREATION
bq.
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/SkeletonHBaseTest.java
PRE-CREATION
bq.
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2060/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. added a unit mini cluster test framework and unit tests.
bq.
bq. since this is a sub-module of hcatalog parent tests shouldn't be affected.
bq.
bq.
bq. Thanks,
bq.
bq. Francis
bq.
bq.
> Output Storage Driver for HBase (Direct PUTs)
> ---------------------------------------------
>
> Key: HCATALOG-73
> URL: https://issues.apache.org/jira/browse/HCATALOG-73
> Project: HCatalog
> Issue Type: Sub-task
> Affects Versions: 0.3
> Reporter: Francis Liu
> Assignee: Francis Liu
> Labels: HBASE
> Attachments: hbase_storage_driver.patch,
> hbase_storage_driver_rev2.patch, hbase_storage_driver_rev3.patch,
> hbase_storage_driver_rev4.patch, hbase_storage_driver_rev5.patch
>
>
> This is the first variation for an HBase Output Storage Driver. Which will be
> doing individual PUTs for each row. The next iteration will involve the use
> of bulk imports.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
