[
https://issues.apache.org/jira/browse/HBASE-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650413#action_12650413
]
stack commented on HBASE-1024:
------------------------------
Good stuff Andrew. I concur. Lets take on these goals. Shall we put up a
wiki page pointing to this issue that talks of rearchitecting project? and
maybe belong in a working document of their own. Perhaps a page up on wiki --
like http://wiki.apache.org/hadoop/Hbase/NewFileFormat -- with perhaps a link
under roadmap or maybe we need an architectural goals section where we stable
these targets (and X-them out as we knock them off)?
> Rearchitect regionserver I/O
> ----------------------------
>
> Key: HBASE-1024
> URL: https://issues.apache.org/jira/browse/HBASE-1024
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: io, regionserver
> Reporter: Andrew Purtell
>
> Consider supporting:
> * 2GB store files
> * 1TB per node (500 store files)
> * Cell values up to ~100MB
> * Typical use case of RS running with 1GB of heap only
> Some ideas:
> * Drop MapFile and make a custom store file format with (competing) design
> goals:
> ** heap efficiency
> ** fast lookups
> ** minimize I/O operations
> ** optimize for typical DFS blocksizes (8MB, 64MB)
> * MRU cache for filehandles and store file indexes
> * Memory mapped store file indexes -- don't hold the indexes in heap; rely on
> the OS blockcache for performance
> * "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
