[
https://issues.apache.org/jira/browse/BOOKKEEPER-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258672#comment-13258672
]
Roger Bush commented on BOOKKEEPER-181:
---------------------------------------
I'd like to weigh in on the concept of accommodating more store types through
having a more abstract interface, without sacrificing the integrity of
BK/Hedwig.
Thinking about the basics that all KV stores have, the first concept is
"table". All KV stores can map a table (even if it is only a filename in an
mdbm). So clearly the table concept is such a common feature, we should have
something that maps to this.
A related question is, do we want to encode the concept of "column families"
into the interface? This is a relatively newer concept that not all kv stores
have. In fact, we should be careful to understand the different concepts here,
as there are a few that are usually bound together. One is that you can have,
in principal, an infinite number of columns. A second concept is that stores
may provide different levels of transactional semantics around the row object
(and thus, may or may not provide multi-column transactional updates). In the
future, NoSQL vendors will probably standardize on a few important concepts in
this area, but in my opinion we a ways off from this happening (to a level of
providing reasonable choice among mature candidate technologies).
However, the current implementation BK and Hedwig do not need these concepts
(since they do without them currently). I would argue that adding column
families restricts the types of K/V stores you can use, unnecessarily.
A simple workaround, which provides ability to work with both store types, is
to instead rely on non-transactional, infinite rows (which many kv stores
have). Using this more abundantly available feature, would map column families
to tables. For example, have a table whose name is = TABLE_NAME::COLUMN_NAME,
for each column family.
Concerning the possible transactional row update semantics: since BK and
Hedwig currently don't rely on this, should they start unnecessarily? Not
assuming column families communicates there is no need for transactional row
update semantics implicitly through the interface.
Relying on less (since Hedwig/BK prove they don't need it) makes our components
interoperate with more choice. It does require a tiny bit of ingenuity on the
plugin implementers part, but not much. And further this can be laid out as
implementation notes somewhere with the source.
Another thing I've been thinking about is SCAN. Scan is extremely useful for
offline administrative tasks (like garbage collection). It is far less useful
for serving operations - at least if you plan on using them without bounds.
The simple example that caught my eye as I was reading Sijie's document was
Bookie recovery. Are we really going to scan through all of the ledgers to
recover? While this seems like a simple, generic scheme, it may not be the one
we'd want to use. Do we really want the recovery process to be somehow related
to a possibly very large scan time? Do we really need scan for recovery? Or
is there an application implementation that does not use scan and that brings
up Bookie's quickly? Scanning big tables is usually one of those application
mistakes that needs to get fixed, regardless of whether you are using a NoSQL
or SQL database (I'm thinking of the many times I have heard someone say
derisively "they were scanning the biggest DB table" - as the "problem" that
had to get fixed). I have to look more at the BK and Hedwig code to understand
this.
> Scale hedwig
> ------------
>
> Key: BOOKKEEPER-181
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-181
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-server, hedwig-server
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: hedwigscale.pdf, hedwigscale.pdf
>
>
> Current implementation of Hedwig and BookKeeper is designed to scale to
> hundreds of thousands of topics, but now we are looking at scaling them to
> tens to hundreds of millions of topics, using a scalable key/value store such
> as HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
