[
https://issues.apache.org/jira/browse/BOOKKEEPER-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260374#comment-13260374
]
Ivan Kelly commented on BOOKKEEPER-181:
---------------------------------------
{quote}
Concerning scan, there definitely needs to be cursors to break this up into
chunks, that is a certainty for scan. However, in the case of BookKeeper, it
seems like the use case is simply for deleting ledgers that have been marked
delete by Hedwig. In that case, couldn't we just use a simple "dequeue
abstraction"? This would avoid the need for scan. Here's a sketch of how I
envision this working: Each bookie has it's own "table" for this. This would
let the trimmer do some nice scheduling of deletes that maps to the physical
layer (we'd want to remove ledgers at just a little bit faster than they are
marked delete - and perhaps there is a knob to make this go faster, in
emergency situations when we need to recover space). The table for the bookie
would have an expanding list (the dequeue) where entries are added at the tail,
and removed from the head. There would be another table that would hold a
reference to both the head and tail. These would be updated in a fashion such
that any system crash simply results in a blank entry just after/below the
head/tail. The entries would have a key "1", "2", ... and the value would be
the ledger id. The trimmer would remove from the earliest entries (and update
it's cursor), and you would add new mark deletes to the tail. You'd need to use
CAS on the cursor update, and do things in such a way that failure doesn't
strand an entry outside of the bounds of the head/tail cursor. There is
wraparound to think of (tail cursor going too high). This must be a technique
that already exists, no?
{quote}
This is adding another abstraction into the metastore interface which only
makes sense for the bookkeeper delete ledger scenario. For hedwig this makes no
sense. Moreover, Hedwig does require scan. To use a deque like this, it would
be better to skip the metastore interface, and instead implement the
LedgerManager interface directly.
It could also create problems with consistency, if after taking from the queue,
the deleter crashes. This could leave a ledger dangling, which we would still
require garbage collection to make sure its cleaned up.
Not having scan also has other implications for BK. For 4.2.0 we want to
implement a "fsck" functionallity, which checks that each bookie contains every
ledger entry it should. This requires that we be able to get a list of ledgers.
> Scale hedwig
> ------------
>
> Key: BOOKKEEPER-181
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-181
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-server, hedwig-server
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: hedwigscale.pdf, hedwigscale.pdf
>
>
> Current implementation of Hedwig and BookKeeper is designed to scale to
> hundreds of thousands of topics, but now we are looking at scaling them to
> tens to hundreds of millions of topics, using a scalable key/value store such
> as HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
