[
https://issues.apache.org/jira/browse/BOOKKEEPER-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261591#comment-13261591
]
Flavio Junqueira commented on BOOKKEEPER-181:
---------------------------------------------
@roger It seems that there are two separate concerns with respect to
scalability, one that originally generated this issue and another that you're
raising about scans. Both sound important.
Our initial concern was about the amount of state a zookeeper ensemble can
handle. A zookeeper server keeps its state in memory, so all zookeeper state
must fit into the memory of a single server. For an application with a large
number of ledgers (tens to hundreds of millions), we can't use zookeeper.
Consequently, we proposed to enable options that can have essentially an
unbounded amount of storage.
The concern you're raising about the cost of scans for garbage-collection
purposes growing with the number of ledgers is valid, and it sounds like a good
idea to rethink the way we are garbage-collecting ledgers in bookies.
> Scale hedwig
> ------------
>
> Key: BOOKKEEPER-181
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-181
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-server, hedwig-server
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: hedwigscale.pdf, hedwigscale.pdf
>
>
> Current implementation of Hedwig and BookKeeper is designed to scale to
> hundreds of thousands of topics, but now we are looking at scaling them to
> tens to hundreds of millions of topics, using a scalable key/value store such
> as HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
