[
https://issues.apache.org/jira/browse/BOOKKEEPER-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157078#comment-13157078
]
[email protected] commented on BOOKKEEPER-39:
---------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2817/#review3508
-----------------------------------------------------------
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java
<https://reviews.apache.org/r/2817/#comment7830>
break line
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java
<https://reviews.apache.org/r/2817/#comment7832>
Hmm, it seems we have an inconsistency in how the configuration keys are
named. Some are camelcase, some are underscored. I think it would be better to
keep them as CamelCase, so ledgerManagerType and zkLedgersRootPath here.
I'll open another JIRA for renaming the others.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java
<https://reviews.apache.org/r/2817/#comment7831>
This is no longer needed.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManager.java
<https://reviews.apache.org/r/2817/#comment7833>
break line
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManagerFactory.java
<https://reviews.apache.org/r/2817/#comment7834>
There's no point in having 2 factory methods here. Merge them.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookkeeperInternalCallbacks.java
<https://reviews.apache.org/r/2817/#comment7835>
This is only used in one class, and isn't quite a callback either, so I
think it should be a private class of HierarchicalLedgerManager.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7836>
typo (existing)
Remove said from these lines. instead, put zkActiveLedgers in brackets. i.e.
Fetch all existing ledgers from zookeeper (*zkActiveLedgers*).
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7837>
Fetch all ledgers currently active within the Bookie (*bkActiveLedger*).
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7838>
to find those ledgers which do not exist in *zkActiveLedgers* and garbage
collect them.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7839>
which avoids the problem of the child list being larger than the maximum
ZooKeeper packet size.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7840>
manages its active ledgers in a sorted map, which simplifies access to
active ledgers in a particular (level1, level2) partition.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7841>
as follows.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7842>
belonging
Again, remove said and put *zkActiveLedgers* in round brackets.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7843>
Fetch all ledgers currently active in the bookie which belong to partition
(level1, level2) (*bkActiveLedgers*).
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7844>
those ledgers which do not exist in *zkActiveLedgers*, and garbage collect
them.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
<https://reviews.apache.org/r/2817/#comment7845>
Hierarchical
- Ivan
On 2011-11-25 09:13:10, Sijie Guo wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2817/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-25 09:13:10)
bq.
bq.
bq. Review request for bookkeeper.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Create a interface LedgerManager to handle 'how to store ledger meta in
zookeeper, how bookie server manages active ledgers and garbage collect those
inactive/deleted ledgers'. Implement this interface using hash mechanism, which
hides detail from client/server code and makes code more clearer.
bq.
bq.
bq. This addresses bug BOOKKEEPER-39.
bq. http://issues.apache.org/jira/browse/BOOKKEEPER-39
bq.
bq.
bq. Diffs
bq. -----
bq.
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/conf/bk_server.conf
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogger.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCache.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerCreateOp.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerDeleteOp.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerOpenOp.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/AbstractZkLedgerManager.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManager.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManager.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManager.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManagerFactory.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookkeeperInternalCallbacks.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/StringUtils.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ZkUtils.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BaseTestCase.java
1206076
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/hierarchical/HierarchicalAsyncLedgerOpsTest.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/hierarchical/HierarchicalBookieFailureTest.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/hierarchical/HierarchicalBookieReadWriteTest.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/hierarchical/HierarchicalBookieRecoveryTest.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/hierarchical/HierarchicalLedgerDeleteTest.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/doc/bookkeeperConfig.textile
1206076
bq.
bq. Diff: https://reviews.apache.org/r/2817/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Sijie
bq.
bq.
> Bookie server failed to restart because of too many ledgers (more than
> ~50,000 ledgers)
> ---------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-39
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-39
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: bookkeeper-39.patch, bookkeeper-39.patch_v2,
> bookkeeper-39.patch_v3, bookkeeper-39.patch_v4
>
>
> If we have ~500,000 topics in hedwig, we might have more than ~500,000
> ledgers in bookkeeper (a topic has more than 1 ledger). So when the bookie
> server restarted, a logfile GC thread is started, which will call
> zk.getChildren to fetch all ledgers, and it failed because of package length
> limitation.
> 2011-08-01 01:18:46,373 - ERROR
> [main-EventThread:EntryLogger$GarbageCollectorThread$1@164] - Error polling
> ZK for the available ledger nodes:
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /ledgers
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1519)
> at
> org.apache.bookkeeper.bookie.EntryLogger$GarbageCollectorThread$1.processResult(EntryLogger.java:162)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:592)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:481)
> 2011-08-01 01:18:46,373 - WARN [main-EventThread:Bookie$1@242] - ZK client
> has been disconnected to the ZK server!
> 2011-08-01 01:18:47,278 - WARN
> [main-SendThread(perf13.platform.mobile.sp2.yahoo.com:2181):ClientCnxn$SendThread@980]
> - Session 0x131833dec850034 for server
> perf13.platform.mobile.sp2.yahoo.com/98.139.43.86:2181, unexpected error,
> closing socket connection and attempting reconnect
> java.io.IOException: Packet len9976413 is out of range!
> at
> org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:78)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:264)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:958)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
