This is an automated email from the ASF dual-hosted git repository.
chenhang pushed a change to branch branch-4.14
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
from c4b944218a [branch-4.14] Fix ByteBuf memory leak problem when
setExplicitLac (#3617)
new 7267058640 Ledger replicate supports throttle (#2778)
new a13dfce431 improve the throttle function (#2991)
new 8405a24bd8 add stats for throttled-write (#3102)
new e3091df85d [Client] Deduplicate error log for SSLException (#3320)
new 6bad86ead5 Autorecovery to rereplicate empty ledgers (#3239)
new 39a88b64bd Fix bookie CI test not run (#3367)
new 8ce6538644 Shut down ReplicationWorker and Auditor on non-recoverable
ZK error (#3374)
new cbaf9f899a fix checkAllLedgersDuration compute (#2970)
new c85d0f62b2 add writeThreadQueuedLatency (#3363)
new 0c888a4db2 use LinkedList to take place of ArrayList. (#3330)
new eeaec84c2e Fix the 3141 revert issue (#3283)
new 3f0437bed5 avoid init WriteSet when waitForWriteSetMs < 0. (#3325)
new 7676215f34 enhance future sync wait. (#3336)
new 7158a67446 fix underReplicatedLedgerTotalSize calculate problem.
(#3337)
new 2d7c0a5616 Fix autoRecover memory leak. (#3361)
new da532fc8b1 Stable testBookieContinueWritingIfMultipleLedgersPresent
test (#3421)
new 83769a669b Pre break loop when self create layoutZNode succeed. (#3335)
new 7e8836659c Fix wrong ledger id parse radix for index relocation file
in IndexPersistenceMgr (#2944)
new bbf80c47be [conf] minorCompactionInterval should be greater than
gcWaitTime (#2116)
new d8484b4976 Make sure the LedgerHandle close callback can be completed
when encounter exception (#2913)
new 523ffcf3ff [ISSUE 2637] Fix jvm_memory_direct_bytes_used metrics when
using jdk11+ (#3252)
new 23355e0912 Issue #2908: Replace unsafe NoEntryException with
IOException (#2909)
new d2d819482c Add latency stats for entry location index lookup so that
possible RocksDB bottleneck can be detected (#3444)
new a57f24ba40 Optimize log for failed to write entry (#3463)
new e7a8af2085 Fix the deadlock when only using io thread to handle
request (#3480)
new 61a4ea710a upgrade hadoop version to 3.2.4 (#3485)
new ebd5e4c11f checkAllLedgers in Auditor supports read throttle (#2973)
new 376040c499 Issue 3070: Fix bug where checkAllLedgers gets stuck when
read throttling is enabled (#3214)
new c5017ec053 zk client config update and bugfix for
ZKMetadataClientDriver (#2958)
new e66cedc522 Return too many requests error when the write cache flush
timeout occurs (#2860)
new 9329e3c69a Fix memory leak when reading entry but the connection
disconnected. (#3528)
new 224254b542 fix close ledgerAuditorManager repeatedly (#3503)
new 969bfc8c5b Rename success with writableResult and update final
writableResult about wait writeSet (#3505)
new 960e801b54 fix cherry-pick issue
new e61a913a7f skipping placementPolicyCheck when ledger replication
disabled (#3561)
new 1eed22a7cc skip replicasCheck when replication disabled (#3563)
new 4fa21138ed Replace sleep with await to avoid flaky test in
SlowBookieTest (#3581)
new 65bdef362d Add example for multiple server list in metadataServiceUri
configuration (#3580)
new 5be36acd27 [test] Fix flaky test testAutoRecoverySessionLoss (#3576)
new 71b36ac61b [security] Upgrade Jetty to 9.4.48.v20220622 to get rid of
CVE-2022-2047 (#3404)
new 2438673d38 Bump jcommander from 1.78 to 1.82 (#3476)
new e657c06cff Upgrade dependencies for CVE-2022-3171 and CVE-2022-42003
(#3579)
new 760cacd38b LedgerOpenOp: Do not call blocking close() in the callback
(#3513)
new cee32aa7a3 When call openLedgerOp, make the timeout ex is a separate
error code (#3562)
new 35d0ecf026 update ALLOCATOR_POOLING_CONCURRENCY default value (#3001)
new 065d93ccb0 Consolidate Netty channel flushes to mitigate syscall
overhead (#3383)
new 8b790bc27a add javadoc comments for test classes (#3587)
new 8e268d2473 update doc about flushInterval config (#3601)
new ebb434b30a ledgerFragment check and results keep order (#3504)
new ab6e15724e Bump snakeyaml from 1.31 to 1.32 to solve CVE-2022-38752
(#3491)
new 1229a8bc1b Adjust log stack output (#3496)
new 82a142fd1a docs: fix bookkeeper command path in decommission doc
(#3499)
new b60c677a4f Reduce unnecessary loop in removeIf if section is empty
(#3512)
new ca032236d4 LedgerHandle: do not complete metadata operation on the
ZookKeeper/Metadata callback thread (#3516)
new 0abc039714 Show content of map (#3538)
new 918796e68e Using a separate thread pool to execute openWithMetadata
(#3548)
new 4d90fe2c20 [configuration] fix typo in bk_server.conf (#3574)
new d10a9e4906 Correct link class name (#3594)
new b0a10f9977 fix cherry-pick issue
new c20c92122e fix cherry-pick issue
new 194a0819d1 Fix flaky AutoRecoveryMainTest (#2881)
new d4c4e21da7 BookieAutoRecoveryTest.testEmptyLedgerLosesQuorumEventually
fix flaky test, ensure that the Auditor is alive (#3149)
new a32d294d5b fix: Stream Storage tests are flaky; statelib test times
out (#2883)
new dda86ad51c fix cherry-pick issue
The 64 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/bookie-tests.yml | 8 +-
.../bookkeeper/common/concurrent/FutureUtils.java | 14 +-
.../apache/bookkeeper/test/TestStatsProvider.java | 4 +
.../src/main/resources/LICENSE-all.bin.txt | 34 ++---
.../src/main/resources/LICENSE-bkctl.bin.txt | 18 +--
.../src/main/resources/LICENSE-server.bin.txt | 34 ++---
.../src/main/resources/NOTICE-all.bin.txt | 18 +--
.../src/main/resources/NOTICE-bkctl.bin.txt | 2 +-
.../src/main/resources/NOTICE-server.bin.txt | 18 +--
bookkeeper-server/pom.xml | 5 +
.../bookkeeper/bookie/BookKeeperServerStats.java | 1 +
.../apache/bookkeeper/bookie/BookieException.java | 2 +-
.../org/apache/bookkeeper/bookie/BookieShell.java | 6 +-
.../apache/bookkeeper/bookie/EntryLogMetadata.java | 4 +-
.../org/apache/bookkeeper/bookie/EntryLogger.java | 17 +--
.../org/apache/bookkeeper/bookie/FileInfo.java | 1 +
.../bookkeeper/bookie/GarbageCollectorThread.java | 4 +-
.../bookkeeper/bookie/IndexPersistenceMgr.java | 5 +-
.../bookie/storage/ldb/DbLedgerStorageStats.java | 10 ++
.../bookie/storage/ldb/EntryLocationIndex.java | 13 +-
.../storage/ldb/EntryLocationIndexStats.java | 9 ++
.../ldb/SingleDirectoryDbLedgerStorage.java | 4 +
.../org/apache/bookkeeper/client/BKException.java | 2 +-
.../apache/bookkeeper/client/BookKeeperAdmin.java | 20 ++-
.../apache/bookkeeper/client/LedgerChecker.java | 83 ++++++++--
.../apache/bookkeeper/client/LedgerCreateOp.java | 12 +-
.../client/LedgerFragmentReplicator.java | 45 +++++-
.../org/apache/bookkeeper/client/LedgerHandle.java | 78 ++++++----
.../apache/bookkeeper/client/LedgerHandleAdv.java | 12 +-
.../org/apache/bookkeeper/client/LedgerOpenOp.java | 70 ++++++---
.../apache/bookkeeper/client/LedgerRecoveryOp.java | 2 +
.../client/ListenerBasedPendingReadOp.java | 4 +-
.../org/apache/bookkeeper/client/PendingAddOp.java | 4 +-
.../apache/bookkeeper/client/PendingReadOp.java | 6 +-
.../RackawareEnsemblePlacementPolicyImpl.java | 4 +-
.../bookkeeper/conf/AbstractConfiguration.java | 49 +++++-
.../bookkeeper/conf/ServerConfiguration.java | 18 +++
.../bookkeeper/meta/AbstractZkLedgerManager.java | 42 ++++++
.../bookkeeper/meta/FlatLedgerManagerFactory.java | 4 +-
.../org/apache/bookkeeper/meta/LedgerManager.java | 27 ++--
.../bookkeeper/meta/LedgerManagerFactory.java | 3 +-
.../LegacyHierarchicalLedgerManagerFactory.java | 4 +-
.../bookkeeper/meta/MSLedgerManagerFactory.java | 3 +-
.../meta/ZkLedgerUnderreplicationManager.java | 68 +++++----
.../bookkeeper/meta/zk/ZKMetadataBookieDriver.java | 2 +-
.../bookkeeper/meta/zk/ZKMetadataClientDriver.java | 2 +-
.../apache/bookkeeper/proto/BookieNettyServer.java | 3 +
.../apache/bookkeeper/proto/BookieProtocol.java | 43 ++++--
.../bookkeeper/proto/PacketProcessorBase.java | 26 +++-
.../bookkeeper/proto/PerChannelBookieClient.java | 4 +-
.../org/apache/bookkeeper/proto/RequestStats.java | 10 ++
.../bookkeeper/proto/WriteEntryProcessor.java | 2 +-
.../bookkeeper/proto/WriteEntryProcessorV3.java | 4 +-
.../org/apache/bookkeeper/replication/Auditor.java | 84 +++++++++--
.../bookkeeper/replication/AuditorElector.java | 7 +-
.../bookkeeper/replication/AutoRecoveryMain.java | 5 +
.../replication/ReplicationException.java | 25 +++
.../bookkeeper/replication/ReplicationWorker.java | 27 +++-
.../autorecovery/ListUnderReplicatedCommand.java | 3 +-
.../QueryAutoRecoveryStatusCommand.java | 3 +-
.../cli/commands/autorecovery/ToggleCommand.java | 3 +-
.../bookie/UpdateBookieInLedgerCommand.java | 2 +-
.../tools/cli/commands/bookies/RecoverCommand.java | 5 +
.../apache/bookkeeper/util/LocalBookKeeper.java | 2 +-
.../util/collections/ConcurrentLongHashMap.java | 2 +-
.../collections/ConcurrentLongLongHashMap.java | 20 ++-
.../util/collections/ConcurrentOpenHashMap.java | 2 +-
.../bookie/ForceAuditorChecksCmdTest.java | 5 +-
.../bookie/GarbageCollectorThreadTest.java | 4 +-
.../bookkeeper/bookie/IndexPersistenceMgrTest.java | 127 ++++++++++++++--
.../apache/bookkeeper/bookie/LedgerCacheTest.java | 9 +-
.../org/apache/bookkeeper/bookie/TestEntryLog.java | 3 +
.../bookie/storage/ldb/EntryLocationIndexTest.java | 29 ++++
.../bookkeeper/client/BookKeeperCloseTest.java | 2 +-
.../bookkeeper/client/BookieDecommissionTest.java | 102 ++++++++++++-
.../client/BookieRecoveryUseIOThreadTest.java | 80 ++++++++++
.../bookkeeper/client/BookieWriteLedgerTest.java | 6 +-
.../apache/bookkeeper/client/MockBookKeeper.java | 17 ++-
.../apache/bookkeeper/client/MockLedgerHandle.java | 5 +
.../apache/bookkeeper/client/SlowBookieTest.java | 40 ++++-
.../bookkeeper/client/TestLedgerChecker.java | 26 ++--
.../bookkeeper/client/UpdateLedgerOpTest.java | 8 +-
.../api/BookKeeperBuildersOpenLedgerTest.java | 167 +++++++++++++++++++++
.../client/api/BookKeeperBuildersTest.java | 42 ------
.../bookkeeper/conf/TestBKConfiguration.java | 8 +
.../bookkeeper/conf/TestServerConfiguration.java | 22 ++-
.../meta/AbstractZkLedgerManagerTest.java | 7 +
.../bookkeeper/proto/WriteEntryProcessorTest.java | 51 ++++++-
.../proto/WriteEntryProcessorV3Test.java | 32 ++++
.../replication/AuditorPeriodicCheckTest.java | 3 +-
.../replication/AuditorReplicasCheckTest.java | 2 +-
.../replication/AutoRecoveryMainTest.java | 76 ++++++----
.../replication/BookieAutoRecoveryTest.java | 7 +-
.../replication/TestReplicationWorker.java | 83 ++++++++--
.../bookkeeper/test/BookKeeperClusterTestCase.java | 14 +-
.../apache/bookkeeper/test/ReadOnlyBookieTest.java | 2 +
.../java/org/apache/bookkeeper/tls/TestTLS.java | 2 +-
.../zookeeper/MockZooKeeperTestCase.java | 32 ++++
.../prometheus/PrometheusMetricsProvider.java | 30 ++--
.../prometheus/TestPrometheusMetricsProvider.java | 40 +++++
.../org/apache/bookkeeper/stats/OpStatsLogger.java | 8 +-
conf/bk_server.conf | 9 +-
.../metadata/etcd/EtcdLedgerManagerFactory.java | 3 +-
pom.xml | 28 +++-
site/docs/4.11.0/admin/decomission.md | 4 +-
.../service/RegistrationServiceProvider.java | 2 +-
.../kv/TestRocksdbKVAsyncStoreWithCheckpoints.java | 5 +-
.../testing/executors/MockExecutorController.java | 26 +++-
.../ListUnderReplicatedCommandTest.java | 8 +-
109 files changed, 1764 insertions(+), 489 deletions(-)
create mode 100644
bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryUseIOThreadTest.java
create mode 100644
bookkeeper-server/src/test/java/org/apache/bookkeeper/client/api/BookKeeperBuildersOpenLedgerTest.java
