Raven888888 commented on issue #1908:
URL: https://github.com/apache/bookkeeper/issues/1908#issuecomment-1288005537
This is what found in logs
Bookie
```
2022-10-20T09:53:23,089+0000 [db-storage-cleanup-10-1] INFO
org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes
for ledgers: [501062, 501893, 501923, 501378, 501922, 501921, 500143, 500142,
501901, 500141, 501898, 501896, 501911, 501047, 501366, 501909, 501908, 501055,
501918, 501054, 501916, 501913, 501912]
2022-10-20T09:53:23,754+0000 [db-storage-cleanup-10-1] WARN
org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage -
Failed to cleanup db indexes
java.io.IOException: Error in RocksDB compact
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.compact(KeyValueStorageRocksDB.java:295)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex.removeOffsetFromDeletedLedgers(EntryLocationIndex.java:256)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.lambda$checkpoint$7(SingleDirectoryDbLedgerStorage.java:638)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_342]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_342]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_342]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_342]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_342]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_342]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: org.rocksdb.RocksDBException: block checksum mismatch: expected
459167422, got 2336252423 in
data/bookkeeper/ledgers/current/locations/000423.sst offset 612461 size 32254
at org.rocksdb.RocksDB.compactRange(Native Method)
~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
at org.rocksdb.RocksDB.compactRange(RocksDB.java:3167)
~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
at org.rocksdb.RocksDB.compactRange(RocksDB.java:3135)
~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.compact(KeyValueStorageRocksDB.java:293)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
... 10 more
2022-10-20T09:54:23,026+0000 [SyncThread-7-1] ERROR
org.apache.bookkeeper.bookie.SyncThread - Exception flushing ledgers
java.io.IOException: Failed to flush RocksDB batch
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB$RocksDBBatch.flush(KeyValueStorageRocksDB.java:451)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.checkpoint(SingleDirectoryDbLedgerStorage.java:621)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.checkpoint(DbLedgerStorage.java:231)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.SyncThread.checkpoint(SyncThread.java:152)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.SyncThread.lambda$doCheckpoint$0(SyncThread.java:99)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_342]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_342]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_342]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_342]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_342]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_342]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: org.rocksdb.RocksDBException: block checksum mismatch: expected
459167422, got 2336252423 in
data/bookkeeper/ledgers/current/locations/000423.sst offset 612461 size 32254
at org.rocksdb.RocksDB.write0(Native Method)
~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
at org.rocksdb.RocksDB.write(RocksDB.java:1586)
~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB$RocksDBBatch.flush(KeyValueStorageRocksDB.java:449)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
... 12 more
2022-10-20T09:54:28,204+0000 [main-EventThread] INFO
org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown
due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged
path:/ledgers/underreplication/locks
2022-10-20T09:54:28,206+0000 [main-EventThread] INFO
org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown
due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged
path:/ledgers/underreplication/locks
2022-10-20T09:55:23,019+0000 [SyncThread-7-1] ERROR
org.apache.bookkeeper.bookie.SyncThread - Exception flushing ledgers
java.io.IOException: Failed to flush RocksDB batch
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB$RocksDBBatch.flush(KeyValueStorageRocksDB.java:451)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.checkpoint(SingleDirectoryDbLedgerStorage.java:621)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.checkpoint(DbLedgerStorage.java:231)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.SyncThread.checkpoint(SyncThread.java:152)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.bookie.SyncThread.lambda$doCheckpoint$0(SyncThread.java:99)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_342]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_342]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_342]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_342]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_342]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_342]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: org.rocksdb.RocksDBException: block checksum mismatch: expected
459167422, got 2336252423 in
data/bookkeeper/ledgers/current/locations/000423.sst offset 612461 size 32254
at org.rocksdb.RocksDB.write0(Native Method)
~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
at org.rocksdb.RocksDB.write(RocksDB.java:1586)
~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB$RocksDBBatch.flush(KeyValueStorageRocksDB.java:449)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
... 12 more
```
broker
```
2022-10-20T13:34:33,566+0000 [pulsar-ph-kafka-79-21] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger
public/kafka/persistent/hourly-datetime-Ingest-partition-0
2022-10-20T13:34:33,569+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[public/kafka/persistent/hourly-datetime-Ingest-partition-0] Closing managed
ledger
2022-10-20T13:34:33,569+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
INFO org.apache.pulsar.broker.PulsarService - No ledger offloader configured,
using NULL instance
2022-10-20T13:34:33,569+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed
ledger public/kafka/persistent/hourly-datetime-Ingest
2022-10-20T13:34:33,574+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[public/kafka/persistent/hourly-datetime-Ingest] Closing managed ledger
2022-10-20T13:34:33,574+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
ERROR io.streamnative.pulsar.handlers.kop.KafkaTopicManager - [[id: 0x864f8b07,
L:/192.168.3.3:9092 - R:/192.168.3.2:33020]]Get empty non-partitioned topic for
name persistent://public/kafka/hourly-datetime-Ingest
2022-10-20T13:34:33,574+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
ERROR io.streamnative.pulsar.handlers.kop.KafkaTopicManager - [[id: 0x54c1cef3,
L:/192.168.3.3:9092 - R:/192.168.3.2:33024]]Get empty non-partitioned topic for
name persistent://public/kafka/hourly-datetime-Ingest
2022-10-20T13:34:33,675+0000 [pulsar-ph-kafka-79-21] INFO
org.apache.pulsar.broker.PulsarService - No ledger offloader configured, using
NULL instance
2022-10-20T13:34:33,675+0000 [pulsar-ph-kafka-79-21] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger
public/kafka/persistent/hourly-datetime-Ingest-partition-0
2022-10-20T13:34:33,679+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[public/kafka/persistent/hourly-datetime-Ingest-partition-0] Closing managed
ledger
2022-10-20T13:34:33,679+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
INFO org.apache.pulsar.broker.PulsarService - No ledger offloader configured,
using NULL instance
2022-10-20T13:34:33,679+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed
ledger public/kafka/persistent/hourly-datetime-Ingest
2022-10-20T13:34:33,684+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[public/kafka/persistent/hourly-datetime-Ingest] Closing managed ledger
2022-10-20T13:34:33,684+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
ERROR io.streamnative.pulsar.handlers.kop.KafkaTopicManager - [[id: 0x864f8b07,
L:/192.168.3.3:9092 - R:/192.168.3.2:33020]]Get empty non-partitioned topic for
name persistent://public/kafka/hourly-datetime-Ingest
2022-10-20T13:34:33,684+0000 [bookkeeper-ml-scheduler-OrderedScheduler-11-0]
ERROR io.streamnative.pulsar.handlers.kop.KafkaTopicManager - [[id: 0x54c1cef3,
L:/192.168.3.3:9092 - R:/192.168.3.2:33024]]Get empty non-partitioned topic for
name persistent://public/kafka/hourly-datetime-Ingest
2022-10-20T13:34:33,685+0000 [pulsar-web-35-9] INFO
org.eclipse.jetty.server.RequestLog - 192.168.3.3 - - [20/Oct/2022:13:34:33
+0000] "GET /admin/v2/persistent/public/kafka/hourly-datetime-Ingest/partitions
HTTP/1.1" 200 16 "-" "Pulsar-Java-v2.10.0" 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
