[
https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129642#comment-16129642
]
ASF GitHub Bot commented on KAFKA-5733:
---------------------------------------
GitHub user guozhangwang opened a pull request:
https://github.com/apache/kafka/pull/3681
KAFKA-5733: RocksDB bulk load with lower number of levels.
This is to complete Bill's PR on KAFKA-5733, incorporating the suggestion
in https://github.com/facebook/rocksdb/issues/2734.
Some minor changes: move `open = true` in `openDB`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guozhangwang/kafka K5733-rocksdb-bulk-load
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3681.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3681
----
commit 2cfc54c8e77b840acc0d176cea8eabda4d1c4b03
Author: Bill Bejeck <[email protected]>
Date: 2017-08-14T14:46:08Z
KAFKA-5733: ensure clean RocksDB directory before setting
prepareForBulkload settings
commit f468147455c9f8eaf640f2821b7849b7b370968d
Author: Bill Bejeck <[email protected]>
Date: 2017-08-14T16:57:29Z
KAFKA-5733: address comments
commit 149634b405cd5d73ec0e27c9a55b16f9971abf1f
Author: Bill Bejeck <[email protected]>
Date: 2017-08-14T18:20:05Z
KAFKA-5733: address additional comments, enforce not opening
and closing if not a clean RocksDB dir on restore
commit b4cd505b5c5cb9e57df4728414b5194a65d14b1c
Author: Guozhang Wang <[email protected]>
Date: 2017-08-16T22:17:17Z
Merge branch 'KAFKA-5733_rocks_db_throws_more_than_num_levels_on_restore'
of https://github.com/bbejeck/kafka into K5733-rocksdb-bulk-load
commit d0ada11eb8e97fbd5efdfb4c66160a81970575e4
Author: Guozhang Wang <[email protected]>
Date: 2017-08-16T22:45:16Z
use compactRange to still enable bulk loading
commit 08da1f16c8aeb587dd8b1ac7b5192f1ab3395cd7
Author: Guozhang Wang <[email protected]>
Date: 2017-08-16T23:53:02Z
minor fixes
----
> System tests get exception RocksDBException: db has more levels than
> options.num_levels
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-5733
> URL: https://issues.apache.org/jira/browse/KAFKA-5733
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.11.0.1
> Reporter: Eno Thereska
> Fix For: 0.11.0.1
>
>
> New system tests as part of KAFKA-5725 with PR
> https://github.com/apache/kafka/pull/3656 consistently fail like this:
> [2017-08-14 10:37:57,216] ERROR User provided listener
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener
> for group SmokeTest failed on partition assignment
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> org.apache.kafka.streams.errors.ProcessorStateException: Error opening store
> cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551)
> at
> org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54)
> at
> org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61)
> at
> org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:736)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:707)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:677)
> Caused by: org.rocksdb.RocksDBException: db has more levels than
> options.num_levels
> at org.rocksdb.RocksDB.open(Native Method)
> at org.rocksdb.RocksDB.open(RocksDB.java:231)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:199)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
