Hi Kafka crew, I'm rolling out a Kafka Streams application on a pretty large dataset, and I see some exceptions that worry me, which I haven't seen with smaller data sets (Using Kafka streams trunk of a few days ago)
2017-02-15 15:25:14,431 | INFO | StreamThread-42 | StreamTask | 86 - com.dexels.kafka.streams - 0.0.53.201702141103 | task [9_4] Initializing state stores 2017-02-15 15:25:14,433 | ERROR | StreamThread-42 | ConsumerCoordinator | 86 - com.dexels.kafka.streams - 0.0.53.201702141103 | User provided listener org.apache.kafka.streams.processor.internals.StreamThread$1 for group KNVB-test-generation-175-person failed on partition assignment org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KNVB-test-generation-175-person-sportlinkkernel-PERSONCOMMUNICATION at location /output/env.managed.repository-master/storage/KNVB-test-generation-175-person/9_4/rocksdb/KNVB-test-generation-175-person-sportlinkkernel-PERSONCOMMUNICATION at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:175) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:145) at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:150) at org.apache.kafka.streams.state.internals.MeteredKeyValueStore$7.run(MeteredKeyValueStore.java:100) at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188) at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:131) at org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:62) at org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:85) at org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:143) at org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834) at org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207) at org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180) at org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937) at org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69) at org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368) Caused by: org.rocksdb.RocksDBException: at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:168) ... 22 more I've checked the folder it points to: [root@842376-demeter1 KNVB-test-generation-175-personcore-PERSONCORE]# ls -l total 223348 -rw-r--r--. 1 431 433 68153335 Feb 15 11:09 000353.sst -rw-r--r--. 1 431 433 68207125 Feb 15 11:09 000355.sst -rw-r--r--. 1 431 433 7241027 Feb 15 11:09 000356.sst -rw-r--r--. 1 431 433 59070938 Feb 15 11:12 000459.sst -rw-r--r--. 1 431 433 18783280 Feb 15 11:13 000489.sst -rw-r--r--. 1 431 433 0 Feb 15 11:38 000502.log -rw-r--r--. 1 431 433 7183476 Feb 15 11:38 000505.sst -rw-r--r--. 1 431 433 16 Feb 15 11:38 CURRENT -rw-r--r--. 1 431 433 37 Feb 15 10:57 IDENTITY -rw-r--r--. 1 431 433 0 Feb 15 10:57 LOCK -rw-r--r--. 1 431 433 13446 Feb 15 11:38 LOG -rw-r--r--. 1 431 433 13130 Feb 15 10:57 LOG.old.1487158687061100 -rw-r--r--. 1 431 433 456 Feb 15 11:38 MANIFEST-000501 -rw-r--r--. 1 431 433 4275 Feb 15 10:57 OPTIONS-000005 -rw-r--r--. 1 431 433 4275 Feb 15 11:38 OPTIONS-000504 [root@842376-demeter1 KNVB-test-generation-175-personcore-PERSONCORE]# What strikes me as odd is that none of the files have been touched in the last few hours (the stacktrace is from 15:25), so there doesn't appear to have been any logging of that error (on the rocksdb side) Any ideas? Or should I post this to RocksDb? regards, Frank