Hello everybody,
we seem to experience very unexpected behaviour from the Kafka Streams 2.2.0
framework, where KeyValueState.get() returns a value of null, although it
really should return some other value.
This seems to be content corruption somewhere at / around the cache layer in
Kafka Streams, with RocksDB delivering the persistent cache.
On the implementation side, we simply
streamsBuilder.addStateStore(Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore("my-topic"),
then retrieve and cache the store in init() of a Transformer, then use the
store in transform().
We see that *very rarely* KeyValueStore.get() returns null, although the
(changelog) topic (on the Kafka broker) has clear evidence that
KeyValueStore.get() should have returned a value.
This was seen at exactly the same time for at least two different topics, upon
starting the streaming application,
a) (changelog) topic with exactly one partition, guaranteed to contain exactly
one key - had reliably returned data for four weeks
b) (changelog) topic over 16 partitions, where for about 30 calls to get(), a
total of 20 returned data as expected, and about 10 returned an incorrect value
of null.
Has anyone seen something like that before, any pointers to root cause?
The following items have made it onto our review agenda:
a) virtual network instability, possibly leading to interesting RocksDB startup
behaviour
(org.apache.kafka.streams.processor.internals.StoreChangelogReader#initialize)
b) on-disk data corruption (Linux -> NFS as persistent cache storage for
RocksDB)
Any insight or pointers greatly appreciated!
Many thanks!
Stefan
