Matthias J. Sax created KAFKA-6999:
--------------------------------------
Summary: Document read-write lock usage of caching enabled stores
Key: KAFKA-6999
URL: https://issues.apache.org/jira/browse/KAFKA-6999
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Matthias J. Sax
>From the mailing list
{quote}Hello again fellow Kafkans,
Yesterday we observed a production deadlock take down one of our instances.
Upon digging, it's clear that our usage of Kafka is the proximate cause, but
the danger of our approach is not clear at all just from the Javadocs.
We have stream processors that read off an incoming KStream, possibly
cross-reference some data from an auxiliary table via
ReadOnlyKeyValueStore.get()
This is done via custom logic rather than a direct KTable join because which
index is consulted may change depending on the shape of incoming data.
The ROKVS docs state,
* A key value store that only supports read operations.
* Implementations should be thread-safe as concurrent reads and writes
* are expected.
They do **not** indicate that the CachingKVS layer uses a ReadWriteLock. If you
have one CachingKVS flush a record cause a read from another CKVS, you are
suddenly vulnerable to classic lock order reversals! Surprise!
A partial stack trace highlighting the problem, with many uninteresting frames
removed, is inline at the bottom of this mail.
You could probably rightly point to us allowing a "observer" pattern to
callback from within streams processing code is dangerous. We might move this
off to an auxiliary thread to alleviate this problem.
But it still remains -- when you go an read that ROKVS documentation, it sure
doesn't prepare you to this possibility!
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
