I just re-read the code for the OffsetStorageWriter, and ran into this comment:
* Note that this only provides write functionality. This is intentional to ensure stale data is * never read. Offset data should only be read during startup or reconfiguration of a task. By * always serving those requests by reading the values from the backing store, we ensure we never * accidentally use stale data. (One example of how this can occur: a task is processing input * partition A, writing offsets; reconfiguration causes partition A to be reassigned elsewhere; * reconfiguration causes partition A to be reassigned to this node, but now the offset data is out * of date). Since these offsets are created and managed by the connector itself, there's no way * for the offset management layer to know which keys are "owned" by which tasks at any given * time. I can't figure out how the KIP avoids the stale-reads problem explained here. Can you talk me through it? I'm cancelling my vote since right now exposing this interface sounds risky and misleading. Gwen On Thu, Sep 7, 2017 at 5:04 PM Gwen Shapira <g...@confluent.io> wrote: > +1 (binding) > > Looking forward to see how connector implementations use this in practice > :) > > On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch <rha...@gmail.com> wrote: > >> I'd like to open the vote for KIP-131: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector >> >> Thanks to Florian for submitting the KIP and the implementation, and to >> everyone else that helped review. >> >> Best regards, >> >> Randall >> >
