The KIP and PR expose the OffsetStorageReader, which is already exposed to the tasks. The OffsetStorageWriter is part of the implementation, but was not and is not exposed thru the API.
> On Sep 7, 2017, at 9:04 PM, Gwen Shapira <g...@confluent.io> wrote: > > I just re-read the code for the OffsetStorageWriter, and ran into this > comment: > > * Note that this only provides write functionality. This is > intentional to ensure stale data is > * never read. Offset data should only be read during startup or > reconfiguration of a task. By > * always serving those requests by reading the values from the backing > store, we ensure we never > * accidentally use stale data. (One example of how this can occur: a > task is processing input > * partition A, writing offsets; reconfiguration causes partition A to > be reassigned elsewhere; > * reconfiguration causes partition A to be reassigned to this node, > but now the offset data is out > * of date). Since these offsets are created and managed by the > connector itself, there's no way > * for the offset management layer to know which keys are "owned" by > which tasks at any given > * time. > > > I can't figure out how the KIP avoids the stale-reads problem explained here. > > Can you talk me through it? I'm cancelling my vote since right now > exposing this interface sounds risky and misleading. > > > Gwen > > >> On Thu, Sep 7, 2017 at 5:04 PM Gwen Shapira <g...@confluent.io> wrote: >> >> +1 (binding) >> >> Looking forward to see how connector implementations use this in practice >> :) >> >>> On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch <rha...@gmail.com> wrote: >>> >>> I'd like to open the vote for KIP-131: >>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector >>> >>> Thanks to Florian for submitting the KIP and the implementation, and to >>> everyone else that helped review. >>> >>> Best regards, >>> >>> Randall >>> >>
