Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Thanks for the feedback, Konstantine. I do agree it's probably better to keep the KIP focused and avoid this kind of baggage, since connectors that *can* optionally use a reader if available can pretty easily deal with the checks. That's 3 binding +1 votes (Gwen, me, and Konstantine), and the

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

I agree with the motivation of this KIP. I think it will enrich the capabilities of source connectors in a quite meaningful way. I'm +1 as well (binding). P.S. Randall I like your idea of enabling connectors to run in old as well as new workers after this KIP. However I think that such

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Status: this KIP has 2 binding +1 votes, but has not yet passed. :-( I'm very interested in getting this into AK 2.6, but have another suggestion to make on the KIP. The current KIP is great and allows connectors using the latest Connect API to get access to the offset reader. Unfortunately, this

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

+1 (binding) On Mon, Feb 25, 2019 at 4:32 AM Florian Hussonnois wrote: > Hi Kafka Team, > > I'd like to bring this thread back at the top of the email stack to get a > chance to see this KIP merge in the next minor/major release. > > Thanks. > > Le ven. 18 janv. 2019 à 01:20, Florian Hussonnois

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Hi Kafka Team, I'd like to bring this thread back at the top of the email stack to get a chance to see this KIP merge in the next minor/major release. Thanks. Le ven. 18 janv. 2019 à 01:20, Florian Hussonnois a écrit : > Hey folks, > > This KIP has start since a while but has never been

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Hey folks, This KIP has start since a while but has never been merged. https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector Would it be possible to restart the vote ? I still think this KIP could be useful to implement connectors (the

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Hi team, Are there any more votes ? Thanks Le 12 sept. 2017 20:18, "Gwen Shapira" a écrit : > Thanks for clarifying. > > +1 again :) > > > > On Sat, Sep 9, 2017 at 6:57 AM Randall Hauch wrote: > > > Gwen, > > > > I've had more time to look into the code.

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Thanks for clarifying. +1 again :) On Sat, Sep 9, 2017 at 6:57 AM Randall Hauch wrote: > Gwen, > > I've had more time to look into the code. First, the OffsetStorageReader > JavaDoc says: "OffsetStorageReader provides access to the offset storage > used by sources. This can

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Gwen, I've had more time to look into the code. First, the OffsetStorageReader JavaDoc says: "OffsetStorageReader provides access to the offset storage used by sources. This can be used by connectors to determine offsets to start consuming data from. This is most commonly used during

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Basically, you are saying that the part where the comment says: "Offset data should only be read during startup or reconfiguration of a task." is incorrect? because the API extension allows reading offset data at any point in the lifecycle, right? On Fri, Sep 8, 2017 at 5:18 AM Florian Hussonnois

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Hi Shapira, We only expose the OffsetStorageReader to connector which relies on KafkaOffsetBackingStore. The store continuesly consumes offsets from kafka so I think we can't have stale data. Le 8 sept. 2017 06:13, "Randall Hauch" a écrit : > The KIP and PR expose the

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

The KIP and PR expose the OffsetStorageReader, which is already exposed to the tasks. The OffsetStorageWriter is part of the implementation, but was not and is not exposed thru the API. > On Sep 7, 2017, at 9:04 PM, Gwen Shapira wrote: > > I just re-read the code for the

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

I just re-read the code for the OffsetStorageWriter, and ran into this comment: * Note that this only provides write functionality. This is intentional to ensure stale data is * never read. Offset data should only be read during startup or reconfiguration of a task. By * always serving those

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

+1 (binding) Looking forward to see how connector implementations use this in practice :) On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch wrote: > I'd like to open the vote for KIP-131: > >

[VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

I'd like to open the vote for KIP-131: https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector Thanks to Florian for submitting the KIP and the implementation, and to everyone else that helped review. Best regards, Randall