Thanks, Luke :) Colin -- I updated the KIP with your feedback. Do you think we would expose the "last stable offset" outside of the controller? Or would it just be an internal concept?
-David On Sun, Sep 18, 2022 at 9:05 AM Luke Chen <show...@gmail.com> wrote: > Hi David, > > Thanks for the KIP! > It's a light-weight transactional proposal for single producer, cool! > +1 for it! > > Luke > > > On Sat, Sep 10, 2022 at 1:14 AM David Arthur <davidart...@apache.org> > wrote: > > > Starting a new thread to avoid issues with mail client threading. > > > > Original thread follows: > > > > Hey folks, I'd like to start a discussion on the idea of adding > > transactions in the KRaft controller. This will allow us to overcome > > the current limitation of atomic batch sizes in Raft which lets us do > > things like create topics with a huge number of partitions. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions > > > > Thanks! > > > > --- > > > > Colin McCabe said: > > > > Thanks for this KIP, David! > > > > In the "motivation" section, it might help to give a concrete example > > of an operation we want to be atomic. My favorite one is probably > > CreateTopics since it's easy to see that we want to create all of a > > topic or none of it, and a topic could be a potentially unbounded > > number of records (although hopefully people have reasonable create > > topic policy classes in place...) > > > > In "broker support", it would be good to clarify that we will buffer > > the records in the MetadataDelta and not publish a new MetadataImage > > until the transaction is over. This is an implementation detail, but > > it's a simple one and I think it will make it easier to understand how > > this works. > > > > In the "Raft Transactions" section of "Rejected Alternatives," I'd add > > that managing buffering in the Raft layer would be a lot less > > efficient than doing it in the controller / broker layer. We would end > > up accumulating big lists of records which would then have to be > > applied when the transaction completed, rather than building up a > > MetadataDelta (or updating the controller state) incrementally. > > > > Maybe we want to introduce the concept of "last stable offset" to be > > the last committed offset that is NOT part of an ongoing transaction? > > Just a nomenclature suggestion... > > > > best, > > Colin > > > -- -David
