Starting a new thread here https://lists.apache.org/thread/895pgb85l08g2l63k99cw5dt2qpjkxb9
On Fri, Sep 9, 2022 at 1:05 PM Colin McCabe <co...@cmccabe.xyz> wrote: > > Also, it looks like someone already claimed KIP-865, so I'd suggest grabbing > a new number. :) > > Colin > > > On Fri, Sep 9, 2022, at 09:38, Colin McCabe wrote: > > Thanks for this KIP, David! > > > > In the "motivation" section, it might help to give a concrete example > > of an operation we want to be atomic. My favorite one is probably > > CreateTopics since it's easy to see that we want to create all of a > > topic or none of it, and a topic could be a potentially unbounded > > number of records (although hopefully people have reasonable create > > topic policy classes in place...) > > > > In "broker support", it would be good to clarify that we will buffer > > the records in the MetadataDelta and not publish a new MetadataImage > > until the transaction is over. This is an implementation detail, but > > it's a simple one and I think it will make it easier to understand how > > this works. > > > > In the "Raft Transactions" section of "Rejected Alternatives," I'd add > > that managing buffering in the Raft layer would be a lot less efficient > > than doing it in the controller / broker layer. We would end up > > accumulating big lists of records which would then have to be applied > > when the transaction completed, rather than building up a MetadataDelta > > (or updating the controller state) incrementally. > > > > Maybe we want to introduce the concept of "last stable offset" to be > > the last committed offset that is NOT part of an ongoing transaction? > > Just a nomenclature suggestion... > > > > best, > > Colin > > > > On Fri, Sep 9, 2022, at 06:42, David Arthur wrote: > >> Hey folks, I'd like to start a discussion on the idea of adding > >> transactions in the KRaft controller. This will allow us to overcome > >> the current limitation of atomic batch sizes in Raft which lets us do > >> things like create topics with a huge number of partitions. > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-865+Metadata+Transactions > >> > >> Thanks! > >> David -- David Arthur
