Re: Streams: clarification needed, checkpoint vs. position files

2022-11-15 Thread Sophie Blee-Goldman
Ah, good thing John showed up to clear things up -- sorry for the misinformation in my first reply (fake news!) Generally speaking only public APIs *need* to be included in the KIP document, anything internal is an implementation detail as you said. In some cases it may still be appropriate to

Re: Streams: clarification needed, checkpoint vs. position files

2022-11-14 Thread Nick Telford
Thank you both, the key point I was missing was that position files track the offsets of the *source* topics, whereas the checkpoint file tracks the offset(s) of the changelog topics. Do you know if I need to include interface/API changes to internal classes/interfaces (those under the

Re: Streams: clarification needed, checkpoint vs. position files

2022-11-11 Thread John Roesler
Hi all, Just to clarify: there actually is a position file. It was a small detail of the IQv2 implementation to add it, otherwise a persistent store's position would be lost after a restart. Otherwise, Sophie is right on the money. The checkpoint refers to an offset in the changelog, while

Re: Streams: clarification needed, checkpoint vs. position files

2022-11-11 Thread Sophie Blee-Goldman
Hey Nick, I haven't been following the new IQv2 work very closely so take this with a grain of salt, but as far as I'm aware there's no such thing as "position files" -- the Position is just an in-memory object and is related to a user's query against the state store, whereas a checkpoint file