beryllw commented on PR #3660: URL: https://github.com/apache/fluss/pull/3660#issuecomment-5009899102
The key fact is that mark-done state can change without a new lake commit (a partition is marked done on idle timeout, with no new data), so it's table-level and doesn't naturally belong to a single snapshot. There are two ways to carry it. Option 1: keep it embedded in the offsets file (current). Simple, no new files, offsets and state published together. But a state-only change needs a new offsets file, which forces reusing the snapshot ID and appending a duplicate entry — the ambiguity you flagged around latest/by-id/readable and retention. Option 2: split it into a dedicated table-level file with its own pointer. State advances independently, so IDs stay 1:1 and retention/GetLakeSnapshot stay clean, V1 never touches state, and the state path can validate tieringEpoch on its own. Cost is one extra file with its own lifecycle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
