JingsongLi commented on PR #8139: URL: https://github.com/apache/paimon/pull/8139#issuecomment-4637973443
I think `restoreAsLatest` can be invisible to streaming readers that handle overwrite snapshots. The new snapshot writes the target snapshot's files into the base manifest list, but writes an empty delta manifest list and marks the commit as `CommitKind.OVERWRITE` (`FileStoreCommitImpl.java:1174-1200`). `DataTableStreamScan` first handles overwrite snapshots via the overwrite-change path, and if the returned plan is empty it advances past the snapshot. Since the restore snapshot has no delta, a streaming reader with `streaming-read-overwrite=true` can skip the restore entirely, missing both files/rows that should be removed from the current latest snapshot and files/rows that should be restored from the target snapshot. Could `restoreAsLatest` produce an overwrite delta relative to the previous latest snapshot (DELETE previous-only files and ADD target-only files), or introduce a dedicated commit kind/streaming-scan handling for restore snapshots? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
