Stephen0421 opened a new pull request, #10083: URL: https://github.com/apache/paimon/pull/10083
### Purpose Add the Spark procedure `sys.remove_unexisting_manifests`, so a missing manifest file can be repaired from Spark the same way as the existing Flink procedure. The repair logic is moved from `RemoveUnexistingManifestsAction` into `RemoveUnexistingManifests` in paimon-core. It drops missing manifest files from the latest snapshot and commits a replacement snapshot. `totalRecordCount` counts rows a scan can still read: ADD entries whose file is not deleted by a remaining manifest. Summing every entry's row count would count DELETE entries, and subtracting DELETE row counts under-counts when the matching ADD was in a dropped manifest. To repair a branch, backtick-quote the table name because `$` is not a valid unquoted Spark identifier: ```sql CALL sys.remove_unexisting_manifests(table => 'mydb.myt'); CALL sys.remove_unexisting_manifests(table => 'mydb.`myt$branch_rt`'); ``` This procedure may cause data loss when used outside the documented repair cases. ### Tests - `RemoveUnexistingManifestsTest` checks the no-snapshot and no-op paths, and that `totalRecordCount` matches readable rows after a compaction manifest is dropped. - `RemoveUnexistingManifestsProcedureTest` checks the Spark procedure, including a branch repair that does not change the main table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
