leaves12138 commented on code in PR #9725: URL: https://github.com/apache/paimon/pull/9725#discussion_r3980137121
########## docs/docs/flink/procedures/repair.md: ########## @@ -0,0 +1,206 @@ +--- +title: "Cleanup and Repair" +sidebar_position: 7 +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Cleanup and Repair + +Inspect orphan files or repair missing files, manifests, and table metadata. + +See [Procedures](../procedures) for Flink version requirements, argument conventions, and catalog selection. + +## remove_orphan_files + +To remove the orphan data files and metadata files. Arguments: + +- `table`: the target table identifier. Cannot be empty, you can use database_name.* to clean whole database. + +- `olderThan`: to avoid deleting newly written files, this procedure only deletes orphan files older than 1 day by default. This argument can modify the interval. + +- `dryRun`: when true, view only orphan files, don't actually remove files. Default is false. + +- `parallelism`: The maximum number of concurrent deleting files. By default is the number of processors available to the Java virtual machine. + +- `mode`: The mode of remove orphan clean procedure (local or distributed) . By default is distributed. + +**Syntax** + +```sql +-- Use named argument +CALL [catalog.]sys.remove_orphan_files( + `table` => 'identifier', + older_than => 'olderThan', + dry_run => 'dryRun', + mode => 'mode' +); + +-- Use indexed argument +CALL [catalog.]sys.remove_orphan_files('identifier'); + +CALL [catalog.]sys.remove_orphan_files('identifier', 'olderThan'); + +CALL [catalog.]sys.remove_orphan_files('identifier', 'olderThan', 'dryRun'); + +CALL [catalog.]sys.remove_orphan_files('identifier', 'olderThan', 'dryRun','parallelism'); + +CALL [catalog.]sys.remove_orphan_files('identifier', 'olderThan', 'dryRun','parallelism','mode'); +``` + +**Example** + +```sql +CALL sys.remove_orphan_files(`table` => 'default.T', older_than => '2023-10-31 12:00:00'); + +CALL sys.remove_orphan_files(`table` => 'default.*', older_than => '2023-10-31 12:00:00'); + +CALL sys.remove_orphan_files(`table` => 'default.T', older_than => '2023-10-31 12:00:00', dry_run => true); + +CALL sys.remove_orphan_files( + `table` => 'default.T', + older_than => '2023-10-31 12:00:00', + dry_run => false, + parallelism => 5 +); + +CALL sys.remove_orphan_files( + `table` => 'default.T', + older_than => '2023-10-31 12:00:00', + dry_run => false, + parallelism => 5, + mode => 'local' +); +``` + +## remove_unexisting_files + +Procedure to remove unexisting data files from manifest entries. See [Java docs](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/RemoveUnexistingFilesAction.html) for detailed use cases. Arguments: + +- `table`: the target table identifier. Cannot be empty, you can use database_name.* to clean whole database. + +- `dry_run` (optional): only check what files will be removed, but not really remove them. Default is false. + +- `parallelism` (optional): number of parallelisms to check files in the manifests. + +Note that user is on his own risk using this procedure, which may cause data loss when used outside from the use cases listed in Java docs. + +**Syntax** + +```sql +-- Use named argument +CALL [catalog.]sys.remove_unexisting_files( + `table` => 'identifier', + dry_run => 'dryRun', + parallelism => parallelism +); + +-- Use indexed argument +CALL [catalog.]sys.remove_unexisting_files('identifier'); + +CALL [catalog.]sys.remove_unexisting_files('identifier', 'dryRun', 'parallelism'); +``` + +**Example** + +```sql +-- remove unexisting data files in the table `mydb.myt` +CALL sys.remove_unexisting_files(`table` => 'mydb.myt'); + +-- only check what files will be removed, but not really remove them (dry run) +CALL sys.remove_unexisting_files(`table` => 'mydb.myt', `dry_run` = true); +``` + +## remove_unexisting_manifests + +Procedure to remove unexisting manifest file from manifset-list. for detailed use cases. Arguments: + +- `table`: the target table identifier. Cannot be empty, you can use database.table$branch_xx to remove branch table unexisting manifest file. + +Note that user is on his own risk using this procedure, which may cause data loss when used outside from the use cases listed in Java docs. + +**Syntax** + +```sql +-- Use named argument +CALL [catalog.]sys.remove_unexisting_files(`table` => 'identifier'); +``` + +**Example** + +```sql +-- remove unexisting manifest file in the table `mydb.myt` +CALL sys.remove_unexisting_manifests(`table` => 'mydb.myt'); Review Comment: Non-blocking, carried over from the old procedure table: please correct the manifest-repair reference in a follow-up. Its Syntax block currently invokes `remove_unexisting_files` instead of `remove_unexisting_manifests`, and the manifest procedure's actual named argument is `tableId`, not `table`. On Flink 1.20.1 I verified that the named call shown here fails validation, while `CALL sys.remove_unexisting_manifests(tableId => 'default.repair_target')` succeeds and preserves the table's rows. Please update the Syntax block and both manifest-repair examples consistently; a positional call also avoids the parameter-name mismatch. This is not a regression introduced by this reorganization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
