0dunay0 opened a new pull request, #10673: URL: https://github.com/apache/paimon/pull/10673
### Purpose Linked issue: close #10672 Add `copy_files` and `recursive_listing` to the `clone` action and procedure for Hive sources. Both are off by default, so existing clones are unchanged. - `copy_files` copies ORC, Parquet and Avro data files as they are instead of rewriting their rows. The formats of all source tables are checked before the job starts. An existing target must have the same fields as the source, nullable, and `prefer_file_format` is rejected. - `recursive_listing` (needs `copy_files`) also lists the sub directories of each partition. Files under directories whose names start with `_` or `.` are skipped, and a file listed by two partitions fails the clone. - `HiveCloneExtractor` gets default methods for both, which reject the options, so other extractors keep working. The file-copy path, which Avro tables already use, also changes in two ways: a partitioned table without partitions in the metastore copies nothing instead of committing its files under an empty partition, and a partition that declares a table column with another type fails the clone. ### Tests - `HiveTableCloneExtractorTest`: recursive listing skips hidden files and directories, only below the partition root, and rejects a file listed by two partitions. - `CloneHiveSchemaFunctionTest`: the field check for an existing target. - `CloneHiveCopyFilesITCase`: clones ORC and Avro tables whose files are nested next to `_temporary` and `.hive-staging` copies, and a partitioned table with an unregistered partition directory, which fails without the fix. It is a new class because `CloneActionITCase` is disabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
