0dunay0 opened a new issue, #10672: URL: https://github.com/apache/paimon/issues/10672
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation `clone` from Hive reads and rewrites every row of ORC and Parquet tables. For large migrations, copying the data files as they are is much cheaper, but it can't be requested. The file-copy path, used for Avro today, only lists the top level of each partition directory, so files in sub directories, such as the `HIVE_UNION_SUBDIR_*` directories that Hive on Tez writes for `UNION ALL`, are not cloned. ### Solution Add two opt-in options to the clone action and procedure, both off by default: - `copy_files`: copy ORC, Parquet and Avro data files as they are. An existing target must have the same fields as the source. - `recursive_listing` (needs `copy_files`): also copy files in sub directories of each partition, skipping files and directories whose names start with `_` or `.`, such as `_temporary` and `.hive-staging`. ### Anything else? While there: a partitioned Hive table with no partitions in the metastore has its files tagged with an empty partition by the file-copy path. It should copy nothing instead. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
