JingsongLi commented on code in PR #10673:
URL: https://github.com/apache/paimon/pull/10673#discussion_r4374426921
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/clone/HiveTableCloneExtractor.java:
##########
@@ -113,7 +152,13 @@ public Map<String, String> extractOptions(Table table) {
}
String format = parseFormat(table);
- if (supportCloneSplits(format)) {
+ if (copyFiles && !COPY_FILES_FORMATS.contains(format)) {
+ throw new UnsupportedOperationException(
+ String.format(
+ "Copying data files is only supported for formats
%s, but table has format '%s'.",
+ COPY_FILES_FORMATS, format));
+ }
+ if (!copyFiles && supportCloneSplits(format)) {
Review Comment:
[P1] Enumerate every registered partition before using direct copying
With `copy_files=true`, this new branch sends ORC/Parquet through
`listFromPureHiveTable`, whose `listPartitions(..., Short.MAX_VALUE)` call
returns at most 32,767 partitions. The WHERE predicate is applied only
afterward, so a larger Hive table is silently truncated; a predicate selecting
a later partition can copy nothing. Using 32,768 registered partition
descriptors and real ORC/Parquet files, `WHERE dt = 32767` selected zero copy
files on this head, while the existing `ListCloneSplitsFunction` path read row
42 from that partition. Forcing unlimited metadata enumeration recovered the
file for both formats. Although this cap predates the PR for the old file-copy
path, enabling it for ORC/Parquet introduces this incomplete-migration
behavior. Fetch all partitions (or page them) before filtering instead of using
`Short.MAX_VALUE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]