JingsongLi commented on code in PR #10673:
URL: https://github.com/apache/paimon/pull/10673#discussion_r4374426921


##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/clone/HiveTableCloneExtractor.java:
##########
@@ -113,7 +152,13 @@ public Map<String, String> extractOptions(Table table) {
         }
 
         String format = parseFormat(table);
-        if (supportCloneSplits(format)) {
+        if (copyFiles && !COPY_FILES_FORMATS.contains(format)) {
+            throw new UnsupportedOperationException(
+                    String.format(
+                            "Copying data files is only supported for formats 
%s, but table has format '%s'.",
+                            COPY_FILES_FORMATS, format));
+        }
+        if (!copyFiles && supportCloneSplits(format)) {

Review Comment:
   [P1] Enumerate every registered partition before using direct copying
   
   With `copy_files=true`, this new branch sends ORC/Parquet through 
`listFromPureHiveTable`, whose `listPartitions(..., Short.MAX_VALUE)` call 
returns at most 32,767 partitions. The WHERE predicate is applied only 
afterward, so a larger Hive table is silently truncated; a predicate selecting 
a later partition can copy nothing. Using 32,768 registered partition 
descriptors and real ORC/Parquet files, `WHERE dt = 32767` selected zero copy 
files on this head, while the existing `ListCloneSplitsFunction` path read row 
42 from that partition. Forcing unlimited metadata enumeration recovered the 
file for both formats. Although this cap predates the PR for the old file-copy 
path, enabling it for ORC/Parquet introduces this incomplete-migration 
behavior. Fetch all partitions (or page them) before filtering instead of using 
`Short.MAX_VALUE`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to