JingsongLi commented on code in PR #10673:
URL: https://github.com/apache/paimon/pull/10673#discussion_r4374391584


##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/clone/HiveTableCloneExtractor.java:
##########
@@ -177,20 +225,85 @@ private static List<HivePartitionFiles> 
listFromPureHiveTable(
                                 valueSetters,
                                 defaultPartitionName);
                 if (predicate == null || predicate.test(partitionRow)) {
-                    results.add(listFiles(fileIO, location, partitionRow, 
format));
+                    checkPartitionColumns(identifier, sourceTable, partition);
+                    results.add(listFiles(fileIO, location, partitionRow, 
format, recursive));
                 }
             }
+            if (recursive) {
+                checkNoSharedFiles(identifier, results);
+            }
             return results;
         }
     }
 
-    private static HivePartitionFiles listFiles(
-            FileIO fileIO, String location, BinaryRow partition, String 
format) throws IOException {
-        List<FileStatus> fileStatuses =
-                Arrays.stream(fileIO.listStatus(new Path(location)))
-                        .filter(s -> !s.isDir())
-                        .filter(HIDDEN_PATH_FILTER)
-                        .collect(Collectors.toList());
+    /**
+     * Copied files are read with the table columns, so a partition must not 
declare a column of the
+     * table with another type. Columns missing from a partition are read as 
null.
+     */
+    private static void checkPartitionColumns(
+            Identifier identifier, Table table, Partition partition) {
+        if (!partition.getSd().isSetCols()) {
+            return;
+        }
+        Map<String, String> tableTypes = new HashMap<>();
+        for (FieldSchema column : table.getSd().getCols()) {
+            tableTypes.put(column.getName(), column.getType());
+        }
+        for (FieldSchema column : partition.getSd().getCols()) {
+            String tableType = tableTypes.get(column.getName());
+            if (tableType != null && 
!tableType.equalsIgnoreCase(column.getType())) {
+                throw new UnsupportedOperationException(
+                        String.format(
+                                "Can not copy the files of partition %s of 
table %s: column '%s' has type '%s' in the partition but '%s' in the table.",
+                                partition.getValues(),
+                                identifier,
+                                column.getName(),
+                                column.getType(),
+                                tableType));
+            }
+        }
+    }
+
+    /**
+     * Listing recursively, a partition located inside another partition would 
have its files cloned
+     * twice.
+     */
+    @VisibleForTesting
+    static void checkNoSharedFiles(Identifier identifier, 
List<HivePartitionFiles> partitions) {
+        Map<Path, BinaryRow> owners = new HashMap<>();
+        for (HivePartitionFiles partitionFiles : partitions) {
+            for (Path path : partitionFiles.paths()) {
+                BinaryRow owner = owners.putIfAbsent(path, 
partitionFiles.partition());
+                if (owner != null) {
+                    throw new IllegalStateException(
+                            String.format(
+                                    "File %s of table %s is listed by more 
than one partition, "
+                                            + "check the partition locations 
or disable recursive listing.",
+                                    path, identifier));
+                }
+            }
+        }
+    }
+
+    @VisibleForTesting
+    static HivePartitionFiles listFiles(
+            FileIO fileIO, String location, BinaryRow partition, String 
format, boolean recursive)
+            throws IOException {
+        Path root = new Path(location);
+        List<FileStatus> fileStatuses;
+        if (recursive) {
+            fileStatuses =
+                    Arrays.stream(fileIO.listFiles(root, true))

Review Comment:
   I rechecked afb6f62c39bed287b596658bc165ccd9e00b937e. The partition-order 
and transactional-source fixes are working; 11 focused extractor/schema and 
real Hive/Flink clone tests pass with normal JDK 8 Maven verify.
   
   The flat listing argument makes sense for object stores, but the 
hierarchical-filesystem issue remains. I independently reproduced it using the 
actual HEAD extractor and HadoopFileIO with real POSIX permissions: the root 
contains readable valid.parquet and an unreadable _temporary directory. 
Nonrecursive listing returns valid.parquet; recursive_listing=true throws 
IOException: Error accessing .../_temporary before the hidden-directory filter 
can run. No mocked ACL or HDFS cluster was used.
   
   This does not protect against copying an incomplete visible dataset: the 
inaccessible path is explicitly excluded staging metadata. Please retain 
efficient flat prefix listing for object stores, while pruning hidden 
directories before descending on hierarchical filesystems (or otherwise 
ensuring those excluded paths cannot block the visible source). The current 
HDFS example and claim that hidden directories are skipped still leave this 
supported use case failing. This confirms the existing P2; no duplicate inline 
finding added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to