Copilot commented on code in PR #4202:
URL: https://github.com/apache/gobblin/pull/4202#discussion_r3440754021


##########
gobblin-utility/src/main/java/org/apache/gobblin/util/HadoopUtils.java:
##########
@@ -570,6 +572,82 @@ public static void renameRecursively(FileSystem 
fileSystem, Path from, Path to)
     }
   }
 
+  /**
+   * Like {@link #renameRecursively(FileSystem, Path, Path)} but renames in 
two phases: every directory subtree whose
+   * root matches {@code deferToLastPhase} is held back and renamed only after 
all other files have been fully
+   * renamed. Both phases use the same concurrent, atomic-subtree-move 
machinery as the unordered variant.
+   *
+   * <p>
+   * Used by Iceberg distcp publishing to move the {@code metadata/} subtree 
only after every other file (the
+   * {@code data/} files and anything else) has landed, since Iceberg metadata 
references data files by path and must
+   * never become visible ahead of the data it points to. {@code 
deferToLastPhase} is evaluated only on directory
+   * nodes encountered during the descent -- not on every file -- so the 
{@code metadata/} subtree is identified by
+   * its directory name and never enumerated file-by-file.

Review Comment:
   The Javadoc says the deferred `metadata/` subtree is "never enumerated 
file-by-file", but the implementation can still enumerate file-by-file in phase 
2 when the destination directory already exists (same behavior as 
`renameRecursively`). Consider rewording this to reflect that the *predicate 
evaluation* is directory-only, rather than making a guarantee about the rename 
strategy.



##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/publisher/CopyDataPublisher.java:
##########
@@ -342,7 +342,8 @@ private void publishFileSet(CopyEntity.DatasetAndPartition 
datasetAndPartition,
 
     if (statesHelper.hasAnyCopyableFile()) {
       // Targets are always absolute, so we start moving from root (will skip 
any existing directories).
-      HadoopUtils.renameRecursively(this.fs, datasetWriterOutputPath, new 
Path("/"));
+      HadoopUtils.renameRecursivelyOrdered(this.fs, datasetWriterOutputPath, 
new Path("/"),
+          HadoopUtils::isIcebergMetadataDir);

Review Comment:
   `CopyDataPublisher` now uses `renameRecursivelyOrdered(..., 
isIcebergMetadataDir)` for every dataset publish. Since this class publishes 
non-Iceberg datasets too, this changes semantics/performance for any dataset 
that happens to contain a `metadata/` directory (it will now always be deferred 
to a second phase). If ordering is only required for Iceberg publishes, gate 
the ordered rename on the presence of an Iceberg commit step (e.g., 
`IcebergOverwritePartitionsStep`) and otherwise keep the previous 
`renameRecursively` behavior.



##########
gobblin-utility/src/main/java/org/apache/gobblin/util/HadoopUtils.java:
##########
@@ -570,6 +572,82 @@ public static void renameRecursively(FileSystem 
fileSystem, Path from, Path to)
     }
   }
 
+  /**
+   * Like {@link #renameRecursively(FileSystem, Path, Path)} but renames in 
two phases: every directory subtree whose
+   * root matches {@code deferToLastPhase} is held back and renamed only after 
all other files have been fully
+   * renamed. Both phases use the same concurrent, atomic-subtree-move 
machinery as the unordered variant.
+   *
+   * <p>
+   * Used by Iceberg distcp publishing to move the {@code metadata/} subtree 
only after every other file (the
+   * {@code data/} files and anything else) has landed, since Iceberg metadata 
references data files by path and must
+   * never become visible ahead of the data it points to. {@code 
deferToLastPhase} is evaluated only on directory
+   * nodes encountered during the descent -- not on every file -- so the 
{@code metadata/} subtree is identified by
+   * its directory name and never enumerated file-by-file.
+   * </p>
+   *
+   * @param fileSystem on which the data needs to be moved
+   * @param from path of the data to be moved
+   * @param to path of the data to be moved
+   * @param deferToLastPhase predicate on a directory {@link Path}; matching 
subtrees are renamed in the last phase
+   */
+  public static void renameRecursivelyOrdered(FileSystem fileSystem, Path 
from, Path to,
+      Predicate<Path> deferToLastPhase) throws IOException {
+
+    log.info(String.format("Recursively renaming (ordered) %s in %s to %s.", 
from, fileSystem.getUri(), to));
+
+    FileSystem throttledFS = getOptionallyThrottledFileSystem(fileSystem, 
10000);
+
+    ExecutorService executorService = 
ScalingThreadPoolExecutor.newScalingThreadPool(1, 100, 100,
+        ExecutorsUtils.newThreadFactory(Optional.of(log), 
Optional.of("rename-thread-%d")));
+    Queue<Future<?>> futures = Queues.newConcurrentLinkedQueue();
+    // Subtrees skipped in phase 1 (their root FileStatus and target path), to 
be renamed in phase 2.
+    Queue<Entry<FileStatus, Path>> deferred = 
Queues.newConcurrentLinkedQueue();
+
+    try {
+      if (!fileSystem.exists(from)) {
+        throw new IOException("Trying to rename a path that does not exist! " 
+ from);
+      }
+
+      // Phase 1: rename everything except the deferred subtrees (which are 
collected into `deferred`).
+      futures.add(executorService.submit(new RenameRecursively(
+          throttledFS, fileSystem.getFileStatus(from), to, executorService, 
futures, deferToLastPhase, deferred)));
+      drainFutures(futures);
+
+      // Phase 2: rename the deferred subtrees, now that everything else is in 
place. No further deferral.
+      for (Entry<FileStatus, Path> entry : deferred) {
+        futures.add(executorService.submit(new RenameRecursively(
+            throttledFS, entry.getKey(), entry.getValue(), executorService, 
futures, null, null)));
+      }
+      drainFutures(futures);
+
+      log.info(String.format("Recursive ordered renaming of %s to %s complete 
(%d deferred subtree(s)).",
+          from, to, deferred.size()));
+
+    } finally {
+      ExecutorsUtils.shutdownExecutorService(executorService, 
Optional.of(log), 1, TimeUnit.SECONDS);
+    }
+  }
+
+  private static void drainFutures(Queue<Future<?>> futures) throws 
IOException {
+    while (!futures.isEmpty()) {
+      try {
+        futures.poll().get();
+      } catch (ExecutionException | InterruptedException ee) {
+        throw new IOException(ee.getCause());
+      }
+    }
+  }

Review Comment:
   `drainFutures` swallows thread interruption and can throw an `IOException` 
with a null cause when an `InterruptedException` occurs (`ee.getCause()` is 
null). It’s better to handle `InterruptedException` separately, restore the 
interrupt flag, and preserve the `ExecutionException` context/message.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to