Copilot commented on code in PR #4202:
URL: https://github.com/apache/gobblin/pull/4202#discussion_r3440754021
##########
gobblin-utility/src/main/java/org/apache/gobblin/util/HadoopUtils.java:
##########
@@ -570,6 +572,82 @@ public static void renameRecursively(FileSystem
fileSystem, Path from, Path to)
}
}
+ /**
+ * Like {@link #renameRecursively(FileSystem, Path, Path)} but renames in
two phases: every directory subtree whose
+ * root matches {@code deferToLastPhase} is held back and renamed only after
all other files have been fully
+ * renamed. Both phases use the same concurrent, atomic-subtree-move
machinery as the unordered variant.
+ *
+ * <p>
+ * Used by Iceberg distcp publishing to move the {@code metadata/} subtree
only after every other file (the
+ * {@code data/} files and anything else) has landed, since Iceberg metadata
references data files by path and must
+ * never become visible ahead of the data it points to. {@code
deferToLastPhase} is evaluated only on directory
+ * nodes encountered during the descent -- not on every file -- so the
{@code metadata/} subtree is identified by
+ * its directory name and never enumerated file-by-file.
Review Comment:
The Javadoc says the deferred `metadata/` subtree is "never enumerated
file-by-file", but the implementation can still enumerate file-by-file in phase
2 when the destination directory already exists (same behavior as
`renameRecursively`). Consider rewording this to reflect that the *predicate
evaluation* is directory-only, rather than making a guarantee about the rename
strategy.
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/publisher/CopyDataPublisher.java:
##########
@@ -342,7 +342,8 @@ private void publishFileSet(CopyEntity.DatasetAndPartition
datasetAndPartition,
if (statesHelper.hasAnyCopyableFile()) {
// Targets are always absolute, so we start moving from root (will skip
any existing directories).
- HadoopUtils.renameRecursively(this.fs, datasetWriterOutputPath, new
Path("/"));
+ HadoopUtils.renameRecursivelyOrdered(this.fs, datasetWriterOutputPath,
new Path("/"),
+ HadoopUtils::isIcebergMetadataDir);
Review Comment:
`CopyDataPublisher` now uses `renameRecursivelyOrdered(...,
isIcebergMetadataDir)` for every dataset publish. Since this class publishes
non-Iceberg datasets too, this changes semantics/performance for any dataset
that happens to contain a `metadata/` directory (it will now always be deferred
to a second phase). If ordering is only required for Iceberg publishes, gate
the ordered rename on the presence of an Iceberg commit step (e.g.,
`IcebergOverwritePartitionsStep`) and otherwise keep the previous
`renameRecursively` behavior.
##########
gobblin-utility/src/main/java/org/apache/gobblin/util/HadoopUtils.java:
##########
@@ -570,6 +572,82 @@ public static void renameRecursively(FileSystem
fileSystem, Path from, Path to)
}
}
+ /**
+ * Like {@link #renameRecursively(FileSystem, Path, Path)} but renames in
two phases: every directory subtree whose
+ * root matches {@code deferToLastPhase} is held back and renamed only after
all other files have been fully
+ * renamed. Both phases use the same concurrent, atomic-subtree-move
machinery as the unordered variant.
+ *
+ * <p>
+ * Used by Iceberg distcp publishing to move the {@code metadata/} subtree
only after every other file (the
+ * {@code data/} files and anything else) has landed, since Iceberg metadata
references data files by path and must
+ * never become visible ahead of the data it points to. {@code
deferToLastPhase} is evaluated only on directory
+ * nodes encountered during the descent -- not on every file -- so the
{@code metadata/} subtree is identified by
+ * its directory name and never enumerated file-by-file.
+ * </p>
+ *
+ * @param fileSystem on which the data needs to be moved
+ * @param from path of the data to be moved
+ * @param to path of the data to be moved
+ * @param deferToLastPhase predicate on a directory {@link Path}; matching
subtrees are renamed in the last phase
+ */
+ public static void renameRecursivelyOrdered(FileSystem fileSystem, Path
from, Path to,
+ Predicate<Path> deferToLastPhase) throws IOException {
+
+ log.info(String.format("Recursively renaming (ordered) %s in %s to %s.",
from, fileSystem.getUri(), to));
+
+ FileSystem throttledFS = getOptionallyThrottledFileSystem(fileSystem,
10000);
+
+ ExecutorService executorService =
ScalingThreadPoolExecutor.newScalingThreadPool(1, 100, 100,
+ ExecutorsUtils.newThreadFactory(Optional.of(log),
Optional.of("rename-thread-%d")));
+ Queue<Future<?>> futures = Queues.newConcurrentLinkedQueue();
+ // Subtrees skipped in phase 1 (their root FileStatus and target path), to
be renamed in phase 2.
+ Queue<Entry<FileStatus, Path>> deferred =
Queues.newConcurrentLinkedQueue();
+
+ try {
+ if (!fileSystem.exists(from)) {
+ throw new IOException("Trying to rename a path that does not exist! "
+ from);
+ }
+
+ // Phase 1: rename everything except the deferred subtrees (which are
collected into `deferred`).
+ futures.add(executorService.submit(new RenameRecursively(
+ throttledFS, fileSystem.getFileStatus(from), to, executorService,
futures, deferToLastPhase, deferred)));
+ drainFutures(futures);
+
+ // Phase 2: rename the deferred subtrees, now that everything else is in
place. No further deferral.
+ for (Entry<FileStatus, Path> entry : deferred) {
+ futures.add(executorService.submit(new RenameRecursively(
+ throttledFS, entry.getKey(), entry.getValue(), executorService,
futures, null, null)));
+ }
+ drainFutures(futures);
+
+ log.info(String.format("Recursive ordered renaming of %s to %s complete
(%d deferred subtree(s)).",
+ from, to, deferred.size()));
+
+ } finally {
+ ExecutorsUtils.shutdownExecutorService(executorService,
Optional.of(log), 1, TimeUnit.SECONDS);
+ }
+ }
+
+ private static void drainFutures(Queue<Future<?>> futures) throws
IOException {
+ while (!futures.isEmpty()) {
+ try {
+ futures.poll().get();
+ } catch (ExecutionException | InterruptedException ee) {
+ throw new IOException(ee.getCause());
+ }
+ }
+ }
Review Comment:
`drainFutures` swallows thread interruption and can throw an `IOException`
with a null cause when an `InterruptedException` occurs (`ee.getCause()` is
null). It’s better to handle `InterruptedException` separately, restore the
interrupt flag, and preserve the `ExecutionException` context/message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]