hudi-agent commented on code in PR #18988:
URL: https://github.com/apache/hudi/pull/18988#discussion_r3519106315
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/callback/common/HoodieWriteCommitCallbackMessage.java:
##########
@@ -69,10 +72,106 @@ public class HoodieWriteCommitCallbackMessage implements
Serializable {
*/
private final Option<Map<String, String>> extraMetadata;
+ /**
+ * Previous base file paths keyed by fileId, resolved lazily. Populated by
the write
+ * client from the cached FileSystemView so that callback implementations
don't have to
+ * rebuild a view. Empty for inserts and for callers that don't pre-resolve.
+ *
+ * <p>Resolution is deferred to the {@code prevFilePathsSupplier} and only
triggered on the
+ * first {@link #getPrevFilePaths()} call: a callback that never reads the
previous paths
+ * pays nothing (no FileSystemView access). The resolved value is memoized
here and is the
+ * form that gets serialized (see {@link #writeObject}).
+ */
+ private Map<String, PrevFilePaths> prevFilePaths;
+
+ /**
+ * Lazily resolves {@link #prevFilePaths}. Transient because it may capture a
+ * FileSystemView (not serializable); excluded from the generated getters so
it never
+ * leaks into JSON/Java serialization. The resolved map is memoized into
+ * {@link #prevFilePaths}.
+ */
+ @Getter(AccessLevel.NONE)
+ private final transient Supplier<Map<String, PrevFilePaths>>
prevFilePathsSupplier;
+
+ /**
+ * Free-form context that producers can attach for downstream callback
consumers.
+ * The OSS write client populates this as empty; specialized callsites or
wrappers
+ * may populate it with whatever context their callbacks need.
+ */
+ private final Map<String, String> extraContext;
+
+ public HoodieWriteCommitCallbackMessage(String commitTime,
+ String tableName,
+ String basePath,
+ List<HoodieWriteStat>
hoodieWriteStat,
+ Option<String> commitActionType,
+ Option<Map<String, String>>
extraMetadata,
+ Supplier<Map<String, PrevFilePaths>>
prevFilePathsSupplier,
+ Map<String, String> extraContext) {
+ this.commitTime = commitTime;
+ this.tableName = tableName;
+ this.basePath = basePath;
+ this.hoodieWriteStat = hoodieWriteStat;
+ this.commitActionType = commitActionType;
+ this.extraMetadata = extraMetadata;
+ this.prevFilePathsSupplier = prevFilePathsSupplier;
+ this.extraContext = extraContext;
+ }
+
public HoodieWriteCommitCallbackMessage(String commitTime,
String tableName,
String basePath,
List<HoodieWriteStat>
hoodieWriteStat) {
- this(commitTime, tableName, basePath, hoodieWriteStat, Option.empty(),
Option.empty());
+ this(commitTime, tableName, basePath, hoodieWriteStat, Option.empty(),
Option.empty(),
+ Collections::emptyMap, Collections.emptyMap());
+ }
+
+ public HoodieWriteCommitCallbackMessage(String commitTime,
+ String tableName,
+ String basePath,
+ List<HoodieWriteStat>
hoodieWriteStat,
+ Option<String> commitActionType,
+ Option<Map<String, String>>
extraMetadata) {
+ this(commitTime, tableName, basePath, hoodieWriteStat, commitActionType,
extraMetadata,
+ Collections::emptyMap, Collections.emptyMap());
+ }
+
+ /**
+ * Returns the previous base file paths keyed by fileId, resolving them on
first access.
+ * A consumer that never calls this triggers no FileSystemView lookup. Never
null.
+ */
+ public Map<String, PrevFilePaths> getPrevFilePaths() {
+ if (prevFilePaths == null) {
+ prevFilePaths = prevFilePathsSupplier != null
+ ? prevFilePathsSupplier.get() : Collections.emptyMap();
+ }
+ return prevFilePaths;
+ }
+
+ /**
+ * Force lazy resolution before serialization so the resolved map (not the
transient
+ * supplier) is what gets written.
+ */
+ private void writeObject(ObjectOutputStream out) throws IOException {
+ getPrevFilePaths();
+ out.defaultWriteObject();
+ }
+
+ /**
+ * Container for previously-existing file paths associated with a single
fileId in a
+ * commit. {@link #prevBaseFilePath} is the base file the new write
replaces, and
+ * {@link #bootstrapBaseFilePath} is the bootstrap-source file the previous
+ * base file referenced (null for non-bootstrap tables).
+ */
+ @Getter
+ public static class PrevFilePaths implements Serializable {
+ private static final long serialVersionUID = 1L;
+ private final String prevBaseFilePath;
Review Comment:
🤖 nit: the `prev` prefix feels redundant here — the enclosing class is
already called `PrevFilePaths`, so `baseFilePath` (and `getBaseFilePath()`)
would read more naturally at call sites instead of
`paths.getPrevBaseFilePath()`.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -287,7 +283,7 @@ public boolean commitStats(String instantTime,
TableWriteStats tableWriteStats,
boolean postCommitStatus = true;
HoodieTimer postCommitTimer = HoodieTimer.start();
try {
- postCommit(table, metadata, instantTime, extraMetadata);
+ postCommit(table, metadata, instantTime, commitActionType,
extraMetadata);
Review Comment:
🤖 Confirmed against the pre-PR code: the callback used to fire after
`emitCommitMetrics`, outside the post-commit try block, so a
`mayBeCleanAndArchive`/`runTableServicesInline` failure with
`canIgnorePostCommitFailures=false` would propagate and skip it. Now that it
fires inside `postCommit` (first, before those two), a consumer can receive a
success callback even when the `commitStats` call ultimately throws. It's
arguably more correct since the commit is already durable at that point, but it
is an observable behavior change worth calling out explicitly.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieClient.java:
##########
@@ -462,4 +478,79 @@ private static Map<String, String>
collectRollingMetadataFromTimeline(
protected Option<Map<String, String>> updateExtraMetadata(Option<Map<String,
String>> extraMetadata) {
return CommitMetadataProperties.enrich(extraMetadata, config, context);
}
+
+ /**
+ * Fire {@link HoodieWriteCommitCallback} for a commit, if enabled. Shared by
+ * {@link BaseHoodieWriteClient#postCommit} (regular auto- and
explicit-commit paths)
+ * and {@link BaseHoodieTableServiceClient} (compaction and clustering
completions).
+ * Lazily constructs the callback instance from {@code
hoodie.write.commit.callback.class}.
+ *
+ * <p>Best-effort: catches and logs any exception from the user-supplied
callback so a
+ * misbehaving observer cannot fail the commit.
+ */
+ protected void fireCommitCallbackIfNecessary(String commitTime,
+ String commitActionType,
+ List<HoodieWriteStat> stats,
+ Supplier<BaseFileOnlyView>
fsViewSupplier,
+ Option<Map<String, String>>
extraMetadata) {
+ if (!config.writeCommitCallbackOn()) {
+ return;
+ }
+ try {
+ if (commitCallback == null) {
+ commitCallback = HoodieCommitCallbackFactory.create(config);
+ }
+ commitCallback.call(new HoodieWriteCommitCallbackMessage(
+ commitTime, config.getTableName(), config.getBasePath(),
+ stats, Option.of(commitActionType), extraMetadata,
+ () -> resolvePrevFilePaths(stats, fsViewSupplier.get()),
+ Collections.emptyMap()));
+ } catch (Exception e) {
+ log.warn("HoodieWriteCommitCallback failed for commit {} ({}); ignoring",
+ commitTime, commitActionType, e);
+ }
+ }
+
+ /**
+ * Pre-resolve the previous base file (and bootstrap base file, if any) for
every
+ * {@link HoodieWriteStat} that represents an update, using a populated
+ * {@link BaseFileOnlyView}. The lookup is O(1) per stat against the cached
view, so
+ * this adds no I/O on top of what the writer already paid.
+ *
+ * <p>Used by {@link #fireCommitCallbackIfNecessary} call sites so the
callback message ships
+ * actual file paths rather than forcing each callback impl to rebuild a
+ * {@code FileSystemView}.
+ */
+ protected static Map<String, PrevFilePaths>
resolvePrevFilePaths(List<HoodieWriteStat> stats,
+
BaseFileOnlyView fsView) {
+ Map<String, PrevFilePaths> out = new HashMap<>();
Review Comment:
🤖 nit: `out` reads a bit like a C-style output-parameter name — something
like `result` or `pathsByFileId` would make it clearer this is just the map
being built and returned.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
