prashantwason commented on code in PR #18084:
URL: https://github.com/apache/hudi/pull/18084#discussion_r2910465576
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -422,6 +426,21 @@ public void releaseResources(String instantTime) {
SparkReleaseResources.releaseCachedData(context, config, basePath,
instantTime);
}
+ /**
+ * Merges engine-specific metadata (e.g., spark_application_id) with the
provided extra metadata.
+ */
+ private Option<Map<String, String>>
mergeEngineCommitMetadata(Option<Map<String, String>> extraMetadata) {
+ Map<String, String> engineMetadata = context.getEngineCommitMetadata();
+ if (engineMetadata.isEmpty()) {
+ return extraMetadata;
+ }
+ Map<String, String> merged = extraMetadata.isPresent()
Review Comment:
Done in commit 6e09f3d.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -329,9 +329,18 @@ protected String getCommitActionType() {
protected void
setCommitMetadata(HoodieWriteMetadata<HoodieData<WriteStatus>> result) {
List<HoodieWriteStat> writeStats =
result.getWriteStatuses().map(WriteStatus::getStat).collectAsList();
result.setWriteStats(writeStats);
+ // Merge engine-specific metadata (e.g., spark_application_id) with extra
metadata
+ Option<Map<String, String>> mergedExtraMetadata = extraMetadata;
+ Map<String, String> engineMetadata = context.getEngineCommitMetadata();
+ if (!engineMetadata.isEmpty()) {
Review Comment:
Done. Extracted `CommitUtils.mergeEngineMetadata()` and use it across all
call sites. See commit 9f6b9e9.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/CommitUtils.java:
##########
@@ -100,6 +100,26 @@ public static HoodieCommitMetadata
buildMetadata(List<HoodieWriteStat> writeStat
return commitMetadata;
}
+ /**
+ * Merges engine-specific metadata (e.g., spark_application_id) with the
provided extra metadata.
+ *
+ * @param extraMetadata the original extra metadata option
+ * @param engineMetadata the engine-specific metadata to merge
+ * @return merged metadata option, or original if engine metadata is empty
+ */
+ public static Option<Map<String, String>> mergeEngineMetadata(
Review Comment:
Added unit tests in `TestCommitUtils` covering: null engine metadata, empty
engine metadata, merge with existing extra metadata, merge with empty extra
metadata (Option.empty()), and key overwrite behavior. See commit af14d08.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
