nsivabalan commented on code in PR #13402:
URL: https://github.com/apache/hudi/pull/13402#discussion_r2136694157
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -239,13 +239,23 @@ public boolean commitStats(String instantTime,
List<HoodieWriteStat> stats,
Option<Map<String, String>> extraMetadata,
String commitActionType, Map<String,
List<String>> partitionToReplaceFileIds,
Option<BiConsumer<HoodieTableMetaClient,
HoodieCommitMetadata>> extraPreCommitFunc) {
+ return commitStats(instantTime, stats, Collections.emptyList(),
extraMetadata, commitActionType, partitionToReplaceFileIds, extraPreCommitFunc,
+ false, Option.empty());
+ }
+
+ public boolean commitStats(String instantTime, List<HoodieWriteStat> stats,
List<HoodieWriteStat> partialMetadataHoodieWriteStatsSoFar,
+ Option<Map<String, String>> extraMetadata,
+ String commitActionType, Map<String,
List<String>> partitionToReplaceFileIds,
+ Option<BiConsumer<HoodieTableMetaClient,
HoodieCommitMetadata>> extraPreCommitFunc,
+ boolean skipStreamingWritesToMetadataTable,
+ Option<HoodieTable> hoodieTableOpt) {
Review Comment:
Note to Reviewer:
skipStreamingWritesToMetadataTable : this is mainly for row writer flows.
With row writer, we can't really do the rdd chaining all the way since custom
data source writes will return an array of WriteStatus and there is no way we
can control the rdd or the triggering of writes.
So, we had to introduce this argument named
"skipStreamingWritesToMetadataTable". So that following call stack will avoid
streaming writes flow and take the legacy write flow.
We do have logic below to enable or disable based on write operations. Just
that we have bulk insert support in both rdd based write client and also based
on row writer. So, I am taking this route so that we keep streaming writes
disabled for row writer paths.
Happy to jam on any better ideas on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
