zhangyue19921010 commented on code in PR #6133:
URL: https://github.com/apache/hudi/pull/6133#discussion_r1029439999
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/marker/WriteMarkers.java:
##########
@@ -52,16 +58,46 @@ public WriteMarkers(String basePath, String
markerFolderPath, String instantTime
this.instantTime = instantTime;
}
+ public Option<Path> create(String partitionPath, String dataFileName, IOType
type) {
+ return create(partitionPath, dataFileName, type, Option.empty());
+ }
+
/**
* Creates a marker without checking if the marker already exists.
*
* @param partitionPath partition path in the table
* @param dataFileName data file name
* @param type write IO type
+ * @param handler could be empty
* @return the marker path
*/
- public Option<Path> create(String partitionPath, String dataFileName, IOType
type) {
- return create(partitionPath, dataFileName, type, false);
+ public Option<Path> create(String partitionPath, String dataFileName, IOType
type, Option<HoodieWriteHandle> handler) {
+ boolean checkIfExists = false;
+
+ if (handler.isPresent()
+ &&
handler.get().getConfig().getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()
+ && handler.get().getConfig().isEarlyConflictDetectionEnable()) {
+
+ HoodieTableMetaClient metaClient =
handler.get().getHoodieTableMetaClient();
+ HoodieActiveTimeline activeTimeline = metaClient.getActiveTimeline();
+
+ HoodieTimeline pendingCompactionTimeline =
activeTimeline.filterPendingCompactionTimeline();
+ HoodieTimeline pendingReplaceTimeline =
activeTimeline.filterPendingReplaceTimeline();
+ // TODO if current is compact or clustering then create marker directly
without early conflict detection.
+ // Need to support early conflict detection between table service and
common writers.
+ if (pendingCompactionTimeline.containsInstant(instantTime) ||
pendingReplaceTimeline.containsInstant(instantTime)) {
Review Comment:
Compaction and Clustering will create markers directly without any early
conflict detection.
Bcz these hoodie table service can be async and not depend on multi-writer
concurrency control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
