yihua commented on code in PR #6133:
URL: https://github.com/apache/hudi/pull/6133#discussion_r1080669381
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/MarkerHandler.java:
##########
@@ -150,7 +167,48 @@ public boolean doesMarkerDirExist(String markerDir) {
* @param markerName marker name
* @return the {@code CompletableFuture} instance for the request
*/
- public CompletableFuture<String> createMarker(Context context, String
markerDir, String markerName) {
+ public CompletableFuture<String> createMarker(Context context, String
markerDir, String markerName, String basePath) {
+ // Step1 do early conflict detection if enable
+ if (timelineServiceConfig.earlyConflictDetectionEnable) {
+ try {
+ synchronized (earlyConflictDetectionLock) {
+ if (earlyConflictDetectionStrategy == null) {
+ earlyConflictDetectionStrategy =
(HoodieTimelineServerBasedEarlyConflictDetectionStrategy)
ReflectionUtils.loadClass(timelineServiceConfig.earlyConflictDetectionStrategy,
+ basePath, markerDir, markerName,
timelineServiceConfig.checkCommitConflict);
+ }
+
+ // markerDir => $base_path/.hoodie/.temp/$instant_time
+ // If markerDir is changed like move to the next instant action, we
need to fresh this earlyConflictDetectionStrategy.
+ // For specific instant related create marker action, we only call
this check/fresh once
+ // instead of starting the conflict detector for every request
+ if (!markerDir.equalsIgnoreCase(currentMarkerDir)) {
+ this.currentMarkerDir = markerDir;
+ Set<String> actions = CollectionUtils.createSet(COMMIT_ACTION,
DELTA_COMMIT_ACTION, REPLACE_COMMIT_ACTION);
+ Set<HoodieInstant> oldInstants = new HashSet<>(
+ viewManager.getFileSystemView(basePath)
+ .getTimeline()
+ .filterCompletedInstants()
+ .filter(instant -> actions.contains(instant.getAction()))
+ .getInstants());
+
+
earlyConflictDetectionStrategy.fresh(timelineServiceConfig.asyncConflictDetectorBatchIntervalMs,
+ timelineServiceConfig.asyncConflictDetectorBatchPeriodMs,
markerDir, basePath, timelineServiceConfig.maxAllowableHeartbeatIntervalInMs,
fileSystem,
+ this, oldInstants);
+ }
+ }
+
+ earlyConflictDetectionStrategy.detectAndResolveConflictIfNecessary();
+
+ } catch (HoodieEarlyConflictDetectionException e) {
+ LOG.warn("Detect write conflict, failed to create marker with early
conflict detection enable", e);
+ return finishCreateMarkerFuture(context, markerDir, markerName);
+ } catch (Exception ex) {
+ LOG.warn("Catch exception during detect and resolve write conflict.");
Review Comment:
nit: Let's make the error message readable.
Also, in this case when early conflict detection fails to execute, we still
allow the marker creation to continue.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/marker/WriteMarkers.java:
##########
@@ -60,8 +67,26 @@ public WriteMarkers(String basePath, String
markerFolderPath, String instantTime
* @param type write IO type
* @return the marker path
*/
- public Option<Path> create(String partitionPath, String dataFileName, IOType
type) {
- return create(partitionPath, dataFileName, type, false);
+ public Option<Path> create(String partitionPath, String dataFileName, IOType
type, Option<HoodieWriteConfig> writeConfig,
+ Option<String> fileId,
Option<HoodieTableMetaClient> metaClient) {
Review Comment:
It's not necessary to use `Option` for the arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
