the-other-tim-brown commented on code in PR #18073:
URL: https://github.com/apache/hudi/pull/18073#discussion_r2822786179
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java:
##########
@@ -174,21 +195,44 @@ public List<MarkerCreationFuture>
fetchPendingMarkerCreationRequests() {
}
/**
- * @param shouldClear Should clear the internal request list or not.
+ * @param shouldClear Should clear the internal request map or not.
* @return futures of pending marker creation requests.
*/
public List<MarkerCreationFuture> getPendingMarkerCreationRequests(boolean
shouldClear) {
- List<MarkerCreationFuture> pendingFutures;
- synchronized (markerCreationFutures) {
- if (markerCreationFutures.isEmpty()) {
- return new ArrayList<>();
- }
- pendingFutures = new ArrayList<>(markerCreationFutures);
+ if (inflightRequestMap.isEmpty()) {
Review Comment:
To keep this consistent with the behavior before, let's also move this in
the synchronized block
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java:
##########
@@ -80,8 +84,9 @@ public class MarkerDirState implements Serializable {
// {@code true} means the file is in use by a {@code
BatchCreateMarkerRunnable}.
// Index of the list is used for the filename, i.e., "1" -> "MARKERS1"
private final List<Boolean> threadUseStatus;
- // A list of pending futures from async marker creation requests
- private final List<MarkerCreationFuture> markerCreationFutures = new
ArrayList<>();
+ // Map of in-flight requests: (markerName, requestId) -> MarkerCreationFuture
+ // Used for BOTH deduplication AND batch marker creation requests queue
+ private final Map<Pair<String, String>, MarkerCreationFuture>
inflightRequestMap = new ConcurrentHashMap<>();
Review Comment:
If we're synchronizing on the object, can we make this a simple HashMap?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
