danny0405 commented on code in PR #20221:
URL: https://github.com/apache/hudi/pull/20221#discussion_r4213665933
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -105,6 +113,29 @@ public class RequestHandler {
private final Registry metricsRegistry =
Registry.getRegistry("TimelineService");
private final ScheduledExecutorService asyncResultService;
+ /**
+ * Client timeline hashes for which the local file system view has already
been synced, per view.
+ *
+ * <p>A client's timeline hash describes a timeline the client loaded from
storage before it sent the request.
+ * Once the server has synced its view after receiving a request carrying
hash {@code H}, the server's view
+ * is at least as recent as any client still holding {@code H}. Further
requests carrying {@code H} can still
+ * fail the hash comparison in {@code isLocalViewBehind} (e.g. a clean
completed after the client loaded its
+ * timeline, so the server timeline has an extra instant the client does not
know about), but syncing again
+ * cannot make the server any less behind. Remembering {@code H} lets us
skip those redundant syncs.
+ *
+ * <p>Without this, every such request does a full timeline reload from
storage under the view lock. With
+ * hundreds of executors issuing view and marker requests, the requests
serialize behind that lock, the
+ * server's request threads are exhausted and clients time out with {@code
NoHttpResponseException}.
+ *
+ * <p>The map is keyed by the view instance (identity) rather than the base
path, so that when a view is
+ * cleared and re-created the new view starts with no remembered hashes and
its first sync is never skipped.
+ * Weak keys let entries go away together with views dropped by the {@link
FileSystemViewManager}.
+ * The map itself is shared across views and is therefore synchronized; each
per-view set is only accessed
+ * while holding the lock on its view.
Review Comment:
a fix for the similar purposes: https://github.com/apache/hudi/pull/19946,
the issue is real, we need to decide which fix is preferable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
