hgudladona commented on issue #13356:
URL: https://github.com/apache/hudi/issues/13356#issuecomment-2964720355
Here is another data point, if we flip the conditions to check for the local
timeline containing the instant from the remote request first, with the above
proposed ReadWriteLock we are seeing only 1 refresh per job, subsequent
requests always pass the `isLocalViewBehind ` check, both timeline instant
check and hash matching check, basically working as expected.
However, I am unsure of the repercussions of doing this.
BEFORE
```
/**
* Determines if local view of table's timeline is behind that of
client's view.
*/
private boolean isLocalViewBehind(Context ctx) {
String basePath =
ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
String lastKnownInstantFromClient = getLastInstantTsParam(ctx);
String timelineHashFromClient = getTimelineHashParam(ctx);
HoodieTimeline localTimeline =
viewManager.getFileSystemView(basePath).getTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
if (LOG.isDebugEnabled()) {
LOG.debug("Client [ LastTs={}, TimelineHash={}],
localTimeline={}",lastKnownInstantFromClient, timelineHashFromClient,
localTimeline.getInstants());
}
if ((!localTimeline.getInstantsAsStream().findAny().isPresent())
&&
HoodieTimeline.INVALID_INSTANT_TS.equals(lastKnownInstantFromClient)) {
return false;
}
String localTimelineHash = localTimeline.getTimelineHash();
// refresh if timeline hash mismatches
if (!localTimelineHash.equals(timelineHashFromClient)) {
return true;
}
// As a safety check, even if hash is same, ensure instant is present
return
!localTimeline.containsOrBeforeTimelineStarts(lastKnownInstantFromClient);
}
```
AFTER
```
/**
* Determines if local view of table's timeline is behind that of
client's view.
*/
private boolean isLocalViewBehind(Context ctx) {
String basePath =
ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
String lastKnownInstantFromClient = getLastInstantTsParam(ctx);
String timelineHashFromClient = getTimelineHashParam(ctx);
HoodieTimeline localTimeline =
viewManager.getFileSystemView(basePath).getTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
if (LOG.isDebugEnabled()) {
LOG.debug("Client [ LastTs={}, TimelineHash={}],
localTimeline={}",lastKnownInstantFromClient, timelineHashFromClient,
localTimeline.getInstants());
}
if ((!localTimeline.getInstantsAsStream().findAny().isPresent())
&&
HoodieTimeline.INVALID_INSTANT_TS.equals(lastKnownInstantFromClient)) {
return false;
}
// As a safety check, even if hash is same, ensure instant is present
if
(!localTimeline.containsOrBeforeTimelineStarts(lastKnownInstantFromClient)) {
return true; <<-- this triggers first refresh; subsequent checks
pass this check
}
String localTimelineHash = localTimeline.getTimelineHash();
// refresh if timeline hash mismatches
return !localTimelineHash.equals(timelineHashFromClient); <<-- After
first refresh; subsequent checks pass this check too.
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
