klsince commented on code in PR #12976:
URL: https://github.com/apache/pinot/pull/12976#discussion_r1576744981
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -1103,6 +1210,48 @@ private static MutableRoaringBitmap
getQueryableDocIdsSnapshotFromSegment(IndexS
return queryableDocIdsSnapshot;
}
+ private void setSegmentContexts(List<SegmentContext> segmentContexts) {
+ for (SegmentContext segmentContext : segmentContexts) {
+ IndexSegment segment = segmentContext.getIndexSegment();
+ if (_trackedSegments.contains(segment)) {
+
segmentContext.setQueryableDocIdsSnapshot(getQueryableDocIdsSnapshotFromSegment(segment));
+ }
+ }
+ }
+
+ private boolean skipUpsertViewRefresh(long upsertViewFreshnessMs) {
+ long nowMs = System.currentTimeMillis();
+ if (upsertViewFreshnessMs >= 0) {
+ return _lastUpsertViewRefreshTimeMs + upsertViewFreshnessMs > nowMs;
+ }
+ return _lastUpsertViewRefreshTimeMs + _upsertViewRefreshIntervalMs > nowMs;
+ }
+
+ private void doBatchRefreshUpsertView(long upsertViewFreshnessMs) {
Review Comment:
Thanks for sharing the code changes, and I read it through. The refreshing
logic is similar with the batch mode here although done at table mgr level. But
your changes didn't allow query threads to force update the view to ensure data
freshness.
I think it's more intuitive to handle data consistency in table partition
mgr, as table partition is the unit for managing upsert states in upsert table.
Today, the table partition mgr `enriches` the segment contexts, which are
provided by table partition mgr, so there shouldn't be limitation to extending
segment contexts later. In fact hiding the consistency logic away from table
mgr would make it easier to iterate in future.
```
public void setSegmentContexts(List<SegmentContext> segmentContexts) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
