klsince commented on code in PR #11020:
URL: https://github.com/apache/pinot/pull/11020#discussion_r1256227999
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BaseTableUpsertMetadataManager.java:
##########
@@ -75,6 +107,127 @@ public void init(TableConfig tableConfig, Schema schema,
TableDataManager tableD
_enableSnapshot = upsertConfig.isEnableSnapshot();
_serverMetrics = serverMetrics;
+ _helixManager = helixManager;
+ _segmentPreloadExecutor = segmentPreloadExecutor;
+ if (_enableSnapshot && upsertConfig.isEnablePreload()) {
+ // Preloading the segments with snapshots for fast upsert metadata
recovery.
+ // Note that there is an implicit waiting logic between the thread doing
the segment preloading here and the
+ // other helix threads about to process segment state transitions (e.g.
taking segments from OFFLINE to ONLINE).
+ // The thread doing the segment preloading here must complete before the
other helix threads start to handle
+ // segment state transitions. This is ensured implicitly because segment
preloading happens here when
+ // initializing this TableUpsertMetadataManager, which happens when
initializing the TableDataManager, which
+ // happens as the lambda of ConcurrentHashMap.computeIfAbsent() method,
which ensures the waiting logic.
+ try {
+ _isPreloading = true;
+ preloadSegments();
+ } catch (Exception e) {
+ // Even if preloading fails, we should continue to complete the
initialization, so that TableDataManager can be
Review Comment:
good point for finer handling, will add.
but at a higher level, we'd better not fail the init() method due to
uncaught preloading failure, so that we can create the tableDataMgr and fall
back to the normal loading flow. If init() keeps failing due to uncaught
preloading failure, creation of tableDataMgr would be retried, leading to
another round of preloading while calling the init() method again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
