Jackie-Jiang commented on code in PR #20106:
URL: https://github.com/apache/pinot/pull/20106#discussion_r4623532476


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java:
##########
@@ -235,43 +244,78 @@ protected void doInit() {
     _enforceConsumptionInOrder = isEnforceConsumptionInOrder();
     _streamMetadataProviderCache = getStreamMetadataProviderCache();
 
-    // For dedup and partial-upsert, need to wait for all segments loaded 
before starting consuming data
-    BooleanSupplier readyForDedupOrPartialUpsert;
-    if (isDedupEnabled() || isPartialUpsertEnabled()) {
-      readyForDedupOrPartialUpsert = new BooleanSupplier() {
-        volatile boolean _allSegmentsLoaded;
-        long _lastCheckTimeMs;
-
-        @Override
-        public boolean getAsBoolean() {
-          if (_allSegmentsLoaded) {
-            return true;
-          } else {
-            synchronized (this) {
-              if (_allSegmentsLoaded) {
-                return true;
-              }
-              long currentTimeMs = System.currentTimeMillis();
-              if (currentTimeMs - _lastCheckTimeMs <= 
READY_TO_CONSUME_DATA_CHECK_INTERVAL_MS) {
-                return false;
-              }
-              _lastCheckTimeMs = currentTimeMs;
-              _allSegmentsLoaded = 
TableStateUtils.isAllSegmentsLoaded(_helixManager, _tableNameWithType);
-              return _allSegmentsLoaded;
-            }
-          }
-        }
-      };
-    } else {
-      readyForDedupOrPartialUpsert = () -> true;
-    }
-    _isTableReadyToConsumeData = () -> 
readyForDedupOrPartialUpsert.getAsBoolean()
-        && _isServerReadyToConsumeData.getAsBoolean();
     _serverIngestionOomProtectionManager = new 
ServerIngestionOomProtectionManager(
         () -> getCachedTableConfigAndSchema().getLeft(), () -> 
isUpsertEnabled() || isDedupEnabled(),
         _serverIngestionOomProtectionThrottleState);
   }
 
+  /// Checks the assigned history again for each consumer, including 
partitions added after table initialization.
+  /// The consumer thread polls this supplier before acquiring its permit, so 
segment loading is never blocked by it.
+  /// TODO: Coordinate history removal (e.g. RESET or manual delete) with an 
already running consumer. This startup
+  /// check does not prevent later removal from discarding a key's base value 
before its next partial update.
+  @VisibleForTesting
+  BooleanSupplier createReadyToConsumeDataSupplier(LLCSegmentName 
llcSegmentName) {
+    if (!requiresHistoryForConsumption()) {
+      return _isServerReadyToConsumeData;
+    }
+    long consumerCreationTimeNs = readinessCheckTimeNs();
+    return () -> {
+      if (!_isServerReadyToConsumeData.getAsBoolean()) {
+        return false;
+      }
+      try {
+        return isReadyToConsume(llcSegmentName, consumerCreationTimeNs);
+      } catch (ZkException e) {

Review Comment:
   Fixed in `9975064432`. The readiness check now catches `HelixException` 
alongside `ZkException`, so a reconnect timeout from `getHelixDataAccessor()` 
or `getHelixPropertyStore()` holds the consumer and retries instead of stopping 
it. `testReadinessRetriesHelixReconnectTimeout` makes `getHelixDataAccessor()` 
throw once: the same consumer stays unready on that check and becomes ready on 
the next. It was not run locally; CI will validate the new head `ebfdeb5896`.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/tablestate/TableStateUtils.java:
##########
@@ -73,64 +81,158 @@ public static List<String> 
getSegmentsInGivenStateForThisInstance(HelixManager h
     return segmentsInGivenState;
   }
 
-  /// Checks if all segments for the given @param tableNameWithType were 
successfully loaded
-  /// This function will get all segments in IDEALSTATE and CURRENTSTATE for 
the given table,
-  /// and then check if all ONLINE segments in IDEALSTATE match with 
CURRENTSTATE.
-  /// @param helixManager helix manager for the server instance
-  /// @param tableNameWithType table name for which segment state is to be 
checked
-  /// @return true if all segments for the given table are successfully 
loaded. False otherwise
-  public static boolean isAllSegmentsLoaded(HelixManager helixManager, String 
tableNameWithType) {
-    List<String> onlineSegments =
-        getSegmentsInGivenStateForThisInstance(helixManager, 
tableNameWithType, SegmentStateModel.ONLINE);
-    if (onlineSegments.isEmpty()) {
-      LOGGER.info("No ONLINE segment found for table: {}", tableNameWithType);
+  /// Checks the current assignment before a consumer starts. Earlier ONLINE 
segments of the same partition must be
+  /// loaded, as must uploaded segments from that partition. Resolve 
partitions from names or segment metadata,
+  /// matching upsert and dedup loading; unresolved segments are 
conservatively included. The consumer and its
+  /// successors are excluded to avoid waiting on themselves.
+  ///
+  /// @param isLocallyReady identifies segments whose metadata is already 
ready locally despite a pending ONLINE
+  ///                      transition, such as a predecessor that finished 
consumption and released its permit
+  public static boolean isReadyToConsume(HelixManager helixManager, String 
tableNameWithType, LLCSegmentName consumer,
+      Predicate<String> isLocallyReady) {
+    return getReadinessSnapshot(helixManager, tableNameWithType, 
isLocallyReady)
+        .isReadyToConsume(consumer, isLocallyReady, true);
+  }
+
+  /// Reads the table assignment and this live instance's current state once 
for evaluation by multiple consumers.
+  /// Only what evaluation reads is kept: this instance's CONSUMING segments, 
and its ONLINE segments that are not
+  /// ONLINE in current state. Partition metadata is read only for 
custom-named segments among the latter that are not
+  /// locally ready. Loaded segments cannot block consumption, so neither they 
nor their partitions are retained.
+  /// A cached snapshot must only reject readiness; callers must perform a 
fresh check before authorizing consumption
+  /// from a cached result.
+  ///
+  /// @param isLocallyReady a thread-safe predicate used only to skip 
unnecessary partition lookups. Its results are
+  ///                      not retained: evaluation must check local readiness 
again in case a manager was replaced.
+  public static ReadinessSnapshot getReadinessSnapshot(HelixManager 
helixManager, String tableNameWithType,
+      Predicate<String> isLocallyReady) {
+    HelixDataAccessor dataAccessor = helixManager.getHelixDataAccessor();
+    IdealState idealState = 
dataAccessor.getProperty(dataAccessor.keyBuilder().idealStates(tableNameWithType));
+    if (idealState == null) {
+      LOGGER.warn("Failed to find ideal state for table: {}", 
tableNameWithType);
+      return new ReadinessSnapshot(Set.of(), Map.of(), Map.of());
+    }
+    String instanceName = helixManager.getInstanceName();
+    CurrentState currentState = getCurrentState(helixManager, 
tableNameWithType);
+    Set<String> consumingSegments = new HashSet<>();
+    Map<String, String> unloadedSegments = new HashMap<>();
+    Map<String, Integer> metadataPartitionIds = new HashMap<>();
+    for (Map.Entry<String, Map<String, String>> entry : 
idealState.getRecord().getMapFields().entrySet()) {
+      String segmentName = entry.getKey();
+      String state = entry.getValue().get(instanceName);
+      if (SegmentStateModel.CONSUMING.equals(state)) {
+        consumingSegments.add(segmentName);
+      } else if (SegmentStateModel.ONLINE.equals(state)) {
+        String actualState = currentState != null ? 
currentState.getState(segmentName) : null;
+        if (!SegmentStateModel.ONLINE.equals(actualState)) {

Review Comment:
   Confirmed: `onBecomeOfflineFromError` leaves the manager registered, and the 
ONLINE retry goes through `replaceSegmentIfCrcMismatch`, which returns early on 
a matching CRC. A segment whose upsert `addSegment` failed therefore comes back 
ONLINE with partial metadata. As you note, this predates this PR, and it also 
affects queries on realtime and offline upsert tables.
   
   I'll fix the cause in a separate PR: the ONLINE retry will rebuild a manager 
whose metadata never finished initializing, regardless of CRC, with a test 
covering the failed add, the reset, and the retry. ONLINE in CurrentState then 
implies initialized metadata, so this check keeps relying on it. Also requiring 
the local flag here would block on loaded segments that are never marked, such 
as a `DuoSegmentDataManager` during a reload.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to