Jackie-Jiang commented on code in PR #20106:
URL: https://github.com/apache/pinot/pull/20106#discussion_r4607166892


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/tablestate/TableStateUtils.java:
##########
@@ -82,30 +90,143 @@ public static List<String> 
getSegmentsInGivenStateForThisInstance(HelixManager h
   public static boolean isAllSegmentsLoaded(HelixManager helixManager, String 
tableNameWithType) {
     List<String> onlineSegments =
         getSegmentsInGivenStateForThisInstance(helixManager, 
tableNameWithType, SegmentStateModel.ONLINE);
-    if (onlineSegments.isEmpty()) {
-      LOGGER.info("No ONLINE segment found for table: {}", tableNameWithType);
-      return true;
-    }
+    return areSegmentsLoaded(helixManager, tableNameWithType, onlineSegments);
+  }
 
-    // Check if ideal state and current state matches for all segments 
assigned to the current instance
+  /// Checks the current assignment before a consumer starts. Earlier ONLINE 
segments of the same partition must be
+  /// loaded, as must uploaded segments from that partition. Resolve 
partitions from names or segment metadata,
+  /// matching upsert and dedup loading; unresolved segments are 
conservatively included. The consumer and its
+  /// successors are excluded to avoid waiting on themselves.
+  ///
+  /// @param isLocallyReady identifies segments whose metadata is already 
ready locally despite a pending ONLINE
+  ///                      transition, such as a predecessor that finished 
consumption and released its permit
+  public static boolean isReadyToConsume(HelixManager helixManager, String 
tableNameWithType, LLCSegmentName consumer,
+      Predicate<String> isLocallyReady) {
+    return getReadinessSnapshot(helixManager, tableNameWithType, 
isLocallyReady)
+        .isReadyToConsume(consumer, isLocallyReady, true);
+  }
+
+  /// Reads the table assignment and this live instance's current state once 
for evaluation by multiple consumers.
+  /// Partition metadata is read only for custom-named ONLINE segments that 
are neither ONLINE in current state nor
+  /// locally ready. Loaded segments cannot block consumption, so their 
partition is irrelevant to this check.
+  /// A cached snapshot must only reject readiness; callers must perform a 
fresh check before authorizing consumption
+  /// from a cached result.
+  ///
+  /// @param isLocallyReady a thread-safe predicate used only to skip 
unnecessary partition lookups. Its results are
+  ///                      not retained: evaluation must check local readiness 
again in case a manager was replaced.
+  public static ReadinessSnapshot getReadinessSnapshot(HelixManager 
helixManager, String tableNameWithType,
+      Predicate<String> isLocallyReady) {
     HelixDataAccessor dataAccessor = helixManager.getHelixDataAccessor();
-    PropertyKey.Builder keyBuilder = dataAccessor.keyBuilder();
+    IdealState idealState = 
dataAccessor.getProperty(dataAccessor.keyBuilder().idealStates(tableNameWithType));
+    if (idealState == null) {
+      LOGGER.warn("Failed to find ideal state for table: {}", 
tableNameWithType);
+      return new ReadinessSnapshot(Map.of(), Map.of(), Map.of());
+    }
     String instanceName = helixManager.getInstanceName();
-    LiveInstance liveInstance = 
dataAccessor.getProperty(keyBuilder.liveInstance(instanceName));
-    if (liveInstance == null) {
-      LOGGER.warn("Failed to find live instance for instance: {}", 
instanceName);
-      return false;
+    Map<String, String> idealStates = new HashMap<>();
+    Map<String, String> currentStates = getCurrentStates(helixManager, 
tableNameWithType);
+    Map<String, Integer> metadataPartitionIds = new HashMap<>();
+    for (Map.Entry<String, Map<String, String>> entry : 
idealState.getRecord().getMapFields().entrySet()) {
+      String segmentName = entry.getKey();
+      String state = entry.getValue().get(instanceName);

Review Comment:
   We already reject `includeConsuming=false` for upsert/dedup in 
[`RealtimeSegmentAssignment.rebalanceTable()`](https://github.com/apache/pinot/blob/b8541a5920/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeSegmentAssignment.java#L297).
 The normal no-downtime REST request must pass a dry run before execution. For 
strict replica-group tables, [successors retain the existing IdealState 
partition 
placement](https://github.com/apache/pinot/blob/b8541a5920/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/BaseSegmentAssignment.java#L166),
 even when the instance partitions change.
   
   There is a separate controller ordering issue worth following up: 
direct/downtime paths can persist reassigned instance partitions before that 
validation. With the allowed but unpinned beta multistage routing mode, a later 
commit can use the changed pool despite the rebalance having failed. This 
predates this PR; I traced the code path, without running a reproduction. The 
fix belongs in early controller validation, before instance-partition writes, 
and I recorded it in the PR description.
   
   I would keep the server gate scoped to assigned history. Requiring all 
unassigned older segments could also block legitimate dedup history moved to 
tiers. The assignment-side follow-up closes the invalid-request path without 
changing that contract.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to