Jackie-Jiang commented on code in PR #19170:
URL: https://github.com/apache/pinot/pull/19170#discussion_r3762570932
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1448,28 +1448,34 @@ public void ensureAllPartitionsConsuming(TableConfig
tableConfig, List<StreamCon
Preconditions.checkState(!_isStopping, "Segment manager is stopping");
String realtimeTableName = tableConfig.getTableName();
+
+ // Fetch all stream offsets BEFORE acquiring the ideal-state update lock.
On tables with many partitions these
+ // stream round-trips can take minutes; doing them here (against a
snapshot of the ideal state) keeps the
+ // per-table ideal-state lock hold-time proportional to the in-memory
ideal-state mutation, not to the offset
+ // I/O. The updater lambda below performs no stream I/O, so it is also
cheap to re-run on ZK CAS retries.
+ IdealState snapshotIdealState =
HelixHelper.getTableIdealState(_helixManager, realtimeTableName);
Review Comment:
Are we fetching the latest consuming segment offset here? If so, we will
lose recent segment commit when using IS snapshot right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
