npawar commented on a change in pull request #6667:
URL: https://github.com/apache/incubator-pinot/pull/6667#discussion_r607349240
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
##########
@@ -448,14 +494,39 @@ private void commitSegmentMetadataInternal(String
realtimeTableName,
// Refresh the Broker routing to reflect the changes in the segment ZK
metadata
_helixResourceManager.sendSegmentRefreshMessage(realtimeTableName,
committingSegmentName, false, true);
- // Step-2
+ // Get current partition groups - this gives current state of latest
segments for each partition
+ // E.g. [A - DONE], [B - IN_PROGRESS], [C - IN_PROGRESS]
+ PartitionLevelStreamConfig streamConfig = new
PartitionLevelStreamConfig(tableConfig.getTableName(),
+ IngestionConfigUtils.getStreamConfigMap(tableConfig));
+ List<PartitionGroupMetadata> currentPartitionGroupMetadataList =
+ getCurrentPartitionGroupMetadataList(idealState, streamConfig);
+
+ // Fetches new partition groups, given current partition groups metadata.
+ // Assume stream has partitions A, B, C, all still consuming. Result will
be A, B, C
+ // Assume A was split into D, E, but messages of A are yet to be consumed,
result will be A, B, C
Review comment:
The `getCurrentPartitionGroupStatus` call will not look at the state in
ideal state. So even if it is OFFLINE, it will be included in the
partitionGroupStatus, with Status=whatever's in the ZK metadata. So it would
still be
0 - seqNum: 0, startOffset: 100, endOffset: 220, status: DONE
1 - seqNum: 0, startOffset: 110, endOffset: -, status: IN_PROGRESS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
