Jackie-Jiang opened a new pull request, #18708:
URL: https://github.com/apache/pinot/pull/18708
## Summary
`RealtimeSegmentDataManager.setPartitionParameters` derived the partition
function divisor (`numPartitions`) from
`computePartitionGroupMetadata(clientId, streamConfig, emptyList(),
timeoutMs).size()`. That call fetches and discards a `StreamPartitionMsgOffset`
for every partition just to take the list size, and — more importantly —
returns the wrong count in two cases:
- **Kafka partition-subset feature** (`stream.kafka.partition.ids`): the
`KafkaStreamMetadataProvider` override returns the *subset* size, not the
topic's full partition count. The subset path also preserves the original Kafka
partition ids as the partition group ids, so a divisor equal to the subset size
cannot even represent those ids (e.g. a subset `{5, 17, 42}` with divisor `3`).
The per-segment partition metadata is then computed inconsistently and
partition pruning can prune valid segments, returning wrong results.
- **Kinesis**: the override excludes expired shards, so `.size()` reflects
active shards rather than the producer's hashing space.
The divisor must match the partition count the upstream producer hashed each
key over. `fetchPartitionCount` returns exactly that with a single call, so
switch to it. For Kafka it resolves via `partitionsFor(topic)`, i.e. the full
topic partition count, independent of any configured subset.
Kinesis partitioning remains unsupported (shards split/merge and ids are not
a contiguous `0..N` range); a `TODO` records this.
Also refreshed the stale surrounding comments (no metric is emitted on
mismatch; it is now logged at `warn`), promoted the fetch-failure log to
`error`, and aligned the failure log message with the actual timeout.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
