shauryachats commented on PR #18433:
URL: https://github.com/apache/pinot/pull/18433#issuecomment-4595321711
> In the above example, the current assignment will assign in a round-robin
manner to balance segment placement. Why 10000%3 = 1 is a wrong assignment
result?
`10000 % 3 = 1` is not a wrong result by itself - if there is no expected
partitioning by a field. The distribution is balanced either way - that's a
fair point. The correctness issue is specifically about partition-aware query
routing, not balance.
When `partitionColumn: "trace_id"` is configured, Pinot uses the instance
partition assignment to prune which servers a query needs to hit. A query like
`WHERE trace_id = 'abc'` determines which logical partition owns that trace_id,
an routes only to the servers holding that partition — skipping the rest. This
is the whole point of the replica-group partition configuration.
This pruning only works if all data for the same logical partition (across
all streams) lives on the same instance group. In this setup, both streams are
co-partitioned by trace_id, so stream 0 partition 0 and stream 1 partition
contain data for the exact same set of trace IDs.
Without the fix:
- Stream 0, partition 0 (Pinot ID 0) → 0 % 3 = 0 → instance group 0
(Server A)
- Stream 1, partition 0 (Pinot ID 10000) → 10000 % 3 = 1 → instance group
1 (Server B)
Now a query for a trace_id in logical partition 0 must hit both Server A
and Server B, because the data is split across
two instance groups. Partition pruning is broken.
With the fix, both map to instance group 0 (Server A), and partition
pruning works as intended.
If the two streams had no relationship between their partition keys, the
colocation argument wouldn't apply — but then
the user also wouldn't be configuring a shared partitionColumn across them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
