Jackie-Jiang opened a new pull request, #19958:
URL: https://github.com/apache/pinot/pull/19958
## Summary
The partition column that segment assignment uses to place OFFLINE segments
can be configured in two legacy places that have nothing to do with segment
assignment: `replicaGroupStrategyConfig` in `segmentsConfig`, and
`partitionColumn` in `replicaGroupPartitionConfig` of
`instanceAssignmentConfigMap`. Instance assignment never reads the column (the
instance partition selectors only use `numPartitions`); its readers are the
replica-group segment assignment strategies, the rebalancer and the offline
upsert upload validation. Neither place is available to a table with
pre-configured `instancePartitionsMap` (table colocation), and neither covers
tiers.
This PR moves the partition column to `SegmentAssignmentConfig`, next to the
strategy it applies to, and deprecates the two legacy places. Both stay
accepted and honored in this release, since the replacement ships with it and a
cluster cannot migrate before upgrading; rejecting them is left for a later
release.
- `SegmentAssignmentConfig` gets an optional `partitionColumn`, and its
`segmentAssignmentStrategy` becomes optional too, so an entry can configure
only the column. The strategy still defaults to the one derived from the
instance partitions, and the column defaults to the single column of
`segmentPartitionConfig`. Entries are keyed per instance partitions type or
tier as before, which covers tiers and colocated tables.
- `TableConfigUtils.getPartitionColumn` resolves the column from the segment
assignment configs first, then from the deprecated instance assignment field,
then from the deprecated `replicaGroupStrategyConfig`, and normalizes an empty
value to `null`. The offline upsert upload validation goes through it instead
of walking the configs itself.
- `replicaGroupStrategyConfig` and its accessors, and the `partitionColumn`
of `InstanceReplicaGroupPartitionConfig`, are marked `@Deprecated(since =
"1.7.0", forRemoval = true)`. The `TableConfigBuilder` setter and the SQL DDL
property for `replicaGroupStrategyConfig` are removed, so new code and new DDL
cannot introduce it, while JSON table configs carrying it keep working.
- Validation of `segmentAssignmentConfigMap`: the strategy must be one of
`balanced`, `replicagroup`, `roundrobin` or `roundrobinreplicagroup`
(`allservers` for a dimension table, as before); the partition column is only
accepted with a partition based strategy (`replicagroup`,
`roundrobinreplicagroup` or the derived one), must be a column of
`segmentPartitionConfig`, must be the same across entries, and must match the
deprecated columns when a table still configures those as well.
- Partition based OFFLINE segment assignment requires a resolvable partition
column, which is what every consumer of a partition already assumes: when the
OFFLINE or tier instance partitions of an OFFLINE table have multiple
partitions, either a column is configured or `segmentPartitionConfig` has
exactly one column. The table config validation checks this for the instance
assignment configs, and the controller checks it against the computed or
fetched OFFLINE instance partitions, which covers pre-configured instance
partitions whose layout is not visible in the table config. Realtime tables are
not checked, since the partition of a realtime segment is encoded in its name.
Tests that configured the column through the legacy configs now use
`SegmentAssignmentConfig` or, where the segment partition config has a single
column, drop it. The quickstart configs drop the column from the realtime
upsert examples, where it was never read, and the legacy block from the
colocated examples, which resolve their single column from
`segmentPartitionConfig`. Unit tests of the runtime handling of the legacy
configs are unchanged.
## Compatibility
Existing table configs keep validating and working. The replacement for
`partitionColumn` in either legacy place is `"segmentAssignmentConfigMap":
{"OFFLINE": {"partitionColumn": "..."}}` (or the tier name as key), which takes
precedence when both are configured and must name the same column. The
replacement for the rest of `replicaGroupStrategyConfig` is an OFFLINE entry in
`instanceAssignmentConfigMap` with a replica-group based
`replicaGroupPartitionConfig` using the same `numInstancesPerPartition` and
`numReplicaGroups` set to the table replication. Tables with partition based
OFFLINE assignment and more than one column in `segmentPartitionConfig` without
a configured partition column now fail validation; they were assigned by a hash
of the segment name rather than by partition. A `segmentAssignmentConfigMap`
entry with an unknown strategy, which the factory silently treated as
`balanced`, is now rejected.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
