xiangfu0 opened a new pull request, #19886:
URL: https://github.com/apache/pinot/pull/19886
## Summary
Migrate the remaining bundled example table configs in `pinot-tools` off
deprecated table-config properties. These 25 configs (SSB, TPC-H, orders,
colocated-join, minion, compression-stats and the docker airlineStats examples)
are the ones apache/pinot#18411 does not touch. #18411 migrates the configs
behind the main quickstarts and starts reporting deprecated keys as
`deprecationWarnings`; with both PRs merged, no bundled sample config uses a
deprecated key. The two PRs touch different config files (plus one identical
hunk in `BootstrapTableTool`) and can merge in either order.
Apart from one three-line null guard in `BootstrapTableTool` (below), this
changes only example configs.
| Deprecated | Replacement | Files |
|---|---|---|
| `segmentsConfig.segmentPushType` (OFFLINE) |
`ingestionConfig.batchIngestionConfig.segmentIngestionType`, same value | 22
(+1 duplicate removed) |
| `segmentsConfig.segmentPushType: APPEND` (REALTIME) | removed; `APPEND` is
the default | 2 |
| `tableIndexConfig.streamConfigs` |
`ingestionConfig.streamIngestionConfig.streamConfigMaps` | 2 |
| `segmentsConfig.replicasPerPartition` | removed; it equals
`segmentsConfig.replication` in both configs | 2 |
| `segmentsConfig.replicaGroupStrategyConfig` |
`segmentAssignmentConfigMap.OFFLINE.segmentAssignmentStrategy: replicaGroup` |
2 |
The minion `airlineStats` offline config already set
`batchIngestionConfig.segmentIngestionType: APPEND`, so its duplicate
`segmentPushType` is just removed.
**Colocated-join examples (`userGroups`, `userFactEvents`).**
`replicaGroupStrategyConfig` was supplying the partition column (`userUUID`)
for segment assignment. Dropping it keeps the same placement:
- Instance partitions come from `instancePartitionsMap`
(`userAttributes_OFFLINE`), so `numInstancesPerPartition` was never used.
- With no configured partition column, `SegmentUtils` takes the partition id
from the segment's partition metadata when exactly one column is partitioned;
`userUUID` is the only one in both tables.
- The replica-group strategy is now stated explicitly in
`segmentAssignmentConfigMap`. Before, it was implied by the shape of the
instance partitions.
**`BootstrapTableTool` fix.** Quickstart bootstrap rewrites
`batchIngestionConfig.batchConfigMaps` input paths and assumed the list is
present. After migration, many configs have a `batchIngestionConfig` that only
sets `segmentIngestionType`, and the colocated-join quickstart failed with an
NPE in `updatedTableConfig`. The guard is the same hunk #18411 adds, so the two
PRs still merge cleanly in either order (checked with `git merge-tree`).
## Testing
- Every changed file parses, and none contains a deprecated key (grep over
all 14 deprecated property names).
- A scratch harness (not committed) loaded each config from master and from
this branch and checked that:
- the deprecation validator from #18411 reports no warnings on the new
config;
- effective settings match: `getReplication()`,
`IngestionConfigUtils.getBatchSegmentIngestionType/Frequency`,
`getStreamConfigMaps` and the resolved partition column;
- after `TableConfigUtils.convertFromLegacyTableConfig` on both sides, the
configs are identical apart from the intended removals above;
- `TableConfigUtils.validate(tableConfig, schema)` passes, for the 24
configs that have a schema next to them.
Negative controls (an unmigrated config, and one with the wrong ingestion
type) fail as expected.
- Ran `pinot-admin.sh QuickStart -type COLOCATED_JOIN` from the packaged
build. It hit the NPE above before the fix, and came up cleanly after it.
- All 20 segments of `userAttributes`, `userGroups` and `userFactEvents`
sit on exactly the servers the shared `userAttributes_OFFLINE` instance
partitions assign to their `userUUID` partition.
- Partition-hinted and inferred-hint multi-stage joins return the same
counts as unhinted joins (2494 and 10000).
- All 7 sample queries ran without exceptions.
- `./mvnw -pl pinot-tools spotless:apply license:format checkstyle:check
license:check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
