xiangfu0 opened a new pull request, #18668:
URL: https://github.com/apache/pinot/pull/18668
## Summary
Deprecates field-level `FieldConfig.encodingType` as the canonical
forward-index encoding source and moves canonical reads/writes to
`fieldConfig.indexes.forward.encodingType`.
This keeps legacy configs backward-compatible:
- `fieldConfig.encodingType` still deserializes and `getEncodingType()`
still defaults missing values to `DICTIONARY` for existing callers.
- `noDictionaryColumns`, `noDictionaryConfig`, and top-level `encodingType`
remain fallback inputs when `indexes.forward.encodingType` is absent.
- When `indexes.forward.encodingType` is present, it wins over the
deprecated field-level value and no-dictionary legacy signals.
## User Manual
For new table configs, configure forward-index encoding under the `forward`
index block:
```json
{
"fieldConfigList": [
{
"name": "message",
"indexes": {
"forward": {
"encodingType": "RAW",
"compressionCodec": "ZSTANDARD"
}
}
}
]
}
```
Existing configs using top-level `encodingType` continue to work, but that
field is now deprecated:
```json
{
"fieldConfigList": [
{
"name": "message",
"encodingType": "RAW"
}
]
}
```
If both are present, `indexes.forward.encodingType` is authoritative:
```json
{
"fieldConfigList": [
{
"name": "message",
"encodingType": "RAW",
"indexes": {
"forward": {
"encodingType": "DICTIONARY"
}
}
}
]
}
```
In that example, Pinot resolves the forward index as `DICTIONARY` encoded.
## Sample Table Config
```json
{
"tableName": "logs_OFFLINE",
"tableType": "OFFLINE",
"segmentsConfig": {
"schemaName": "logs"
},
"tableIndexConfig": {},
"fieldConfigList": [
{
"name": "logLine",
"indexes": {
"forward": {
"encodingType": "RAW",
"compressionCodec": "ZSTANDARD"
},
"text": {}
}
},
{
"name": "statusCode",
"indexes": {
"forward": {
"encodingType": "DICTIONARY"
},
"inverted": {}
}
}
]
}
```
## Validation
- `./mvnw -pl pinot-segment-local
-Dtest=ForwardIndexTypeTest,DictionaryIndexTypeTest,TableConfigUtilsTest test`
- `./mvnw -pl pinot-segment-local -Dtest=DictionaryIndexTypeTest test`
- `./mvnw -pl pinot-segment-spi -Dtest=ForwardIndexConfigTest test`
- `./mvnw -pl pinot-spi -Dtest=FieldConfigTest,OpenStructIndexConfigTest
test`
- `./mvnw spotless:apply -pl pinot-spi,pinot-segment-spi,pinot-segment-local`
- `./mvnw checkstyle:check -pl
pinot-spi,pinot-segment-spi,pinot-segment-local`
- `./mvnw license:format -pl pinot-spi,pinot-segment-spi,pinot-segment-local`
- `./mvnw license:check -pl pinot-spi,pinot-segment-spi,pinot-segment-local`
`license:check` passes with pre-existing unknown-extension warnings for
Lucene binary test resources under
`pinot-segment-local/src/test/resources/data/lucene_80_index`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
