xiangfu0 opened a new pull request, #19244:
URL: https://github.com/apache/pinot/pull/19244
## Summary
Allow table-level ingestion transforms on `OFFLINE` tables to use registered
scalar functions from any volatility category. New or changed `REALTIME` table
ingestion transforms continue to require `IMMUTABLE` functions.
The rest of transform validation remains in place, including expression
parsing, destination-column validation, self-reference checks, and the Groovy
policy. Schema-level transforms and `postPartialUpsertTransformConfigs` also
retain their existing volatility checks.
## Motivation
An offline upsert table needs a comparison value to choose the latest row
for each primary key. Daily Parquet inputs can contain only a subset of the key
space and may not contain a physical timestamp or row-version column. A value
that is fixed for every ingestion run cannot order updates across those daily
drops.
This change lets an offline table declare a generated comparison column in
the Pinot schema and populate it with `now()` or another explicitly selected
non-immutable transform, without first rewriting the Parquet files. The policy
change is scoped to finite, task-driven `OFFLINE` ingestion and is not applied
to `REALTIME` ingestion.
## Usage
Declare the generated comparison column in the Pinot schema even though it
is absent from the source Parquet file:
```json
{
"schemaName": "dailyDimension",
"dimensionFieldSpecs": [
{
"name": "id",
"dataType": "STRING"
}
],
"metricFieldSpecs": [
{
"name": "ingestionTime",
"dataType": "LONG"
}
],
"primaryKeyColumns": ["id"]
}
```
Configure full upsert and populate the comparison column during offline
ingestion:
```json
{
"tableName": "dailyDimension",
"tableType": "OFFLINE",
"segmentsConfig": {
"segmentPushType": "APPEND",
"replication": "1"
},
"routing": {
"instanceSelectorType": "strictReplicaGroup"
},
"tableIndexConfig": {
"segmentPartitionConfig": {
"columnPartitionMap": {
"id": {
"functionName": "Murmur",
"numPartitions": 1
}
}
}
},
"upsertConfig": {
"mode": "FULL",
"comparisonColumns": ["ingestionTime"]
},
"ingestionConfig": {
"transformConfigs": [
{
"columnName": "ingestionTime",
"transformFunction": "now()"
}
]
}
}
```
The Parquet rows must contain `id`, but they do not need to contain
`ingestionTime`. Offline upsert still requires primary-key partitioning and
strict-replica-group routing.
## Compatibility
- `OFFLINE` table-level ingestion transforms accept `IMMUTABLE`, `STABLE`,
and `VOLATILE` functions.
- New or changed `REALTIME` table-level ingestion transforms continue to
reject non-immutable functions.
- An identical non-immutable transform already stored on a `REALTIME` table
remains grandfathered so unrelated config updates are not blocked.
- Schema-level and post-partial-upsert transforms continue to require
immutable functions.
- Unknown functions, invalid expressions, and disabled Groovy transforms
remain rejected.
## Validation
- `TableConfigUtilsTest`: 78 tests passed.
- `SchemaUtilsTest`: 15 tests passed.
- `PinotTableRestletResourceTest` and `TableConfigsRestletResourceTest`: 45
tests passed.
- Spotless, Checkstyle, license formatting/checking, and `git diff --check`
passed for all affected modules.
## Draft status and known correctness gap
This PR changes validation policy only; it does not make volatile functions
batch-stable. Row-based offline segment generation currently evaluates
transforms once while gathering statistics and again while indexing after
rewinding the input. A function such as `now()` or `rand()` can therefore
produce different values between the two passes. Dictionary-encoded columns can
resolve a second-pass value that was absent from the first-pass dictionary to
an incorrect dictionary ID, while no-dictionary columns can store values
inconsistent with first-pass min/max metadata. Either case can select the wrong
offline-upsert winner.
Values can also differ across records, retries, rebuilds, and backfills, and
registered functions that depend on unavailable runtime context can validate
but fail during batch evaluation.
This should remain a draft discussion PR until segment generation either
materializes and reuses transformed rows or provides a defined build-stable
evaluation mode, with an end-to-end segment-generation/readback regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
