raghavyadav01 opened a new pull request, #20217:
URL: https://github.com/apache/pinot/pull/20217
## Summary
Derived columns declared in `ingestionConfig.transformConfigs` can be
materialised when a segment loads, by `BaseDefaultColumnHandler` (#19581,
#19955). On a later reload, `computeDefaultColumnActionMap` compares an
existing auto-generated column with the schema only by field type, data type,
single/multi-value and default value. It never compares the transform function.
So after changing a derived column's definition, for example `times(amount,
100)` → `times(amount, 1000)`, a reload is a no-op and the segment keeps
serving the old values with no error. The only way out today is a
force-download reload.
## Changes
- When a derived column is created or rebuilt at load, the handler records a
fingerprint of its transform function in the segment custom map, as
`custom.derivedColumnTransform.<column>`. Removing the column clears the entry.
- It uses the custom map because it is carried by both
`SegmentMetadataImpl` and the mapped metadata path, so neither the
`ColumnMetadata` SPI nor the mapped binary format changes.
- The value is a fingerprint, `String.hashCode()` in hex, which is
specified by the JLS and stable, rather than the raw expression. Segment
properties split comma-separated values into lists, so a raw expression like
`times(amount, 100)` would be cut at the comma.
- On reload, a new `UPDATE_TRANSFORM_FUNCTION` action rebuilds an
auto-generated column whose recorded fingerprint no longer matches the table
config, including one whose transform was removed. It goes through the existing
update path: remove the indexes, then create them again.
- Every auto-generated derived column that reads a column being rebuilt is
rebuilt too, transitively. The existing dependency ordering creates them after
their sources, from the sources' new raw values.
- A column with no recorded fingerprint, i.e. materialised before this
change, is left as it is. Upgrading does not rebuild every derived column of
every existing segment; such a column picks up a changed definition the next
time it is rebuilt.
## Testing
- `testChangedTransformFunctionRebuildsDerivedColumnAndItsDependents` (v1
and v3): a chain `a = plus(column1, 1)`, `b = plus(a, 1)`. Changing `a` to
`plus(column1, 10)` makes `needProcess()` true and rebuilds both `a` and `b`,
and the recorded fingerprint is updated. A further reload has nothing to do.
With the check disabled the test fails with "a changed transform function must
trigger a reload".
- `testDerivedColumnWithoutRecordedTransformIsNotRebuilt` (v1 and v3): a
segment without the recorded fingerprint keeps its values.
- `SegmentPreProcessorTest` 91/91 and `DefaultColumnHandlerTest` 1/1 pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]