Vamsi-klu opened a new pull request, #18977: URL: https://github.com/apache/pinot/pull/18977
Fixes #9989 ## What changed This PR persists derived-column transform function metadata into segment column metadata and uses that metadata during default-column reload decisions. The implementation adds a `transformFunction` column metadata property and exposes it through `ColumnMetadata`. Segment generation now records the transform function associated with a generated column, and default-column creation records the transform function for auto-generated derived columns as well. During reload, `BaseDefaultColumnHandler` now compares the transform function stored in segment metadata with the transform function currently configured for the column. If the value changed, or if a legacy auto-generated derived column has no stored transform metadata but the table config now declares a transform, Pinot regenerates the column instead of leaving stale derived values in the segment. The default-value fallback path is covered as well: if a derived column's transform argument is missing from the segment and Pinot has to materialize the default value instead of evaluated values, the transform function is still persisted in metadata. That prevents the segment from repeatedly looking like it has no transform metadata on subsequent reloads. ## Why it matters Derived columns are produced from transform expressions. If a table config changes the expression, reloaded segments need to reflect the new expression for auto-generated derived columns. Without tracking the expression in segment metadata, Pinot can know that the column exists, but not whether it was created from the current transform function. That creates a correctness risk: a segment can continue serving values computed by an older transform even after the table config has been updated. Persisting and comparing the transform function gives reload a stable, segment-local way to decide whether an auto-generated derived column must be rebuilt. The legacy metadata handling matters because existing segments created before this metadata existed will naturally have no transform function recorded. Those segments should not be permanently stuck with stale values once they are reloaded under a table config that defines a transform for the auto-generated column. ## Impact Auto-generated derived columns are regenerated when their transform function changes. Legacy auto-generated derived columns that do not yet have stored transform metadata are brought forward on reload and receive the current transform metadata. Segment-generation metadata now includes transform-function provenance for generated columns, which improves observability and future reload decisions. The metadata API remains backward compatible: `ColumnMetadata#getTransformFunction()` defaults to `null`, and older segment metadata without the property still loads successfully. ## Testing - `./mvnw -Dmaven.repo.local=/tmp/m2-pinot-9989 -pl pinot-segment-spi,pinot-segment-local -am -Dtest=ColumnMetadataImplTest,DefaultColumnHandlerTest -Dsurefire.failIfNoSpecifiedTests=false test` - `./mvnw -Dmaven.repo.local=/tmp/m2-pinot-9989 -pl pinot-segment-spi,pinot-segment-local spotless:apply checkstyle:check license:check` - `git diff --check` The tests cover metadata round-trip, missing metadata backward compatibility, transform-function action-map decisions, regeneration when a legacy auto-generated derived column lacks stored transform metadata, and persistence of transform metadata when Pinot falls back to default values because a transform argument is absent. --- Drafted-by: Codex (GPT-5) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
