raghavyadav01 opened a new pull request, #19955: URL: https://github.com/apache/pinot/pull/19955
## Problem A column derived at segment load from `ingestionConfig.transformConfigs` reads its source column through `PinotSegmentColumnReader` in `BaseDefaultColumnHandler.ValueReader`. The reader was created without the source's null value vector, so a null source value reaches the transform function as the default value stored in its place. With null handling enabled, `times(source, 2)` over a null `source` therefore produces a non-null `0.0`, while ingestion produces null. Measured on 5M rows with every 7th source value null: 714,290 rows have `source IS NULL` and 0 have `derived IS NULL`. `IS NULL` filters, `COUNT(derived)` and `AVG(derived)` all differ from the same table ingested with the transform. ## Change `ValueReader` opens the source's null value vector when the segment has one and returns `null` for those docs, so the function sees `null` exactly as it does at ingestion. A function that returns `null` for a null input then marks the derived doc null through the existing null-vector path. When the source has no null vector (null handling disabled), nothing changes. ## Note for reviewers With null handling enabled, a default column added in the same reload is null on every doc, so a column derived from it is now null too, matching ingestion, where the transform runs before defaults are filled. If #19581 lands first, its tests that derive from such a column will need null handling off to keep asserting the derived value; I will adjust whichever of the two merges second. ## Tests `SegmentPreProcessorTest.testDerivedColumnFromNullableSourceKeepsNulls` (v1 and v3): builds a segment whose nullable source has nulls, adds `derived = times(source, 2)` on reload, and checks every doc is null exactly where the source is and `2 * source` elsewhere. It fails on master (`Derived value must be null where the source is null, doc: 1`). `SegmentPreProcessorTest`, the default-column handler tests, `ForwardIndexHandlerTest` and `ImmutableSegmentLoaderTest` pass (129 tests). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
