xiangfu0 opened a new pull request, #18435: URL: https://github.com/apache/pinot/pull/18435
## Summary - Add `TransformFunction#getDataSource()` so transform functions can explicitly expose a backing segment `DataSource` when one exists. - Wire `ColumnContext.fromTransformFunction(...)` to preserve exposed data sources across transform operator chains. - Update identifier, literal, array literal, generated array, base, and map item transform implementations to implement the new method explicitly. - Switch `jsonExtractIndex` and `TEXT_MATCH` to consume the source from their input transform instead of re-looking up raw identifiers in `columnContextMap`. ## Why Some index-backed transform functions need access to the segment `DataSource` for the expression they consume. Before this change, that access was only available through `ColumnContext` for raw columns, which made transform-to-transform chains lose the data source even when a transform result still mapped directly to a segment source. ## User Manual For transform function authors: - Implement `getDataSource()` explicitly on every `TransformFunction` implementation. - Return the backing `DataSource` only when the transform output maps directly to a segment data source, such as an identifier or a map item key source. - Return `null` for computed expressions and literals. - Prefer `argument.getDataSource()` when a function needs indexes from its input expression. ## Sample Queries These existing index-backed forms continue to work, with the data source now flowing through the transform-function API: ```sql SELECT TEXT_MATCH(skills, 'sewing') AS match FROM testTable; ``` ```sql SELECT jsonExtractIndex(jsonSV, '$.intVal', 'INT') FROM testTable; ``` ## Validation - `./mvnw -pl pinot-core -am -DskipTests compile` - `./mvnw -pl pinot-core -am -Dtest=IdentifierTransformFunctionTest,DateTimeConversionTransformFunctionTest,JsonExtractIndexTransformFunctionTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test` - `./mvnw -pl pinot-core -am -Dtest=TextMatchTransformFunctionTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
