mbutrovich commented on issue #3602: URL: https://github.com/apache/datafusion-comet/issues/3602#issuecomment-3992204882
Investigated this with instrumentation in `adapt_batch_with_expressions`. After logging all projection expressions across the full `CometIcebergNativeSuite` (including schema evolution, partitioned tables, MOR, and complex types), every expression is either a column reference (`name@N`) or a cast wrapping a column reference (`Cast [child: name@N]` / `COMET_CAST_COLUMN(name@N AS ...)`). No literal expressions appeared. The two scenarios where we expected literals: 1. Partition columns — iceberg-rust's RecordBatchTransformer injects identity partition values into the batch as data columns before it reaches us. 2. Schema evolution (missing columns) — iceberg-rust's `RecordBatchTransformer` adds missing columns filled with nulls. In the "schema evolution - add column" test, old files written with (id, name) arrive with `file_schema=["id", "name", "age"]` already populated — the adapter just sees `age@2`. Since iceberg-rust handles both cases internally, our adapter never produces literal expressions, and there's nothing to cache. Closing as not applicable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
