andygrove opened a new issue, #3320: URL: https://github.com/apache/datafusion-comet/issues/3320
## Summary Several tests crash with `CometRuntimeException: CometNativeExec should not be executed directly without a serialized plan` when native_datafusion is used in auto mode. ## Failing Tests - `ParquetFilterSuite`: "SPARK-31026: Parquet predicate pushdown for fields having dots in the names" - `ParquetFilterSuite`: "Filters should be pushed down for Parquet readers at row group level" - `ParquetFilterSuite`: "filter pushdown - StringPredicate" - `FileBasedDataSourceSuite`: "SPARK-41017: filter pushdown with nondeterministic predicates" ## Error ``` CometRuntimeException: CometNativeExec should not be executed directly without a serialized plan: CometNativeScan parquet [...] PushedFilters: [IsNotNull(...), EqualTo(...)], ... ``` ## Root Cause These tests execute queries that go through a code path where `CometNativeScan` is included in a plan but the plan serialization to the native side didn't happen (likely because the plan was executed in a test-specific way that bypasses the normal `CometSparkSessionExtensions` hooks). The `CometNativeScan` node's `doExecuteColumnar` throws because it requires a serialized native plan. This may also indicate cases where `CometNativeScan` is placed in a plan that isn't fully wrapped by `CometExec`, possibly because filter pushdown inspection tests run plans in a non-standard way. ## Related Discovered in CI for #3307 (enable native_datafusion in auto scan mode). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
