baibaichen opened a new pull request, #11546: URL: https://github.com/apache/incubator-gluten/pull/11546
## What changes are proposed in this pull request? This PR fixes a bug where deprecated dataset aggregators (e.g., `typed.sum`) were failing with "Couldn't find value#X in [...]" errors during attribute binding in Spark 4.0+. **Root Cause**: This issue is caused by a behavior change in Spark 4.0 and later versions. In Spark 3.5, `BindReferences.bindReference` throws `IllegalStateException` when it cannot find an attribute, while Spark 4.x throws `SparkException`. The original code in `ExpressionConverter` only caught `IllegalStateException`, causing the exception to propagate uncaught in Spark 4.x and preventing proper fallback behavior. The issue was discovered through Spark UT `DeprecatedDatasetAggregatorSuite`. **Changes**: 1. Modified `ExpressionConverter.scala` to validate attribute existence **before** attempting to bind, rather than relying on exception catching 2. This approach is more robust and works across all Spark versions 3. Improved error messages for binding failures 4. Added `GlutenDeprecatedDatasetAggregatorSuite` for Spark 4.0 and 4.1 to validate the fix Fixes #11403 ## How was this patch tested? - Added new test suite `GlutenDeprecatedDatasetAggregatorSuite` for both Spark 4.0 and 4.1 - The test suite includes cases for deprecated dataset aggregators that previously failed - Enabled the test suites in `VeloxTestSettings` for both Spark versions ## Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
