LuciferYang opened a new pull request, #12541: URL: https://github.com/apache/gluten/pull/12541
### What changes were proposed in this pull request? Two small Spark 3.2-era workarounds in the ClickHouse backend that are no longer justified now that Gluten supports only Spark 3.3+: **`ExtendedGeneratorNestedColumnAliasing` (`backends-clickhouse/.../ExtendedColumnPruning.scala`)** Previously reimplemented `NestedColumnAliasing.getAttributeToExtractValues` locally because the Spark 3.2 version only accepted two arguments. Every currently supported branch (3.3, 3.4, 3.5, 4.0, 4.1) exposes the three-argument overload with the `extractor` parameter, so both call sites now delegate to upstream and the private reimplementation (`collectRootReferenceAndExtractValue`, `getAttributeToExtractValues`, `totalFieldNum`) is removed. The Gluten-specific `collectNestedGetStructFields` extractor is kept and the call site now has a comment explaining why it is needed. **`HadoopMapReduceAdapter.isBucketWrite` (`backends-clickhouse/.../CHColumnarWrite.scala`)** Previously used reflection to read `WriteJobDescription.bucketSpec`, guarding against Spark 3.2 where the field did not exist. `bucketSpec` is a public `val` on every currently supported branch, so the reflection helper is replaced with a direct `description.bucketSpec.isDefined` check. No behavior change on any supported Spark version. ### How was this patch tested? - `./build/mvn -Pbackends-clickhouse -Pspark-3.3 -Pdelta -pl backends-clickhouse -am test-compile -DskipTests`: SUCCESS - `./dev/format-scala-code.sh`: no diff - Cross-checked `NestedColumnAliasing.getAttributeToExtractValues` bodies on `upstream/branch-3.3`, `branch-3.4`, `branch-3.5`, `branch-4.0`, `branch-4.1`: three-argument overload present on every branch. The internal `canAlias` guard reshuffled between 3.4 and 3.5, but for the Gluten call site using `collectNestedGetStructFields` the guard is a semantic no-op (single-attribute chains, never traverses lambda variables). - Cross-checked `WriteJobDescription.bucketSpec` on the same five branches: `val bucketSpec: Option[WriterBucketSpec]` on all of them. - ClickHouse-specific runtime tests (`GlutenClickHouseHiveTableSuite`'s "Nested column pruning for Project(Filter(Generate))" cases, `GlutenClickHouseTPCHParquetBucketSuite`) rely on native artifacts and Linux, so they run in CI rather than locally. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude claude-opus-4-7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
