beliefer opened a new pull request, #12438: URL: https://github.com/apache/gluten/pull/12438
## What changes are proposed in this pull request? Fixies https://github.com/apache/gluten/issues/12436. Gluten decides the ORC column-mapping mode from a single global config: `spark.hadoop.orc.force.positional.evolution` flips `orcUseColumnNames`, which selects name-based vs position-based mapping for the whole query. This breaks any query that reads a **mix** of ORC tables where some files carry real physical column names and some carry Hive placeholder names (`_col0`, `_col1`, …): - With `orcUseColumnNames=true` (default): the `_col*` files have no real names to match, so every column reads back **null** → a filtered join side becomes empty and AQE folds the plan to `LocalTableScan rows=0` (silently wrong result). - With `orcUseColumnNames=false` (positional): the real-name files are mapped by ordinal and land on the wrong column, failing with `SCHEMA_MISMATCH` (`From Kind: INTEGER, To Kind: VARCHAR`) or, when a string filter is pushed down, `Filter(BytesValues, ...): testInt64Range() is not supported`. No single global value can read both tables correctly in one query. Vanilla Spark makes this decision **per file** in `OrcUtils.requestedColumnIds`: ```scala if (forcePositionalEvolution || orcFieldNames.forall(_.startsWith("_col"))) { // map physical schema -> data schema by INDEX } else { // map by NAME } ``` Gluten already honors the `forcePositionalEvolution` half (#12234). The missing half is `orcFieldNames.forall(_.startsWith("_col"))`: even in name mode, an ORC file whose physical schema is entirely `_col*` must be mapped by position, because those placeholder names carry no identity — the real names live only in the table schema. That per-file decision can only be made in the native reader, which is the only layer that sees each file's physical field names. This PR contains the Gluten-side change; the native-reader change is submitted separately to Velox (`facebookincubator/velox`), which adds the `_col*` detection to `DwrfReader` and falls back to positional mapping in name mode. **Gluten change** — `VeloxIteratorApi.setFileSchemaForLocalFiles`: Always attach the table (data) schema to the split for ORC/DWRF, instead of only when `orcUseColumnNames=false`. The native reader needs the table schema in name mode too, so it can remap the `_col*` file columns to the real names by position. Parquet behavior is unchanged (still only attached when mapping by position). This is additive: the existing positional path and the explicit `orc.force.positional.evolution=true` behavior are unaffected. ## How was this patch tested? Added a regression test to `GlutenHiveSQLQuerySuite` for all supported Spark shims (spark33/34/35/40/41). The test: - Creates a Hive ORC table with columns literally named `_col0`/`_col1` so the physical ORC field names are guaranteed to be placeholders (independent of the embedded Hive version; mirrors Spark's SPARK-34897 setup), plus a second metastore table with real names over the same files, and a separate real-name ORC table. - **Without** setting `spark.hadoop.orc.force.positional.evolution` (default `orcUseColumnNames=true`), asserts that the `_col*` table reads correct values via the real column names, that the real-name table still reads correctly by name in the same session, and that a join of the two returns a non-empty result (the original failure folded it to an empty `LocalTableScan`). - Confirms the scans stay native via `checkOperatorMatch[HiveTableScanExecTransformer]`. Note: the test passes end-to-end only together with the corresponding Velox change; the Gluten-side change alone attaches the schema but relies on the native reader to perform the `_col*` positional fallback. ## Was this patch authored or co-authored using generative AI tooling? co-authored with Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
