uros-b opened a new pull request, #57929:
URL: https://github.com/apache/spark/pull/57929
### What changes were proposed in this pull request?
Uses `StructType.getFieldIndex` for the column lookups in the Parquet and
ORC aggregate push down paths:
```scala
index = dataSchema.getFieldIndex(colName).getOrElse(-1) // was
fieldNames.toList.indexOf(colName)
if (partitionSchema.getFieldIndex(colName).isDefined) { // was
fields.map(_.name).toSet.contains(colName)
```
Three sites in `ParquetUtils` and one in `OrcUtils`.
### Why are the changes needed?
`fieldNames` is a `def` that allocates a fresh `Array` on every call, and
the Parquet sites then convert it to a `List`, so each lookup allocated twice
and scanned linearly; the `partitionSchema` check built a whole `Set` per call.
`getFieldIndex` reads the cached `nameToIndex` map.
To be accurate about the size of this: **it is a readability and allocation
cleanup, not a measurable speedup.** Aggregate push down is off by default, and
each surrounding loop iteration reads Parquet or ORC footer statistics, which
dominates any name lookup. The `OrcUtils` site is included because the two push
down implementations are deliberate mirrors that share `AggregatePushDownUtils`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing aggregate push down tests (`FileSourceAggregatePushDownSuite`)
cover both paths.
`getOrElse(-1)` matches `indexOf`'s not-found contract, and the exact-match
`getFieldIndex` is used rather than `getFieldIndexCaseInsensitive`, preserving
case sensitivity. The one case where the two forms would differ is a schema
with duplicate column names, where `indexOf` returns the first match and a
name-keyed map the last; `FileTable.schema` runs
`checkSchemaColumnNameDuplication` on both the data and partition schemas
before any scan is built, and both `ParquetTable` and `OrcTable` extend
`FileTable`, so that case is unreachable.
Note: open PR #57030 touches the same two `ParquetUtils` lines. There is no
logical conflict, and I am happy to rebase if that one lands first.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
