voonhous opened a new pull request, #19891:
URL: https://github.com/apache/hudi/pull/19891
### Change Logs
`rebuildFilterFromParquet` -- the recursive walk that re-spells a
pushed-down Spark filter onto the
column names an evolved parquet file actually carries -- existed five times:
once as a private helper
inside `ParquetSchemaEvolutionUtils` (hudi-spark-common) and once, byte for
byte, in the companion
object of each of `Spark3LegacyHoodieParquetFileFormat`,
`Spark40LegacyHoodieParquetFileFormat`,
`Spark41LegacyHoodieParquetFileFormat` and
`Spark42LegacyHoodieParquetFileFormat`. A normalized diff
of all four copies against the shared helper is empty once the recursive
call's name is aligned.
This moves the 3-arg helper out of the `ParquetSchemaEvolutionUtils` class
body into its companion
object as a public function and deletes the four copies. The four legacy
formats now call
`ParquetSchemaEvolutionUtils.rebuildFilterFromParquet(...)`; the six
`SparkNNParquetReader`s keep
calling the 1-arg instance method, whose signature does not change -- it now
delegates to the
companion. No behaviour change: the body is verbatim, and all four version
modules already declared
`hudi-spark-common`, so no pom edits.
This is not dead code being tidied. The legacy formats are what
`HoodieBaseRelation` and
`HoodieDataSourceHelper` build readers from, so the path is live for
metadata-table reads and for
any table with `hoodie.file.group.reader.enabled=false`.
`Spark33/34/35LegacyHoodieParquetFileFormat` never carried a copy and are
untouched. The rest of the
duplication in these files (`typeChangeInfos`, `getHadoopConfClone`, the
predicate-pushdown preamble)
is deliberately left alone -- it is a separate change with its own drift to
reconcile.
<details>
<summary>Detail: what moved, and what the test pins</summary>
**hudi-spark-common `ParquetSchemaEvolutionUtils`**
- `private def rebuildFilterFromParquetHelper(oldFilter, fileSchema,
querySchema)` becomes
`object ParquetSchemaEvolutionUtils.rebuildFilterFromParquet(oldFilter,
fileSchema, querySchema)`.
The helper read no instance state, so this is a move, not a refactor.
- `def rebuildFilterFromParquet(filter: Filter)` on the class keeps its
signature and body shape and
now delegates to the companion with the same two schemas it passed before.
**The four legacy formats**
- The private copy is deleted from the companion object; the one call site
becomes
`filters.map(ParquetSchemaEvolutionUtils.rebuildFilterFromParquet(_,
fileSchema, querySchemaOption.orElse(null)))`.
Same package, so no import is needed for the call.
- Imports that the deletion leaves unused are dropped: `InternalSchemaUtils`
(the only user was the
copy) and `InternalSchema` (only the copy named the type; the remaining
`InternalSchema*` hits are
`SparkInternalSchemaConverter`, `InternalSchemaMerger` and
`InternalSchemaCache`). The wildcard
`org.apache.spark.sql.sources._` narrows to `sources.Filter`, the one
member still referenced.
**Test**
`TestParquetSchemaEvolutionUtils` gains one case for the now-public
companion function, built on the
same `InternalSchemaConverter.convert` pair the file's existing helpers use.
Field ids are positional,
so a query schema of `(id, renamed, added)` over a file schema of `(id,
original)` gives all four
arms in one fixture:
- `renamed` (id 1) is re-spelled to the file's `original`, `id` (id 0) is
left alone;
- `added` (id 2) has no field in the file, so a filter on it collapses to
`AlwaysTrue` rather than
skipping row groups it cannot evaluate;
- `And`/`Or`/`Not` rebuild their children;
- a null file schema or a null query schema returns the very same filter
instance.
**Verification**
- `TestParquetSchemaEvolutionUtils` run green in hudi-spark-common.
- Compile: hudi-spark3-common (`-Dspark3.5`), hudi-spark4.0.x,
hudi-spark4.1.x and hudi-spark4.2.x.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
