snmvaughan opened a new pull request, #7262:
URL: https://github.com/apache/datafusion-comet/pull/7262
## Which issue does this PR close?
Backport of #7159 (squash commit 01fc37a80) to `branch-1.2`, for #7158.
## Rationale for this change
`branch-1.2` has the bug #7159 fixes on `main`. With
`spark.comet.parquet.rowFilterPushdown.enabled=true` on Spark 3.x, a
pushed-down filter that contains a cast Comet's planner inserted, such as the
Int64 positions of `substring(s, 1, 2)`, fails with `ParquetSchemaConvert {
physical_type: "INT32", spark_type: "bigint" }`. The native scan's expression
adapter treats every cast in the predicate as a Parquet schema conversion. See
#7158 for the details.
## What changes are included in this PR?
A clean cherry-pick of #7159 with `-x`. The changed lines are identical to
the source commit. `replace_with_spark_cast` now rewrites only the casts
DataFusion's schema adapter builds, which set the explicit-target flag that
`CastExpr::has_explicit_metadata()` returns, and leaves the planner's casts
alone.
Backports to other branches: `branch-1.1`, in a separate PR.
## How are these changes tested?
The tests come with the cherry-pick, in
`native/core/src/parquet/schema_adapter.rs`. On this branch all 98
`schema_adapter` tests pass, and `cargo fmt --check` and `cargo clippy
--all-targets --workspace -- -D warnings` are clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
