sunchao commented on PR #5365:
URL:
https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5418790494
Consolidating the three outstanding issues from the review of
`ef46cb8dd45891b7c2ab892ee9c35adaf05d4ce5` in one top-level comment:
1. **[P2] Column-mapping name collisions can silently return wrong data.**
Write a Delta table `(a BIGINT, p BIGINT)` partitioned by `p`, enable name
mapping, rename data column `a` to `b`, then rename partition column `p` to
`a`. The scan mixes physical data names with logical partition names, so
DataFusion replaces the data column with the partition constant. With an
asserted native Delta scan, `SELECT b, a` returned `(100,100), (100,100)`
instead of Spark's `(1,100), (2,100)`. Use consistent physical partition names,
or fall back for ambiguous scans.
2. **[P2] Scalar-subquery partition filters can fail in a fused native
operator.** After the scalar subquery resolves, `outputPartitioning` still
reports zero partitions. A native parent reads that count and fails with `All
per-partition arrays must have length 0`. Reproduced with AQE disabled using
`SELECT id + 1, p FROM fact WHERE p = (SELECT max(p) FROM dim WHERE country =
'yes')`, with both a native Delta scan and native projection present. The
partition count must reflect the resolved execution data.
3. **[P2] Hadoop credential references are forwarded unresolved.** With
`SimpleAWSCredentialsProvider` and access/secret keys referencing
`${review.access}` / `${review.secret}`, Hadoop resolves the configured values,
but Delta admission accepts the scan and native option extraction retains the
literal placeholders. Native credential lookup does not expand them. Resolve
the values consistently with Hadoop, or fall back. This was verified against
the actual Hadoop provider and current admission/extraction helpers plus the
production call path; this credential probe did not make a live S3 request.
The two query failures above were reproduced on `chao-reviews-1` with Spark
4.0.4 / Delta 4.0.1. These are bounded P2 correctness/compatibility issues, not
broad P0/P1 failures; the silent wrong-result case is the highest priority. The
existing 43 native tests and 100 contrib tests passed, but do not cover these
cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
