liujiwen-up opened a new pull request, #10659:
URL: https://github.com/apache/paimon/pull/10659
### Purpose
Fixes #10657.
Quote primary-key names before passing them to `Expressions.identity` in
`PaimonScan.outputOrdering`. Spark parses the argument as a multipart SQL
identifier: a valid top-level name such as `#id` or ``a`b`` fails to parse,
while `a.b` is interpreted as a nested field and fails resolution. The existing
bucket-expression fix in #5439 does not cover this separate ordering path.
On the tested master baseline `021ca8c10cf35444e5a79dafdb0a729a812f0b81`,
this is reachable through real SQL when both
`spark.paimon.scan.preserve-data-grouping` and
`spark.sql.sources.v2.bucketing.enabled` are true and the scan can report
ordering. A single INSERT producing one file is sufficient; merge reads are not
required. Reuse the existing `BucketExpression.quote` helper, preserving
identity transforms, primary-key order, ascending direction and null ordering.
The default scan-layout policy is unchanged.
### Tests
- Added four SQL regression cases to `BucketedTableQueryTest` for ordinary,
hash-prefixed, dotted and backtick-containing primary-key names, using a
regular bucket key to isolate ordering from writer/bucket-key handling.
- Cover compound primary keys and all four combinations of the two scan
settings. Assert a real bucketed input partition with one split and one data
file, complete single-component field references and ascending direction.
Verify bucketed joins still eliminate shuffle and sort.
- Before the production fix: the suite ran 10 tests, with only the 3
special-name regressions failing (2 ParseExceptions and 1 AnalysisException).
After the fix: `BucketedTableQueryTest` and `BucketedScanPolicyTest` ran 14
tests with no failures on the project's Spark 3.5.8 / Scala 2.12.18 / Java 8
configuration.
- Ran checks without `fast-build` (checkstyle, Spotless and enforcer), plus
the root `verify`/RAT check.
- Independently rebuilt the Spark 3.5 shaded artifact and ran synthetic SQL
on standard Spark 3.5.2 / Scala 2.12.18 / Java 8 / local[2]: the tested master
baseline went from 48/60 successful queries to 60/60; historical community
commit `7615188e04f197266d404f96050b3e5f07473505` went from 26/46 to 46/46.
Checked single-file scans, merge reads, multi-split early returns,
projected-out bucket keys, actual plans, and runtime class origins. Class
digests match across common, Spark3 common and the final shaded jar after
forcing jar recreation.
- Dotted/backtick-containing names used as bucket keys expose separate
pre-existing writer failures in the standalone matrix. These remain unchanged
and are outside this patch; the regression cases use a regular bucket key.
Commands for the focused test and repository checks:
```sh
mvn -pl
paimon-spark/paimon-spark-common,paimon-spark/paimon-spark3-common,paimon-spark/paimon-spark-ut,paimon-spark/paimon-spark-3.5
\
-Pspark3
-DwildcardSuites=org.apache.paimon.spark.sql.BucketedTableQueryTest,org.apache.paimon.spark.sql.BucketedScanPolicyTest
\
-Dtest=none -DfailIfNoTests=false -DCodeCacheSize=512m test
mvn -N -Pspark3 -DskipTests verify
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
