liujiwen-up opened a new issue, #10657:
URL: https://github.com/apache/paimon/issues/10657
### Problem and environment
`PaimonScan.outputOrdering` passes raw primary-key names to
`Expressions.identity`. Spark parses this argument as a multipart SQL
identifier, so valid top-level names containing `#` or a backtick cause a
`ParseException`, and a name such as `a.b` is interpreted as a nested field and
causes an `AnalysisException`.
Reproduced independently on current community master
`021ca8c10cf35444e5a79dafdb0a729a812f0b81` (2.2-SNAPSHOT), using standard Spark
3.5.2, Scala 2.12.18, Java 8 and local[2]. Also reproduced in the project's
Spark 3.5.8 test configuration and on historical community commit
`7615188e04f197266d404f96050b3e5f07473505`.
This differs from #5439: that fix quotes the bucket expression, but not the
primary-key identity expressions used for output ordering. I searched the
existing public issues/PRs and did not find an ordering-specific fix.
### Minimal reproduction
Configure a Paimon catalog named `paimon`, then run:
```sql
SET spark.sql.sources.v2.bucketing.enabled=true;
SET spark.paimon.scan.preserve-data-grouping=true;
CREATE TABLE paimon.default.ordering_repro (
seq INT,
`#event_id` STRING
) USING paimon
TBLPROPERTIES (
'primary-key'='seq,#event_id',
'bucket-key'='seq',
'bucket'='1'
);
INSERT INTO paimon.default.ordering_repro VALUES (1, 'b'), (1, 'a'), (2,
'a');
SELECT seq, `#event_id` FROM paimon.default.ordering_repro;
```
Expected: the three inserted rows. Actual: query planning fails in
`Expressions.identity -> PaimonScan.outputOrdering` with a syntax error at `#`.
The current default `scan.preserve-data-grouping=false` avoids this path.
Either scan setting disabled also avoids it. A single INSERT and a single data
file are sufficient. The regression test verifies a real bucketed input
partition with one split and one file; it does not fabricate internal scan
state. Multiple splits in a grouped input partition can make `outputOrdering`
return empty, so a successful query alone is not proof of branch coverage.
### Proposed fix and validation
Reuse the already imported `BucketExpression.quote` before
`Expressions.identity`:
```scala
primaryKeys
.map(quote)
.map(Expressions.identity)
```
This keeps the identity transform and sorting semantics, and handles
embedded backticks through the existing escaping helper. It follows the
project's existing quoted-identity and bucket-expression handling.
A focused implementation and regression tests are ready on my fork:
https://github.com/liujiwen-up/paimon/commit/4882a565c9
The new SQL tests cover ordinary, hash-prefixed, dotted and
backtick-containing names, compound keys, all combinations of the two scan
settings, exact field references, and bucketed join shuffle/sort elimination.
Before the fix, 3 special-name tests fail; after it, all 14 tests in
`BucketedTableQueryTest` and `BucketedScanPolicyTest` pass. Formatting,
checkstyle, enforcer and RAT checks pass.
Independent final-jar verification on Spark 3.5.2: current master improves
from 48/60 successful queries to 60/60; the historical baseline improves from
26/46 to 46/46. Runtime class origins and class digests across all shading
layers were checked. Separate existing writer problems for
dotted/backtick-containing bucket-key names are outside this fix; the ordering
regressions use a regular bucket key to isolate this path.
I would like to contribute this small fix. Could this issue be assigned to
me so I can open the pull request in accordance with the contribution guide?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
