deepakpanda93 opened a new pull request, #19555:
URL: https://github.com/apache/hudi/pull/19555
### Describe the issue this Pull Request addresses
Closes #17357. (JIRA: HUDI-8827, a subtask of HUDI-9109 "Bridging Hudi Spark
SQL behavior gaps".)
Spark moves partition columns to the end of the table schema. If a partition
column is declared anywhere else in
`CREATE TABLE`, the stored column order silently differs from what was
written, and a positional
`INSERT INTO ... SELECT` assigns values to the wrong columns. The reporter
hit this as a baffling cast error on a `ts`
column they had explicitly cast to `bigint`. Nothing on the SQL DDL page
mentions the constraint.
The example in the **Create partitioned table** section had exactly that
shape, so following it produced the bug.
### Summary and Changelog
Two changes to the *Create partitioned table* section:
1. **Fixed the example.** It declared `(id, name, dt, hh)` with `PARTITIONED
BY (dt)`, leaving `hh` after the partition
column. Changed the clause to `PARTITIONED BY (dt, hh)`. The schema is
untouched — with both columns partitioned they
are already the trailing columns, in declaration order, so nothing is
reordered.
2. **Added a `:::caution`** stating that partition columns must be declared
last, with the resulting schema, both
failure modes, and the explicit-column-list alternative.
The existing multi-field `:::note` is unchanged; testing confirmed it was
already correct.
Applied to `website/docs/sql_ddl.md` (next) and
`website/versioned_docs/version-1.2.0/sql_ddl.md` (current released
docs), per the next-plus-current convention used in #19473 and #19551. The
section is byte-identical in 1.0.0–1.1.1 and
carries the same example; those were left alone — happy to widen if
preferred.
### Reproduction
Spark 3.5.7, `hudi-spark3.5-bundle_2.12:1.2.0`.
**The issue as reported.** Declaring `(id, name, price, dt, ts)` with
`PARTITIONED BY (dt)` stores the table as
`(id, name, price, ts, dt)`, and a positional insert fails:
```
insert into t select 1, 'a1', 10.0, '2021-03-21', 1L;
[INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_SAFELY_CAST]
Cannot safely cast `ts` "STRING" to "BIGINT"
```
**The worse case, which the issue does not mention.** When the shifted
columns happen to be type-compatible there is no
error at all. Declaring `(id, a, dt, b)` as strings and inserting
`1,'VALUE_A','VALUE_DT','VALUE_B'` returns exit 0 and
stores `b='VALUE_DT'` with `dt='VALUE_B'` — the partition value written into
a data column and a data value used as the
partition. That is why the new text is a `caution` and calls out the silent
case explicitly.
**All four variants of the example's own schema** `(id BIGINT, name STRING,
dt STRING, hh STRING)`, inserting
`1,'n1','2024-01-01','10'`:
| `PARTITIONED BY` | Stored column order | Result |
|---|---|---|
| `(dt)` — what the page had | `id, name, **hh, dt**` | :x: silently stores
`hh='2024-01-01'`, `dt='10'`, exit 0 |
| `(dt, hh)` — what the page now has | `id, name, dt, hh` (unchanged) |
:white_check_mark: `dt='2024-01-01'`, `hh='10'` |
| `(hh)` | `id, name, dt, hh` (unchanged) | :white_check_mark: correct —
`hh` is already last |
| `(hh, dt)` | — | :x: rejected at analysis: `Partition schema fields order
does not match the table schema fields order, tableSchemaFields: (dt, hh),
partitionFields: (hh, dt)` (`HoodieSchemaUtils.checkPartitionSchemaOrder`) |
Two things follow. Only the first row was broken, which is why this PR
changes one line rather than restructuring the
section. And the `(hh)` row shows the constraint is "partition columns must
be trailing" in general, not something
specific to `dt`.
Also verified that naming the columns explicitly avoids the mismatch even on
a mis-declared table:
`INSERT INTO t (id, name, price, dt, ts) SELECT ...` lands every value
correctly. That is why the caution offers it as an
alternative.
Worth noting the last row for contrast: getting the `PARTITIONED BY` *order*
wrong fails loudly, whereas getting the
*declaration* order wrong — the subject of this PR — is checked by nothing.
### A note on scope
The referenced PR #12577 was closed unmerged; it only added a test
reproducing the problem, with all three insert
variants commented out as "None of these queries work". So there is no code
fix to describe here — this documents
current behaviour, which is what the issue asks for.
### Site verification
`npm run build` passes with the warning set byte-identical to a baseline
build of the same base commit — no new broken
links or anchors. `/docs/sql_ddl` and `/docs/next/sql_ddl` were loaded from
`npm run serve` and the code block renders as
`PARTITIONED BY (dt, hh);` with the schema unchanged, the caution renders as
an admonition, and the section anchors are
intact. `/docs/1.1.1/sql_ddl` still shows the old text, as intended.
### Impact
Documentation only. No code, config, or behaviour change.
### Risk Level
none
### Documentation Update
This PR is the documentation update — the SQL DDL page, `/docs/sql_ddl` and
`/docs/next/sql_ddl`.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
