ad1happy2go opened a new pull request, #20063:
URL: https://github.com/apache/hudi/pull/20063
### Describe the issue this Pull Request addresses
Follow-up to #19304. After it, an explicit upgrade of a table below version
8 that uses a single-field `ComplexKeyGenerator` fails: `CALL
upgrade_table(table => 't', to_version => 'TEN')` returns `false`, the table
stays at its old version, and no `hoodie.table.complex.keygenerator.encoding`
is recorded. The driver log shows:
```
ERROR UpgradeOrDowngradeProcedure: Failed: Could not upgrade/downgrade table
at file:/…/t to version TEN.
org.apache.hudi.exception.HoodieIOException: Failed to scan metadata
at BaseHoodieTimeline.getInstantsFromFileSystem(...)
at ActiveTimelineV2.<init>(...)
at HoodieTableMetaClient.getActiveTimeline(...)
at KeyGenUtils.deduceComplexKeyGenEncodingFromData(...)
at KeyGenUtils.resolveComplexKeyGenEncodingForWrite(...)
at UpgradeDowngrade.resolveComplexKeygenEncoding(...)
at UpgradeDowngrade.run(...)
Caused by: java.io.FileNotFoundException: File file:/…/t/.hoodie/timeline
does not exist
```
Root cause: `UpgradeOrDowngradeProcedure` (and hudi-cli's `SparkMain`
upgrade path) build their meta client with
`setLayoutVersion(config.getTimelineLayoutVersion())`, i.e. timeline layout 2,
while a version 6 table is still on layout 1.
`UpgradeDowngrade.resolveComplexKeygenEncoding`, added by #19304, reads the
table's data before any hop runs, through that meta client, so it looks for the
V2 timeline directory the table does not have yet. Upgrades performed by an
ordinary write are not affected, because the writer's meta client takes the
layout from `hoodie.properties`.
### Summary and Changelog
`CALL upgrade_table` (and hudi-cli `upgrade table`) work again on pre-v8
single-field complex key generator tables, and record the encoding the table's
data carries.
- `UpgradeDowngrade.resolveComplexKeygenEncoding`: deduce the encoding
through a meta client built from the same storage and base path, which takes
the timeline layout from `hoodie.properties`. The caller's meta client is not
modified, so the upgrade/downgrade hops behave as before.
-
`TestUpgradeDowngrade.testUpgradeWithPinnedTimelineLayoutRecordsComplexKeygenEncoding`:
upgrades the v6 complex keygen fixtures (0.14.0 `field:value` keys and 0.14.1
bare keys) through a meta client with a pinned layout, as the procedure builds
it, and checks the table reaches the current version with `FIELD_PREFIXED` /
`VALUE_ONLY` recorded. Both cases fail on master with the error above and pass
with this change.
### Impact
Fixes `upgrade_table` / hudi-cli upgrade for pre-v8 single-field complex key
generator tables. No behaviour change for any other table, or for upgrades
performed by a write. No API or config change.
### Risk Level
low
Only the meta client used to read data for the encoding deduction changes,
and only for single-field complex key generator tables without the encoding
property. Verification:
- `TestUpgradeDowngrade` (hudi-spark): all pass, including the two new cases.
- End-to-end on a Spark 3.5 bundle built from this branch, each case on a
pristine copy of a table written by a genuine older release, one fresh session
per case:
- `upgrade_table` then an upsert of the same keys — before this change
5/13 cases failed (the five upgrades of pre-v8 tables), after it 13/13 pass:
0.14.1 bare COW, MOR, non-partitioned (to TEN and to NINE) record `VALUE_ONLY`,
0.14.0 prefixed records `FIELD_PREFIXED`; unchanged: SimpleKeyGenerator,
two-field ComplexKeyGenerator (not tracked), a 1.0.2 v8 table, a v9 table
written by 1.1.1, an already-current table (no-op), and three `downgrade_table`
to EIGHT cases. Every follow-up upsert matched all 100 existing keys with no
duplicates.
- Write path unchanged: the same 14-case write matrix (upsert, insert,
bulk_insert, insert_overwrite[_table], delete, MOR + inline compaction,
non-partitioned, new table, global RLI point lookup on bare keys) gives
identical results with and without this change.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
