SEPURI-SAI-KRISHNA commented on issue #19666:
URL: https://github.com/apache/hudi/issues/19666#issuecomment-5371535622
I'm working on this one, but it is blocked behind #19648, so I wanted to
leave a note rather than
have it look unclaimed.
The dependency is real rather than procedural. #19648 settles what the write
path does for a
multi-field table: `PartitionPathFormatterBase#combine` stops
slash-separating every field so that
it matches `KeyGenUtils#getRecordPartitionPath`, which has always guarded
the substitution on a
single partition field. Until that lands, "what a multi-field slash table
looks like on disk"
depends on which write path produced it, and a read-side fix would be aiming
at a moving target.
There is also a fork in what fixing this should mean, and I would rather
agree on it before writing
the patch:
1. **Make the layout readable.** Teach
`HoodieSparkUtils#doParsePartitionColumnValues` to line the
extra path fragments up with the partition columns. This keeps existing
multi-field slash tables
queryable, but the layout is genuinely ambiguous — with `date_col` and
`city` and a path
`2026/01/05/san-francisco`, nothing in the path says where the date stops
and the city starts,
and it gets worse once a value legitimately contains a dash.
2. **Reject the combination at write time.** Fail a multi-field table with
`slash.separated.date.partitioning` enabled, the way the config's own
documentation already
treats hive-style partitioning as incompatible. This is honest about the
ambiguity but breaks
any pipeline currently writing such a table.
Option 2 also overlaps #19669, since both would land in
`HoodieWriterUtils.validateTableConfig` and
would want the same answer on how hard to fail.
My inclination is 2 for new tables, because option 1 cannot be made
unambiguous in general. Happy
to go the other way if a committer prefers keeping existing tables readable
— please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
