rangareddy commented on issue #17283:
URL: https://github.com/apache/hudi/issues/17283#issuecomment-5505511983
Correction to my previous comment. One claim in it was wrong, and since it
points at another ticket I want it retracted before someone acts on it.
I wrote that "the separator is irrelevant" and that **#17282 (HUDI-8311) is
the same defect**. Both are wrong. I generalised from this ticket's failure
without reading the error text in #17282, which turns out to be a different
exception at a different place:
```
java.lang.IllegalArgumentException: Partition key parts [segment, ts] does
not match with partition values [cat1, 2024, 01, 01]. Check partition strategy.
at
org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.getPartitionClause(QueryBasedDDLExecutor.java:191)
```
The separator is not incidental there, it is the entire mechanism: `/` is
Hudi's partition-path separator, so a `yyyy/MM/DD` output format writes
`cat1/2024/01/01`, four path segments against two declared partition fields.
That trips an arity check inside Hudi and never reaches Hive. This ticket's
`yyyy-MM-DD` writes `cat1/2024-10-01`, two segments, so the arity check passes,
the DDL is emitted, and Hive is what rejects it.
| | this ticket (`yyyy-MM-DD`) | #17282 (`yyyy/MM/DD`) |
| --- | --- | --- |
| Partition path | `cat1/2024-10-01` | `cat1/2024/01/01` |
| Segments vs declared fields | 2 vs 2 | 4 vs 2 |
| Where it fails | at Hive, on the emitted `ALTER TABLE` | inside Hudi,
before any SQL |
| Cause | partition column typed numeric, value is a date string |
partition-path arity mismatch |
Different causes, different fix sites. They should not be merged, and a fix
for one will not resolve the other. I have left the mechanism for #17282 on
that ticket.
What stands from my previous comment is everything about **this** ticket:
`HiveSchemaUtil.getPartitionKeyType` typing the Hive partition column from the
table column, the resulting numeric-column-versus-date-string mismatch, and the
`hoodie.datasource.write.drop.partition.columns=true` workaround to try. None
of that depended on the comparison I got wrong.
I also owe the original triage note a correction: I dismissed its "a fix
handling one separator and not the other would be arbitrary" framing as wrong.
It was closer to right than my dismissal, in that both formats do need
handling. Where it needs adjusting is only "almost certainly the same fix",
since the two fail for unrelated reasons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
