rangareddy commented on issue #17283: URL: https://github.com/apache/hudi/issues/17283#issuecomment-5504704784
Correcting my earlier triage: I said this was "not decidable from source". It is. The mechanism is in the Hive sync DDL builder, and it is a **partition column type mismatch, not a separator problem**. ### Why the ALTER TABLE fails When Hive sync builds the partition clause, it types each partition column by looking the field name up in the table schema: https://github.com/apache/hudi/blob/a2788eef922e2375432ead3fe50af7fe5f55faf7/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveSchemaUtil.java#L449-L457 If the partition field is also a column in the table schema, the Hive partition column inherits **that column's type**. Only names absent from the schema fall through to `STRING`. The lookup does match here: the schema map is re-keyed with backticks by `convertMapSchemaToHiveSchema` -> `hiveCompatibleFieldName(..., doFormat = true)`, and the call site at `:396` looks up with `partitionKeyWithTicks`, so both sides are tick-surrounded. In the reproduction, `ts` is a real column holding epoch seconds (`1704121827`), so Hive gets `ts` typed as a numeric column. `TimestampBasedKeyGenerator` then writes the *formatted* value into the partition path, so the sync emits ``` ADD IF NOT EXISTS PARTITION (`segment`='cat1',`ts`='2024-10-01') ``` a non-numeric literal against a numeric partition column, which Hive rejects. That is the failing statement in the report. ### What this means for triage - The separator is irrelevant. Any `hoodie.keygen.timebased.output.dateformat` whose output is not parseable as the source column's type fails identically, which is why `YYYYMMDD` would appear to "work" (it parses as a number). So #17282 (HUDI-8311) is the same defect, and my earlier suggestion that a fix handling one separator but not the other would be "arbitrary" was the wrong framing. - The trigger is a partition field that is **both** a table column and re-formatted by the key generator. A timestamp partition field that is not retained in the written schema does not hit this. ### Workaround to try `hoodie.datasource.write.drop.partition.columns=true` keeps `ts` out of the written schema, which should make `getPartitionKeyType` fall through to `STRING` and let the partition register. Consistent with the code path above, but I have **not** run it against a live metastore. ### Fix direction Where a partition field is produced by `TimestampBasedKeyGenerator` with an output date format, the Hive partition column wants to be typed from the formatted output rather than from the source column. That changes emitted DDL, so it needs a compatibility call for tables already synced with a numeric partition column, where a re-sync would disagree with the registered type. Everything above is read from source; I have not reproduced it against Hive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
