rangareddy commented on issue #17282: URL: https://github.com/apache/hudi/issues/17282#issuecomment-5505512201
Re-checked against `master` at `a2788eef922e`. This is decidable from source, so amending my earlier note that it was not. Correcting one thing in that note first: I suggested this was "almost certainly the same fix" as #17283 (HUDI-8312). It is not, and I have posted a retraction there. The two fail for unrelated reasons at different places, and merging them would hide one behind the other. ### Why the sync fails `/` is Hudi's partition-path separator, so a `yyyy/MM/DD` output format does not produce one partition value containing slashes, it produces **three extra path segments**. The written path for the reproduction is `cat1/2024/01/01`: four segments, against the two fields declared by `--partitioned-by segment,ts`. The DDL builder resolves the path through the configured extractor and then asserts the counts agree: https://github.com/apache/hudi/blob/a2788eef922e2375432ead3fe50af7fe5f55faf7/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java#L221-L237 `MultiPartKeysValueExtractor` splits flat on `/`, so it returns `[cat1, 2024, 01, 01]`, four values for two keys, and `checkArgument` throws with exactly the message in the report. Nothing is sent to Hive: this fails inside Hudi while the statement is still being built. (The stack trace in the description shows `:191`; the check is at `:223` on current master.) Contrast #17283, where `yyyy-MM-DD` yields `cat1/2024-10-01`, two segments for two keys. The arity check passes there, the `ALTER TABLE` is emitted, and Hive is what rejects it, for an unrelated reason (the partition column is typed from the table column, which is numeric). ### The actual gap The extractors available today each handle one shape, and this configuration falls between them: - `MultiPartKeysValueExtractor` - N flat segments for N fields. Cannot absorb a multi-segment value. - `SlashEncodedDayPartitionValueExtractor` - collapses `yyyy/mm/dd` into a single value, but requires the path to be **exactly** three segments and throws otherwise, so it cannot cope with the leading `cat1`. - `SlashEncodedHourPartitionValueExtractor`, `SinglePartPartitionValueExtractor`, `HiveStylePartitionValueExtractor`, `NonPartitionedExtractor` - none applies. So the missing case is the **mixed** layout: a `CustomKeyGenerator` with `segment:simple,ts:timestamp` and a slash-bearing date format produces one single-segment field followed by one three-segment field, and no extractor knows that a given field spans more than one segment. ### Fix direction The information needed is already in the write config: `hoodie.datasource.write.partitionpath.field` says which fields are `timestamp` type, and the output format says how many segments each contributes. An extractor driven by that per-field segment count would handle this and any other multi-segment format without a new class per shape. The cheaper interim option is to reject a slash-bearing `hoodie.keygen.timebased.output.dateformat` at write time when the sync cannot represent it, so the failure lands at configuration time rather than at sync time. Read from source; I have not reproduced this against a live metastore. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
