SEPURI-SAI-KRISHNA opened a new issue, #19647: URL: https://github.com/apache/hudi/issues/19647
### Bug Description **What happened:** With `hoodie.datasource.write.slash.separated.date.partitioning=true`, any write that goes through Spark's row writer fails with: ``` java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class java.lang.String ``` and a `null` value in the partition column fails with a `NullPointerException` on both the row writer and the `Row` write paths. This affects bulk insert, `INSERT INTO` with `hoodie.sql.bulk.insert.enable=true`, and anything else reaching `SparkKeyGeneratorInterface#getPartitionPath(InternalRow, StructType)` — that is, the default write path for bulk insert. Writes that go through the Avro key generator (`getKey(GenericRecord)`) are unaffected, which is why the feature appears to work in the common `INSERT INTO` case. Root cause is in [`PartitionPathFormatterBase#combine`](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/PartitionPathFormatterBase.java#L60-L70). The formatter is generic over the string representation `S` it builds — `String` for the Avro and `Row` write paths, `UTF8String` for the row writer — but the slash-separated-date branch hard-casts to `java.lang.String`: ```java if (!useHiveStylePartitioning && partitionPathParts.length == 1) { if (slashSeparatedDatePartitioning) { return ((S) ((String) toString(partitionPathParts[0])).replace('-', '/')); // <-- CCE } else { return tryEncode(handleEmpty(toString(partitionPathParts[0]))); } } ``` Three separate defects in that branch: 1. The `(String)` cast blows up whenever `S` is `UTF8String` (the row writer). 2. It skips `handleEmpty`, so a null partition value NPEs on `null.replace(...)` instead of landing in `__HIVE_DEFAULT_PARTITION__`. 3. It skips `tryEncode`, so URL encoding is silently dropped for this branch. The multi-field loop carries the same hard-cast, and additionally applies the `-` -> `/` substitution to *every* partition field. The Avro write path (`KeyGenUtils#getRecordPartitionPath`) only substitutes when the table is partitioned by a single column, so the two write paths derive different partition paths for the same record. Existing coverage for this feature only exercises the Avro key-generator path, which is why none of this was caught. **What you expected:** The row writer should produce the same partition path as the Avro write path — `2026/01/05` for a partition value of `2026-01-05`, and `__HIVE_DEFAULT_PARTITION__` for a null value — rather than throwing. **Steps to reproduce:** 1. Start `spark-sql` (or `spark-shell`) with the Hudi bundle and `--conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension`. 2. Create a table with slash-separated date partitioning and write to it through the row writer: ```sql set hoodie.sql.bulk.insert.enable=true; set hoodie.sql.insert.mode=non-strict; create table slash_part_repro ( id string, name string, ts bigint, datestr string ) using hudi tblproperties ( 'primaryKey' = 'id', 'type' = 'COW', 'preCombineField' = 'ts', 'hoodie.datasource.write.slash.separated.date.partitioning' = 'true' ) partitioned by (datestr); -- fails with ClassCastException insert into slash_part_repro values (1, 'a1', 1000, '2026-01-05'); -- with the row writer disabled this one fails with NullPointerException instead insert into slash_part_repro values (2, 'a2', 2000, null); ``` 3. Both statements fail. Expected `_hoodie_partition_path` values are `2026/01/05` and `__HIVE_DEFAULT_PARTITION__`. The same two failures reproduce as a plain unit test against `SimpleKeyGenerator`, with no Spark job involved: ```java TypedProperties props = new TypedProperties(); props.put(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "_row_key"); props.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), "ts_ms"); props.put(KeyGeneratorOptions.SLASH_SEPARATED_DATE_PARTITIONING.key(), "true"); props.put(KeyGeneratorOptions.HIVE_STYLE_PARTITIONING_ENABLE.key(), "false"); SimpleKeyGenerator keyGenerator = new SimpleKeyGenerator(props); // ts_ms = "2020-03-21" keyGenerator.getPartitionPath(internalRow, row.schema()); // ClassCastException ``` ### Environment **Hudi version:** 1.2.0 and master (1.3.0-SNAPSHOT). The feature was introduced by #17787 (merged 2026-01-30), which is an ancestor of `release-1.2.0`, so every release carrying the config is affected. **Query engine:** Spark (Spark SQL and the DataSource writer; verified on Spark 3.5 / Scala 2.12) **Relevant configs:** - `hoodie.datasource.write.slash.separated.date.partitioning=true` - `hoodie.datasource.write.hive_style_partitioning=false` - `hoodie.datasource.write.row.writer.enable=true` (default) - `hoodie.sql.bulk.insert.enable=true` to reach the row writer from Spark SQL ### Logs and Stack Trace Row-writer path (`getPartitionPath(InternalRow, StructType)`): ``` java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class java.lang.String (org.apache.spark.unsafe.types.UTF8String is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap') at org.apache.hudi.keygen.PartitionPathFormatterBase.combine(PartitionPathFormatterBase.java:66) at org.apache.hudi.keygen.BuiltinKeyGenerator.combinePartitionPathUnsafe(BuiltinKeyGenerator.java:143) at org.apache.hudi.keygen.SimpleKeyGenerator.getPartitionPath(SimpleKeyGenerator.java:111) ``` Null partition value (`getPartitionPath(Row)`): ``` java.lang.NullPointerException at org.apache.hudi.keygen.PartitionPathFormatterBase.combine(PartitionPathFormatterBase.java:66) at org.apache.hudi.keygen.BuiltinKeyGenerator.combinePartitionPath(BuiltinKeyGenerator.java:135) at org.apache.hudi.keygen.SimpleKeyGenerator.getPartitionPath(SimpleKeyGenerator.java:105) ``` Surfaced through Spark as a stage failure during the bulk insert: ``` org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 51.0 failed 1 times, most recent failure: Lost task 0.0 in stage 51.0: java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class java.lang.String ``` I have a fix and will open a PR referencing this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
