SEPURI-SAI-KRISHNA opened a new issue, #19651:
URL: https://github.com/apache/hudi/issues/19651
### Bug Description
**What happened:**
A table partitioned by a `date` column whose partition path does not line up
one-to-one with the
partition columns is written successfully but **cannot be read at all**.
Every query against it —
including an unfiltered `select *` — fails with:
```
java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String
cannot be cast to class java.lang.Integer
```
When the partition fragments do not line up with the partition columns,
`HoodieSparkUtils#doParsePartitionColumnValues` falls back onto
`castStringToType` to recover the
partition values. That method returns values in their *external*
representation rather than
Catalyst's *internal* one:
```scala
case _: DecimalType => new java.math.BigDecimal(value) // Catalyst expects
Decimal
case StringType => UTF8String.fromString(value)
case _: TimestampType => UTF8String.fromString(value) // Catalyst expects
Long (micros)
case _: DateType => UTF8String.fromString(value) // Catalyst expects
Int (epoch days)
```
Those values go straight into the `InternalRow` of partition values that
partition pruning
(`SparkHoodieTableFileIndex`) and the file index (`HoodieFileIndex`)
evaluate against, so Spark
reads a `UTF8String` through `InternalRow#getInt` and throws.
Five distinct problems in that one method, all verified:
| input | expected | actual |
|---|---|---|
| `"2023-03-01"` as `DateType` | `Int` 19417 (epoch days) | `UTF8String`
`2023-03-01` |
| `"2023-03-01 01:02:03"` as `TimestampType` | `Long` micros | `UTF8String` |
| `"1.50"` as `DecimalType(10,2)` | `org.apache.spark.sql.types.Decimal` |
`java.math.BigDecimal` |
| `"__HIVE_DEFAULT_PARTITION__"` as any type | `null` | the literal sentinel
string |
| `"a%3Db"` as `StringType` | `a=b` | `a%3Db` (never unescaped) |
The default-partition and unescaping cases also diverge from the
non-fallback path: when the
fragments *do* line up, parsing goes through Spark's partition parser, which
maps the default
partition to `null` and unescapes values. So the same table yields different
partition values
depending on which branch parses the path.
**What you expected:**
Partition values recovered from the partition path should be in Catalyst's
internal representation
and should match what the aligned path produces, so the table is readable
and partition pruning
works.
**Steps to reproduce:**
```sql
create table date_part_repro (
id int,
name string,
ts long,
grass_date date
) using hudi
tblproperties (
primaryKey = 'id',
type = 'cow',
orderingFields = 'ts',
'hoodie.datasource.write.slash.separated.date.partitioning' = 'true'
)
partitioned by (grass_date);
insert into date_part_repro values(1, 'a1', 1000, date'2023-02-27');
insert into date_part_repro values(2, 'a2', 1000, date'2023-03-01');
-- writes fine; partition directories 2023/02/27 and 2023/03/01 are created
select * from date_part_repro; -- ClassCastException
```
The write succeeds and the data is on disk; only reads fail. The same
failure reproduces without
that config whenever the fragments do not line up — for example a hive-style
table
`partitioned by (dt date, city string)` where a `city` value contains an
unencoded `/`, giving
`dt=2023-03-01/city=san/francisco` (three fragments, two columns).
It also reproduces as a plain unit test with no Spark job:
```scala
HoodieSparkUtils.castStringToType("2023-03-01", DateType)
// => UTF8String "2023-03-01", where Catalyst requires an Int
```
### Environment
**Hudi version:** 1.2.0 and master (1.3.0-SNAPSHOT)
**Query engine:** Spark (verified on Spark 3.5 / Scala 2.12)
**Relevant configs:** any table whose partition path fragments do not line
up with the partition
columns and that has a `date`, `timestamp` or `decimal` partition column,
e.g.
`hoodie.datasource.write.slash.separated.date.partitioning=true` with a
single `date` partition
column.
### Logs and Stack Trace
```
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
stage 45.0 failed 1
times, most recent failure: Lost task 0.0 in stage 45.0:
java.lang.ClassCastException: class
org.apache.spark.unsafe.types.UTF8String cannot be cast to class
java.lang.Integer
(org.apache.spark.unsafe.types.UTF8String is in unnamed module of loader
'app'; java.lang.Integer
is in module java.base of loader 'bootstrap')
```
I have a fix and will open a PR referencing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
