SEPURI-SAI-KRISHNA opened a new pull request, #19652:
URL: https://github.com/apache/hudi/pull/19652
### Describe the issue this Pull Request addresses
Closes #19651.
A table partitioned by a `date` column whose partition path does not line up
one-to-one with the
partition columns writes successfully but cannot be read at all — every
query, including an
unfiltered `select *`, fails with `ClassCastException: UTF8String cannot be
cast to
java.lang.Integer`.
When the partition fragments do not line up with the partition columns,
`HoodieSparkUtils#doParsePartitionColumnValues` falls back onto
`castStringToType`, which returned
partition values in their *external* representation instead of Catalyst's
*internal* one. Those
values are placed directly into the `InternalRow` of partition values that
`SparkHoodieTableFileIndex`
and `HoodieFileIndex` evaluate against, so Spark reads a `UTF8String`
through `InternalRow#getInt`
and throws.
Five problems in that one method:
| input | expected | before this PR |
|---|---|---|
| `"2023-03-01"` as `DateType` | `Int` epoch days | `UTF8String` |
| `"2023-03-01 01:02:03"` as `TimestampType` | `Long` micros | `UTF8String` |
| `"1.50"` as `DecimalType(10,2)` | `Decimal` | `java.math.BigDecimal` |
| `"__HIVE_DEFAULT_PARTITION__"` | `null` | the literal sentinel string |
| `"a%3Db"` as `StringType` | `a=b` | `a%3Db` |
The last two also diverged from the non-fallback path: when the fragments do
line up, parsing goes
through Spark's partition parser, which maps the default partition to `null`
and unescapes values.
The same table therefore produced different partition values depending on
which branch parsed the
path.
### Summary and Changelog
Partition values recovered from a partition path are now in Catalyst's
internal representation, so
tables with a `date`/`timestamp`/`decimal` partition column whose path does
not line up with the
partition columns are readable and prune correctly.
- `HoodieSparkUtils#castStringToType` now delegates to
`SparkParsePartitionUtil#castPartValueToDesiredType` — the same function
already backing
`parsePartitionPath` on the aligned path, so both paths converge by
construction instead of
through a second hand-rolled conversion. This is what fixes all five rows
of the table above; the
hand-rolled `match` is deleted.
- Threaded the configured `timeZoneId` down to the conversion, so timestamp
partition values are
resolved against the session/reader time zone (as the aligned path already
did) rather than the
JVM default. `castStringToType` keeps its original two-argument form as an
overload defaulting to
the session-local time zone, so the existing signature stays binary
compatible.
- Kept the pre-existing lenient fallback: a value that cannot be converted
is still logged and left
in its string representation, so genuinely malformed paths behave as
before.
- Tests: new `TestHoodieSparkUtilsPartitionValues` covering the internal
representations, the
default-partition-to-null mapping, unescaping, time-zone handling, and
both fallback shapes
(single date column laid out as `yyyy/MM/dd`, and a hive-style value
spilling over a `/`); new
end-to-end `TestTypedPartitionValues` reading a `date`-partitioned table
and pruning on it.
Verified that the new tests fail on `master` with the `ClassCastException`
above and pass with the
fix.
### Impact
Fixes a total read failure for affected tables. Data already written is fine
— only the read path
was broken, so the fix makes existing tables readable again without any
migration.
Affects 1.2.0 and master.
Behavior changes, both bringing the fallback path in line with the aligned
path:
- A partition value of `__HIVE_DEFAULT_PARTITION__` now reads back as `null`
rather than as the
literal string.
- String partition values are now URL-unescaped, so a value written as
`a%3Db` reads back as `a=b`.
`castStringToType` is public; its existing two-argument signature is
preserved as an overload.
### Risk Level
low
The change replaces a hand-rolled conversion with the conversion helper
Spark and Hudi already use
for the aligned path, in the same module. The previously broken paths threw
before producing any
result, and the two behavior changes above make the fallback agree with the
parser that already
governs the majority of tables. Verified against the new unit and end-to-end
tests plus the
`hudi-spark-client` key-generator/datasource suites and the
`TestHoodieFileIndex`,
`TestLazyPartitionPathFetching`, `TestSlashSeparatedPartitionValue` and
`TestShowPartitions` Spark
suites, all green.
### Documentation Update
none — no new configs and no change to any documented behavior.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
