rangareddy commented on issue #17367:
URL: https://github.com/apache/hudi/issues/17367#issuecomment-5351239829
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-8909).
**Findings: confirmed, still open - and the test suite itself records the
workaround.**
`hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/ColumnStatIndexTestBase.scala:69`:
```scala
.add("c7", StringType) // HUDI-8909. To support Byte w/ partition stats
index.
```
That is the same `c7` column your reproduction steps ask the reader to
switch to `Byte`. It was deliberately typed as `StringType` to route around
this bug, with a comment pointing back at this ticket - so the workaround is
in-tree and the underlying problem is untouched.
(There is a second schema in the same file at `:403` using `.add("c7",
BinaryType)` with `Array(item).map(_.toByte)` at `:416`, but that is
bytes-as-binary, not the Byte/TinyInt scalar this ticket is about. Worth not
confusing the two when reproducing.)
The read side does handle the type -
`hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/ColumnStatsIndexSupport.scala:470`
and `:522` both do `case ByteType => colStatsValue.asInstanceOf[Int].toByte`.
So the gap is where you said it is: merging min/max values across files, not
the transposition back to Spark types.
Also for the record, PR #12671 (`[HUDI-8680] Enabling partition stats by
default`), referenced in the description as where this surfaced, was **closed
without merging**.
Reproduction is cheap because the harness is already there: change `c7` back
to `ByteType` at `ColumnStatIndexTestBase.scala:69` and run
`TestColumnStatsIndex.testMetadataColumnStatsIndex` for COW.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
