voonhous commented on issue #18974:
URL: https://github.com/apache/hudi/issues/18974#issuecomment-6599997950
The Flink path is worse than a `ClassCastException`: with `BinaryRowData`
input it silently stores wrong values. The writers build their `RowType` from
`HoodieSchema` (`INT`) and read the field with `getInt`, so the one- or
two-byte value is zero extended:
| Written (SMALLINT) | Stored |
|---|---|
| -1 | 65535 |
| 300 | 300 |
| -300 | 65236 |
| 32767 | 32767 |
| -32768 | 32768 |
`TINYINT` -1 is stored as 255. This covers the Parquet, Lance and Avro log
block writers. `GenericRowData` input fails instead:
```
java.lang.RuntimeException: java.lang.ClassCastException: class
java.lang.Short cannot be cast to class java.lang.Integer (java.lang.Short and
java.lang.Integer are in module java.base of loader 'bootstrap')
at
org.apache.hudi.io.storage.row.RowDataParquetWriteSupport.write(RowDataParquetWriteSupport.java:71)
at
org.apache.hudi.io.storage.row.RowDataParquetWriteSupport.write(RowDataParquetWriteSupport.java:37)
at
org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:152)
at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:425)
at
org.apache.hudi.benchmarks.rowwriter.SmallIntCheck.write(SmallIntCheck.java:125)
at
org.apache.hudi.benchmarks.rowwriter.SmallIntCheck.lambda$main$1(SmallIntCheck.java:81)
at
org.apache.hudi.benchmarks.rowwriter.SmallIntCheck.report(SmallIntCheck.java:99)
at
org.apache.hudi.benchmarks.rowwriter.SmallIntCheck.main(SmallIntCheck.java:81)
Caused by: java.lang.ClassCastException: class java.lang.Short cannot be
cast to class java.lang.Integer (java.lang.Short and java.lang.Integer are in
module java.base of loader 'bootstrap')
at
org.apache.flink.table.data.GenericRowData.getInt(GenericRowData.java:150)
at
org.apache.hudi.io.storage.row.parquet.ParquetRowDataWriter$IntWriter.write(ParquetRowDataWriter.java:316)
at
org.apache.hudi.io.storage.row.parquet.ParquetRowDataWriter.write(ParquetRowDataWriter.java:97)
at
org.apache.hudi.io.storage.row.RowDataParquetWriteSupport.write(RowDataParquetWriteSupport.java:69)
... 7 more
```
Reproduced on master (b0d16f9), Flink 2.2.1, by writing the values through
`RowDataParquetWriteSupport` and reading the file back. Other engines read the
stored numbers, the column stats hold them, and Flink returns no rows for
`WHERE small < 0` because Parquet row-group filtering prunes on those stats.
Separately, MOR and Lance reads of a `TINYINT` / `SMALLINT` column whose table
schema says `INT` fail with `ClassCastException` (`Integer` to `Byte`).
A fix for new writes (widen these columns to the write schema's integer type
in `HoodieTableSink`), with `ITTestNarrowIntStoredValues` covering stored
values, stats and filters, is on voonhous/hudi@2773e79. Existing files keep the
unsigned values until rewritten through Flink. I can open it as a PR against
this issue.
Spark-side symptoms of the same collapse, reproduced on the same commit:
#20503 (`bulk_insert` of `ShortType` / `ByteType`) and #20504 (MERGE INTO with
a `tinyint` source).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
