Jiayi-Wang-db opened a new issue, #58495:
URL: https://github.com/apache/spark/issues/58495
### Problem
Spark's row-based Parquet reader and vectorized Parquet reader handle some
incompatible requested
schemas differently. The vectorized reader raises
`SchemaColumnConvertNotSupportedException`, while
the row-based reader silently interprets the physical value as the requested
Catalyst type.
Two examples are:
| Parquet file type | Requested Spark type | Row-based reader | Vectorized
reader |
| --- | --- | --- | --- |
| `FIXED_LEN_BYTE_ARRAY(4)` | `STRING` | Returns the raw bytes as UTF-8 |
Rejects the conversion |
| `INT32 (DATE)` | `DECIMAL(10, 0)` | Returns the day count as a decimal |
Rejects the conversion |
The row reader's `ParquetRowConverter` currently accepts every binary-like
primitive as a string
and treats any `INT32` or `INT64` without decimal metadata as an unannotated
integer-backed
decimal. The latter ignores semantic logical annotations such as `DATE`.
This is a correctness problem because changing
`spark.sql.parquet.enableVectorizedReader` can
change a query from failing cleanly to returning incorrectly interpreted
data.
### Expected behavior
Both readers should reject these unsupported conversions with
`FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`.
Supported conversions should remain unchanged, including Parquet `BINARY` to
Spark `STRING` and
unannotated or signed-integer `INT32`/`INT64` to a sufficiently compatible
Spark decimal type.
### Related discussions
* #31319 / SPARK-34212 established the decimal conversion rules and
discussed consistency between
the row-based and vectorized readers.
* #44368 / SPARK-40876 added lossless widening conversions to both readers
and required logical
annotation checks for semantic types such as `DATE`.
* #29600 / SPARK-32317 proposed a broader configurable conversion matrix but
was not merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
