Jiayi-Wang-db opened a new pull request, #58496: URL: https://github.com/apache/spark/pull/58496
### What changes were proposed in this pull request? Make the Parquet row-based reader reject two incompatible primitive type conversions that the vectorized reader already rejects: * Parquet `FIXED_LEN_BYTE_ARRAY` to Spark SQL `STRING` * Parquet `INT32` annotated as `DATE` to Spark SQL `DECIMAL` The row converter now accepts `INT32` and `INT64` as decimals only when their logical annotation is decimal, absent, or a signed integer annotation. Unsupported conversions raise `SchemaColumnConvertNotSupportedException`, consistently with the vectorized reader. This follows the conversion semantics established by SPARK-34212 and SPARK-40876. The earlier SPARK-32317 / PR #29600 discussion proposed a broader configurable conversion matrix but was not merged. Closes #58495. ### Why are the changes needed? The row-based reader currently ignores incompatible logical and physical type combinations. It interprets a `DATE` value as an unannotated integer-backed decimal and interprets arbitrary fixed-length bytes as UTF-8. This can silently return incorrect data, while reading the same file and requested schema with the vectorized reader fails with a Parquet column type mismatch. Both readers should reject these unsupported conversions instead of producing reader-dependent results. ### Does this PR introduce _any_ user-facing change? Yes. When the vectorized Parquet reader is disabled, the two unsupported schema conversions above now fail with `FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH` instead of returning incorrectly interpreted values. Supported binary-to-string and integer-to-decimal conversions are unchanged. ### How was this patch tested? Added an end-to-end test in `ParquetIOSuite` that writes raw Parquet schemas for both incompatible conversions and verifies that the row-based and vectorized readers return the same error condition. The focused test command was attempted locally: ``` build/sbt 'sql/core/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite -- -z "Parquet readers reject incompatible primitive type conversions consistently"' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
