sankalpsthakur opened a new pull request, #59268:
URL: https://github.com/apache/spark/pull/59268
### What changes were proposed in this pull request?
`ParquetVectorUpdaterFactory.getUpdater` accepted a DECIMAL-annotated INT32
column for a `ByteType`, `ShortType`, `IntegerType` or `LongType` read type
(the last through the INT32 -> long widening path), and a DECIMAL-annotated
INT64 column for `LongType`. It never checked the annotation, so it returned
the stored unscaled value.
This PR adds `canReadAsIntegral(descriptor)` and requires it on those five
plain integral branches. It is false only for a `DecimalLogicalTypeAnnotation`
with a non-zero scale. A rejected read falls through to the existing
`constructConvertNotSupportedException`, the path that already rejects
DECIMAL(10, 2) (INT64) read as int. The user sees
`FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`. The decimal branches
(`canReadAs*Decimal`) are unchanged.
A DECIMAL with scale 0 stores the integral value itself, so it can still be
read as an integral type. The row reader returns the same values for those
inputs, both on master and with #59080.
This is the vectorized counterpart of #59080 (SPARK-59251). That PR makes
the row-based reader (`ParquetRowConverter`) reject incompatible primitive
conversions with the same error. This PR does not change the row reader.
### Why are the changes needed?
Reading Parquet data with a user-specified schema silently returns wrong
values:
```scala
spark.sql("SELECT CAST(123.45 AS DECIMAL(9, 2)) AS
v").write.parquet("/tmp/dec92")
spark.read.schema("v INT").parquet("/tmp/dec92").show()
// before: 12345, the unscaled value
// after: [FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH] ... Data
type mismatches when reading
// Parquet column [v]. Expected Spark type int, actual Parquet type
INT32.
```
The same read of a DECIMAL(10, 2) column (INT64) already failed with that
error, so the outcome depended on the decimal's precision. Reading DECIMAL(10,
2) as bigint also returned 12345.
### Does this PR introduce _any_ user-facing change?
Yes. This applies to the vectorized Parquet reader, which is the default.
Two reads of a DECIMAL with a non-zero scale used to return the unscaled value
and now fail with `FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`:
- a DECIMAL stored as INT32 (precision <= 9), read as tinyint, smallint, int
or bigint
- a DECIMAL stored as INT64, read as bigint
DECIMAL with scale 0 and reads as a decimal type are unaffected.
Notes for reviewers:
- A query that depended on the old (wrong) result now fails. Reviewers may
want a migration guide note or a legacy flag. This PR adds neither.
- For these inputs the two readers now disagree. The parquet-mr row reader
is used when the vectorized reader is disabled or not applicable. It still
returns the unscaled value, on master and with #59080 as it stands. Aligning it
is a follow-up, possibly within #59080's scope.
- The same blind spot exists for other non-decimal read types of a
DECIMAL-annotated INT32/INT64 column: double, date and ANSI intervals. They are
left out to keep this PR to the integral types in the JIRA.
### How was this patch tested?
New tests:
- `ParquetTypeWideningSuite`, `SPARK-60010: unsupported parquet conversion
DecimalType(9,2) / DecimalType(18,2) -> ByteType / ShortType / IntegerType /
LongType` (8 tests):
- Files are written with dictionary encoding on and off, using the Parquet
v1 and v2 writers, and read with the vectorized reader.
- Each test checks `FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`
with column `[a]`, the expected Spark type, and actual type INT32 or INT64.
- `ParquetTypeWideningSuite`, `SPARK-60010: parquet conversion
DecimalType(9,0) -> ByteType / ShortType / IntegerType / LongType` and
`DecimalType(18,0) -> LongType` (5 tests): both readers still read scale-0
decimals as integral values.
- `ParquetVectorUpdaterSuite`, `SPARK-60010: factory rejects a DECIMAL with
a non-zero scale read as an integral type`:
- `getUpdater` throws `SchemaColumnConvertNotSupportedException` for INT32
DECIMAL(9, 2), INT64 DECIMAL(9, 2) and INT64 DECIMAL(18, 2) with all four
integral types.
- It still returns an updater for INT32 DECIMAL(9, 0) and INT64
DECIMAL(18, 0).
No existing test or SQL golden file depended on the old behavior. I checked
the Parquet-related tests in sql/core and sql/hive and the golden files, so
none were updated.
Nothing was built or run locally. Validation is the fork's GitHub Actions
run for this branch (the Build check on this PR).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
