Cosmin Dumitru created SPARK-46056:
--------------------------------------
Summary: Vectorized parquet reader throws NPE when reading files
with DecimalType default values
Key: SPARK-46056
URL: https://issues.apache.org/jira/browse/SPARK-46056
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.5.0, 3.4.0
Reporter: Cosmin Dumitru
The scenario is a bit more complicated than what the title says but it's not
that far fetched.
# Write a parquet file with one column
# Evolve the schema and add a new column with DecimalType wide enough that it
doesn't fit in a long and has a default value.
# Try to read the file with the new schema
# NPE
The issue lies in how the column vector stores DecimalTypes. It incorrectly
assumes that they fit in a long and try to write it to associated long array.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
