Brian Wu created SPARK-60010:
--------------------------------
Summary: Reading a Parquet DECIMAL column with an integer schema
returns the raw unscaled integer instead of honoring the decimal scale
Key: SPARK-60010
URL: https://issues.apache.org/jira/browse/SPARK-60010
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1
Reporter: Brian Wu
When a Parquet column physically stored as DECIMAL is read with an integer
read-schema, Spark's behavior is unexpected: for low-precision decimals it
silently returns the raw unscaled stored integer (ignoring the scale), while
for higher-precision decimals it throws a type-mismatch error.
Reproduction:
{code:java}
import org.apache.spark.sql.types._
val intSchema = StructType(Seq(StructField("v", IntegerType)))
Case A – DECIMAL(9,2), physically stored as INT32:
spark.sql("SELECT CAST(123.45 AS DECIMAL(9,2)) AS
v").write.mode("overwrite").parquet("/tmp/dec92")
spark.read.schema(intSchema).parquet("/tmp/dec92").show()
Result - silently returns the unscaled stored value (scale ignored)
+-----+
| v|
+-----+
|12345|
+-----+
Case B – DECIMAL(10,2), physically stored as INT64:
spark.sql("SELECT CAST(123.45 AS DECIMAL(10,2)) AS
v").write.mode("overwrite").parquet("/tmp/dec102")
spark.read.schema(intSchema).parquet("/tmp/dec102").show()
Result - throws
org.apache.spark.SparkException:
[FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH] Encountered error while
reading file
file:///tmp/dec102/part-00000-a59fef00-bb30-4788-a195-2f4c6173117e-c000.snappy.parquet.
Data type mismatches when reading Parquet column [v]. Expected Spark type int,
actual Parquet type INT64{code}
[https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#numeric-types]
Parquet standard does explicitly not define behavior here. However, it seems
Spark behavior does not make sense in these cases. Suggest Spark should honor
the schema type and not interpret the physical value as unscaled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
