This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7713565 [SPARK-34786][SQL][FOLLOWUP] Explicitly declare
DecimalType(20, 0) for Parquet UINT_64
7713565 is described below
commit 771356555c1110b898ff09ea23fe0b00749caefd
Author: Kent Yao <[email protected]>
AuthorDate: Thu Apr 29 04:51:27 2021 +0000
[SPARK-34786][SQL][FOLLOWUP] Explicitly declare DecimalType(20, 0) for
Parquet UINT_64
### What changes were proposed in this pull request?
Explicitly declare DecimalType(20, 0) for Parquet UINT_64, avoid use
DecimalType.LongDecimal which only happens to have 20 as precision.
https://github.com/apache/spark/pull/31960#discussion_r622691560
### Why are the changes needed?
fix ambiguity
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
not needed, just current CI pass
Closes #32390 from yaooqinn/SPARK-34786-F.
Authored-by: Kent Yao <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../sql/execution/datasources/parquet/ParquetSchemaConverter.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
index 8c4e088..e751c97 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
@@ -141,7 +141,9 @@ class ParquetToSparkSchemaConverter(
originalType match {
case INT_64 | null => LongType
case DECIMAL => makeDecimalType(Decimal.MAX_LONG_DIGITS)
- case UINT_64 => DecimalType.LongDecimal
+ // The precision to hold the largest unsigned long is:
+ // `java.lang.Long.toUnsignedString(-1).length` = 20
+ case UINT_64 => DecimalType(20, 0)
case TIMESTAMP_MICROS => TimestampType
case TIMESTAMP_MILLIS => TimestampType
case _ => illegalType()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
