dongjoon-hyun commented on a change in pull request #31319:
URL: https://github.com/apache/spark/pull/31319#discussion_r564264345
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
##########
@@ -268,18 +268,27 @@ private[parquet] class ParquetRowConverter(
}
// For INT32 backed decimals
- case t: DecimalType if
parquetType.asPrimitiveType().getPrimitiveTypeName == INT32 =>
- new ParquetIntDictionaryAwareDecimalConverter(t.precision, t.scale,
updater)
+ case _: DecimalType if
parquetType.asPrimitiveType().getPrimitiveTypeName == INT32 =>
+ val metadata = parquetType.asPrimitiveType().getDecimalMetadata
+ val precision = metadata.getPrecision()
+ val scale = metadata.getScale()
+ new ParquetIntDictionaryAwareDecimalConverter(precision, scale,
updater)
Review comment:
1. For object handling, after creating decimal object, the upper layer
uses the object's precision and scale to handle the object's value. Do you have
some usual-suspect code?
2. For the user-facing side, this doesn't change the user-provided schema.
So, there is no change from user's perspective.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
