dongjoon-hyun commented on a change in pull request #31329:
URL: https://github.com/apache/spark/pull/31329#discussion_r564307326
##########
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
##########
@@ -169,16 +169,16 @@ private[sql] class AvroDeserializer(
}
updater.set(ordinal, bytes)
- case (FIXED, d: DecimalType) => (updater, ordinal, value) =>
- val bigDecimal =
decimalConversions.fromFixed(value.asInstanceOf[GenericFixed], avroType,
- LogicalTypes.decimal(d.precision, d.scale))
- val decimal = createDecimal(bigDecimal, d.precision, d.scale)
+ case (FIXED, _: DecimalType) => (updater, ordinal, value) =>
+ val d = avroType.getLogicalType.asInstanceOf[LogicalTypes.Decimal]
+ val bigDecimal =
decimalConversions.fromFixed(value.asInstanceOf[GenericFixed], avroType, d)
+ val decimal = createDecimal(bigDecimal, d.getPrecision, d.getScale)
Review comment:
BTW, @cloud-fan . I want to make it sure the following.
1. This PR have no side-effect when the physical schema matches the catalyst
schema.
2. This PR fixes the incorrect result when the physical schema mismatches
the catalyst schema.
This is much better than before. And, we can fix the inconsistency in the
upper-layer still because we have the original decimal's precision and scale
still.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
