Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205912718
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -87,17 +88,33 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
case d: DecimalType =>
(getter, ordinal) => getter.getDecimal(ordinal, d.precision,
d.scale).toString
case StringType =>
- (getter, ordinal) => new
Utf8(getter.getUTF8String(ordinal).getBytes)
+ (getter, ordinal) =>
+ if (avroType.getType == Type.ENUM) {
+ new GenericData.EnumSymbol(avroType,
getter.getUTF8String(ordinal).toString)
+ } else {
+ new Utf8(getter.getUTF8String(ordinal).getBytes)
+ }
case BinaryType =>
- (getter, ordinal) => ByteBuffer.wrap(getter.getBinary(ordinal))
+ (getter, ordinal) =>
+ val data = getter.getBinary(ordinal)
+ if (avroType.getType == Type.FIXED) {
+ // Handles fixed-type fields in output schema. Test case is
included in test.avro
+ // as it includes several fixed fields that would fail if we
specify schema
+ // on-write without this condition
+ val fixed = new GenericData.Fixed(avroType)
+ fixed.bytes(data)
+ fixed
+ } else {
+ ByteBuffer.wrap(data)
+ }
--- End diff --
Thanks for this hint! I reversed the expression as you suggested. I don't
think its related to the poor performance of the particular schema I tested,
but I think the issue is more general and not related to my changes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
