Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22037#discussion_r209140501
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala
---
@@ -139,7 +152,22 @@ object SchemaConverters {
case FloatType => builder.floatType()
case DoubleType => builder.doubleType()
- case _: DecimalType | StringType => builder.stringType()
+ case StringType => builder.stringType()
+ case d: DecimalType =>
+ val avroType = LogicalTypes.decimal(d.precision, d.scale)
+ val fixedSize = minBytesForPrecision(d.precision)
+ // Use random name to avoid conflict in naming of fixed field.
+ // Field names must start with [A-Za-z_], while the charset of
Random.alphanumeric contains
+ // [0-9]. So add a single character "f" to ensure the name is
valid.
+ val name = "f" + Random.alphanumeric.take(32).mkString("")
+ if (nullable) {
+ val schema = avroType.addToSchema(
+ SchemaBuilder.builder().fixed(name).size(fixedSize))
+ builder.`type`(schema)
+ } else {
+ avroType.addToSchema(builder.fixed(name).size(fixedSize))
--- End diff --
Here we can add the schema to `builder` directly. If the builder is
nullable, we need to create schema with logical type and then add it to the
nullable builder (complete the type as `union` with `null`)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
