dbtsai commented on issue #24635: [SPARK-27762][SQL] Support user provided avro schema for writing fields with different ordering URL: https://github.com/apache/spark/pull/24635#issuecomment-494198592 @cloud-fan This feature was already added in 2.4.0 (by us, Apple). The reason is avro and catalyst schemas are very different, and in avro, it has enum and union types which are not supported in catalyst. To ensure the compatibility with our online runtime (which is not Spark base), we need to have a way to write enum or union types from Spark. Thus, we submitted this PR which allows Spark to specify the avro schema while writing. For parquet or orc formats, they match catalyst schema well, I think we can implement a new feature to support write with custom catalyst schema. Note that this is different from the use-case of supporting custom avro schema.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
