bdrillard edited a comment on issue #22878: [SPARK-25789][SQL] Support for Dataset of Avro URL: https://github.com/apache/spark/pull/22878#issuecomment-447359187 I agree with @gengliangwang, as far as I can see from my look at the [external data source module](https://github.com/apache/spark/tree/master/external/avro), the original `DataFrame` support for Avro that used to be in Spark-Avro has been rolled into Spark-proper. This PR that @xuanyuanking has taken the time to drive and respond to would roll in the support I had written for `Dataset` of Avro, but that had never been committed to Spark-Avro (it sat in PR without review), I imagine for the same reason that @srowen mentions. Our users would no longer have any need for Spark-Avro after this PR. As far as I can tell, the entire Spark-Avro project would be subsumed by Spark after this. On the other note that @gengliangwang mentions, refactoring `InitializeAvroObject` to `NewInstance` is a bit tough, since it entails minor, passive changes to the `TreeNode` class, see [here](https://github.com/apache/spark/pull/21348/files#diff-eac5b02bb450a235fef5e902a2671254R361). I'm happy to submit a follow-up PR that cleans up the `Expression` if that's easier, but I'll leave that up to you and @xuanyuanking.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
