[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17567759#comment-17567759 ]
Ivan Sadikov commented on SPARK-39802: -------------------------------------- [~Gengliang.Wang] Would you be able to comment on this ticket? Thanks. > Support Avro recursive schemas in Spark > --------------------------------------- > > Key: SPARK-39802 > URL: https://issues.apache.org/jira/browse/SPARK-39802 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.4.0 > Reporter: Ivan Sadikov > Priority: Major > > This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718. > It would be great if Spark could support recursive Avro schemas as currently > the following exception is thrown: > {code:java} > org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive > reference in Avro schema, which can not be processed by Spark: { > "type": "record", > "name": "Struct", > "fields": [ > { > "name": "fields", > "type": [ > "null", > { > "type": "array", > "items": { > "type": "record", > "name": "fields", > "fields": [ > { > "name": "value", > "type": [ > "null", > "string" > ], > "default": null > } > ] > } > } > ], > "default": null > } > ] > }{code} > I would appreciate it if someone could comment on the overall effort to make > recursive Avro schemas work in Spark and whether or not there are any API > restrictions that could potentially hinder the implementation, thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org