[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Sadikov updated SPARK-39802: --------------------------------- Description: This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718. It would be great if Spark could support recursive Avro schemas as currently the following exception is thrown: {code:java} org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive reference in Avro schema, which can not be processed by Spark: { "type": "record", "name": "Struct", "fields": [ { "name": "fields", "type": [ "null", { "type": "array", "items": { "type": "record", "name": "fields", "fields": [ { "name": "value", "type": [ "null", "string" ], "default": null } ] } } ], "default": null } ] }{code} I would appreciate it if someone could comment on the overall effort to make recursive Avro schemas work in Spark, thanks. was: This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718. It would be great if Spark could support recursive Avro schemas as currently the following exception is thrown: {code:java} org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive reference in Avro schema, which can not be processed by Spark: { "type": "record", "name": "Struct", "fields": [ { "name": "fields", "type": [ "null", { "type": "array", "items": { "type": "record", "name": "fields", "fields": [ { "name": "value", "type": [ "null", "string" ], "default": null } ] } } ], "default": null } ] }{code} > Support Avro recursive schemas in Spark > --------------------------------------- > > Key: SPARK-39802 > URL: https://issues.apache.org/jira/browse/SPARK-39802 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.4.0 > Reporter: Ivan Sadikov > Priority: Major > > This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718. > It would be great if Spark could support recursive Avro schemas as currently > the following exception is thrown: > {code:java} > org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive > reference in Avro schema, which can not be processed by Spark: { > "type": "record", > "name": "Struct", > "fields": [ > { > "name": "fields", > "type": [ > "null", > { > "type": "array", > "items": { > "type": "record", > "name": "fields", > "fields": [ > { > "name": "value", > "type": [ > "null", > "string" > ], > "default": null > } > ] > } > } > ], > "default": null > } > ] > }{code} > I would appreciate it if someone could comment on the overall effort to make > recursive Avro schemas work in Spark, thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org