[
https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Sadikov updated SPARK-39802:
---------------------------------
Description:
This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
It would be great if Spark could support recursive Avro schemas as currently
the following exception is thrown:
{code:java}
org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive
reference in Avro schema, which can not be processed by Spark: {
"type": "record",
"name": "Struct",
"fields": [
{
"name": "fields",
"type": [
"null",
{
"type": "array",
"items": {
"type": "record",
"name": "fields",
"fields": [
{
"name": "value",
"type": [
"null",
"string"
],
"default": null
}
]
}
}
],
"default": null
}
]
}{code}
> Support Avro recursive schemas in Spark
> ---------------------------------------
>
> Key: SPARK-39802
> URL: https://issues.apache.org/jira/browse/SPARK-39802
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Ivan Sadikov
> Priority: Major
>
> This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
> It would be great if Spark could support recursive Avro schemas as currently
> the following exception is thrown:
> {code:java}
> org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive
> reference in Avro schema, which can not be processed by Spark: {
> "type": "record",
> "name": "Struct",
> "fields": [
> {
> "name": "fields",
> "type": [
> "null",
> {
> "type": "array",
> "items": {
> "type": "record",
> "name": "fields",
> "fields": [
> {
> "name": "value",
> "type": [
> "null",
> "string"
> ],
> "default": null
> }
> ]
> }
> }
> ],
> "default": null
> }
> ]
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
