[
https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Sadikov updated SPARK-39802:
---------------------------------
Description:
This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
It would be great if Spark could support recursive references in Avro schemas
as currently this is disabled and the following exception is thrown:
{code:java}
org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive
reference in Avro schema, which can not be processed by Spark: {
"type": "record",
"name": "Struct",
"fields": [
{
"name": "fields",
"type": [
"null",
{
"type": "array",
"items": {
"type": "record",
"name": "fields",
"fields": [
{
"name": "value",
"type": [
"null",
"string"
],
"default": null
}
]
}
}
],
"default": null
}
]
}{code}
I would appreciate it if someone could comment on the overall effort to make
recursive Avro schemas work in Spark and whether or not there are any API
restrictions that could potentially hinder the implementation, thanks.
was:
This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
It would be great if Spark could support recursive Avro schemas as currently
the following exception is thrown:
{code:java}
org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive
reference in Avro schema, which can not be processed by Spark: {
"type": "record",
"name": "Struct",
"fields": [
{
"name": "fields",
"type": [
"null",
{
"type": "array",
"items": {
"type": "record",
"name": "fields",
"fields": [
{
"name": "value",
"type": [
"null",
"string"
],
"default": null
}
]
}
}
],
"default": null
}
]
}{code}
I would appreciate it if someone could comment on the overall effort to make
recursive Avro schemas work in Spark and whether or not there are any API
restrictions that could potentially hinder the implementation, thanks.
> Support recursive references in Avro schemas in Spark
> -----------------------------------------------------
>
> Key: SPARK-39802
> URL: https://issues.apache.org/jira/browse/SPARK-39802
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Ivan Sadikov
> Priority: Major
>
> This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
> It would be great if Spark could support recursive references in Avro schemas
> as currently this is disabled and the following exception is thrown:
> {code:java}
> org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive
> reference in Avro schema, which can not be processed by Spark: {
> "type": "record",
> "name": "Struct",
> "fields": [
> {
> "name": "fields",
> "type": [
> "null",
> {
> "type": "array",
> "items": {
> "type": "record",
> "name": "fields",
> "fields": [
> {
> "name": "value",
> "type": [
> "null",
> "string"
> ],
> "default": null
> }
> ]
> }
> }
> ],
> "default": null
> }
> ]
> }{code}
> I would appreciate it if someone could comment on the overall effort to make
> recursive Avro schemas work in Spark and whether or not there are any API
> restrictions that could potentially hinder the implementation, thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
