[
https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Sadikov updated SPARK-39802:
---------------------------------
Description:
This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
It would be great if Spark could support recursive Avro schemas as currently
the following exception is thrown:
{code:java}
org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive
reference in Avro schema, which can not be processed by Spark: {
"type": "record",
"name": "Struct",
"fields": [
{
"name": "fields",
"type": [
"null",
{
"type": "array",
"items": {
"type": "record",
"name": "fields",
"fields": [
{
"name": "value",
"type": [
"null",
"string"
],
"default": null
}
]
}
}
],
"default": null
}
]
}{code}
I would appreciate it if someone could comment on the overall effort to make
recursive Avro schemas work in Spark and whether or not there are any API
restrictions that could potentially hinder the implementation, thanks
was:
This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
It would be great if Spark could support recursive Avro schemas as currently
the following exception is thrown:
{code:java}
org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive
reference in Avro schema, which can not be processed by Spark: {
"type": "record",
"name": "Struct",
"fields": [
{
"name": "fields",
"type": [
"null",
{
"type": "array",
"items": {
"type": "record",
"name": "fields",
"fields": [
{
"name": "value",
"type": [
"null",
"string"
],
"default": null
}
]
}
}
],
"default": null
}
]
}{code}
I would appreciate it if someone could comment on the overall effort to make
recursive Avro schemas work in Spark, thanks.
> Support Avro recursive schemas in Spark
> ---------------------------------------
>
> Key: SPARK-39802
> URL: https://issues.apache.org/jira/browse/SPARK-39802
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Ivan Sadikov
> Priority: Major
>
> This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
> It would be great if Spark could support recursive Avro schemas as currently
> the following exception is thrown:
> {code:java}
> org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive
> reference in Avro schema, which can not be processed by Spark: {
> "type": "record",
> "name": "Struct",
> "fields": [
> {
> "name": "fields",
> "type": [
> "null",
> {
> "type": "array",
> "items": {
> "type": "record",
> "name": "fields",
> "fields": [
> {
> "name": "value",
> "type": [
> "null",
> "string"
> ],
> "default": null
> }
> ]
> }
> }
> ],
> "default": null
> }
> ]
> }{code}
> I would appreciate it if someone could comment on the overall effort to make
> recursive Avro schemas work in Spark and whether or not there are any API
> restrictions that could potentially hinder the implementation, thanks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
