[
https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17567759#comment-17567759
]
Ivan Sadikov commented on SPARK-39802:
--------------------------------------
[~Gengliang.Wang] Would you be able to comment on this ticket? Thanks.
> Support Avro recursive schemas in Spark
> ---------------------------------------
>
> Key: SPARK-39802
> URL: https://issues.apache.org/jira/browse/SPARK-39802
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Ivan Sadikov
> Priority: Major
>
> This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.
> It would be great if Spark could support recursive Avro schemas as currently
> the following exception is thrown:
> {code:java}
> org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive
> reference in Avro schema, which can not be processed by Spark: {
> "type": "record",
> "name": "Struct",
> "fields": [
> {
> "name": "fields",
> "type": [
> "null",
> {
> "type": "array",
> "items": {
> "type": "record",
> "name": "fields",
> "fields": [
> {
> "name": "value",
> "type": [
> "null",
> "string"
> ],
> "default": null
> }
> ]
> }
> }
> ],
> "default": null
> }
> ]
> }{code}
> I would appreciate it if someone could comment on the overall effort to make
> recursive Avro schemas work in Spark and whether or not there are any API
> restrictions that could potentially hinder the implementation, thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
