[
https://issues.apache.org/jira/browse/SPARK-24855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-24855.
----------------------------------
Resolution: Fixed
Resolving as Fixed: this change was merged to master. Implementing commit:
0cea9e3cd0a9 [SPARK-24855][SQL][EXTERNAL] Built-in AVRO support should support
specified schema on write. (Triaged as a stranded ticket — the fix landed but
the JIRA was left unresolved. Verified present on current master and not
reverted.)
> Built-in AVRO support should support specified schema on write
> --------------------------------------------------------------
>
> Key: SPARK-24855
> URL: https://issues.apache.org/jira/browse/SPARK-24855
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Brian Lindblom
> Assignee: Brian Lindblom
> Priority: Minor
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> spark-avro appears to have been brought in from an upstream project,
> [https://github.com/databricks/spark-avro.] I opened a PR a while ago to
> enable support for 'forceSchema', which allows us to specify an AVRO schema
> with which to write our records to handle some use cases we have. I didn't
> get this code merged but would like to add this feature to the AVRO
> reader/writer code that was brought in. The PR is here and I will follow up
> with a more formal PR/Patch rebased on spark master branch:
> https://github.com/databricks/spark-avro/pull/222
>
> This change allows us to specify a schema, which should be compatible with
> the schema generated by spark-avro from the dataset definition. This allows
> a user to do things like specify default values, change union ordering, or...
> in the case where you're reading in an AVRO data set, doing some sort of
> in-line field cleansing, then writing out with the original schema, preserve
> that original schema in the output container files. I've had several use
> cases where this behavior was desired and there were several other asks for
> this in the spark-avro project.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
