[
https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570756#comment-16570756
]
Dongjoon Hyun commented on SPARK-24924:
---------------------------------------
1. Theoretically, Spark 2.4 should handle both Hive tables simultaneously if
the jars co-exist.
2. `ALTER TABLE` is technically possible, but it seems not a good way for users
because `spark.sql.sources.provider` is a Spark-generated metadata.
3. For now, there is another issue with `FileFormat` trait. In Spark 2.4,
SPARK-24691 adds `FileFormat.supportDataType` and uses it to verify data types.
Currently, it's a breaking change because the latest 3rd-party file format like
databricks avro 4.0.0 doesn't have that method. The current Spark 2.4 master
branch raises `java.lang.AbstractMethodError`. I think we had better fix this
in Spark-side for compatibility.
> Add mapping for built-in Avro data source
> -----------------------------------------
>
> Key: SPARK-24924
> URL: https://issues.apache.org/jira/browse/SPARK-24924
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Dongjoon Hyun
> Assignee: Dongjoon Hyun
> Priority: Minor
> Fix For: 2.4.0
>
>
> This issue aims to the followings.
> # Like `com.databricks.spark.csv` mapping, we had better map
> `com.databricks.spark.avro` to built-in Avro data source.
> # Remove incorrect error message, `Please find an Avro package at ...`.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
