GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/14075
Data Source API: Extend RelationProvider and CreatableRelationProvider
without Extending SchemaRelationProvider
#### What changes were proposed in this pull request?
When users try to implement a data source API with extending only
`RelationProvider` and `CreatableRelationProvider`, they will hit an error when
resolving the relation.
```
spark.read
.format("org.apache.spark.sql.test.DefaultSourceWithoutUserSpecifiedSchema")
.load()
.write.
format("org.apache.spark.sql.test.DefaultSourceWithoutUserSpecifiedSchema")
.save()
```
The error they hit is like
```
xyzDataSource does not allow user-specified schemas.;
org.apache.spark.sql.AnalysisException: xyzDataSource does not allow
user-specified schemas.;
at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:319)
at
org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:494)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
```
Actually, the bug fix is simple.
`DataSource.createRelation(sparkSession.sqlContext, mode, options, data)`
already returns a BaseRelation. We should not assign schema to
`userSpecifiedSchema`. That schema assignment only makes sense for the data
sources that extend `FileFormat`.
#### How was this patch tested?
Added a test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark dataSource
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14075.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14075
----
commit dd644f8117e889cebd6caca58702a7c7e3d88bef
Author: gatorsmile <[email protected]>
Date: 2016-07-06T19:43:43Z
fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
