GitHub user rdblue opened a pull request:

    https://github.com/apache/spark/pull/20603

    [SPARK-23418][SQL]: Fail DataSourceV2 reads when user schema is passed, but 
not supported.

    ## What changes were proposed in this pull request?
    
    DataSourceV2 initially allowed user-supplied schemas when a source doesn't 
implement `ReadSupportWithSchema`, as long as the schema was identical to the 
source's schema. This is confusing behavior because changes to an underlying 
table can cause a previously working job to fail with an exception that 
user-supplied schemas are not allowed.
    
    This reverts commit adcb25a0624, which was added to #20387 so that it could 
be removed in a separate JIRA issue and PR.
    
    ## How was this patch tested?
    
    Existing tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rdblue/spark SPARK-23418-revert-adcb25a0624

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20603.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20603
    
----
commit 8ab22313a30dc86ad6464248f99fa704ec82ef39
Author: Ryan Blue <blue@...>
Date:   2018-01-17T21:58:12Z

    SPARK-22386: DataSourceV2: Use immutable logical plans.

commit 6ac33cd3b3daf37ff46797f43c7227f56cb3dad0
Author: Ryan Blue <blue@...>
Date:   2018-01-24T19:34:42Z

    SPARK-23203: Fix scala style check.

commit 466dfd1e695f346fcf986f8292d8a387ab1fb957
Author: Ryan Blue <blue@...>
Date:   2018-01-24T19:54:10Z

    SPARK-23203: Fix Kafka tests, use StreamingDataSourceV2Relation.
    
    This also removes unused imports.

commit c9e226de93d6a461272f817cbc4f8d5fb703f46d
Author: Ryan Blue <blue@...>
Date:   2018-02-02T20:30:33Z

    SPARK-23204: DataFrameReader: Remove v2 table identifier parsing.

commit 580243fe2d43662015cc9bd4ef715a6b7d162628
Author: Ryan Blue <blue@...>
Date:   2018-02-02T21:48:29Z

    SPARK-23203: Remove import changes from DataSourceV2Utils.

commit b97379d7f57d16aff032fecd1915c32070332b90
Author: Ryan Blue <blue@...>
Date:   2018-02-06T19:21:08Z

    SPARK-23203: Remove TableIdentifier from DataSourceV2Relation.

commit c7326a37e2c737f94bb46b73b6050cfa4fdcebf9
Author: Ryan Blue <blue@...>
Date:   2018-02-07T17:22:10Z

    Remove path from DataSourceV2Relation.

commit df00376f12f5aeb5de40d1c7aa0105bddae12983
Author: Ryan Blue <blue@...>
Date:   2018-02-09T19:09:55Z

    Implement doCanonicalize in DataSourceV2Relation.

commit 98814910b843121fb974ed4cf637292cda41a7f1
Author: Ryan Blue <blue@...>
Date:   2018-02-09T19:12:41Z

    Remove write methods from DataSourceV2Relation.

commit 6e88a9fbb35ed9687e042e643d83e28633406eed
Author: Ryan Blue <blue@...>
Date:   2018-02-09T21:04:16Z

    Remove unnecessary imports.

commit 57e05c2babbcaec3ed3aa69765e1145539879c97
Author: Ryan Blue <blue@...>
Date:   2018-02-09T23:52:42Z

    Add DataSourceV2Relation.create to ensure projection is always set.

commit aaa1e0395dccf69b51ee540173eceddb9cb8e267
Author: Ryan Blue <blue@...>
Date:   2018-02-13T21:24:16Z

    Rename userSchema -> userSpecifiedSchema.

commit adcb25a06240dc413f58b2d1240405b0a5485578
Author: Ryan Blue <blue@...>
Date:   2018-02-13T21:57:42Z

    Allow some user-supplied schemas without ReadSupportWithSchema.
    
    This is going to be removed in a follow-up commit, so it is as
    self-contained as possible. This allows user-supplied schemas when
    ReadSupportWithSchema is not implemented as long as the supplied schema
    is identical to the reader's schema.

commit f623080309dcf75525af6b8369e7427c31589afa
Author: Ryan Blue <blue@...>
Date:   2018-02-13T22:05:30Z

    Revert "Allow some user-supplied schemas without ReadSupportWithSchema."
    
    This reverts commit adcb25a06240dc413f58b2d1240405b0a5485578.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to