cloud-fan commented on issue #24233: [SPARK-26356][SQL] remove SaveMode from data source v2 URL: https://github.com/apache/spark/pull/24233#issuecomment-495131562 Let me quote something from your meeting notes: > PR #24233: Remove SaveMode > Ryan: This PR has a hack that passes SaveMode directly to file sources. Because the behavior of v1 is unknown, this should not be included in a release. > Wenchen: This is needed in order to use v2 file sources. Intent is migrate the file source to provide a reference implementation. Without the hack, v2 sources can’t be used (by default) because we know it [v2] is broken [i.e., can’t run CTAS]. > Ryan: v2 isn’t broken. The only problem is path-based tables because we don’t know how they should behave [Gengliang is looking into this]. Catalog tables aren’t broken so we should focus on getting that support done. > Russel/Matt: is this needed for current tests to pass? > Wenchen: no, tests run using v1. > Russel: +1 for not including the hack. > Ryan: Also, tests for the new plans will require an in-memory source that will be a cleaner reference implementation. At that time, I didn't realize that we do have test cases relying on file source v2 writing. That's why I think it's better to do it in another PR and update the tests. I've created a ticket to track it: https://issues.apache.org/jira/browse/SPARK-27815
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
