Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r223982672
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -169,15 +174,16 @@ object DataSourceV2Relation {
options: Map[String, String],
tableIdent: Option[TableIdentifier] = None,
userSpecifiedSchema: Option[StructType] = None):
DataSourceV2Relation = {
- val reader = source.createReader(options, userSpecifiedSchema)
+ val readSupport = source.createReadSupport(options,
userSpecifiedSchema)
--- End diff --
Looks not directly related with this PR but I think this is a good place to
ask. Why do we make a readsupport in write path?
https://github.com/apache/spark/blob/e06da95cd9423f55cdb154a2778b0bddf7be984c/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L249
Retrieving the physical schema of the underlying storage is potentially
expensive. Actually even worse: it looks odd that write path requires read
side's schema. Which schema should we expect here in write path?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
