Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22413#discussion_r217736452
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -241,10 +241,12 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
val source = cls.newInstance().asInstanceOf[DataSourceV2]
source match {
case provider: BatchWriteSupportProvider =>
- val options = extraOptions ++
- DataSourceV2Utils.extractSessionConfigs(source,
df.sparkSession.sessionState.conf)
+ val sessionOptions = DataSourceV2Utils.extractSessionConfigs(
+ source,
+ df.sparkSession.sessionState.conf)
+ val options = sessionOptions ++ extraOptions
- val relation = DataSourceV2Relation.create(source, options.toMap)
+ val relation = DataSourceV2Relation.create(source, options)
--- End diff --
I wrote a test for read path since I was able to grab options propagated to
DataSource but I have no idea so far for write path, only mocking probably.
Does it make sense to do that?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
