[
https://issues.apache.org/jira/browse/SPARK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606527#comment-16606527
]
Dale Richardson commented on SPARK-22387:
-----------------------------------------
We've missed shared configs like what is required for Kerberos
> propagate session configs to data source read/write options
> -----------------------------------------------------------
>
> Key: SPARK-22387
> URL: https://issues.apache.org/jira/browse/SPARK-22387
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Wenchen Fan
> Assignee: Jiang Xingbo
> Priority: Major
> Fix For: 2.3.0
>
>
> This is an open discussion. The general idea is we should allow users to set
> some common configs in session conf so that they don't need to type them
> again and again for each data source operations.
> Proposal 1:
> propagate every session config which starts with {{spark.datasource.config.}}
> to data source options. The downside is, users may only want to set some
> common configs for a specific data source.
> Proposal 2:
> propagate session config which starts with
> {{spark.datasource.config.myDataSource.}} only to {{myDataSource}}
> operations. One downside is, some data source may not have a short name and
> makes the config key pretty long, e.g.
> {{spark.datasource.config.com.company.foo.bar.key1}}.
> Proposal 3:
> Introduce a trait `WithSessionConfig` which defines session config key
> prefix. Then we can pick session configs with this key-prefix and propagate
> it to this particular data source.
> One another thing also worth to think: sometimes it's really annoying if
> users have a typo in the config key and spend a lot of time to figure out why
> things don't work as expected. We should allow data source to validate the
> given options and throw exception if an option can't be recognized.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
