[ https://issues.apache.org/jira/browse/SPARK-32439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-32439. ---------------------------------- Resolution: Won't Fix > Override datasource implementation during look up via configuration > ------------------------------------------------------------------- > > Key: SPARK-32439 > URL: https://issues.apache.org/jira/browse/SPARK-32439 > Project: Spark > Issue Type: New Feature > Components: SQL > Affects Versions: 3.1.0 > Reporter: Terry Kim > Priority: Minor > > We need a mechanism to override the datasource implementation via > configuration. > For example, suppose I have a custom CSV datasource implementation called > "my_csv". One way to use it is: > {code} > val df = spark.read.format("my_csv").load(...) > {code} > Since the source data is the same format (CSV), you should be able to > override the default implementation. > One proposal is to do the following: > {code} > spark.conf.set("spark.sql.datasource.override.csv", "my_csv") val df = > spark.read.csv(...) > {code} > This has a benefit that the user doesn't have to change the application code > to try out a new datasource implementation for the same source format. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org