cloud-fan commented on a change in pull request #24025: [SPARK-27106][SQL]
merge CaseInsensitiveStringMap and DataSourceOptions
URL: https://github.com/apache/spark/pull/24025#discussion_r263985486
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala
##########
@@ -35,4 +40,18 @@ trait FileDataSourceV2 extends TableProvider with
DataSourceRegister {
def fallBackFileFormat: Class[_ <: FileFormat]
lazy val sparkSession = SparkSession.active
+
+ protected def getPaths(map: CaseInsensitiveStringMap): Seq[String] = {
Review comment:
IIRC there was a proprosal to offically define array options (not sure it's
from @HyukjinKwon or @MaxGekk ). But the problem is, even if we define a
general array option getter in the `CaseInsensitiveStringMap`, we still lack a
general array option builder.
I'd like to leave this as a special logic in file source, and generalize the
array option later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
