cloud-fan commented on a change in pull request #23383: [SPARK-23817][SQL]
Migrate ORC file format read path to data source V2
URL: https://github.com/apache/spark/pull/23383#discussion_r246346765
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
##########
@@ -90,8 +93,18 @@ case class DataSource(
case class SourceInfo(name: String, schema: StructType, partitionColumns:
Seq[String])
- lazy val providingClass: Class[_] =
- DataSource.lookupDataSource(className, sparkSession.sessionState.conf)
+ lazy val providingClass: Class[_] = {
+ val cls = DataSource.lookupDataSource(className,
sparkSession.sessionState.conf)
+ // `providingClass` is used for resolving data source relation for catalog
tables.
+ // As now catalog for data source V2 is under development, here we fall
back all the
+ // [[FileDataSourceV2]] to [[FileFormat]] to guarantee the current catalog
works.
+ // [[FileDataSourceV2]] will still be used if we call the load()/save()
method in
+ // [[DataFrameReader]]/[[DataFrameWriter]].
Review comment:
..., because they call lookupDataSource directly instead of providingClass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
