rdblue commented on issue #25291: [SPARK-28554][SQL] implement basic catalog functionalities for JDBC v2 with a DS v1 fallback API URL: https://github.com/apache/spark/pull/25291#issuecomment-518351200 > have a special version of the v2 Table interface (DataSourceV1Table), which can return a v1 BaseRelation [CatalogTableAsV2](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala#L175) wraps a `CatalogTable` so that it can be converted to an `UnresolvedCatalogRelation`. Why is a second way to do this necessary? > have a special version of the v2 TableCatalog interface (DataSourceV1TableCatalog), which can return a DataSourceV1Table Any catalog can return a table instance with this type, if it is public. Why does this require changing the TableCatalog interface? > The v1 analyzer rule catches v2 plans and converts them to v1 plans, if the table is DataSourceV1Table I don't think v2 plans should be converted to v1 plans. As we discussed in the last sync, v1 read and write interfaces can be called from v2, but v2 should always provide v2 behavior guarantees. Converting back to v1 plans is not the right way to add compatibility.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
