cloud-fan commented on a change in pull request #26750: [SPARK-28948][SQL]
Support passing all Table metadata in TableProvider
URL: https://github.com/apache/spark/pull/26750#discussion_r354692676
##########
File path:
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableProvider.java
##########
@@ -36,26 +38,47 @@
public interface TableProvider {
/**
- * Return a {@link Table} instance to do read/write with user-specified
options.
+ * Return a {@link Table} instance with specified table properties to do
read/write.
+ * Implementations should infer the table schema and partitioning.
+ *
+ * @param properties The specified table properties. It's case preserving
(contains exactly what
Review comment:
Yes, it's used as table properties directly, not read/write options. The
read/write options are case-insensitive and passed through
`Table.newScanBuilder`/`Table.newWriteBuilder`.
Table properties doesn't have to be case-insensitive. So here I just define
it as case-preserving. The implementation is free to interpret it
case-sensitive or case-insensitive. (Spark can't control it anyway)
If you read a table with `DataFrameReader`, then the options will be passed
to the data source twice: once with `TableProvider.getTable`, once with
`Table.newScanOptions`.
If you create the table first with `CREATE TABLE USING v2Provider`, and then
read the table with `DataFrameReader`, then table properties and read options
are different.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
