szehon-ho opened a new pull request, #59036: URL: https://github.com/apache/spark/pull/59036
### What changes were proposed in this pull request? Add `Table.displayProperties()` for additional display-only table properties, defaulting to an empty map. The map and its keys and values must be non-null. Spark merges these entries with non-reserved stored properties for V2 `DESCRIBE TABLE EXTENDED`, `SHOW TBLPROPERTIES`, and `SHOW TABLE EXTENDED`. Stored values take precedence, and reserved keys in the display map are ignored. Table type, location, provider, owner, and other reserved metadata continue to come from `properties()`. Support metadata-only tables through `TableInfo.Builder.withDisplayProperties()` and `DelegatingTable`. Keep `SHOW CREATE TABLE` on `properties()`. The API documentation also guides `createTableLike` implementations to exclude display-only entries when copying source metadata, notes that implementations may compute them on demand, and explains that Spark redacts them like stored properties. ### Why are the changes needed? Catalogs may expose transient metadata such as labels that should be visible when inspecting a table without appearing in the DDL used to recreate it. The additive API lets connectors supply these entries separately from stored properties. JIRA: https://issues.apache.org/jira/browse/SPARK-59778 ### Does this PR introduce _any_ user-facing change? Yes. Table implementations can opt in to display-only properties. Existing connectors retain their behavior because the new method defaults to an empty map. Mockito mocks that only stub `properties()` also retain their existing results. `spark.catalog.getTableProperties()` remains equivalent to `SHOW TBLPROPERTIES`, including in PySpark and Spark Connect, so it also returns display-only entries. The Scala, Connect Scala, and Python API documentation explains that these entries may be transient and are excluded from the DDL returned by `getCreateTableString()`. `ALTER TABLE ... SET/UNSET TBLPROPERTIES` forwards changes to the catalog without consulting display properties. The API documentation guides catalogs to apply these changes to stored properties and keep display-only entries unchanged. After a colliding stored non-reserved property is unset, the display value is shown if it remains in the display map. Reserved display keys, including `comment`, remain ignored. The shared display helper applies the existing redaction rules and reuses the stored property map fetched for reserved metadata. `TableInfo` carries display properties as returned metadata. Spark leaves them empty when calling the table creation and staging methods. V2 views retain their existing behavior. The SQL references for `SHOW TBLPROPERTIES` and `ALTER TABLE ... UNSET TBLPROPERTIES` explain display-only entries and stored-property precedence. For now, `SHOW TABLE EXTENDED` omits display-only entries for the built-in session catalog and its `CatalogExtension`s. `DESCRIBE TABLE EXTENDED` and `SHOW TBLPROPERTIES` include them when an extension returns a V2 table. ### How was this patch tested? For this revision, all 34 selected tests passed using `SPARK_LOCAL_IP=localhost build/sbt`: - Catalyst `CatalogV2UtilSuite`: 24 tests. - Catalyst `CatalogSuite -- -z "display properties"`: 2 tests. - V2 `DescribeTableSuite`, `ShowTablesSuite`, `ShowTblPropertiesSuite`, `DataSourceV2MetadataTableSuite`, and `DataSourceV2DataFrameSessionCatalogSuite`, filtered by `"display properties"`: 7 tests. - V2 `ShowCreateTableSuite -- -z "display-only properties"`: 1 test. The added assertions cover reserved-key lookup and the session-catalog `SHOW TABLE EXTENDED` limitation. The helper redaction test checks that mutable input maps remain unchanged. External-table fixtures now use `CREATE EXTERNAL TABLE`, and display fixture setup uses the underlying live table. Catalyst Java Checkstyle, Catalyst/SQL test Scalastyle, and `git diff --check` also passed for this revision. The previous revision passed all 214 selected tests; one pre-existing test was ignored: - Catalyst `CatalogSuite` and `CatalogV2UtilSuite`: 103 tests. - V2 `DescribeTableSuite`, `ShowTblPropertiesSuite`, `ShowTablesSuite`, and `ShowCreateTableSuite`, plus `DataSourceV2MetadataTableSuite`: 100 tests. - `DataSourceV2DataFrameSessionCatalogSuite -- -z "display properties"`: 1 test. - `org.apache.spark.sql.internal.CatalogSuite -- -z "catalog API"`: 10 tests. Ran with `SPARK_LOCAL_IP=localhost build/sbt`, using `catalyst/testOnly` and `sql/testOnly` for the suites above. Coverage includes stored/display collisions, SET and UNSET, every reserved display key with and without a stored location, redaction, metadata-only tables, exclusion from generated DDL, and preservation across copies, pinning, and ALTER TABLE in both V2 and session-catalog fixtures. The previous revision also passed Catalyst Java Checkstyle, Catalyst/SQL main and test Scalastyle, and `git diff --check`. SQL API/Connect Common scalafmt validation and Python syntax checks passed in an earlier revision; those files are unchanged in this update. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
