This is an automated email from the ASF dual-hosted git repository.
szehon-ho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 918645bce996 [SPARK-59778][SQL] Add display-only properties to the
Table API
918645bce996 is described below
commit 918645bce996329a1c840361f16a54d2c747eac1
Author: Szehon Ho <[email protected]>
AuthorDate: Mon Oct 26 13:29:56 2026 -0700
[SPARK-59778][SQL] Add display-only properties to the Table API
### What changes were proposed in this pull request?
Add `Table.displayProperties()` for additional display-only table
properties, defaulting to an empty map. The map and its keys and values must be
non-null. Spark merges these entries with non-reserved stored properties for V2
`DESCRIBE TABLE EXTENDED`, `SHOW TBLPROPERTIES`, and `SHOW TABLE EXTENDED`.
Stored values take precedence, and reserved keys in the display map are
ignored. Table type, location, provider, owner, and other reserved metadata
continue to come from `properties()`.
Support metadata-only tables through
`TableInfo.Builder.withDisplayProperties()` and `DelegatingTable`. Keep `SHOW
CREATE TABLE` on `properties()`. The API documentation also guides
`createTableLike` implementations to exclude display-only entries when copying
source metadata, notes that implementations may compute them on demand, and
explains that Spark redacts them like stored properties.
### Why are the changes needed?
Catalogs may expose transient metadata such as labels that should be
visible when inspecting a table without appearing in the DDL used to recreate
it. The additive API lets connectors supply these entries separately from
stored properties.
JIRA: https://issues.apache.org/jira/browse/SPARK-59778
### Does this PR introduce _any_ user-facing change?
Yes. Table implementations can opt in to display-only properties. Existing
connectors retain their behavior because the new method defaults to an empty
map. Mockito mocks that only stub `properties()` also retain their existing
results.
`spark.catalog.getTableProperties()` remains equivalent to `SHOW
TBLPROPERTIES`, including in PySpark and Spark Connect, so it also returns
display-only entries. The Scala, Connect Scala, and Python API documentation
explains that these entries may be transient and are excluded from the DDL
returned by `getCreateTableString()`.
`ALTER TABLE ... SET/UNSET TBLPROPERTIES` forwards changes to the catalog
without consulting display properties. The API documentation guides catalogs to
apply these changes to stored properties and keep display-only entries
unchanged. After a colliding stored non-reserved property is unset, the display
value is shown if it remains in the display map. Reserved display keys,
including `comment`, remain ignored. The shared display helper applies the
existing redaction rules and reuses t [...]
`TableInfo` carries display properties as returned metadata. Spark leaves
them empty when calling the table creation and staging methods. V2 views retain
their existing behavior. The SQL references for `SHOW TBLPROPERTIES` and `ALTER
TABLE ... UNSET TBLPROPERTIES` explain display-only entries and stored-property
precedence.
For now, `SHOW TABLE EXTENDED` omits display-only entries for the built-in
session catalog and its `CatalogExtension`s. `DESCRIBE TABLE EXTENDED` and
`SHOW TBLPROPERTIES` include them when an extension returns a V2 table.
### How was this patch tested?
For this revision, all 34 selected tests passed using
`SPARK_LOCAL_IP=localhost build/sbt`:
- Catalyst `CatalogV2UtilSuite`: 24 tests.
- Catalyst `CatalogSuite -- -z "display properties"`: 2 tests.
- V2 `DescribeTableSuite`, `ShowTablesSuite`, `ShowTblPropertiesSuite`,
`DataSourceV2MetadataTableSuite`, and
`DataSourceV2DataFrameSessionCatalogSuite`, filtered by `"display properties"`:
7 tests.
- V2 `ShowCreateTableSuite -- -z "display-only properties"`: 1 test.
The added assertions cover reserved-key lookup and the session-catalog
`SHOW TABLE EXTENDED` limitation. The helper redaction test checks that mutable
input maps remain unchanged. External-table fixtures now use `CREATE EXTERNAL
TABLE`, and display fixture setup uses the underlying live table.
Catalyst Java Checkstyle, Catalyst/SQL test Scalastyle, and `git diff
--check` also passed for this revision.
The previous revision passed all 214 selected tests; one pre-existing test
was ignored:
- Catalyst `CatalogSuite` and `CatalogV2UtilSuite`: 103 tests.
- V2 `DescribeTableSuite`, `ShowTblPropertiesSuite`, `ShowTablesSuite`, and
`ShowCreateTableSuite`, plus `DataSourceV2MetadataTableSuite`: 100 tests.
- `DataSourceV2DataFrameSessionCatalogSuite -- -z "display properties"`: 1
test.
- `org.apache.spark.sql.internal.CatalogSuite -- -z "catalog API"`: 10
tests.
Ran with `SPARK_LOCAL_IP=localhost build/sbt`, using `catalyst/testOnly`
and `sql/testOnly` for the suites above. Coverage includes stored/display
collisions, SET and UNSET, every reserved display key with and without a stored
location, redaction, metadata-only tables, exclusion from generated DDL, and
preservation across copies, pinning, and ALTER TABLE in both V2 and
session-catalog fixtures.
The previous revision also passed Catalyst Java Checkstyle, Catalyst/SQL
main and test Scalastyle, and `git diff --check`. SQL API/Connect Common
scalafmt validation and Python syntax checks passed in an earlier revision;
those files are unchanged in this update.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)
Closes #59036 from szehon-ho/codex/table-display-properties.
Authored-by: Szehon Ho <[email protected]>
Signed-off-by: Szehon Ho <[email protected]>
---
docs/sql-ref-syntax-aux-show-tblproperties.md | 8 ++++
docs/sql-ref-syntax-ddl-alter-table.md | 7 +++
python/pyspark/sql/catalog.py | 6 +++
.../org/apache/spark/sql/catalog/Catalog.scala | 3 ++
.../sql/connector/catalog/DelegatingTable.java | 5 ++
.../apache/spark/sql/connector/catalog/Table.java | 32 +++++++++++++
.../spark/sql/connector/catalog/TableInfo.java | 35 +++++++++++++-
.../sql/connector/catalog/CatalogV2Util.scala | 10 ++++
.../spark/sql/connector/catalog/CatalogSuite.scala | 53 ++++++++++++++++++++++
.../sql/connector/catalog/CatalogV2UtilSuite.scala | 40 ++++++++++++++++
.../sql/connector/catalog/InMemoryBaseTable.scala | 16 ++++++-
.../catalog/InMemoryRowLevelOperationTable.scala | 2 +-
.../InMemoryRowLevelOperationTableCatalog.scala | 4 +-
.../sql/connector/catalog/InMemoryTable.scala | 2 +-
.../connector/catalog/InMemoryTableCatalog.scala | 3 +-
.../catalog/MixedColumnIdTableCatalog.scala | 4 +-
.../catalog/NullColumnIdInMemoryTableCatalog.scala | 4 +-
...ableIdAndNullColumnIdInMemoryTableCatalog.scala | 4 +-
.../TypeChangeResetsColIdTableCatalog.scala | 2 +-
.../org/apache/spark/sql/connect/Catalog.scala | 3 ++
.../datasources/v2/DescribeTableExec.scala | 12 ++---
.../datasources/v2/ShowTablePropertiesExec.scala | 6 +--
.../datasources/v2/ShowTablesExtendedExec.scala | 16 ++++---
.../DataSourceV2DataFrameSessionCatalogSuite.scala | 25 +++++++++-
.../connector/DataSourceV2MetadataTableSuite.scala | 50 ++++++++++++++++++--
.../execution/command/v2/CommandSuiteBase.scala | 31 +++++++++++++
.../execution/command/v2/DescribeTableSuite.scala | 27 ++++++++++-
.../command/v2/ShowCreateTableSuite.scala | 15 ++++++
.../sql/execution/command/v2/ShowTablesSuite.scala | 28 ++++++++++++
.../command/v2/ShowTblPropertiesSuite.scala | 39 ++++++++++++++++
.../apache/spark/sql/internal/CatalogSuite.scala | 28 +++++++++++-
31 files changed, 481 insertions(+), 39 deletions(-)
diff --git a/docs/sql-ref-syntax-aux-show-tblproperties.md
b/docs/sql-ref-syntax-aux-show-tblproperties.md
index 5b7ddcbcd953..cf2186c7998e 100644
--- a/docs/sql-ref-syntax-aux-show-tblproperties.md
+++ b/docs/sql-ref-syntax-aux-show-tblproperties.md
@@ -24,6 +24,14 @@ license: |
This statement returns the value of a table property given an optional value
for
a property key. If no key is specified then all the properties are returned.
+For tables in [Data Source V2](sql-v2-data-sources.html) catalogs, the output
may also include
+display-only properties supplied by the catalog. Spark itself does not remove
these entries on
+`ALTER TABLE ... UNSET TBLPROPERTIES`; whether they remain depends on the
catalog.
+`SHOW CREATE TABLE` does not include them. Stored properties take precedence
over display-only
+entries with the same key. After unsetting a stored property other than
`comment`, Spark displays
+the display-only value with that key if the catalog still supplies it.
Display-only `comment`
+entries are ignored, even after unsetting the stored `comment`.
+
### Syntax
```sql
diff --git a/docs/sql-ref-syntax-ddl-alter-table.md
b/docs/sql-ref-syntax-ddl-alter-table.md
index 28ecc44a5bf7..156a279da909 100644
--- a/docs/sql-ref-syntax-ddl-alter-table.md
+++ b/docs/sql-ref-syntax-ddl-alter-table.md
@@ -278,6 +278,13 @@ ALTER TABLE table_identifier SET TBLPROPERTIES ( key1 =
val1, key2 = val2, ... )
`ALTER TABLE UNSET` command is used to drop the table property.
+For tables in [Data Source V2](sql-v2-data-sources.html) catalogs, catalogs
may supply display-only
+properties in metadata commands such as `SHOW TBLPROPERTIES`. Spark itself
does not remove these
+entries on `ALTER TABLE ... UNSET TBLPROPERTIES`; whether they remain depends
on the catalog.
+After unsetting a stored property other than `comment`, Spark displays the
display-only value
+with that key if the catalog still supplies it. Display-only `comment` entries
are ignored,
+even after unsetting the stored `comment`.
+
**Note:** If the specified property key does not exist, whether specify `IF
EXISTS` or not, the command will ignore it and finally succeed.
##### Syntax
diff --git a/python/pyspark/sql/catalog.py b/python/pyspark/sql/catalog.py
index ed7d6ac7482b..6d0b3d6e5b7e 100644
--- a/python/pyspark/sql/catalog.py
+++ b/python/pyspark/sql/catalog.py
@@ -393,6 +393,12 @@ class Catalog:
dict
Map of property key to value.
+ Notes
+ -----
+ For V2 tables, the returned dict also includes display-only
properties, which may be
+ transient. These entries are excluded from the table DDL returned by
+ :meth:`getCreateTableString`.
+
Examples
--------
>>> _ = spark.sql("DROP TABLE IF EXISTS tbl_props_doc")
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala
index 1bb88be33d8e..836eebb0695b 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala
@@ -846,6 +846,9 @@ abstract class Catalog {
/**
* Returns all table properties as a map (same as `SHOW TBLPROPERTIES`).
*
+ * For V2 tables, the returned map also includes display-only properties,
which may be
+ * transient. These entries are excluded from the table DDL returned by
`getCreateTableString`.
+ *
* @param tableName
* table or view name; may be qualified with catalog and database
(namespace).
* @since 4.2.0
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/DelegatingTable.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/DelegatingTable.java
index 2e8d63eb16d9..af215c350460 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/DelegatingTable.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/DelegatingTable.java
@@ -74,6 +74,11 @@ public class DelegatingTable implements Table {
return Collections.unmodifiableMap(info.properties());
}
+ @Override
+ public Map<String, String> displayProperties() {
+ return Collections.unmodifiableMap(info.displayProperties());
+ }
+
@Override
public Transform[] partitioning() {
return info.partitions();
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Table.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Table.java
index ec27bcf6c82e..2a9299d297c0 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Table.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Table.java
@@ -95,6 +95,38 @@ public interface Table extends Relation {
return Collections.emptyMap();
}
+ /**
+ * Returns additional, display-only table properties.
+ *
+ * <p>Spark adds these entries to the non-reserved properties displayed by
V2 table metadata
+ * commands and {@code Catalog.getTableProperties}. Entries in {@link
#properties()} take
+ * precedence, and reserved keys such as {@link TableCatalog#PROP_EXTERNAL},
+ * {@link TableCatalog#PROP_LOCATION}, and {@link TableCatalog#PROP_COMMENT}
are ignored in
+ * this map. Table type and other reserved metadata are read from {@code
properties()}.
+ * After a stored non-reserved property is removed, Spark displays the entry
for that key if
+ * it remains in this map. Spark redacts these entries in the same way as
stored properties.
+ *
+ * <p>Spark passes {@code ALTER TABLE ... SET/UNSET TBLPROPERTIES} to
+ * {@link TableCatalog#alterTable} without consulting this map. Catalogs
should apply these
+ * changes to {@code properties()} and keep display-only entries unchanged.
+ *
+ * <p>For now, {@code SHOW TABLE EXTENDED} omits these entries for the
built-in session catalog
+ * and its {@link CatalogExtension}s.
+ *
+ * <p>These entries may be transient and are excluded from {@code SHOW
CREATE TABLE}.
+ * Implementations of {@link TableCatalog#createTableLike(Identifier,
TableInfo, Table)} should
+ * not copy them from the source table. Spark reads these entries only to
display table
+ * metadata, so implementations may compute them on demand.
+ *
+ * <p>The default implementation returns an empty map.
+ *
+ * @return a non-null map with non-null keys and values
+ * @since 4.4.0
+ */
+ default Map<String, String> displayProperties() {
+ return Collections.emptyMap();
+ }
+
/**
* Returns the set of capabilities for this table.
*/
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableInfo.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableInfo.java
index 1a3261e3085a..72f3586d2f6a 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableInfo.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableInfo.java
@@ -16,6 +16,7 @@
*/
package org.apache.spark.sql.connector.catalog;
+import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
@@ -24,7 +25,8 @@ import org.apache.spark.sql.connector.expressions.Transform;
import org.apache.spark.sql.types.StructType;
/**
- * Metadata describing a data-source table: its columns, properties,
partitioning and constraints.
+ * Metadata describing a data-source table: its columns, stored and
display-only properties,
+ * partitioning and constraints.
* Spark realizes a {@code TableInfo} into a {@link Table} via {@link
DelegatingTable}; a catalog
* that has its own {@link Table} object returns that instead. Views are
described by the sibling
* {@link View}, which -- unlike a table -- is itself a {@link Relation}
because Spark never builds
@@ -34,6 +36,7 @@ public class TableInfo {
private final Column[] columns;
private final Map<String, String> properties;
+ private final Map<String, String> displayProperties;
private final Transform[] partitions;
private final Constraint[] constraints;
@@ -43,6 +46,7 @@ public class TableInfo {
protected TableInfo(Builder builder) {
this.columns = builder.columns;
this.properties = builder.properties;
+ this.displayProperties = builder.displayProperties;
this.partitions = builder.partitions;
this.constraints = builder.constraints;
}
@@ -59,6 +63,22 @@ public class TableInfo {
return properties;
}
+ /**
+ * Returns the additional display-only properties described by {@link
Table#displayProperties()}.
+ *
+ * <p>These properties describe metadata returned by a catalog. Spark leaves
them empty when
+ * calling {@link TableCatalog#createTable(Identifier, TableInfo)},
+ * {@link TableCatalog#createTableLike(Identifier, TableInfo, Table)},
+ * {@link StagingTableCatalog#stageCreate(Identifier, TableInfo)},
+ * {@link StagingTableCatalog#stageReplace(Identifier, TableInfo)}, and
+ * {@link StagingTableCatalog#stageCreateOrReplace(Identifier, TableInfo)}.
+ *
+ * @since 4.4.0
+ */
+ public Map<String, String> displayProperties() {
+ return displayProperties;
+ }
+
public Transform[] partitions() {
return partitions;
}
@@ -66,12 +86,25 @@ public class TableInfo {
public Constraint[] constraints() { return constraints; }
public static class Builder extends RelationBuilder<Builder> {
+ protected Map<String, String> displayProperties = new HashMap<>();
protected Transform[] partitions = new Transform[0];
protected Constraint[] constraints = new Constraint[0];
@Override
protected Builder self() { return this; }
+ /**
+ * Sets additional display-only properties.
+ *
+ * @param displayProperties a non-null map with non-null keys and values
+ * @see TableInfo#displayProperties()
+ * @since 4.4.0
+ */
+ public Builder withDisplayProperties(Map<String, String>
displayProperties) {
+ this.displayProperties = new HashMap<>(displayProperties);
+ return this;
+ }
+
public Builder withPartitions(Transform[] partitions) {
this.partitions = partitions;
return this;
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
index de170375effb..68ddd2b34360 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
@@ -692,6 +692,16 @@ private[sql] object CatalogV2Util {
builder
}
+ /** Returns redacted, non-reserved table properties with additional
display-only metadata. */
+ def tablePropertiesForDisplay(
+ properties: util.Map[String, String],
+ displayProperties: util.Map[String, String],
+ conf: SQLConf): Map[String, String] = {
+ val merged = (displayProperties.asScala.toMap ++ properties.asScala)
+ .filterNot { case (key, _) => TABLE_RESERVED_PROPERTIES.contains(key) }
+ conf.redactOptions(merged)
+ }
+
def convertTableProperties(t: TableSpec): Map[String, String] = {
val props = convertTableProperties(
t.properties, t.options, t.serde, t.location, t.comment,
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogSuite.scala
index 8c59b2ef4598..9a784f3c383e 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogSuite.scala
@@ -77,6 +77,59 @@ class CatalogSuite extends SparkFunSuite {
assert(loaded.getClass == catalog.getClass)
}
+ test("TableInfo and DelegatingTable carry display-only properties
separately") {
+ val displayProperties = new util.HashMap[String, String]()
+ displayProperties.put("catalog-label", "catalog-value")
+ val info = new TableInfo.Builder()
+ .withProperties(util.Map.of("persisted", "stored"))
+ .withDisplayProperties(displayProperties)
+ .build()
+ displayProperties.put("catalog-label", "changed")
+ val table = new DelegatingTable(info, "table")
+
+ assert(table.displayProperties() === util.Map.of("catalog-label",
"catalog-value"))
+ assert(table.properties() === util.Map.of("persisted", "stored"))
+ intercept[UnsupportedOperationException] {
+ table.displayProperties().put("catalog-label", "changed")
+ }
+ val defaultTable = new DelegatingTable(new TableInfo.Builder().build(),
"default")
+ assert(defaultTable.displayProperties().isEmpty)
+ }
+
+ gridTest("display properties survive copies, pinning and alterations")(
+ Seq("InMemoryTableCatalog", "InMemoryRowLevelOperationTableCatalog")) {
catalogName =>
+ val catalog = if (catalogName == "InMemoryRowLevelOperationTableCatalog") {
+ new InMemoryRowLevelOperationTableCatalog
+ } else {
+ new InMemoryTableCatalog
+ }
+ // The row-level catalog always copies on load; enable this for
InMemoryTableCatalog too.
+ catalog.initialize("test", new
CaseInsensitiveStringMap(util.Map.of("copyOnLoad", "true")))
+ val table = catalog.createTable(testIdent, new TableInfo.Builder()
+ .withColumns(columns)
+ .withProperties(util.Map.of("persisted", "stored"))
+ .build()).asInstanceOf[InMemoryBaseTable]
+ table.setDisplayProperties(util.Map.of("catalog-label", "catalog-value"))
+ table.setVersion("3")
+ table.setValidatedVersion("2")
+ val loaded = catalog.loadTable(testIdent).asInstanceOf[InMemoryBaseTable]
+ assert(loaded ne table)
+ assert(loaded.displayProperties() === util.Map.of("catalog-label",
"catalog-value"))
+ assert(loaded.version() === "3")
+ assert(loaded.validatedVersion() === "2")
+ catalog.pinTable(testIdent, "snapshot")
+
+ val altered = catalog.alterTable(testIdent,
TableChange.setProperty("persisted", "updated"))
+ .asInstanceOf[InMemoryBaseTable]
+ assert(altered.validatedVersion() === "2")
+ val current = catalog.loadTable(testIdent)
+ assert(current.properties() === util.Map.of("persisted", "updated"))
+ assert(current.displayProperties() === util.Map.of("catalog-label",
"catalog-value"))
+ val snapshot = catalog.loadTable(testIdent, "snapshot")
+ assert(snapshot.properties() === util.Map.of("persisted", "stored"))
+ assert(snapshot.displayProperties() === util.Map.of("catalog-label",
"catalog-value"))
+ }
+
test("listTables") {
val catalog = newCatalog()
val ident1 = Identifier.of(Array("ns"), "test_table_1")
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogV2UtilSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogV2UtilSuite.scala
index acbc2aefe212..d0891e86c38d 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogV2UtilSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogV2UtilSuite.scala
@@ -35,6 +35,46 @@ import org.apache.spark.sql.util.CaseInsensitiveStringMap
class CatalogV2UtilSuite extends SparkFunSuite {
+ test("tablePropertiesForDisplay preserves stored properties with no display
properties") {
+ val properties = java.util.Map.of(
+ "persisted", "stored", TableCatalog.PROP_EXTERNAL, "true")
+
+ assert(CatalogV2Util.tablePropertiesForDisplay(
+ properties, java.util.Map.of[String, String](), new SQLConf) ===
+ Map("persisted" -> "stored"))
+ }
+
+ test("tablePropertiesForDisplay ignores reserved display keys and preserves
stored values") {
+ val displayProperties = new java.util.HashMap[String, String]()
+ CatalogV2Util.TABLE_RESERVED_PROPERTIES.foreach(displayProperties.put(_,
"display-value"))
+ displayProperties.put("persisted", "display-value")
+ displayProperties.put("catalog-label", "catalog-value")
+ val properties = java.util.Map.of("persisted", "stored")
+
+ assert(CatalogV2Util.tablePropertiesForDisplay(
+ properties, displayProperties, new SQLConf) ===
+ Map("persisted" -> "stored", "catalog-label" -> "catalog-value"))
+ }
+
+ test("tablePropertiesForDisplay redacts stored and display-only values") {
+ val conf = new SQLConf
+ conf.setConf(SQLConf.SQL_OPTIONS_REDACTION_PATTERN, "(?i)catalog-label".r)
+ val properties = new java.util.HashMap[String, String](
+ java.util.Map.of("password", "stored-password", "persisted", "stored"))
+ val displayProperties = new java.util.HashMap[String,
String](java.util.Map.of(
+ "catalog-label", "catalog-value", "secret", "display-secret",
"persisted", "display-value"))
+
+ assert(CatalogV2Util.tablePropertiesForDisplay(properties,
displayProperties, conf) === Map(
+ "password" -> "*********(redacted)",
+ "catalog-label" -> "*********(redacted)",
+ "secret" -> "*********(redacted)",
+ "persisted" -> "stored"))
+ assert(properties === java.util.Map.of(
+ "password", "stored-password", "persisted", "stored"))
+ assert(displayProperties === java.util.Map.of(
+ "catalog-label", "catalog-value", "secret", "display-secret",
"persisted", "display-value"))
+ }
+
private def catalogWithStateOptions(keys: java.util.Set[String]):
TableCatalog = {
val catalog = mock(classOf[TableCatalog])
when(catalog.tableStateOptionKeys()).thenReturn(keys)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala
index adf1e655ab10..08ce1090dae5 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala
@@ -76,6 +76,8 @@ abstract class InMemoryBaseTable(
// Stores the table version validated during the last `ALTER TABLE ... ADD
CONSTRAINT` operation.
private var validatedTableVersion: String = null
+ private var tableDisplayProperties: util.Map[String, String] = null
+
// Assign column IDs to columns that do not have one, including nested
struct fields within
// arrays and maps. This simulates connectors that support column identity
tracking.
private var tableColumns: Array[Column] =
InMemoryBaseTable.assignMissingIds(initialColumns)
@@ -86,6 +88,15 @@ abstract class InMemoryBaseTable(
tableColumns = newColumns
}
+ override def displayProperties(): util.Map[String, String] = {
+ Option(tableDisplayProperties).map(util.Collections.unmodifiableMap(_))
+ .getOrElse(super.displayProperties())
+ }
+
+ def setDisplayProperties(displayProperties: util.Map[String, String]): Unit
= {
+ tableDisplayProperties = new util.HashMap[String,
String](displayProperties)
+ }
+
override def version(): String = tableVersion.toString
def setVersion(version: String): Unit = {
@@ -93,7 +104,7 @@ abstract class InMemoryBaseTable(
}
/**
- * Copies version and validated version from another table.
+ * Copies version, validated version and display-only metadata from another
table.
*
* Some test catalogs (e.g. [[NullColumnIdInMemoryTableCatalog]],
* [[NullTableIdAndNullColumnIdInMemoryTableCatalog]]) create a new table
object
@@ -104,7 +115,8 @@ abstract class InMemoryBaseTable(
* [[InMemoryTable]].copy, validated-version propagation, and the
join-refresh
* tests in [[DSv2IncrementallyConstructedQueryTests]]).
*/
- def setVersionAndValidatedVersionFrom(sourceTable: InMemoryBaseTable): Unit
= {
+ def copyTableStateFrom(sourceTable: InMemoryBaseTable): Unit = {
+ setDisplayProperties(sourceTable.displayProperties())
setVersion(sourceTable.version())
if (sourceTable.validatedVersion() != null) {
setValidatedVersion(sourceTable.validatedVersion())
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTable.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTable.scala
index 3e04ab719b7b..7fb11ea8be26 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTable.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTable.scala
@@ -124,7 +124,7 @@ class InMemoryRowLevelOperationTable private (
}
}
copied.commits ++= commits.map(_.copy())
- copied.setVersionAndValidatedVersionFrom(this)
+ copied.copyTableStateFrom(this)
copied.replacedPartitions = replacedPartitions
copied.lastWriteInfo = lastWriteInfo
copied.lastWriteLog = lastWriteLog
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTableCatalog.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTableCatalog.scala
index 5c9060fcca6c..15260d6e1b28 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTableCatalog.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTableCatalog.scala
@@ -100,7 +100,7 @@ class InMemoryRowLevelOperationTableCatalog
tableId = table.id,
schemaAlignmentConfig = table.schemaAlignmentConfig)
newTable.alterTableWithData(table.data, schema)
- newTable.setVersionAndValidatedVersionFrom(table)
+ newTable.copyTableStateFrom(table)
tables.put(ident, newTable)
@@ -143,7 +143,7 @@ class PartialSchemaEvolutionCatalog extends
InMemoryRowLevelOperationTableCatalo
constraints = table.constraints,
schemaAlignmentConfig = table.schemaAlignmentConfig)
newTable.alterTableWithData(table.data, table.schema)
- newTable.setVersionAndValidatedVersionFrom(table)
+ newTable.copyTableStateFrom(table)
tables.put(ident, newTable)
newTable
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala
index 20ce2af03ac6..e5657f9d88bf 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala
@@ -161,7 +161,7 @@ class InMemoryTable(
copiedTable.commits ++= commits.map(_.copy())
- copiedTable.setVersionAndValidatedVersionFrom(this)
+ copiedTable.copyTableStateFrom(this)
copiedTable
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableCatalog.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableCatalog.scala
index dc9253e42540..50930031d2b3 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableCatalog.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableCatalog.scala
@@ -263,7 +263,6 @@ class BasicInMemoryTableCatalog extends TableCatalog {
}
table.increaseVersion()
- val currentVersion = table.version()
val columnsWithIds = InMemoryBaseTable.assignMissingIds(
CatalogV2Util.structTypeToV2Columns(schema))
val reconstructedId =
Option(table.id()).getOrElse(util.UUID.randomUUID().toString)
@@ -278,7 +277,7 @@ class BasicInMemoryTableCatalog extends TableCatalog {
throw new UnsupportedOperationException(
s"Unsupported InMemoryBaseTable subclass: ${other.getClass.getName}")
}
- newTable.setVersion(currentVersion)
+ newTable.copyTableStateFrom(table)
changes.foreach {
case a: TableChange.AddConstraint =>
newTable.setValidatedVersion(a.validatedTableVersion())
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/MixedColumnIdTableCatalog.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/MixedColumnIdTableCatalog.scala
index f77cad3c077d..4c38aa0dcf92 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/MixedColumnIdTableCatalog.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/MixedColumnIdTableCatalog.scala
@@ -50,7 +50,7 @@ class MixedColumnIdTableCatalog extends InMemoryTableCatalog {
id = table.id,
nullIdNames = snapshot)
mixedTable.alterTableWithData(table.data, table.schema)
- mixedTable.setVersionAndValidatedVersionFrom(table)
+ mixedTable.copyTableStateFrom(table)
mixedTable
}
@@ -121,7 +121,7 @@ class MixedColumnIdInMemoryTable(
dataMap.synchronized {
copiedTable.alterTableWithData(data, schema)
}
- copiedTable.setVersionAndValidatedVersionFrom(this)
+ copiedTable.copyTableStateFrom(this)
copiedTable
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/NullColumnIdInMemoryTableCatalog.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/NullColumnIdInMemoryTableCatalog.scala
index fead2e6fba79..0185a6b0a893 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/NullColumnIdInMemoryTableCatalog.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/NullColumnIdInMemoryTableCatalog.scala
@@ -45,7 +45,7 @@ class NullColumnIdInMemoryTableCatalog extends
InMemoryTableCatalog {
constraints = table.constraints,
id = table.id)
nullColIdTable.alterTableWithData(table.data, table.schema)
- nullColIdTable.setVersionAndValidatedVersionFrom(table)
+ nullColIdTable.copyTableStateFrom(table)
nullColIdTable
}
@@ -101,7 +101,7 @@ class NullColumnIdInMemoryTable(
dataMap.synchronized {
copiedTable.alterTableWithData(data, schema)
}
- copiedTable.setVersionAndValidatedVersionFrom(this)
+ copiedTable.copyTableStateFrom(this)
copiedTable
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/NullTableIdAndNullColumnIdInMemoryTableCatalog.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/NullTableIdAndNullColumnIdInMemoryTableCatalog.scala
index c1d45e42fc46..09d7036956b6 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/NullTableIdAndNullColumnIdInMemoryTableCatalog.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/NullTableIdAndNullColumnIdInMemoryTableCatalog.scala
@@ -41,7 +41,7 @@ class NullTableIdAndNullColumnIdInMemoryTableCatalog extends
InMemoryTableCatalo
properties = table.properties,
constraints = table.constraints)
nullTable.alterTableWithData(table.data, table.schema)
- nullTable.setVersionAndValidatedVersionFrom(table)
+ nullTable.copyTableStateFrom(table)
nullTable
}
@@ -94,7 +94,7 @@ class NullTableIdAndNullColumnIdInMemoryTable(
dataMap.synchronized {
copiedTable.alterTableWithData(data, schema)
}
- copiedTable.setVersionAndValidatedVersionFrom(this)
+ copiedTable.copyTableStateFrom(this)
copiedTable
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/TypeChangeResetsColIdTableCatalog.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/TypeChangeResetsColIdTableCatalog.scala
index 83effcef2b26..09b057d0f7a0 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/TypeChangeResetsColIdTableCatalog.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/TypeChangeResetsColIdTableCatalog.scala
@@ -62,7 +62,7 @@ class TypeChangeResetsColIdTableCatalog extends
InMemoryTableCatalog {
alteredTable.constraints,
id = alteredTable.id)
tableWithResetIds.alterTableWithData(alteredTable.data,
alteredTable.schema)
- tableWithResetIds.setVersionAndValidatedVersionFrom(alteredTable)
+ tableWithResetIds.copyTableStateFrom(alteredTable)
tables.put(ident, tableWithResetIds)
tableWithResetIds
}
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Catalog.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Catalog.scala
index ce7a10c4026c..807f9a1523f6 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Catalog.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Catalog.scala
@@ -853,6 +853,9 @@ class Catalog(sparkSession: SparkSession) extends
catalog.Catalog {
/**
* Returns all table properties as a map (same as `SHOW TBLPROPERTIES`).
*
+ * For V2 tables, the returned map also includes display-only properties,
which may be
+ * transient. These entries are excluded from the table DDL returned by
`getCreateTableString`.
+ *
* @param tableName
* qualified or unqualified table or view name
* @since 4.2.0
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
index 68f4620eaf0d..d35aeb32cdbc 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
@@ -18,7 +18,6 @@
package org.apache.spark.sql.execution.datasources.v2
import scala.collection.mutable.ArrayBuffer
-import scala.jdk.CollectionConverters._
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.catalog.{CatalogTableType, ClusterBySpec}
@@ -189,7 +188,8 @@ case class DescribeTableExec(
rows += toCatalystRow("# Detailed Table Information", "", "")
addIdentifierRows(rows, catalogName, identifier, entityLabel = "Table")
- val tableType = if
(table.properties().containsKey(TableCatalog.PROP_EXTERNAL)) {
+ val tableProperties = table.properties()
+ val tableType = if
(tableProperties.containsKey(TableCatalog.PROP_EXTERNAL)) {
CatalogTableType.EXTERNAL.name
} else {
CatalogTableType.MANAGED.name
@@ -198,13 +198,13 @@ case class DescribeTableExec(
CatalogV2Util.TABLE_RESERVED_PROPERTIES
.filterNot(_ == TableCatalog.PROP_EXTERNAL)
.foreach(propKey => {
- if (table.properties.containsKey(propKey)) {
- rows += toCatalystRow(propKey.capitalize,
table.properties.get(propKey), "")
+ if (tableProperties.containsKey(propKey)) {
+ rows += toCatalystRow(propKey.capitalize,
tableProperties.get(propKey), "")
}
})
val properties =
- conf.redactOptions(table.properties.asScala.toMap).toList
- .filter(kv => !CatalogV2Util.TABLE_RESERVED_PROPERTIES.contains(kv._1))
+ CatalogV2Util.tablePropertiesForDisplay(
+ tableProperties, table.displayProperties(), conf).toList
.sortBy(_._1).map {
case (key, value) => key + "=" + value
}.mkString("[", ",", "]")
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablePropertiesExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablePropertiesExec.scala
index 3a957e043393..b561f98825b1 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablePropertiesExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablePropertiesExec.scala
@@ -31,11 +31,9 @@ case class ShowTablePropertiesExec(
propertyKey: Option[String]) extends LeafV2CommandExec {
override protected def run(): Seq[InternalRow] = {
- import scala.jdk.CollectionConverters._
-
// The reserved properties are accessible through DESCRIBE
- val properties = conf.redactOptions(catalogTable.properties.asScala.toMap)
- .filter { case (k, _) =>
!CatalogV2Util.TABLE_RESERVED_PROPERTIES.contains(k) }
+ val properties = CatalogV2Util.tablePropertiesForDisplay(
+ catalogTable.properties(), catalogTable.displayProperties(), conf)
propertyKey match {
case Some(p) =>
val propValue = properties
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExtendedExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExtendedExec.scala
index f9ca033310c7..3dbe3644dd90 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExtendedExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExtendedExec.scala
@@ -82,7 +82,8 @@ case class ShowTablesExtendedExec(
results.put("Catalog", catalogName)
results.put("Namespace", identifier.namespace().quoted)
results.put("Table", identifier.name())
- val tableType = if
(table.properties().containsKey(TableCatalog.PROP_EXTERNAL)) {
+ val tableProperties = table.properties()
+ val tableType = if
(tableProperties.containsKey(TableCatalog.PROP_EXTERNAL)) {
CatalogTableType.EXTERNAL
} else {
CatalogTableType.MANAGED
@@ -92,15 +93,16 @@ case class ShowTablesExtendedExec(
CatalogV2Util.TABLE_RESERVED_PROPERTIES
.filterNot(_ == TableCatalog.PROP_EXTERNAL)
.foreach(propKey => {
- if (table.properties.containsKey(propKey)) {
- results.put(propKey.capitalize, table.properties.get(propKey))
+ if (tableProperties.containsKey(propKey)) {
+ results.put(propKey.capitalize, tableProperties.get(propKey))
}
})
- val displayedTableProperties =
conf.redactOptions(table.properties.asScala.toMap).toList
- .filterNot { case (key, _) =>
CatalogV2Util.TABLE_RESERVED_PROPERTIES.contains(key) }
- .sortBy(_._1)
- .map { case (key, value) => key + "=" + value }
+ val displayedTableProperties =
+ CatalogV2Util.tablePropertiesForDisplay(
+ tableProperties, table.displayProperties(), conf).toList
+ .sortBy(_._1)
+ .map { case (key, value) => key + "=" + value }
if (displayedTableProperties.nonEmpty) {
results.put("Table Properties", displayedTableProperties.mkString("[",
", ", "]"))
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala
index d10460181329..881baf2fb5b3 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala
@@ -21,7 +21,7 @@ import java.util.Locale
import org.scalatest.BeforeAndAfter
-import org.apache.spark.sql.{AnalysisException, DataFrame, SaveMode}
+import org.apache.spark.sql.{AnalysisException, DataFrame, Row, SaveMode}
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
import org.apache.spark.sql.connector.catalog._
@@ -53,6 +53,28 @@ class DataSourceV2DataFrameSessionCatalogSuite
override protected val catalogAndNamespace: String = ""
+ test("display properties survive ALTER TABLE in the session catalog") {
+ val t = "display_properties"
+ withTable(t) {
+ sql(s"CREATE TABLE $t (id bigint) USING $v2Format")
+ val table = sessionCatalog.loadTable(Identifier.of(Array("default"), t))
+ .asInstanceOf[InMemoryTable]
+ table.setDisplayProperties(java.util.Map.of("catalog-label",
"catalog-value"))
+
+ sql(s"ALTER TABLE $t SET TBLPROPERTIES ('k' = 'v')")
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $t"),
+ Seq(Row("catalog-label", "catalog-value"), Row("k", "v")))
+ checkAnswer(sql(s"DESCRIBE TABLE EXTENDED $t").where("col_name = 'Table
Properties'"),
+ Row("Table Properties", "[catalog-label=catalog-value,k=v]", ""))
+
+ assert(!sql(s"SHOW TABLE EXTENDED LIKE '$t'")
+ .head().getString(3).contains("catalog-label"))
+
+ sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('k')")
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $t"), Row("catalog-label",
"catalog-value"))
+ }
+ }
+
test("saveAsTable: Append mode should not fail if the table already exists "
+
"and a same-name temp view exist") {
withTable("same_name") {
@@ -177,6 +199,7 @@ class InMemoryTableSessionCatalog extends
TestV2SessionCatalogBase[InMemoryTable
val newTable = new InMemoryTable(table.name, schema,
table.partitioning, properties)
.alterTableWithData(table.data, schema)
+ newTable.setDisplayProperties(table.displayProperties())
tables.put(ident, newTable)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataTableSuite.scala
index 3a7e5322d620..b3b2f2d1efea 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataTableSuite.scala
@@ -17,6 +17,8 @@
package org.apache.spark.sql.connector
+import java.util
+
import org.apache.spark.SparkConf
import org.apache.spark.sql.Row
import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
@@ -39,6 +41,34 @@ class DataSourceV2MetadataTableSuite extends
SharedSparkSession {
"spark.sql.catalog.table_catalog",
classOf[TestingDataSourceTableCatalog].getName)
+ test("metadata commands include display properties from DelegatingTable") {
+ val tableName = "table_catalog.display_properties.test_display_properties"
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tableName"),
+ Seq(Row("catalog-label", "catalog-value"), Row("persisted", "stored")))
+ checkAnswer(
+ sql(s"DESCRIBE TABLE EXTENDED $tableName").where("col_name = 'Table
Properties'"),
+ Row("Table Properties",
"[catalog-label=catalog-value,persisted=stored]", ""))
+ checkAnswer(
+ sql("SHOW TABLE EXTENDED IN table_catalog.display_properties " +
+ "LIKE 'test_display_properties'"),
+ Row("display_properties", "test_display_properties", false,
+ """Catalog: table_catalog
+ |Namespace: display_properties
+ |Table: test_display_properties
+ |Type: MANAGED
+ |Provider: parquet
+ |Table Properties: [catalog-label=catalog-value, persisted=stored]
+ |Schema: root
+ | |-- id: long (nullable = true)
+ |
+ |""".stripMargin))
+ assert(spark.catalog.getTableProperties(tableName) ===
+ util.Map.of("catalog-label", "catalog-value", "persisted", "stored"))
+ val ddl = spark.catalog.getCreateTableString(tableName)
+ assert(ddl.contains("'persisted' = 'stored'"))
+ assert(!ddl.contains("catalog-label"))
+ }
+
test("file source table") {
withTempPath { path =>
val loc = path.getCanonicalPath
@@ -113,10 +143,21 @@ class DataSourceV2MetadataTableSuite extends
SharedSparkSession {
/**
* A read-only [[TableCatalog]] that returns [[DelegatingTable]] for a small
set of canned
* table fixtures. Used to drive the data-source-table read path (file source
+ v2 provider)
- * through Spark's V1 data-source machinery.
+ * through Spark's V1 data-source machinery. The table
+ * `display_properties.test_display_properties` backs the V2 metadata-command
test and is the
+ * only table returned by `listTables`.
*/
class TestingDataSourceTableCatalog extends TableCatalog {
override def loadTable(ident: Identifier): Table = ident.name() match {
+ case "test_display_properties" =>
+ val info = new TableInfo.Builder()
+ .withSchema(new StructType().add("id", "long"))
+ .withProperties(util.Map.of("persisted", "stored"))
+ .withProvider("parquet")
+ .withDisplayProperties(util.Map.of(
+ "catalog-label", "catalog-value", "persisted", "display-value"))
+ .build()
+ new DelegatingTable(info, ident.toString)
case "test_json" =>
val info = new TableInfo.Builder()
.withSchema(new StructType().add("col", "string"))
@@ -152,8 +193,11 @@ class TestingDataSourceTableCatalog extends TableCatalog {
throw new RuntimeException("shouldn't be called")
override def renameTable(oldIdent: Identifier, newIdent: Identifier): Unit =
throw new RuntimeException("shouldn't be called")
- override def listTables(namespace: Array[String]): Array[Identifier] =
- throw new RuntimeException("shouldn't be called")
+ override def listTables(namespace: Array[String]): Array[Identifier] =
namespace match {
+ case Array("display_properties") =>
+ Array(Identifier.of(namespace, "test_display_properties"))
+ case _ => throw new RuntimeException("shouldn't be called")
+ }
private var catalogName = ""
override def initialize(name: String, options: CaseInsensitiveStringMap):
Unit = {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CommandSuiteBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CommandSuiteBase.scala
index 548e15f9e783..123575eb2dd7 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CommandSuiteBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CommandSuiteBase.scala
@@ -17,6 +17,8 @@
package org.apache.spark.sql.execution.command.v2
+import scala.jdk.CollectionConverters._
+
import org.apache.spark.SparkConf
import org.apache.spark.sql.catalyst.analysis.ResolvePartitionSpec
import org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
@@ -77,4 +79,33 @@ trait CommandSuiteBase extends SharedSparkSession {
catalogPlugin.asTableCatalog.loadTable(Identifier.of(Array(schema), table))
.asInstanceOf[InMemoryTable]
}
+
+ def createTableWithDisplayProperties(
+ namespace: String,
+ table: String,
+ external: Boolean = false,
+ withLocation: Boolean = false): Unit = {
+ val location = if (withLocation) "LOCATION 'file:/tmp/display-properties'
" else ""
+ val tableType = if (external) "EXTERNAL " else ""
+ sql(s"CREATE ${tableType}TABLE $catalog.$namespace.$table (id bigint)
$defaultUsing " +
+ s"COMMENT 'table comment' $location" +
+ "TBLPROPERTIES ('persisted' = 'stored')")
+ val tableCatalog = spark.sessionState.catalogManager.catalog(catalog)
+ .asInstanceOf[BasicInMemoryTableCatalog]
+ val liveTable = tableCatalog.liveTable(Identifier.of(Array(namespace),
table))
+ .asInstanceOf[InMemoryTable]
+ setDisplayProperties(liveTable, external)
+ }
+
+ /**
+ * Sets display-only test metadata.
+ */
+ def setDisplayProperties(table: InMemoryTable, external: Boolean = false):
Unit = {
+ val reserved = CatalogV2Util.TABLE_RESERVED_PROPERTIES.map(key => key ->
s"display-$key")
+ table.setDisplayProperties((reserved.toMap ++ Map(
+ "catalog-label" -> "catalog-value",
+ "password" -> "display-password",
+ "persisted" -> "display-value",
+ TableCatalog.PROP_EXTERNAL -> (!external).toString)).asJava)
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DescribeTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DescribeTableSuite.scala
index 2e8fe7a4c023..b2385236e9b2 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DescribeTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DescribeTableSuite.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.execution.command.v2
import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
-import org.apache.spark.sql.connector.catalog.TableCatalog
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, TableCatalog}
import org.apache.spark.sql.execution.command
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.StringType
@@ -220,6 +220,31 @@ class DescribeTableSuite extends
command.DescribeTableSuiteBase
}
}
+ gridTest("display properties preserve table type and reserved metadata")(
+ Seq((false, false), (true, true))) {
+ case (external, withLocation) =>
+ withNamespaceAndTable("ns", "table") { tbl =>
+ createTableWithDisplayProperties("ns", "table", external, withLocation)
+
+ val reservedRows =
CatalogV2Util.TABLE_RESERVED_PROPERTIES.map(_.capitalize)
+ checkAnswer(
+ sql(s"DESCRIBE TABLE EXTENDED $tbl").where(
+ $"col_name".isin((reservedRows ++ Seq("Type", "Table
Properties")): _*)),
+ Seq(
+ Row("Type", if (external) "EXTERNAL" else "MANAGED", ""),
+ Row("Comment", "table comment", ""),
+ Row("Provider", "_", ""),
+ Row("Owner", Utils.getCurrentUserName(), ""),
+ Row("Table Properties",
+
"[catalog-label=catalog-value,password=*********(redacted),persisted=stored]",
"")) ++
+ (if (withLocation) {
+ Seq(Row("Location", "file:/tmp/display-properties", ""))
+ } else {
+ Nil
+ }))
+ }
+ }
+
test("DESCRIBE TABLE EXTENDED emits structured Catalog/Namespace/Table
rows") {
// Pin that DescribeTableExec emits the resolved-identifier components as
separate
// rows under the `# Detailed Table Information` block, so consumers can
read each
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala
index 961ea612e3ae..425b359036d4 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala
@@ -33,6 +33,21 @@ class ShowCreateTableSuite extends
command.ShowCreateTableSuiteBase with Command
override def fullName: String = s"$catalog.$ns.$table"
+ test("show create table excludes display-only properties") {
+ withNamespaceAndTable(ns, table) { t =>
+ createTableWithDisplayProperties(ns, table, withLocation = true)
+
+ assert(getShowCreateDDL(t, false) === Array(
+ s"CREATE TABLE $t (",
+ "id BIGINT)",
+ defaultUsing,
+ "COMMENT 'table comment'",
+ "LOCATION 'file:/tmp/display-properties'",
+ "TBLPROPERTIES (",
+ "'persisted' = 'stored')"))
+ }
+ }
+
test("SPARK-33898: show create table as serde") {
withNamespaceAndTable(ns, table) { t =>
spark.sql(s"CREATE TABLE $t (id bigint, data string) $defaultUsing")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTablesSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTablesSuite.scala
index e0888fee6ce2..91f71b75160e 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTablesSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTablesSuite.scala
@@ -69,6 +69,34 @@ class ShowTablesSuite extends command.ShowTablesSuiteBase
with CommandSuiteBase
}
}
+ gridTest("show table extended preserves reserved metadata with display
properties")(
+ Seq((false, false), (true, true))) {
+ case (external, withLocation) =>
+ withNamespaceAndTable("ns", "tbl") { _ =>
+ createTableWithDisplayProperties("ns", "tbl", external, withLocation)
+
+ val information = sql(s"SHOW TABLE EXTENDED IN $catalog.ns LIKE 'tbl'")
+ .head().getString(3)
+ val tableType = if (external) "EXTERNAL" else "MANAGED"
+ val locationRow = if (withLocation) "Location:
file:/tmp/display-properties\n" else ""
+ val properties =
+ "[catalog-label=catalog-value, password=*********(redacted),
persisted=stored]"
+ assert(information ===
+ s"""Catalog: $catalog
+ |Namespace: ns
+ |Table: tbl
+ |Type: $tableType
+ |Comment: table comment
+ |${locationRow}Provider: _
+ |Owner: ${Utils.getCurrentUserName()}
+ |Table Properties: $properties
+ |Schema: root
+ | |-- id: long (nullable = true)
+ |
+ |""".stripMargin)
+ }
+ }
+
test("show table extended omits empty table properties") {
withNamespaceAndTable("ns", "tbl") { t =>
sql(s"CREATE TABLE $t (id bigint) $defaultUsing")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTblPropertiesSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTblPropertiesSuite.scala
index ace1e30a26b2..b96dc8415a5e 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTblPropertiesSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTblPropertiesSuite.scala
@@ -17,10 +17,49 @@
package org.apache.spark.sql.execution.command.v2
+import org.apache.spark.sql.Row
import org.apache.spark.sql.execution.command
/**
* The class contains tests for the `SHOW TBLPROPERTIES` command to check V2
table catalogs.
*/
class ShowTblPropertiesSuite extends command.ShowTblPropertiesSuiteBase with
CommandSuiteBase {
+ test("display properties supplement current properties and exclude reserved
keys") {
+ withNamespaceAndTable("ns", "table") { tbl =>
+ createTableWithDisplayProperties("ns", "table")
+
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl"),
+ Seq(Row("catalog-label", "catalog-value"), Row("password",
"*********(redacted)"),
+ Row("persisted", "stored")))
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl ('persisted')"),
Row("persisted", "stored"))
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl ('catalog-label')"),
+ Row("catalog-label", "catalog-value"))
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl ('password')"),
+ Row("password", "*********(redacted)"))
+
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl ('location')"),
+ Row("location", s"Table $tbl does not have property: location"))
+
+ sql(s"ALTER TABLE $tbl SET TBLPROPERTIES ('persisted' = 'updated')")
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl"),
+ Seq(Row("catalog-label", "catalog-value"), Row("password",
"*********(redacted)"),
+ Row("persisted", "updated")))
+
+ sql(s"ALTER TABLE $tbl UNSET TBLPROPERTIES ('persisted',
'catalog-label')")
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl"),
+ Seq(Row("catalog-label", "catalog-value"), Row("password",
"*********(redacted)"),
+ Row("persisted", "display-value")))
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl ('persisted')"),
+ Row("persisted", "display-value"))
+ assert(spark.catalog.getTableProperties(tbl).get("persisted") ===
"display-value")
+ assert(!spark.catalog.getCreateTableString(tbl).contains("persisted"))
+
+ sql(s"ALTER TABLE $tbl SET TBLPROPERTIES ('catalog-label' = 'updated')")
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl ('catalog-label')"),
+ Row("catalog-label", "updated"))
+ sql(s"ALTER TABLE $tbl UNSET TBLPROPERTIES ('catalog-label')")
+ checkAnswer(sql(s"SHOW TBLPROPERTIES $tbl ('catalog-label')"),
+ Row("catalog-label", "catalog-value"))
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala
index 40184a16a658..762fd0cd7a65 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala
@@ -34,7 +34,7 @@ import
org.apache.spark.sql.catalyst.expressions.GenericInternalRow
import org.apache.spark.sql.catalyst.plans.logical.Range
import org.apache.spark.sql.classic.Catalog
import org.apache.spark.sql.connector.{FakeV2Provider,
InMemoryTableSessionCatalog}
-import org.apache.spark.sql.connector.catalog.{CatalogManager, CatalogV2Util,
Identifier, InMemoryCatalog}
+import org.apache.spark.sql.connector.catalog.{CatalogManager, CatalogV2Util,
Identifier, InMemoryBaseTable, InMemoryCatalog, TableCatalog}
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.CatalogHelper
import org.apache.spark.sql.connector.catalog.functions._
import org.apache.spark.sql.test.SharedSparkSession
@@ -1201,6 +1201,32 @@ class CatalogSuite extends SharedSparkSession with
AnalysisTest with BeforeAndAf
spark.catalog.dropTable(t)
}
+ test("catalog API: getTableProperties includes V2 display-only properties") {
+ val t = "testcat.ns.table"
+ withTable(t) {
+ sql(s"CREATE TABLE $t (id INT) USING _ TBLPROPERTIES ('persisted' =
'stored')")
+ val catalog =
spark.sessionState.catalogManager.catalog("testcat").asTableCatalog
+ val table = catalog.loadTable(Identifier.of(Array("ns"), "table"))
+ .asInstanceOf[InMemoryBaseTable]
+ table.setDisplayProperties(Map(
+ "catalog-label" -> "catalog-value",
+ "password" -> "display-password",
+ "persisted" -> "display-value",
+ TableCatalog.PROP_EXTERNAL -> "true",
+ TableCatalog.PROP_COMMENT -> "display comment").asJava)
+
+ assert(spark.catalog.getTableProperties(t).asScala.toMap ===
+ Map("catalog-label" -> "catalog-value", "password" ->
"*********(redacted)",
+ "persisted" -> "stored"))
+ assert(spark.catalog.getTable(t).tableType === "MANAGED")
+ assert(spark.catalog.getTable(t).description == null)
+ val ddl = spark.catalog.getCreateTableString(t)
+ assert(ddl.contains("'persisted' = 'stored'"))
+ assert(!ddl.contains("catalog-label"))
+ assert(!ddl.contains("password"))
+ }
+ }
+
test("catalog API: getCreateTableString") {
val t = "catalog_api_ext_ddl_t"
spark.sql(s"DROP TABLE IF EXISTS $t")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
