cloud-fan commented on a change in pull request #31245:
URL: https://github.com/apache/spark/pull/31245#discussion_r570789600
##########
File path: docs/sql-migration-guide.md
##########
@@ -40,6 +40,10 @@ license: |
- In Spark 3.2, script transform default FIELD DELIMIT is `\u0001` for no
serde mode, serde property `field.delim` is `\t` for Hive serde mode when user
specifies serde. In Spark 3.1 or earlier, the default FIELD DELIMIT is `\t`,
serde property `field.delim` is `\u0001` for Hive serde mode when user
specifies serde.
- In Spark 3.2, the auto-generated `Cast` (such as those added by type
coercion rules) will be stripped when generating column alias names. E.g.,
`sql("SELECT floor(1)").columns` will be `FLOOR(1)` instead of `FLOOR(CAST(1 AS
DOUBLE))`.
+
+ - In Spark 3.2, the output schema of `SHOW TABLES` becomes `namespace:
string, tableName: string, isTemporary: boolean`. In Spark 3.1 or earlier, the
`namespace` field was named `database` for the builtin catalog, and there is no
`isTemporary` field for v2 catalogs. Since Spark 3.2, you can restore the old
schema by setting `spark.sql.legacy.keepCommandOutputSchema` to `true`.
Review comment:
The last sentence can be: `To restore the old schema, you can set ...`
##########
File path: docs/sql-migration-guide.md
##########
@@ -40,6 +40,10 @@ license: |
- In Spark 3.2, script transform default FIELD DELIMIT is `\u0001` for no
serde mode, serde property `field.delim` is `\t` for Hive serde mode when user
specifies serde. In Spark 3.1 or earlier, the default FIELD DELIMIT is `\t`,
serde property `field.delim` is `\u0001` for Hive serde mode when user
specifies serde.
- In Spark 3.2, the auto-generated `Cast` (such as those added by type
coercion rules) will be stripped when generating column alias names. E.g.,
`sql("SELECT floor(1)").columns` will be `FLOOR(1)` instead of `FLOOR(CAST(1 AS
DOUBLE))`.
+
+ - In Spark 3.2, the output schema of `SHOW TABLES` becomes `namespace:
string, tableName: string, isTemporary: boolean`. In Spark 3.1 or earlier, the
`namespace` field was named `database` for the builtin catalog, and there is no
`isTemporary` field for v2 catalogs. Since Spark 3.2, you can restore the old
schema by setting `spark.sql.legacy.keepCommandOutputSchema` to `true`.
+
+ - In Spark 3.2, the output schema of `SHOW TABLE EXTENDED` becomes
`namespace: string, tableName: string, isTemporary: boolean, information:
string`. In Spark 3.1 or earlier, the `namespace` field was named `database`
for the builtin catalog, and no change for the v2 catalogs. Since Spark 3.2,
you can restore the old schema by setting
`spark.sql.legacy.keepCommandOutputSchema` to `true`.
Review comment:
ditto
##########
File path: docs/sql-migration-guide.md
##########
@@ -40,6 +40,10 @@ license: |
- In Spark 3.2, script transform default FIELD DELIMIT is `\u0001` for no
serde mode, serde property `field.delim` is `\t` for Hive serde mode when user
specifies serde. In Spark 3.1 or earlier, the default FIELD DELIMIT is `\t`,
serde property `field.delim` is `\u0001` for Hive serde mode when user
specifies serde.
- In Spark 3.2, the auto-generated `Cast` (such as those added by type
coercion rules) will be stripped when generating column alias names. E.g.,
`sql("SELECT floor(1)").columns` will be `FLOOR(1)` instead of `FLOOR(CAST(1 AS
DOUBLE))`.
+
+ - In Spark 3.2, the output schema of `SHOW TABLES` becomes `namespace:
string, tableName: string, isTemporary: boolean`. In Spark 3.1 or earlier, the
`namespace` field was named `database` for the builtin catalog, and there is no
`isTemporary` field for v2 catalogs. Since Spark 3.2, you can restore the old
schema by setting `spark.sql.legacy.keepCommandOutputSchema` to `true`.
Review comment:
We don't have the problem that config is added after the behavior change
is released.
##########
File path: docs/sql-migration-guide.md
##########
@@ -40,6 +40,10 @@ license: |
- In Spark 3.2, script transform default FIELD DELIMIT is `\u0001` for no
serde mode, serde property `field.delim` is `\t` for Hive serde mode when user
specifies serde. In Spark 3.1 or earlier, the default FIELD DELIMIT is `\t`,
serde property `field.delim` is `\u0001` for Hive serde mode when user
specifies serde.
- In Spark 3.2, the auto-generated `Cast` (such as those added by type
coercion rules) will be stripped when generating column alias names. E.g.,
`sql("SELECT floor(1)").columns` will be `FLOOR(1)` instead of `FLOOR(CAST(1 AS
DOUBLE))`.
+
+ - In Spark 3.2, the output schema of `SHOW TABLES` becomes `namespace:
string, tableName: string, isTemporary: boolean`. In Spark 3.1 or earlier, the
`namespace` field was named `database` for the builtin catalog, and there is no
`isTemporary` field for v2 catalogs. Since Spark 3.2, you can restore the old
schema by setting `spark.sql.legacy.keepCommandOutputSchema` to `true`.
Review comment:
The last sentence can be: `To restore the old schema with the builtin
catalog, you can set ...`
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -825,22 +825,10 @@ case class DescribeColumnCommand(
case class ShowTablesCommand(
databaseName: Option[String],
tableIdentifierPattern: Option[String],
+ override val output: Seq[Attribute] = Seq.empty,
Review comment:
nit: shall we add the new parameter to the end?
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTablesSuite.scala
##########
@@ -119,6 +102,34 @@ trait ShowTablesSuiteBase extends
command.ShowTablesSuiteBase {
assert(errMsg.contains("Database from v1 session catalog is not
specified"))
}
}
+
+ test("SPARK-34157 Unify output of SHOW TABLES and pass output attributes
properly") {
+ withNamespace(s"$catalog.ns") {
+ sql(s"CREATE NAMESPACE $catalog.ns")
+ sql(s"USE $catalog.ns")
+ withTable("tbl") {
+ sql("CREATE TABLE tbl(col1 int, col2 string) USING parquet")
+ checkAnswer(sql("show tables"), Row("ns", "tbl", false))
+ checkAnswer(sql("show tables")
+ .select(col("namespace"), col("tableName"), col("isTemporary")),
Review comment:
instead of using `select`, shall we check `sql("show
tables").schema.fieldNames`?
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTablesSuite.scala
##########
@@ -119,6 +102,34 @@ trait ShowTablesSuiteBase extends
command.ShowTablesSuiteBase {
assert(errMsg.contains("Database from v1 session catalog is not
specified"))
}
}
+
+ test("SPARK-34157 Unify output of SHOW TABLES and pass output attributes
properly") {
+ withNamespace(s"$catalog.ns") {
+ sql(s"CREATE NAMESPACE $catalog.ns")
+ sql(s"USE $catalog.ns")
+ withTable("tbl") {
+ sql("CREATE TABLE tbl(col1 int, col2 string) USING parquet")
+ checkAnswer(sql("show tables"), Row("ns", "tbl", false))
+ checkAnswer(sql("show tables")
+ .select(col("namespace"), col("tableName"), col("isTemporary")),
+ Row("ns", "tbl", false))
+ assert(sql("show table extended like 'tbl'").collect()(0).length == 4)
+ assert(sql("show table extended like 'tbl'").select(col("namespace"),
col("tableName"),
Review comment:
ditto
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTablesSuite.scala
##########
@@ -119,6 +102,34 @@ trait ShowTablesSuiteBase extends
command.ShowTablesSuiteBase {
assert(errMsg.contains("Database from v1 session catalog is not
specified"))
}
}
+
+ test("SPARK-34157 Unify output of SHOW TABLES and pass output attributes
properly") {
+ withNamespace(s"$catalog.ns") {
+ sql(s"CREATE NAMESPACE $catalog.ns")
+ sql(s"USE $catalog.ns")
+ withTable("tbl") {
+ sql("CREATE TABLE tbl(col1 int, col2 string) USING parquet")
+ checkAnswer(sql("show tables"), Row("ns", "tbl", false))
+ checkAnswer(sql("show tables")
+ .select(col("namespace"), col("tableName"), col("isTemporary")),
+ Row("ns", "tbl", false))
+ assert(sql("show table extended like 'tbl'").collect()(0).length == 4)
+ assert(sql("show table extended like 'tbl'").select(col("namespace"),
col("tableName"),
+ col("isTemporary"), col("information")).collect()(0).length == 4)
+
+ // Keep the legacy output schema
+ withSQLConf(SQLConf.LEGACY_KEEP_COMMAND_OUTPUT_SCHEMA.key -> "true") {
+ checkAnswer(sql("show tables"), Row("ns", "tbl", false))
+ checkAnswer(sql("show tables")
+ .select(col("database"), col("tableName"), col("isTemporary")),
Review comment:
ditto
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTablesSuite.scala
##########
@@ -119,6 +102,34 @@ trait ShowTablesSuiteBase extends
command.ShowTablesSuiteBase {
assert(errMsg.contains("Database from v1 session catalog is not
specified"))
}
}
+
+ test("SPARK-34157 Unify output of SHOW TABLES and pass output attributes
properly") {
+ withNamespace(s"$catalog.ns") {
+ sql(s"CREATE NAMESPACE $catalog.ns")
+ sql(s"USE $catalog.ns")
+ withTable("tbl") {
+ sql("CREATE TABLE tbl(col1 int, col2 string) USING parquet")
+ checkAnswer(sql("show tables"), Row("ns", "tbl", false))
+ checkAnswer(sql("show tables")
+ .select(col("namespace"), col("tableName"), col("isTemporary")),
+ Row("ns", "tbl", false))
+ assert(sql("show table extended like 'tbl'").collect()(0).length == 4)
+ assert(sql("show table extended like 'tbl'").select(col("namespace"),
col("tableName"),
+ col("isTemporary"), col("information")).collect()(0).length == 4)
+
+ // Keep the legacy output schema
+ withSQLConf(SQLConf.LEGACY_KEEP_COMMAND_OUTPUT_SCHEMA.key -> "true") {
+ checkAnswer(sql("show tables"), Row("ns", "tbl", false))
+ checkAnswer(sql("show tables")
+ .select(col("database"), col("tableName"), col("isTemporary")),
+ Row("ns", "tbl", false))
+ assert(sql("show table extended like 'tbl'").collect()(0).length ==
4)
+ assert(sql("show table extended like 'tbl'").select(col("database"),
col("tableName"),
+ col("isTemporary"), col("information")).collect()(0).length == 4)
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
