[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r322740354
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala
##
@@ -178,6 +178,24 @@ case class DataSourceResolution(
val aliased = delete.tableAlias.map(SubqueryAlias(_,
relation)).getOrElse(relation)
DeleteFromTable(aliased, delete.condition)
+case ShowNamespacesStatement(None, pattern) =>
+ defaultCatalog match {
Review comment:
@imback82 are you working on it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r322737478
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala
##
@@ -178,6 +178,24 @@ case class DataSourceResolution(
val aliased = delete.tableAlias.map(SubqueryAlias(_,
relation)).getOrElse(relation)
DeleteFromTable(aliased, delete.condition)
+case ShowNamespacesStatement(None, pattern) =>
+ defaultCatalog match {
Review comment:
Let's implement switching the current catalog first, otherwise we are not
able to test it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r322536288
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala
##
@@ -178,6 +178,27 @@ case class DataSourceResolution(
val aliased = delete.tableAlias.map(SubqueryAlias(_,
relation)).getOrElse(relation)
DeleteFromTable(aliased, delete.condition)
+case ShowNamespacesStatement(None, pattern) =>
+ defaultCatalog match {
+case Some(catalog: SupportsNamespaces) =>
+ ShowNamespaces(catalog, None, pattern)
+case Some(_) =>
+ throw new AnalysisException(
+"The default v2 catalog doesn't support showing namespaces.")
+case None =>
+ throw new AnalysisException("No default v2 catalog is set.")
+ }
+
+case ShowNamespacesStatement(Some(namespace), pattern) =>
+ val CatalogNamespace(maybeCatalog, ns) = namespace
+ maybeCatalog match {
+case Some(catalog: SupportsNamespaces) =>
+ ShowNamespaces(catalog, Some(ns), pattern)
+case _ =>
+ throw new AnalysisException(
Review comment:
If the catalog name is specified, but catalog doesn't support namespace, I
think we should fail instead of falling back to the current catalog.
It's similar to: if the catalog name is specified, but doesn't contain the
table we need, we should fail instead of falling back to the current catalog.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r322535605
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala
##
@@ -178,6 +178,24 @@ case class DataSourceResolution(
val aliased = delete.tableAlias.map(SubqueryAlias(_,
relation)).getOrElse(relation)
DeleteFromTable(aliased, delete.condition)
+case ShowNamespacesStatement(None, pattern) =>
+ defaultCatalog match {
Review comment:
yea
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320746204
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/InMemoryTableCatalog.scala
##
@@ -30,12 +30,9 @@ import org.apache.spark.sql.sources.v2.Table
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.util.CaseInsensitiveStringMap
-class InMemoryTableCatalog extends TableCatalog with SupportsNamespaces {
+class InMemoryTableCatalogBase extends TableCatalog {
Review comment:
maybe `BasicInMemoryTableCatalog`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320746067
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/InMemoryTableCatalog.scala
##
@@ -30,12 +30,9 @@ import org.apache.spark.sql.sources.v2.Table
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.util.CaseInsensitiveStringMap
-class InMemoryTableCatalog extends TableCatalog with SupportsNamespaces {
+class InMemoryTableCatalogBase extends TableCatalog {
Review comment:
fine to me
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#discussion_r320744597 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/sql/ShowNamespacesStatement.scala ## @@ -0,0 +1,24 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.catalyst.plans.logical.sql + +/** + * A SHOW NAMESPACES statement, as parsed from SQL. + */ +case class ShowNamespacesStatement(namespace: Option[Seq[String]], pattern: Option[String]) Review comment: ditto This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320744482
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -560,6 +560,17 @@ object OverwritePartitionsDynamic {
}
}
+/**
+ * The logical plan of the SHOW NAMESPACES command that works for v2 catalogs.
+ */
+case class ShowNamespaces(
+catalog: SupportsNamespaces,
+namespace: Option[Seq[String]],
Review comment:
After reading the code, it's actually `catalogAndNamespace`, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320744055
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
##
@@ -747,6 +747,81 @@ class DataSourceV2SQLSuite
assert(expected === df.collect())
}
+ test("ShowNamespaces: show root namespaces with default v2 catalog") {
+spark.conf.set("spark.sql.default.catalog", "testcat")
+
+runShowDatabasesSql("SHOW NAMESPACES", Seq())
+
+spark.sql("CREATE TABLE testcat.ns1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns1.ns1_1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns2.table (id bigint) USING foo")
+
+runShowDatabasesSql("SHOW NAMESPACES", Seq("ns1", "ns2"))
+runShowDatabasesSql("SHOW NAMESPACES LIKE '*1*'", Seq("ns1"))
+
+// Try to look up only with catalog name, which should list root
namespaces.
+runShowDatabasesSql("SHOW NAMESPACES IN testcat", Seq("ns1", "ns2"))
+ }
+
+ test("ShowNamespaces: show sub-namespaces") {
+spark.sql("CREATE TABLE testcat.ns.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns.ns1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns.ns2.table (id bigint) USING foo")
+
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns", Seq("ns.ns1",
"ns.ns2"))
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns LIKE '*2*'",
Seq("ns.ns2"))
+
+// Try to look up namespace that doesn't exist.
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns.ns3", Seq())
+ }
+
+ test("ShowNamespaces: default v2 catalog is not set") {
+spark.sql("CREATE TABLE testcat.ns.table (id bigint) USING foo")
+
+val exception = intercept[AnalysisException] {
+ runShowDatabasesSql("SHOW NAMESPACES", Seq(""))
+}
+
+assert(exception.getMessage.contains("No default v2 catalog is set."))
+ }
+
+ test("ShowNamespaces: default v2 catalog doesn't support namespace") {
+spark.conf.set(
+ "spark.sql.catalog.testcat_no_namspace",
+ classOf[InMemoryTableCatalogBase].getName)
+spark.conf.set("spark.sql.default.catalog", "testcat_no_namspace")
+
+val exception = intercept[AnalysisException] {
+ runShowDatabasesSql("SHOW NAMESPACES", Seq(""))
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320744142
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
##
@@ -747,6 +747,81 @@ class DataSourceV2SQLSuite
assert(expected === df.collect())
}
+ test("ShowNamespaces: show root namespaces with default v2 catalog") {
+spark.conf.set("spark.sql.default.catalog", "testcat")
+
+runShowDatabasesSql("SHOW NAMESPACES", Seq())
+
+spark.sql("CREATE TABLE testcat.ns1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns1.ns1_1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns2.table (id bigint) USING foo")
+
+runShowDatabasesSql("SHOW NAMESPACES", Seq("ns1", "ns2"))
+runShowDatabasesSql("SHOW NAMESPACES LIKE '*1*'", Seq("ns1"))
+
+// Try to look up only with catalog name, which should list root
namespaces.
+runShowDatabasesSql("SHOW NAMESPACES IN testcat", Seq("ns1", "ns2"))
+ }
+
+ test("ShowNamespaces: show sub-namespaces") {
+spark.sql("CREATE TABLE testcat.ns.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns.ns1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns.ns2.table (id bigint) USING foo")
+
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns", Seq("ns.ns1",
"ns.ns2"))
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns LIKE '*2*'",
Seq("ns.ns2"))
+
+// Try to look up namespace that doesn't exist.
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns.ns3", Seq())
+ }
+
+ test("ShowNamespaces: default v2 catalog is not set") {
+spark.sql("CREATE TABLE testcat.ns.table (id bigint) USING foo")
+
+val exception = intercept[AnalysisException] {
+ runShowDatabasesSql("SHOW NAMESPACES", Seq(""))
+}
+
+assert(exception.getMessage.contains("No default v2 catalog is set."))
+ }
+
+ test("ShowNamespaces: default v2 catalog doesn't support namespace") {
+spark.conf.set(
+ "spark.sql.catalog.testcat_no_namspace",
+ classOf[InMemoryTableCatalogBase].getName)
+spark.conf.set("spark.sql.default.catalog", "testcat_no_namspace")
+
+val exception = intercept[AnalysisException] {
+ runShowDatabasesSql("SHOW NAMESPACES", Seq(""))
+}
+
+assert(exception.getMessage.contains(
+ "The default v2 catalog doesn't support showing namespaces."))
+ }
+
+ test("ShowNamespaces: v2 catalog doesn't support namespace") {
+spark.conf.set(
+ "spark.sql.catalog.testcat_no_namspace",
+ classOf[InMemoryTableCatalogBase].getName)
+
+val exception = intercept[AnalysisException] {
+ runShowDatabasesSql("SHOW NAMESPACES in testcat_no_namspace", Seq(""))
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320743486
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
##
@@ -747,6 +747,81 @@ class DataSourceV2SQLSuite
assert(expected === df.collect())
}
+ test("ShowNamespaces: show root namespaces with default v2 catalog") {
+spark.conf.set("spark.sql.default.catalog", "testcat")
+
+runShowDatabasesSql("SHOW NAMESPACES", Seq())
+
+spark.sql("CREATE TABLE testcat.ns1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns1.ns1_1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns2.table (id bigint) USING foo")
+
+runShowDatabasesSql("SHOW NAMESPACES", Seq("ns1", "ns2"))
+runShowDatabasesSql("SHOW NAMESPACES LIKE '*1*'", Seq("ns1"))
+
+// Try to look up only with catalog name, which should list root
namespaces.
+runShowDatabasesSql("SHOW NAMESPACES IN testcat", Seq("ns1", "ns2"))
+ }
+
+ test("ShowNamespaces: show sub-namespaces") {
+spark.sql("CREATE TABLE testcat.ns.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns.ns1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns.ns2.table (id bigint) USING foo")
+
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns", Seq("ns.ns1",
"ns.ns2"))
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns LIKE '*2*'",
Seq("ns.ns2"))
+
+// Try to look up namespace that doesn't exist.
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns.ns3", Seq())
+ }
+
+ test("ShowNamespaces: default v2 catalog is not set") {
+spark.sql("CREATE TABLE testcat.ns.table (id bigint) USING foo")
+
+val exception = intercept[AnalysisException] {
+ runShowDatabasesSql("SHOW NAMESPACES", Seq(""))
Review comment:
nit: we can write `sql("SHOW NAMESPACES")` here directly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320742819
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
##
@@ -747,6 +747,81 @@ class DataSourceV2SQLSuite
assert(expected === df.collect())
}
+ test("ShowNamespaces: show root namespaces with default v2 catalog") {
+spark.conf.set("spark.sql.default.catalog", "testcat")
+
+runShowDatabasesSql("SHOW NAMESPACES", Seq())
+
+spark.sql("CREATE TABLE testcat.ns1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns1.ns1_1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns2.table (id bigint) USING foo")
+
+runShowDatabasesSql("SHOW NAMESPACES", Seq("ns1", "ns2"))
+runShowDatabasesSql("SHOW NAMESPACES LIKE '*1*'", Seq("ns1"))
+
+// Try to look up only with catalog name, which should list root
namespaces.
+runShowDatabasesSql("SHOW NAMESPACES IN testcat", Seq("ns1", "ns2"))
+ }
+
+ test("ShowNamespaces: show sub-namespaces") {
+spark.sql("CREATE TABLE testcat.ns.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns.ns1.table (id bigint) USING foo")
+spark.sql("CREATE TABLE testcat.ns.ns2.table (id bigint) USING foo")
+
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns", Seq("ns.ns1",
"ns.ns2"))
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns LIKE '*2*'",
Seq("ns.ns2"))
+
+// Try to look up namespace that doesn't exist.
+runShowDatabasesSql("SHOW NAMESPACES IN testcat.ns.ns3", Seq())
+ }
+
+ test("ShowNamespaces: default v2 catalog is not set") {
+spark.sql("CREATE TABLE testcat.ns.table (id bigint) USING foo")
+
+val exception = intercept[AnalysisException] {
+ runShowDatabasesSql("SHOW NAMESPACES", Seq(""))
+}
+
+assert(exception.getMessage.contains("No default v2 catalog is set."))
+ }
+
+ test("ShowNamespaces: default v2 catalog doesn't support namespace") {
+spark.conf.set(
+ "spark.sql.catalog.testcat_no_namspace",
+ classOf[InMemoryTableCatalogBase].getName)
+spark.conf.set("spark.sql.default.catalog", "testcat_no_namspace")
+
+val exception = intercept[AnalysisException] {
+ runShowDatabasesSql("SHOW NAMESPACES", Seq(""))
+}
+
+assert(exception.getMessage.contains(
+ "The default v2 catalog doesn't support showing namespaces."))
+ }
+
+ test("ShowNamespaces: v2 catalog doesn't support namespace") {
+spark.conf.set(
+ "spark.sql.catalog.testcat_no_namspace",
+ classOf[InMemoryTableCatalogBase].getName)
+
+val exception = intercept[AnalysisException] {
+ runShowDatabasesSql("SHOW NAMESPACES in testcat_no_namspace", Seq(""))
+}
+
+assert(exception.getMessage.contains(
+ "No v2 catalog with showing namespaces is available"))
+ }
+
+ private def runShowDatabasesSql(
Review comment:
nit: `testShowNamespaces`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320741537
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala
##
@@ -178,6 +178,27 @@ case class DataSourceResolution(
val aliased = delete.tableAlias.map(SubqueryAlias(_,
relation)).getOrElse(relation)
DeleteFromTable(aliased, delete.condition)
+case ShowNamespacesStatement(None, pattern) =>
+ defaultCatalog match {
+case Some(catalog: SupportsNamespaces) =>
+ ShowNamespaces(catalog, None, pattern)
+case Some(_) =>
+ throw new AnalysisException(
+"The default v2 catalog doesn't support showing namespaces.")
+case None =>
+ throw new AnalysisException("No default v2 catalog is set.")
+ }
+
+case ShowNamespacesStatement(Some(namespace), pattern) =>
+ val CatalogNamespace(maybeCatalog, ns) = namespace
+ maybeCatalog match {
+case Some(catalog: SupportsNamespaces) =>
+ ShowNamespaces(catalog, Some(ns), pattern)
Review comment:
maybe better to write `if (ns.nonEmpty) Some(ns) else None`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320583736
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowNamespacesExec.scala
##
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.NamespaceHelper
+import org.apache.spark.sql.catalog.v2.SupportsNamespaces
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.util.StringUtils
+import org.apache.spark.sql.execution.LeafExecNode
+
+/**
+ * Physical plan node for showing namespaces.
+ */
+case class ShowNamespacesExec(
+output: Seq[Attribute],
+catalog: SupportsNamespaces,
+namespace: Option[Seq[String]],
+pattern: Option[String])
+extends LeafExecNode {
+ override protected def doExecute(): RDD[InternalRow] = {
+val namespaces = namespace.map{ ns =>
+if (ns.nonEmpty) {
Review comment:
> should we list the root namespaces or call listNamespaces with an empty
array?
I think these 2 are the same?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320583051
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##
@@ -779,6 +779,21 @@ class DDLParserSuite extends AnalysisTest {
ShowTablesStatement(Some(Seq("tbl")), Some("*dog*")))
}
+ test("show namespaces") {
Review comment:
cc @xianyinxin can you add similar parser tests for DELETE/UPDATE as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#discussion_r320578981 ## File path: docs/sql-keywords.md ## @@ -179,6 +179,7 @@ Below is a list of all the keywords in Spark SQL. MONTHreservednon-reservedreserved MONTHSnon-reservednon-reservednon-reserved MSCKnon-reservednon-reservednon-reserved + NAMESPACESnon-reservednon-reservednon-reserved Review comment: cc @xianyinxin , we should also add DELETE and UPDATE. Can you open a PR to do it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r319815174
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowDatabasesExec.scala
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.NamespaceHelper
+import org.apache.spark.sql.catalog.v2.SupportsNamespaces
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.util.StringUtils
+import org.apache.spark.sql.execution.LeafExecNode
+
+/**
+ * Physical plan node for showing databases.
+ */
+case class ShowDatabasesExec(
+output: Seq[Attribute],
+catalog: SupportsNamespaces,
+pattern: Option[String])
+extends LeafExecNode {
+ override protected def doExecute(): RDD[InternalRow] = {
+val namespaces = catalog.listNamespaces().flatMap(getNamespaces(catalog,
_))
Review comment:
Yea that's exactly what I mentioned before, with addition of `SHOW
NAMESPACES IN ns1`, +1
> Another idea is: SHOW NAMESPACES should list the root namespaces of the
current catalog, no matter what the current namespace is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r319815174
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowDatabasesExec.scala
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.NamespaceHelper
+import org.apache.spark.sql.catalog.v2.SupportsNamespaces
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.util.StringUtils
+import org.apache.spark.sql.execution.LeafExecNode
+
+/**
+ * Physical plan node for showing databases.
+ */
+case class ShowDatabasesExec(
+output: Seq[Attribute],
+catalog: SupportsNamespaces,
+pattern: Option[String])
+extends LeafExecNode {
+ override protected def doExecute(): RDD[InternalRow] = {
+val namespaces = catalog.listNamespaces().flatMap(getNamespaces(catalog,
_))
Review comment:
Yea that's exactly what I mentioned before, +1
> Another idea is: SHOW NAMESPACES should list the root namespaces of the
current catalog, no matter what the current namespace is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
cloud-fan commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r319379649
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowDatabasesExec.scala
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.NamespaceHelper
+import org.apache.spark.sql.catalog.v2.SupportsNamespaces
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.util.StringUtils
+import org.apache.spark.sql.execution.LeafExecNode
+
+/**
+ * Physical plan node for showing databases.
+ */
+case class ShowDatabasesExec(
+output: Seq[Attribute],
+catalog: SupportsNamespaces,
+pattern: Option[String])
+extends LeafExecNode {
+ override protected def doExecute(): RDD[InternalRow] = {
+val namespaces = catalog.listNamespaces().flatMap(getNamespaces(catalog,
_))
Review comment:
> Add SHOW NAMESPACES that behaves differently than SHOW DATABASES
I prefer this.
Another idea is: `SHOW NAMESPACES` should list the root namespaces of the
current catalog, no matter what the current namespace is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
