[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r308519178
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/TestInMemoryTableCatalog.scala
##
@@ -120,6 +119,16 @@ class InMemoryTable(
override val properties: util.Map[String, String])
extends Table with SupportsRead with SupportsWrite {
+ def this(
Review comment:
where do we use this new constructor?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r308518254
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##
@@ -24,7 +24,7 @@ import
org.apache.spark.sql.catalog.v2.expressions.{ApplyTransform, BucketTransf
import org.apache.spark.sql.catalyst.analysis.{AnalysisTest,
UnresolvedRelation, UnresolvedStar}
import org.apache.spark.sql.catalyst.catalog.BucketSpec
import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
-import
org.apache.spark.sql.catalyst.plans.logical.sql.{AlterTableAddColumnsStatement,
AlterTableAlterColumnStatement, AlterTableDropColumnsStatement,
AlterTableRenameColumnStatement, AlterTableSetLocationStatement,
AlterTableSetPropertiesStatement, AlterTableUnsetPropertiesStatement,
AlterViewSetPropertiesStatement, AlterViewUnsetPropertiesStatement,
CreateTableAsSelectStatement, CreateTableStatement, DropTableStatement,
DropViewStatement, InsertIntoStatement, QualifiedColType,
ReplaceTableAsSelectStatement, ReplaceTableStatement}
+import
org.apache.spark.sql.catalyst.plans.logical.sql.{AlterTableAddColumnsStatement,
AlterTableAlterColumnStatement, AlterTableDropColumnsStatement,
AlterTableRenameColumnStatement, AlterTableSetLocationStatement,
AlterTableSetPropertiesStatement, AlterTableUnsetPropertiesStatement,
AlterViewSetPropertiesStatement, AlterViewUnsetPropertiesStatement,
CreateTableAsSelectStatement, CreateTableStatement, DescribeColumnStatement,
DescribeTableStatement, DropTableStatement, DropViewStatement,
InsertIntoStatement, QualifiedColType, ReplaceTableAsSelectStatement,
ReplaceTableStatement}
Review comment:
+1 for wildcard here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r306591887
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
##
@@ -237,6 +237,9 @@ object DataSourceV2Strategy extends Strategy with
PredicateHelper {
Nil
}
+case DescribeTable(catalog, ident, _, isExtended) =>
+ DescribeTableExec(catalog, ident, isExtended) :: Nil
Review comment:
how about
```
case DescribeTable(catalog, ident, r: DataSourceV2Relation, isExtended) =>
DescribeTableExec(r.table, ident, isExtended) :: Nil
```
Then we don't need to lookup the table again in `DescribeTableExec`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r304445635
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeTableSchemas.scala
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans
+
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.types.{MetadataBuilder, StringType, StructField,
StructType}
+
+private[sql] object DescribeTableSchemas {
+ val DESCRIBE_TABLE_ATTRIBUTES = Seq(
Review comment:
```
scala> val df1 = sql("desc t1")
df1: org.apache.spark.sql.DataFrame = [col_name: string, data_type: string
... 1 more field]
scala> val df2 = sql("desc t2")
df2: org.apache.spark.sql.DataFrame = [col_name: string, data_type: string
... 1 more field]
scala> df1.crossJoin(df2).show
++-+---++-+---+
|col_name|data_type|comment|col_name|data_type|comment|
++-+---++-+---+
| i| int| null| j| int| null|
++-+---++-+---+
```
This is not a common use case but we don't have to break it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302917683
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.{Identifier, TableCatalog}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.plans.DescribeTableSchemas
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.sources.v2.Table
+
+case class DescribeTableExec(
+catalog: TableCatalog,
+ident: Identifier,
+isExtended: Boolean) extends LeafExecNode {
+
+ import DescribeTableExec._
+
+ override def output: Seq[AttributeReference] =
DescribeTableSchemas.DESCRIBE_TABLE_ATTRIBUTES
+
+ override protected def doExecute(): RDD[InternalRow] = {
+val rows = new ArrayBuffer[InternalRow]()
+if (catalog.tableExists(ident)) {
+ val table = catalog.loadTable(ident)
+ addSchema(rows, table)
+
+ if (isExtended) {
+addPartitioning(rows, table)
+addProperties(rows, table)
+ }
+
+} else {
+ rows += toCatalystRow(s"Table $ident does not exist.", "", "")
+}
+sparkContext.parallelize(rows)
+ }
+
+ private def addSchema(rows: ArrayBuffer[InternalRow], table: Table): Unit = {
+rows ++= table.schema.map{ column =>
+ toCatalystRow(
+column.name, column.dataType.simpleString,
column.getComment().getOrElse(""))
+}
+ }
+
+ private def addPartitioning(rows: ArrayBuffer[InternalRow], table: Table):
Unit = {
+rows += EMPTY_ROW
+rows += toCatalystRow(" Partitioning", "", "")
+rows += toCatalystRow("--", "", "")
+if (table.partitioning.isEmpty) {
+ rows += toCatalystRow("Not partitioned", "", "")
+} else {
+ rows ++= table.partitioning.zipWithIndex.map {
+case (transform, index) => toCatalystRow(s"Part $index",
transform.describe(), "")
+ }
+}
+ }
+
+ private def addProperties(rows: ArrayBuffer[InternalRow], table: Table):
Unit = {
+rows += EMPTY_ROW
+rows += toCatalystRow(" Table Property", " Value", "")
+rows += toCatalystRow("", "---", "")
+rows ++= table.properties.asScala.toList.sortBy(_._1).map {
+ case (key, value) => toCatalystRow(key, value, "")
+}
+ }
+}
+
+private object DescribeTableExec {
+ private val ENCODER = RowEncoder(DescribeTableSchemas.DESCRIBE_TABLE_SCHEMA)
+ private val EMPTY_ROW = toCatalystRow("", "", "")
+
+ private def toCatalystRow(strs: String*): InternalRow = {
+ENCODER.resolveAndBind().toRow(
Review comment:
the encoder only need to call `resolveAndBind` once
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302915123
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.{Identifier, TableCatalog}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.plans.DescribeTableSchemas
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.sources.v2.Table
+
+case class DescribeTableExec(
+catalog: TableCatalog,
+ident: Identifier,
+isExtended: Boolean) extends LeafExecNode {
+
+ import DescribeTableExec._
+
+ override def output: Seq[AttributeReference] =
DescribeTableSchemas.DESCRIBE_TABLE_ATTRIBUTES
+
+ override protected def doExecute(): RDD[InternalRow] = {
+val rows = new ArrayBuffer[InternalRow]()
+if (catalog.tableExists(ident)) {
+ val table = catalog.loadTable(ident)
+ addSchema(rows, table)
+
+ if (isExtended) {
+addPartitioning(rows, table)
+addProperties(rows, table)
+ }
+
+} else {
+ rows += toCatalystRow(s"Table $ident does not exist.", "", "")
Review comment:
I think we can follow the https://github.com/apache/spark/pull/24937: The
`DescribeTable` should contain an `UnresolvedRelation`, so that analyzer can
check table existence for us.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302914263
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.{Identifier, TableCatalog}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.plans.DescribeTableSchemas
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.sources.v2.Table
+
+case class DescribeTableExec(
+catalog: TableCatalog,
+ident: Identifier,
+isExtended: Boolean) extends LeafExecNode {
+
+ import DescribeTableExec._
+
+ override def output: Seq[AttributeReference] =
DescribeTableSchemas.DESCRIBE_TABLE_ATTRIBUTES
+
+ override protected def doExecute(): RDD[InternalRow] = {
+val rows = new ArrayBuffer[InternalRow]()
+if (catalog.tableExists(ident)) {
+ val table = catalog.loadTable(ident)
+ addSchema(rows, table)
+
+ if (isExtended) {
+addPartitioning(rows, table)
+addProperties(rows, table)
+ }
+
+} else {
+ rows += toCatalystRow(s"Table $ident does not exist.", "", "")
Review comment:
shouldn't we throw exception when table not found?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302897660
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeTableSchemas.scala
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans
+
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.types.{MetadataBuilder, StringType, StructField,
StructType}
+
+private[sql] object DescribeTableSchemas {
+ val DESCRIBE_TABLE_ATTRIBUTES = Seq(
Review comment:
We shouldn't define attributes in an object. `AttributeReference` will be
assigned a unique ID when created, and in general we should create new
attributes when creating a new logical plan.
For example, if you do `df1 = sql("desc table t1"); df2 = sql("desc table
");`, `df1.join(df2)` would fail.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302911446
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeTableSchemas.scala
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans
+
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.types.{MetadataBuilder, StringType, StructField,
StructType}
+
+private[sql] object DescribeTableSchemas {
+ val DESCRIBE_TABLE_ATTRIBUTES = Seq(
Review comment:
or, we call follow `DescribeCommandBase`: create attributes in an abstract
class instead of object.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302912624
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeTableSchemas.scala
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans
+
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.types.{MetadataBuilder, StringType, StructField,
StructType}
+
+private[sql] object DescribeTableSchemas {
+ val DESCRIBE_TABLE_ATTRIBUTES = Seq(
+AttributeReference("col_name", StringType, nullable = false,
+ new MetadataBuilder().putString("comment", "name of the
column").build())(),
+AttributeReference("data_type", StringType, nullable = false,
+ new MetadataBuilder().putString("comment", "data type of the
column").build())(),
+AttributeReference("comment", StringType, nullable = true,
+ new MetadataBuilder().putString("comment", "comment of the
column").build())()
+ )
+
+ val DESCRIBE_TABLE_SCHEMA = StructType(
+DESCRIBE_TABLE_ATTRIBUTES.map(attr => StructField(attr.name,
attr.dataType, attr.nullable)))
Review comment:
nit: `StructType.fromAttributes(DESCRIBE_TABLE_ATTRIBUTES)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302911446
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeTableSchemas.scala
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans
+
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.types.{MetadataBuilder, StringType, StructField,
StructType}
+
+private[sql] object DescribeTableSchemas {
+ val DESCRIBE_TABLE_ATTRIBUTES = Seq(
Review comment:
or, we call follow `DescribeCommandBase`: create attributes in an abstract
class instead of object.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302898436
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -499,6 +501,14 @@ object OverwritePartitionsDynamic {
}
}
+case class DescribeTable(
+catalog: TableCatalog,
+ident: Identifier,
+isExtended: Boolean) extends Command {
+ override lazy val output = DescribeTableSchemas.DESCRIBE_TABLE_ATTRIBUTES
+ override lazy val schema = DescribeTableSchemas.DESCRIBE_TABLE_SCHEMA
Review comment:
by default schema is `StructType.fromAttributes(output)`, so we don't need
to override it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302898286
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -499,6 +501,14 @@ object OverwritePartitionsDynamic {
}
}
+case class DescribeTable(
+catalog: TableCatalog,
+ident: Identifier,
+isExtended: Boolean) extends Command {
+ override lazy val output = DescribeTableSchemas.DESCRIBE_TABLE_ATTRIBUTES
Review comment:
we don't need lazy val here, as it's not a heavy computing
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
cloud-fan commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r302897660
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/DescribeTableSchemas.scala
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans
+
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.types.{MetadataBuilder, StringType, StructField,
StructType}
+
+private[sql] object DescribeTableSchemas {
+ val DESCRIBE_TABLE_ATTRIBUTES = Seq(
Review comment:
We shouldn't define attributes in an object. `AttributeReference` will be
assigned a unique ID when created, and in general we should create new
attributes when creating a new logical plan.
I'm not sure what will be broken, but this is not a recommended pattern.
I would just define a schema here, and when I need attributes, just call
`schema.toAttributes`, which creates new `AttributeReference`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
