[GitHub] spark pull request #14921: [SPARK-17361][SQL] file-based external table with...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14921 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14921: [SPARK-17361][SQL] file-based external table with...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14921#discussion_r77430470
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala ---
@@ -305,6 +306,26 @@ class CatalogSuite
columnFields.foreach { f => assert(columnString.contains(f.toString)) }
}
+ test("createExternalTable should fail if path and schema are both not
given " +
+"for file-based data source") {
+val e = intercept[AnalysisException] {
+ spark.catalog.createExternalTable("tbl", "json", Map.empty[String,
String])
+}
+assert(e.message.contains("Unable to infer schema"))
+ }
+
+ test("createExternalTable should not fail if path is not given but
schema is given " +
--- End diff --
this behaviour is consistent with hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14921: [SPARK-17361][SQL] file-based external table with...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/14921#discussion_r77300888
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -314,12 +314,8 @@ case class DataSource(
/**
* Create a resolved [[BaseRelation]] that can be used to read data from
or write data into this
* [[DataSource]]
- *
- * @param checkPathExist A flag to indicate whether to check the
existence of path or not.
- * This flag will be set to false when we create
an empty table (the
- * path of the table does not exist).
*/
- def resolveRelation(checkPathExist: Boolean = true): BaseRelation = {
+ def resolveRelation(): BaseRelation = {
--- End diff --
To clarify it, `RelationProvider` is not only for read path.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14921: [SPARK-17361][SQL] file-based external table with...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14921#discussion_r77295817
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -314,12 +314,8 @@ case class DataSource(
/**
* Create a resolved [[BaseRelation]] that can be used to read data from
or write data into this
* [[DataSource]]
- *
- * @param checkPathExist A flag to indicate whether to check the
existence of path or not.
- * This flag will be set to false when we create
an empty table (the
- * path of the table does not exist).
*/
- def resolveRelation(checkPathExist: Boolean = true): BaseRelation = {
+ def resolveRelation(): BaseRelation = {
--- End diff --
After a discussion with Wenchen, `resolveRelation` will be invoked by
`CREATE TABLE ... USING...`, although the write path in `DataFrameWriter`APIs
does not invoke it. Thanks! @clockfly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14921: [SPARK-17361][SQL] file-based external table with...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14921#discussion_r77292804
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -314,12 +314,8 @@ case class DataSource(
/**
* Create a resolved [[BaseRelation]] that can be used to read data from
or write data into this
* [[DataSource]]
- *
- * @param checkPathExist A flag to indicate whether to check the
existence of path or not.
- * This flag will be set to false when we create
an empty table (the
- * path of the table does not exist).
*/
- def resolveRelation(checkPathExist: Boolean = true): BaseRelation = {
+ def resolveRelation(): BaseRelation = {
--- End diff --
Based on my understanding, `resolveRelation` is not invoked by the write
path of the non-file based data sources.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
