[GitHub] spark issue #16868: [SPARK-19115] [SQL] Supporting Create External Table Lik...

[tejasapatil]

Github user tejasapatil commented on the issue:

https://github.com/apache/spark/pull/16868
  
BTW: Hive can allow location of external table to be same as the source 
table. Can you please try that ? It should work. In the test case validations I 
see this check which will fail in such case:

`assert(sourceTable.storage.locationUri != targetTable.storage.locationUri, 
`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16868: [SPARK-19115] [SQL] Supporting Create External Table Lik...

[tejasapatil]

Github user tejasapatil commented on the issue:

https://github.com/apache/spark/pull/16868
  
You have good coverage for DDL statements. How about a test case for 
reading from external table creating using the command this PR enables ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert/read data to a not exist locati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16672
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72759/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert/read data to a not exist locati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16672
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert/read data to a not exist locati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16672
  
**[Test build #72759 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72759/testReport)**
 for PR 16672 at commit 
[`c3439ff`](https://github.com/apache/spark/commit/c3439ffecfcde7ecc06b6dd40e1d085c433eea94).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16868: [SPARK-19115] [SQL] Supporting Create External Table Lik...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16868
  
Please update the PR description. This statement can be used for data 
source tables too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685218
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -969,15 +1053,19 @@ class HiveDDLSuite
 val targetTable = spark.sessionState.catalog.getTableMetadata(
   TableIdentifier(targetTabName, Some("default")))
 
-checkCreateTableLike(sourceView, targetTable)
+checkCreateTableLike(sourceView, targetTable, tableType)
   }
 }
   }
 
-  private def checkCreateTableLike(sourceTable: CatalogTable, targetTable: 
CatalogTable): Unit = {
-// The created table should be a MANAGED table with empty view text 
and original text.
-assert(targetTable.tableType == CatalogTableType.MANAGED,
-  "the created table must be a Hive managed table")
+  private def checkCreateTableLike(
+sourceTable: CatalogTable,
+targetTable: CatalogTable,
+tableType: CatalogTableType): Unit = {
+// The created table should be a MANAGED table or EXTERNAL table with 
empty view text
+// and original text.
+assert(targetTable.tableType == tableType,
+  s"the created table must be a Hive ${tableType.name} table")
--- End diff --

For a Hive table? It could be a data source table too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16777: [SPARK-19435][SQL] Type coercion between ArrayTypes

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16777
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72760/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16777: [SPARK-19435][SQL] Type coercion between ArrayTypes

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16777
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16777: [SPARK-19435][SQL] Type coercion between ArrayTypes

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16777
  
**[Test build #72760 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72760/testReport)**
 for PR 16777 at commit 
[`10afdcb`](https://github.com/apache/spark/commit/10afdcb006327e689794e7ddf8df4744634d842e).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685186
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -926,38 +993,55 @@ class HiveDDLSuite
  |CREATE EXTERNAL TABLE $sourceTabName (key INT comment 
'test', value STRING)
  |COMMENT 'Apache Spark'
  |PARTITIONED BY (ds STRING, hr STRING)
- |LOCATION '$basePath'
-   """.stripMargin)
+ |LOCATION '$basePath1'
+ """.stripMargin)
 for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12")) 
{
   sql(
 s"""
|INSERT OVERWRITE TABLE $sourceTabName
|partition (ds='$ds',hr='$hr')
|SELECT 1, 'a'
- """.stripMargin)
+   """.stripMargin)
 }
-sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
 
-val sourceTable = 
catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
+val locationClause = if (location.nonEmpty) s"LOCATION 
'${location.getOrElse("")}'" else ""
+sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName 
$locationClause")
+
+val sourceTable = catalog.getTableMetadata(
+  TableIdentifier(sourceTabName, Some("default")))
 assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
 assert(sourceTable.comment == Option("Apache Spark"))
-val targetTable = 
catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
+val targetTable = catalog.getTableMetadata(
+  TableIdentifier(targetTabName, Some("default")))
 
-checkCreateTableLike(sourceTable, targetTable)
+checkCreateTableLike(sourceTable, targetTable, tableType)
   }
 }
   }
 
   test("CREATE TABLE LIKE a view") {
+// CREATE TABLE LIKE a view.
+withCreateTableLikeView(None)
--- End diff --

The same here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685175
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -926,38 +993,55 @@ class HiveDDLSuite
  |CREATE EXTERNAL TABLE $sourceTabName (key INT comment 
'test', value STRING)
  |COMMENT 'Apache Spark'
  |PARTITIONED BY (ds STRING, hr STRING)
- |LOCATION '$basePath'
-   """.stripMargin)
+ |LOCATION '$basePath1'
+ """.stripMargin)
--- End diff --

revert it back


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685183
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -891,32 +932,58 @@ class HiveDDLSuite
 assert(DDLUtils.isDatasourceTable(sourceTable))
 assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
 
-checkCreateTableLike(sourceTable, targetTable)
+checkCreateTableLike(sourceTable, targetTable, tableType)
   }
 }
   }
 
   test("CREATE TABLE LIKE a managed Hive serde table") {
-val catalog = spark.sessionState.catalog
+// CREATE TABLE LIKE a managed Hive serde table.
+withCreateTableLikeManagedHiveTable(None)
+
+// CREATE TABLE LIKE a managed Hive serde table location ...
+withTempDir { tmpDir =>
+  withCreateTableLikeManagedHiveTable(Some(tmpDir.toURI.toString))
+}
+  }
+
+  private def withCreateTableLikeManagedHiveTable(location : 
Option[String]): Unit = {
 val sourceTabName = "tab1"
 val targetTabName = "tab2"
+val tableType = if (location.isDefined) CatalogTableType.EXTERNAL else 
CatalogTableType.MANAGED
+val catalog = spark.sessionState.catalog
 withTable(sourceTabName, targetTabName) {
   sql(s"CREATE TABLE $sourceTabName TBLPROPERTIES('prop1'='value1') AS 
SELECT 1 key, 'a'")
-  sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
 
-  val sourceTable = 
catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
+  val locationClause = if (location.nonEmpty) s"LOCATION 
'${location.getOrElse("")}'" else ""
+  sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName 
$locationClause")
+
+  val sourceTable = catalog.getTableMetadata(
+TableIdentifier(sourceTabName, Some("default")))
   assert(sourceTable.tableType == CatalogTableType.MANAGED)
   assert(sourceTable.properties.get("prop1").nonEmpty)
-  val targetTable = 
catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
+  val targetTable = catalog.getTableMetadata(
+TableIdentifier(targetTabName, Some("default")))
 
-  checkCreateTableLike(sourceTable, targetTable)
+  checkCreateTableLike(sourceTable, targetTable, tableType)
 }
   }
 
   test("CREATE TABLE LIKE an external Hive serde table") {
+// CREATE TABLE LIKE an external Hive serde table.
+withCreateTableLikeExtHiveTable(None)
+
+// CREATE TABLE LIKE an external Hive serde table location ...
+withTempDir { tmpDir =>
+  withCreateTableLikeExtHiveTable(Some(tmpDir.toURI.toString))
+}
+  }
+
+  private def withCreateTableLikeExtHiveTable(location : Option[String]): 
Unit = {
 val catalog = spark.sessionState.catalog
+val tableType = if (location.isDefined) CatalogTableType.EXTERNAL else 
CatalogTableType.MANAGED
 withTempDir { tmpDir =>
-  val basePath = tmpDir.toURI
+  val basePath1 = tmpDir.toURI
--- End diff --

revert it back?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685179
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -926,38 +993,55 @@ class HiveDDLSuite
  |CREATE EXTERNAL TABLE $sourceTabName (key INT comment 
'test', value STRING)
  |COMMENT 'Apache Spark'
  |PARTITIONED BY (ds STRING, hr STRING)
- |LOCATION '$basePath'
-   """.stripMargin)
+ |LOCATION '$basePath1'
+ """.stripMargin)
 for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12")) 
{
   sql(
 s"""
|INSERT OVERWRITE TABLE $sourceTabName
|partition (ds='$ds',hr='$hr')
|SELECT 1, 'a'
- """.stripMargin)
+   """.stripMargin)
--- End diff --

revert it back


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16870: [SPARK-19496][SQL]to_date udf to return null when input ...

[tejasapatil]

Github user tejasapatil commented on the issue:

https://github.com/apache/spark/pull/16870
  
`format` could also be invalid. Since the model we are going with is to 
return `null` for bad inputs, the same could be done for `format`. Please add a 
test case for this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685148
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -833,54 +833,95 @@ class HiveDDLSuite
   }
 
   test("CREATE TABLE LIKE a temporary view") {
+// CREATE TABLE LIKE a temporary view.
+withCreateTableLikeTempView(None)
+
+// CREATE TABLE LIKE a temporary view location ...
+withTempDir {tmpDir =>
+  withCreateTableLikeTempView(Some(tmpDir.toURI.toString))
+}
+  }
+
+  private def withCreateTableLikeTempView(location : Option[String]): Unit 
= {
 val sourceViewName = "tab1"
 val targetTabName = "tab2"
+val tableType = if (location.isDefined) CatalogTableType.EXTERNAL else 
CatalogTableType.MANAGED
 withTempView(sourceViewName) {
   withTable(targetTabName) {
 spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
   .createTempView(sourceViewName)
-sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName")
+
+val locationClause = if (location.nonEmpty) s"LOCATION 
'${location.getOrElse("")}'" else ""
+sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName 
$locationClause")
 
 val sourceTable = 
spark.sessionState.catalog.getTempViewOrPermanentTableMetadata(
   TableIdentifier(sourceViewName))
 val targetTable = spark.sessionState.catalog.getTableMetadata(
   TableIdentifier(targetTabName, Some("default")))
 
-checkCreateTableLike(sourceTable, targetTable)
+checkCreateTableLike(sourceTable, targetTable, tableType)
   }
 }
   }
 
   test("CREATE TABLE LIKE a data source table") {
+// CREATE TABLE LIKE a data source table.
+withCreateTableLikeDSTable(None)
--- End diff --

The same here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685152
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -833,54 +833,95 @@ class HiveDDLSuite
   }
 
   test("CREATE TABLE LIKE a temporary view") {
+// CREATE TABLE LIKE a temporary view.
+withCreateTableLikeTempView(None)
+
+// CREATE TABLE LIKE a temporary view location ...
+withTempDir {tmpDir =>
+  withCreateTableLikeTempView(Some(tmpDir.toURI.toString))
+}
+  }
+
+  private def withCreateTableLikeTempView(location : Option[String]): Unit 
= {
 val sourceViewName = "tab1"
 val targetTabName = "tab2"
+val tableType = if (location.isDefined) CatalogTableType.EXTERNAL else 
CatalogTableType.MANAGED
 withTempView(sourceViewName) {
   withTable(targetTabName) {
 spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
   .createTempView(sourceViewName)
-sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName")
+
+val locationClause = if (location.nonEmpty) s"LOCATION 
'${location.getOrElse("")}'" else ""
+sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName 
$locationClause")
 
 val sourceTable = 
spark.sessionState.catalog.getTempViewOrPermanentTableMetadata(
   TableIdentifier(sourceViewName))
 val targetTable = spark.sessionState.catalog.getTableMetadata(
   TableIdentifier(targetTabName, Some("default")))
 
-checkCreateTableLike(sourceTable, targetTable)
+checkCreateTableLike(sourceTable, targetTable, tableType)
   }
 }
   }
 
   test("CREATE TABLE LIKE a data source table") {
+// CREATE TABLE LIKE a data source table.
+withCreateTableLikeDSTable(None)
+
+// CREATE TABLE LIKE a data source table location ...
+withTempDir { tmpDir =>
+  withCreateTableLikeDSTable(Some(tmpDir.toURI.toString))
+}
+  }
+
+  private def withCreateTableLikeDSTable(location : Option[String]): Unit 
= {
 val sourceTabName = "tab1"
 val targetTabName = "tab2"
+val tableType = if (location.isDefined) CatalogTableType.EXTERNAL else 
CatalogTableType.MANAGED
 withTable(sourceTabName, targetTabName) {
   spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
 .write.format("json").saveAsTable(sourceTabName)
-  sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+  val locationClause = if (location.nonEmpty) s"LOCATION 
'${location.getOrElse("")}'" else ""
+  sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName 
$locationClause")
 
   val sourceTable =
-
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName, 
Some("default")))
+spark.sessionState.catalog.getTableMetadata(
+  TableIdentifier(sourceTabName, Some("default")))
   val targetTable =
-
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName, 
Some("default")))
+spark.sessionState.catalog.getTableMetadata(
+  TableIdentifier(targetTabName, Some("default")))
   // The table type of the source table should be a Hive-managed data 
source table
   assert(DDLUtils.isDatasourceTable(sourceTable))
   assert(sourceTable.tableType == CatalogTableType.MANAGED)
 
-  checkCreateTableLike(sourceTable, targetTable)
+  checkCreateTableLike(sourceTable, targetTable, tableType)
 }
   }
 
   test("CREATE TABLE LIKE an external data source table") {
+// CREATE TABLE LIKE an external data source table.
+withCreateTableLikeExtDSTable(None)
--- End diff --

The same here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685143
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -833,54 +833,95 @@ class HiveDDLSuite
   }
 
   test("CREATE TABLE LIKE a temporary view") {
+// CREATE TABLE LIKE a temporary view.
+withCreateTableLikeTempView(None)
+
+// CREATE TABLE LIKE a temporary view location ...
+withTempDir {tmpDir =>
--- End diff --

`{tmpDir` -> `{ tmpDir`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685138
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -833,54 +833,95 @@ class HiveDDLSuite
   }
 
   test("CREATE TABLE LIKE a temporary view") {
+// CREATE TABLE LIKE a temporary view.
+withCreateTableLikeTempView(None)
--- End diff --

`withCreateTableLikeTempView(None)` -> 
`withCreateTableLikeTempView(location = None)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685120
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -70,12 +71,19 @@ case class CreateTableLikeCommand(
   sourceTableDesc.provider
 }
 
+// If location is specified, we create an external table internally.
+// Else create managed table.
+val tblType = if (location.isEmpty) {
+  CatalogTableType.MANAGED
+} else {
+  CatalogTableType.EXTERNAL
+}
--- End diff --

Shorten it to one line?
```Scala
  val tblType = if (location.isEmpty) CatalogTableType.MANAGED else 
CatalogTableType.EXTERNAL
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16890: when colum is use alias ,the order by result is wrong

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16890
  
@muyannian Could you click the "Close pull request" button below?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685105
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -70,12 +71,19 @@ case class CreateTableLikeCommand(
   sourceTableDesc.provider
 }
 
+// If location is specified, we create an external table internally.
+// Else create managed table.
--- End diff --

`Otherwise, create a`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16868: [SPARK-19115] [SQL] Supporting Create External Ta...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16868#discussion_r100685099
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -70,12 +71,19 @@ case class CreateTableLikeCommand(
   sourceTableDesc.provider
 }
 
+// If location is specified, we create an external table internally.
--- End diff --

-> `If the`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16882: [SPARK-19544][SQL] Improve error message when some colum...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16882
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16674#discussion_r100685083
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala
 ---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, 
SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, 
CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+  protected override val spark: SparkSession = TestHive.sparkSession
+
+  override def beforeAll(): Unit = {
+super.beforeAll()
+// Create a simple table with two columns: id and id1
+spark.range(1, 10).selectExpr("id", "id 
id1").write.format("json").saveAsTable("jt")
--- End diff --

Yes, we can remove both by doing something like 
```Scala
abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
  import testImplicits._

  protected override def beforeAll(): Unit = {
super.beforeAll()
// Create a simple table with two columns: id and id1
spark.range(1, 10).selectExpr("id", "id 
id1").write.format("json").saveAsTable("jt")
  }

  protected override def afterAll(): Unit = {
try {
  spark.sql(s"DROP TABLE IF EXISTS jt")
} finally {
  super.afterAll()
}
  }
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16882: [SPARK-19544][SQL] Improve error message when some colum...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16882
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72756/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16882: [SPARK-19544][SQL] Improve error message when some colum...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16882
  
**[Test build #72756 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72756/testReport)**
 for PR 16882 at commit 
[`03ec9de`](https://github.com/apache/spark/commit/03ec9dea7ddb923b2095c8e373f21d640a1300fd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16674#discussion_r100684983
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +534,96 @@ class SQLViewSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
 }
   }
 
-  test("create hive view for joined tables") {
-// make sure the new flag can handle some complex cases like join and 
schema change.
-withTable("jt1", "jt2") {
-  spark.range(1, 
10).toDF("id1").write.format("json").saveAsTable("jt1")
-  spark.range(1, 
10).toDF("id2").write.format("json").saveAsTable("jt2")
-  sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == 
id2")
-  checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 
9).map(i => Row(i, i)))
-
-  val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
-  df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
-  checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 
9).map(i => Row(i, i)))
-
-  sql("DROP VIEW testView")
-}
-  }
-
-  test("SPARK-14933 - create view from hive parquet table") {
--- End diff --

The following test cases are moved to `HiveSQLViewSuite `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16620: [WIP][SPARK-19263] DAGScheduler should avoid sending con...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16620
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72757/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16620: [WIP][SPARK-19263] DAGScheduler should avoid sending con...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16620
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16620: [WIP][SPARK-19263] DAGScheduler should avoid sending con...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16620
  
**[Test build #72757 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72757/testReport)**
 for PR 16620 at commit 
[`c898148`](https://github.com/apache/spark/commit/c8981482c8c0c6e0a062babcf747f861038a1279).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16832
  
**[Test build #72763 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72763/testReport)**
 for PR 16832 at commit 
[`5e91877`](https://github.com/apache/spark/commit/5e91877ba0e14faa2920f4f4121be361a78b685f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16674#discussion_r100684516
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -617,13 +617,17 @@ class Analyzer(
 private def lookupTableFromCatalog(
 u: UnresolvedRelation,
 defaultDatabase: Option[String] = None): LogicalPlan = {
+  val tableIdentWithDb = u.tableIdentifier.copy(
+database = u.tableIdentifier.database.orElse(defaultDatabase))
--- End diff --

```Scala
val db = 
u.tableIdentifier.database.orElse(defaultDatabase).orElse(Some(conf.defaultDB))
```
Then we can use it in the error message.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16787: [SPARK-19448][SQL]optimize some duplication funct...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16787


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16787
  
Thanks! Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16832
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72758/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16832
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16832
  
**[Test build #72758 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72758/testReport)**
 for PR 16832 at commit 
[`358769c`](https://github.com/apache/spark/commit/358769c6a5bedbbe4e94a940d55896e6a1b74096).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16787
  
Thank you very much! LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert/read data to a not exist locati...

[windpiger]

Github user windpiger commented on the issue:

https://github.com/apache/spark/pull/16672
  
yes, let me add these test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16672: [SPARK-19329][SQL]insert/read data to a not exist...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16672#discussion_r100684307
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -1431,4 +1431,30 @@ class HiveDDLSuite
   }
 }
   }
+
+  test("insert data to a table which has altered the table location " +
+"to an not exist location should success") {
+withTable("t") {
+  withTempDir { dir =>
+spark.sql(
+  s"""create table t(a string, b int)
--- End diff --

General style suggestions. Please use upper case for SQL keywords. For 
example, in this SQL statement can be improved to
```
CREATE TABLE t(a STRING, b INT)
USING parquet
OPTIONS(path "xyz")
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16787
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16787
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72754/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16787
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72753/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16787
  
**[Test build #72754 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72754/testReport)**
 for PR 16787 at commit 
[`d44ff38`](https://github.com/apache/spark/commit/d44ff38cc71296e48e3b6fc4a2e4f3908c7508a6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16787
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16787
  
**[Test build #72753 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72753/testReport)**
 for PR 16787 at commit 
[`547d1da`](https://github.com/apache/spark/commit/547d1da142a4814e16b36d1cc868e07bab01f38a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert/read data to a not exist locati...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16672
  
The test case also can check another INSERT mode. INSERT OVERWRITE? Also 
verifying the behaviors for Hive Serde tables? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert/read data to a not exist locati...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16672
  
More test cases for non pre-existing locations? For example, INESRT without 
an ALTER LOCATION? You can simply drop the directory. This scenario is 
reasonable when the table is external. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16724: [SPARK-19352][SQL] Keep sort order of rows after externa...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16724
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72751/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16724: [SPARK-19352][SQL] Keep sort order of rows after externa...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16724
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16672: [SPARK-19329][SQL]insert/read data to a not exist...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16672#discussion_r100684204
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -1431,4 +1431,30 @@ class HiveDDLSuite
   }
 }
   }
+
+  test("insert data to a table which has altered the table location " +
+"to an not exist location should success") {
--- End diff --

Test case names are not accurate after you add new test cases.  Actually, 
could you split the test cases? 





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16724: [SPARK-19352][SQL] Keep sort order of rows after externa...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16724
  
**[Test build #72751 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72751/testReport)**
 for PR 16724 at commit 
[`b1ce030`](https://github.com/apache/spark/commit/b1ce0308cf44ca5bad60a4e954f6169a3c80967e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert/read data to a not exist locati...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16672
  
: ) `success` is a noun and `exist` is a verb. 

`insert/read data to a not exist location datasource table should success` 
-> `Reading from or writing to a data-source table with a non pre-existing 
location should succeed`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16832
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72755/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16832
  
**[Test build #72755 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72755/testReport)**
 for PR 16832 at commit 
[`b734646`](https://github.com/apache/spark/commit/b7346460babc8d1c9422ff8a0acf969e5b180568).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16832
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16777: [SPARK-19435][SQL] Type coercion between ArrayTypes

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16777
  
**[Test build #72762 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72762/testReport)**
 for PR 16777 at commit 
[`180f3c1`](https://github.com/apache/spark/commit/180f3c10250288363eee5fcf87e11585f15db597).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16777: [SPARK-19435][SQL] Type coercion between ArrayTypes

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16777
  
@cloud-fan, I just addressed your comments and test a build with Scala 2.10.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16776: [SPARK-19436][SQL] Add missing tests for approxQu...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16776#discussion_r100684044
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala ---
@@ -63,44 +63,49 @@ final class DataFrameStatFunctions private[sql](df: 
DataFrame) {
*   Note that values greater than 1 are accepted but give the same 
result as 1.
* @return the approximate quantiles at the given probabilities
*
-   * @note NaN values will be removed from the numerical column before 
calculation
+   * @note null and NaN values will be removed from the numerical column 
before calculation
*
* @since 2.0.0
*/
   def approxQuantile(
   col: String,
   probabilities: Array[Double],
   relativeError: Double): Array[Double] = {
-StatFunctions.multipleApproxQuantiles(df.select(col).na.drop(),
-  Seq(col), probabilities, relativeError).head.toArray
+val res = approxQuantile(Array(col), probabilities, relativeError)
+if (res != null) {
+  res.head
+} else {
+  null
+}
   }
 
   /**
* Calculates the approximate quantiles of numerical columns of a 
DataFrame.
-   * @see [[DataFrameStatsFunctions.approxQuantile(col:Str* 
approxQuantile]] for
-   * detailed description.
+   * @see `DataFrameStatsFunctions.approxQuantile` for detailed 
description.
*
-   * Note that rows containing any null or NaN values values will be 
removed before
-   * calculation.
* @param cols the names of the numerical columns
* @param probabilities a list of quantile probabilities
*   Each number must belong to [0, 1].
*   For example 0 is the minimum, 0.5 is the median, 1 is the maximum.
-   * @param relativeError The relative target precision to achieve (>= 0).
+   * @param relativeError The relative target precision to achieve 
(greater or equal to 0).
*   If set to zero, the exact quantiles are computed, which could be 
very expensive.
*   Note that values greater than 1 are accepted but give the same 
result as 1.
* @return the approximate quantiles at the given probabilities of each 
column
*
-   * @note Rows containing any NaN values will be removed before 
calculation
+   * @note Rows containing any null or NaN values will be removed before 
calculation
*
* @since 2.2.0
*/
   def approxQuantile(
   cols: Array[String],
   probabilities: Array[Double],
   relativeError: Double): Array[Array[Double]] = {
-StatFunctions.multipleApproxQuantiles(df.select(cols.map(col): 
_*).na.drop(), cols,
-  probabilities, relativeError).map(_.toArray).toArray
+try {
+  StatFunctions.multipleApproxQuantiles(df.select(cols.map(col): 
_*).na.drop(), cols,
--- End diff --

Let us add a TODO comment above this function and create a JIRA for 
tracking this issue. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16776: [SPARK-19436][SQL] Add missing tests for approxQu...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16776#discussion_r100684027
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala ---
@@ -58,49 +58,56 @@ final class DataFrameStatFunctions private[sql](df: 
DataFrame) {
* @param probabilities a list of quantile probabilities
*   Each number must belong to [0, 1].
*   For example 0 is the minimum, 0.5 is the median, 1 is the maximum.
-   * @param relativeError The relative target precision to achieve 
(greater or equal to 0).
+   * @param relativeError The relative target precision to achieve 
(greater than or equal to 0).
*   If set to zero, the exact quantiles are computed, which could be 
very expensive.
*   Note that values greater than 1 are accepted but give the same 
result as 1.
* @return the approximate quantiles at the given probabilities
*
-   * @note NaN values will be removed from the numerical column before 
calculation
+   * @note null and NaN values will be removed from the numerical column 
before calculation. If
+   *   the dataframe is empty or all rows contain null or NaN, null is 
returned.
*
* @since 2.0.0
*/
   def approxQuantile(
   col: String,
   probabilities: Array[Double],
   relativeError: Double): Array[Double] = {
-StatFunctions.multipleApproxQuantiles(df.select(col).na.drop(),
-  Seq(col), probabilities, relativeError).head.toArray
+val res = approxQuantile(Array(col), probabilities, relativeError)
+if (res != null) {
+  res.head
+} else {
+  null
+}
--- End diff --

The above five lines can be shorten to `Option(res).map(_.head).orNull`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16620: [WIP][SPARK-19263] DAGScheduler should avoid sending con...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16620
  
**[Test build #72761 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72761/testReport)**
 for PR 16620 at commit 
[`0cd3188`](https://github.com/apache/spark/commit/0cd31886f157794aafc31008308632efa5fc725b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16882: [SPARK-19544][SQL] Improve error message when som...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/16882#discussion_r100683913
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
 ---
@@ -116,7 +116,7 @@ object TypeCoercion {
* i.e. the main difference with [[findTightestCommonType]] is that here 
we allow some
* loss of precision when widening decimal and double, and promotion to 
string.
*/
-  private def findWiderTypeForTwo(t1: DataType, t2: DataType): 
Option[DataType] = (t1, t2) match {
--- End diff --

(Added)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16777: [SPARK-19435][SQL] Type coercion between ArrayTypes

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16777
  
**[Test build #72760 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72760/testReport)**
 for PR 16777 at commit 
[`10afdcb`](https://github.com/apache/spark/commit/10afdcb006327e689794e7ddf8df4744634d842e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16870: [SPARK-19496][SQL]to_date udf to return null when...

[windpiger]

Github user windpiger commented on a diff in the pull request:

https://github.com/apache/spark/pull/16870#discussion_r100683849
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
 ---
@@ -95,9 +95,10 @@ object DateTimeUtils {
 sdf
   }
 
-  def newDateFormat(formatString: String, timeZone: TimeZone): DateFormat 
= {
+  def newDateFormat(formatString: String, timeZone: TimeZone, isLenient: 
Boolean): DateFormat = {
 val sdf = new SimpleDateFormat(formatString, Locale.US)
 sdf.setTimeZone(timeZone)
+sdf.setLenient(isLenient)
--- End diff --

we can test it with lenient false. this is  a util func, if test is ok, 
should we always set it to false?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert/read data to a not exist locati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16672
  
**[Test build #72759 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72759/testReport)**
 for PR 16672 at commit 
[`c3439ff`](https://github.com/apache/spark/commit/c3439ffecfcde7ecc06b6dd40e1d085c433eea94).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15125: [SPARK-5484][GraphX] Periodically do checkpoint in Prege...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15125
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15125: [SPARK-5484][GraphX] Periodically do checkpoint in Prege...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15125
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72748/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15125: [SPARK-5484][GraphX] Periodically do checkpoint in Prege...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15125
  
**[Test build #72748 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72748/testReport)**
 for PR 15125 at commit 
[`a25d00c`](https://github.com/apache/spark/commit/a25d00c0807e72a8b29d6dcece6c1469ef919e81).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16832
  
**[Test build #72758 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72758/testReport)**
 for PR 16832 at commit 
[`358769c`](https://github.com/apache/spark/commit/358769c6a5bedbbe4e94a940d55896e6a1b74096).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16895: [SPARK-15615][SQL] Add an API to load DataFrame from Dat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16895
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72749/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16895: [SPARK-15615][SQL] Add an API to load DataFrame from Dat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16895
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16895: [SPARK-15615][SQL] Add an API to load DataFrame from Dat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16895
  
**[Test build #72749 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72749/testReport)**
 for PR 16895 at commit 
[`bb304de`](https://github.com/apache/spark/commit/bb304de56e43c8e9e49dd77ca45d883b8c907fc3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16620: [WIP][SPARK-19263] DAGScheduler should avoid sending con...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16620
  
**[Test build #72757 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72757/testReport)**
 for PR 16620 at commit 
[`c898148`](https://github.com/apache/spark/commit/c8981482c8c0c6e0a062babcf747f861038a1279).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16882: [SPARK-19544][SQL] Improve error message when some colum...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16882
  
**[Test build #72756 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72756/testReport)**
 for PR 16882 at commit 
[`03ec9de`](https://github.com/apache/spark/commit/03ec9dea7ddb923b2095c8e373f21d640a1300fd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16832
  
**[Test build #72755 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72755/testReport)**
 for PR 16832 at commit 
[`b734646`](https://github.com/apache/spark/commit/b7346460babc8d1c9422ff8a0acf969e5b180568).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16882: [SPARK-19544][SQL] Improve error message when som...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/16882#discussion_r100683242
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
 ---
@@ -116,7 +116,7 @@ object TypeCoercion {
* i.e. the main difference with [[findTightestCommonType]] is that here 
we allow some
* loss of precision when widening decimal and double, and promotion to 
string.
*/
-  private def findWiderTypeForTwo(t1: DataType, t2: DataType): 
Option[DataType] = (t1, t2) match {
--- End diff --

Sure!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert data to a not exist location da...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16672
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert data to a not exist location da...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16672
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72746/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert data to a not exist location da...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16672
  
**[Test build #72746 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72746/testReport)**
 for PR 16672 at commit 
[`abc57dd`](https://github.com/apache/spark/commit/abc57ddedde78cfd8e94125416423cbcd4e56f71).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16787
  
**[Test build #72754 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72754/testReport)**
 for PR 16787 at commit 
[`d44ff38`](https://github.com/apache/spark/commit/d44ff38cc71296e48e3b6fc4a2e4f3908c7508a6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16787
  
**[Test build #72753 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72753/testReport)**
 for PR 16787 at commit 
[`547d1da`](https://github.com/apache/spark/commit/547d1da142a4814e16b36d1cc868e07bab01f38a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16787
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16787
  
**[Test build #72752 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72752/testReport)**
 for PR 16787 at commit 
[`0918461`](https://github.com/apache/spark/commit/09184616951e90f25d7f653c009b41b3f79eeea1).
 * This patch **fails Scala style tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16787
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72752/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16787: [SPARK-19448][SQL]optimize some duplication functions be...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16787
  
**[Test build #72752 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72752/testReport)**
 for PR 16787 at commit 
[`0918461`](https://github.com/apache/spark/commit/09184616951e90f25d7f653c009b41b3f79eeea1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16868: [SPARK-19115] [SQL] Supporting Create External Table Lik...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16868
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16868: [SPARK-19115] [SQL] Supporting Create External Table Lik...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16868
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72745/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16868: [SPARK-19115] [SQL] Supporting Create External Table Lik...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16868
  
**[Test build #72745 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72745/testReport)**
 for PR 16868 at commit 
[`81feda4`](https://github.com/apache/spark/commit/81feda4e9f90554b5c22c089e1037be9cebf6392).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16832
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72750/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16832
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16832: [WIP][SPARK-19490][SQL] ignore case sensitivity when fil...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16832
  
**[Test build #72750 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72750/testReport)**
 for PR 16832 at commit 
[`7214b8a`](https://github.com/apache/spark/commit/7214b8a9c3fd7b5d4c352b3dd908f54089a68bc1).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16897: Remove two useless blank lines in HashJoin.scala

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16897
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16897: Remove two useless blank lines in HashJoin.scala

[yucai]

GitHub user yucai opened a pull request:

https://github.com/apache/spark/pull/16897

Remove two useless blank lines in HashJoin.scala

## What changes were proposed in this pull request?

Remove two useless blank lines in HashJoin.scala

## How was this patch tested?

compile codes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yucai/spark blank_lines

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16897.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16897


commit 796b1c755348e848a42a826b058c77be9e75fea0
Author: Yucai 
Date:   2017-02-12T03:36:48Z

Remove blank lines in HashJoin.scala




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16724: [SPARK-19352][SQL] Keep sort order of rows after externa...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16724
  
**[Test build #72751 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72751/testReport)**
 for PR 16724 at commit 
[`b1ce030`](https://github.com/apache/spark/commit/b1ce0308cf44ca5bad60a4e954f6169a3c80967e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16878#discussion_r100682391
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala 
---
@@ -1690,6 +1690,16 @@ class DDLSuite extends QueryTest with 
SharedSQLContext with BeforeAndAfterEach {
 }
   }
 
+  test("block creating duplicate temp table") {
+withView("t_temp") {
+  sql("CREATE TEMPORARY VIEW t_temp AS SELECT 1, 2")
+  val e = intercept[TempTableAlreadyExistsException] {
+sql("CREATE TEMPORARY TABLE t_temp (c3 int, c4 string) USING JSON")
+  }.getMessage
+  assert(e.contains("already exists"))
--- End diff --

Please capture the whole error message.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16878#discussion_r100682360
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -425,7 +425,9 @@ class SparkSqlAstBuilder(conf: SQLConf) extends 
AstBuilder {
 
 logWarning(s"CREATE TEMPORARY TABLE ... USING ... is deprecated, 
please use " +
   "CREATE TEMPORARY VIEW ... USING ... instead")
-CreateTempViewUsing(table, schema, replace = true, global = false, 
provider, options)
+// Since we don't support IF NOT EXISTS for temp table, we should 
not allow
+// replacing existing temp table, that may accidentally remove a 
temp view in use.
--- End diff --

Just got your points... How about update the description to
```
Unlike CREATE TEMPORARY VIEW USING, CREATE TEMPORARY TABLE USING does not 
support IF NOT EXISTS, we should not allow users replacing the existing temp 
table; otherwise, it may accidentally remove a temp view in use.
```

Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16878#discussion_r100682316
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -425,7 +425,9 @@ class SparkSqlAstBuilder(conf: SQLConf) extends 
AstBuilder {
 
 logWarning(s"CREATE TEMPORARY TABLE ... USING ... is deprecated, 
please use " +
   "CREATE TEMPORARY VIEW ... USING ... instead")
-CreateTempViewUsing(table, schema, replace = true, global = false, 
provider, options)
+// Since we don't support IF NOT EXISTS for temp table, we should 
not allow
+// replacing existing temp table, that may accidentally remove a 
temp view in use.
--- End diff --

The same here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16878#discussion_r100682314
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -425,7 +425,9 @@ class SparkSqlAstBuilder(conf: SQLConf) extends 
AstBuilder {
 
 logWarning(s"CREATE TEMPORARY TABLE ... USING ... is deprecated, 
please use " +
   "CREATE TEMPORARY VIEW ... USING ... instead")
-CreateTempViewUsing(table, schema, replace = true, global = false, 
provider, options)
+// Since we don't support IF NOT EXISTS for temp table, we should 
not allow
--- End diff --

`temp table` -> `temp view`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16672: [SPARK-19329][SQL]insert data to a not exist location da...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16672
  
The changes in this PR affects both read and write paths. Please update the 
PR description and title. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org