[GitHub] spark pull request #14204: [SPARK-16520] [WEBUI] Link executors to correspon...

[nblintao]

Github user nblintao commented on a diff in the pull request:

https://github.com/apache/spark/pull/14204#discussion_r71830047
  
--- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala 
---
@@ -73,13 +73,15 @@ class JsonProtocolSuite extends SparkFunSuite {
   BlockManagerId("Scarce", "to be counted...", 100))
 val unpersistRdd = SparkListenerUnpersistRDD(12345)
 val logUrlMap = Map("stderr" -> "mystderr", "stdout" -> 
"mystdout").toMap
+val workerUrlMap = Map("url" -> "spark://Worker@192.168.1.104:32790",
+  "ui_url" -> "http://192.168.1.104:46445;).toMap
--- End diff --

Good idea. I've updated referring to the test cases in `ClientSuite.scala`. 
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14204: [SPARK-16520] [WEBUI] Link executors to corresponding wo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14204
  
**[Test build #62719 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62719/consoleFull)**
 for PR 14204 at commit 
[`48ae86f`](https://github.com/apache/spark/commit/48ae86f630465ff134da792eeba2007966b5cecf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14315: [HOTFIX][BUILD][SPARK-16287][SQL] Fix annotation argumen...

[techaddict]

Github user techaddict commented on the issue:

https://github.com/apache/spark/pull/14315
  
@jaceklaskowski thanks for finding this out. Its weird it passed locally 
too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14298: [SPARK-16283][SQL] Implement `percentile_approx` SQL fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14298
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62706/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14298: [SPARK-16283][SQL] Implement `percentile_approx` SQL fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14298
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14298: [SPARK-16283][SQL] Implement `percentile_approx` SQL fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14298
  
**[Test build #62706 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62706/consoleFull)**
 for PR 14298 at commit 
[`da87bfd`](https://github.com/apache/spark/commit/da87bfdb0ab91a265f74df01ec87221651583919).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class StringToMap(text: Expression, pairDelim: Expression, 
keyValueDelim: Expression)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14302: [SPARK-16663][SQL] desc table should be consistent betwe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14302
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14302: [SPARK-16663][SQL] desc table should be consistent betwe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14302
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62709/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14302: [SPARK-16663][SQL] desc table should be consistent betwe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14302
  
**[Test build #62709 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62709/consoleFull)**
 for PR 14302 at commit 
[`1ffa49d`](https://github.com/apache/spark/commit/1ffa49d6f2b7e7ac4114e215b56002feaf4fde09).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14313: [SPARK-16674][SQL] Avoid per-record type dispatch in JDB...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14313
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62708/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14313: [SPARK-16674][SQL] Avoid per-record type dispatch in JDB...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14313
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14313: [SPARK-16674][SQL] Avoid per-record type dispatch in JDB...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14313
  
**[Test build #62708 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62708/consoleFull)**
 for PR 14313 at commit 
[`5eae0e6`](https://github.com/apache/spark/commit/5eae0e6b7d68a7781b1849441a000cf3ff7fe804).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14315: [HOTFIX][BUILD][SPARK-16287][SQL] Fix annotation argumen...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14315
  
I'm wondering why that PR passed all tests...

LGTM, pending jenkins, thanks for finding this out!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14210: [SPARK-16556] [SPARK-16559] [SQL] Fix Two Bugs in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/14210


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14315: [HOTFIX][BUILD][SPARK-16287][SQL] Fix annotation argumen...

[jaceklaskowski]

Github user jaceklaskowski commented on the issue:

https://github.com/apache/spark/pull/14315
  
/cc @cloud-fan @techaddict 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14210: [SPARK-16556] [SPARK-16559] [SQL] Fix Two Bugs in Bucket...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14210
  
thanks, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14315: [HOTFIX][BUILD][SPARK-16287][SQL] Fix annotation argumen...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14315
  
**[Test build #62718 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62718/consoleFull)**
 for PR 14315 at commit 
[`ee34380`](https://github.com/apache/spark/commit/ee34380a532ca8b5df5f938a0eba51c94a973dff).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13373
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62704/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13373
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62704 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62704/consoleFull)**
 for PR 13373 at commit 
[`4a3e72e`](https://github.com/apache/spark/commit/4a3e72e6321fcbc449eb2fad3c903bd790cea2cb).
 * This patch **fails Spark unit tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14315: [HOTFIX] Fix annotation argument needs to be a co...

[jaceklaskowski]

GitHub user jaceklaskowski opened a pull request:

https://github.com/apache/spark/pull/14315

[HOTFIX] Fix annotation argument needs to be a constant

## What changes were proposed in this pull request?

Fix for compilation error:

```

/Users/jacek/dev/oss/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala:402:
 error: annotation argument needs to be a constant; found: "_FUNC_(text[, 
pairDelim, keyValueDelim]) - Creates a map after splitting the text ".+("into 
key/value pairs using delimiters. ").+("Default delimiters are \',\' for 
pairDelim and \':\' for keyValueDelim.")
"into key/value pairs using delimiters. " +
  ^
```

## How was this patch tested?

Local build


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jaceklaskowski/spark 
build-fix-complexTypeCreator

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14315.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14315


commit ee34380a532ca8b5df5f938a0eba51c94a973dff
Author: Jacek Laskowski 
Date:   2016-07-22T05:24:08Z

[HOTFIX] Fix annotation argument needs to be a constant




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14210: [SPARK-16556] [SPARK-16559] [SQL] Fix Two Bugs in...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14210#discussion_r71828372
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/sources/CreateTableAsSelectSuite.scala
 ---
@@ -212,7 +213,23 @@ class CreateTableAsSelectSuite extends DataSourceTest 
with SharedSQLContext with
   )
   val table = catalog.getTableMetadata(TableIdentifier("t"))
   assert(DDLUtils.getBucketSpecFromTableProperties(table) ==
-Some(BucketSpec(5, Seq("a"), Seq("b"
+Option(BucketSpec(5, Seq("a"), Seq("b"
+}
+  }
+
+  test("create table using as select - with zero buckets") {
+withTable("t") {
+  val e = intercept[AnalysisException] {
+sql(
+  s"""
+ |CREATE TABLE t USING PARQUET
+ |OPTIONS (PATH '${path.toString}')
+ |CLUSTERED BY (a) SORTED BY (b) INTO 0 BUCKETS
--- End diff --

hm? so we allow `CLUSTERED BY` when schema is not specified for data source 
table?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14313: [SPARK-16674][SQL] Avoid per-record type dispatch in JDB...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14313
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62705/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14313: [SPARK-16674][SQL] Avoid per-record type dispatch in JDB...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14313
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14314: [SPARK-16678] [SPARK-16677] [SQL] Fix two View-related b...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14314
  
**[Test build #62717 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62717/consoleFull)**
 for PR 14314 at commit 
[`c64092c`](https://github.com/apache/spark/commit/c64092c6aaec42663278343a27467c1c8c165b92).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14313: [SPARK-16674][SQL] Avoid per-record type dispatch in JDB...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14313
  
**[Test build #62705 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62705/consoleFull)**
 for PR 14313 at commit 
[`ad64483`](https://github.com/apache/spark/commit/ad64483e2263485f843a469310fb8d252824d09e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14285: [SPARK-16649][SQL] Push partition predicates down into m...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14285
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62703/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14285: [SPARK-16649][SQL] Push partition predicates down into m...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14285
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14285: [SPARK-16649][SQL] Push partition predicates down into m...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14285
  
**[Test build #62703 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62703/consoleFull)**
 for PR 14285 at commit 
[`de3ca68`](https://github.com/apache/spark/commit/de3ca686ce54109dc3564de75b6cb6411ead7d5e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14314: [SPARK-16678] [SPARK-16677] [SQL] Fix two View-related b...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14314
  
**[Test build #62716 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62716/consoleFull)**
 for PR 14314 at commit 
[`c698a4a`](https://github.com/apache/spark/commit/c698a4af68907f8f9eb874ae6bd0b69d7e9fee59).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62715 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62715/consoleFull)**
 for PR 13373 at commit 
[`bf74b0e`](https://github.com/apache/spark/commit/bf74b0e7c76eb2462183e76c3e3c4f8405ff82f1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14132
  
**[Test build #62714 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62714/consoleFull)**
 for PR 14132 at commit 
[`2ee7d21`](https://github.com/apache/spark/commit/2ee7d21889490f05b591a276fa33df96693caf05).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14314: [SPARK-16678] [SPARK-16677] [SQL] Fix two View-re...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14314#discussion_r71827097
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -105,6 +105,12 @@ case class CreateViewCommand(
   val tableIdentifier = tableDesc.identifier.copy(database = 
Option(database))
 
   if (sessionState.catalog.tableExists(tableIdentifier)) {
+val tableMetadata = 
sessionState.catalog.getTableMetadata(tableIdentifier)
+if (tableMetadata.tableType != CatalogTableType.VIEW) {
+  throw new AnalysisException(
--- End diff --

Yeah, a good question. Just did a check using Hive. It is allowed by Hive. 
```
hive> CREATE VIEW IF NOT EXISTS tab1 AS SELECT * FROM t1;
OK
Time taken: 0.678 seconds
```
Let me move this a little bit down. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62713 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62713/consoleFull)**
 for PR 13373 at commit 
[`b573919`](https://github.com/apache/spark/commit/b5739193ecb4c50aacc9d450f37e7be0d49939c7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[lianhuiwang]

Github user lianhuiwang commented on the issue:

https://github.com/apache/spark/pull/13373
  
cc @cloud-fan @rxin @hvanhovell


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14285: [SPARK-16649][SQL] Push partition predicates down into m...

[lianhuiwang]

Github user lianhuiwang commented on the issue:

https://github.com/apache/spark/pull/14285
  
cc @cloud-fan @rxin @hvanhovell 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14210: [SPARK-16556] [SPARK-16559] [SQL] Fix Two Bugs in...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14210#discussion_r71826808
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -344,6 +344,11 @@ class SparkSqlAstBuilder(conf: SQLConf) extends 
AstBuilder {
 table, provider, partitionColumnNames, bucketSpec, mode, options, 
query)
 } else {
   val struct = Option(ctx.colTypeList()).map(createStructType)
+  if (struct.isEmpty && bucketSpec.nonEmpty) {
+throw new ParseException(
+  "Expected explicit specification of table schema when using 
CLUSTERED BY clause.", ctx)
--- End diff --

`DataFrameWriter` does not allow it. 
- First, `save()` API has an 
[`assertNotBucketed("save")`](https://github.com/apache/spark/blob/864b764eafa57a1418b683ccf6899b01bab28fba/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L203)
- Second, `insertInto(tableName: String)` API also has an 
[`assertNotBucketed("save")`](https://github.com/apache/spark/blob/864b764eafa57a1418b683ccf6899b01bab28fba/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L243)
- Third, `saveAsTable(tableName: String)` API relies on 
`CreateTableUsingAsSelect`. This will be converted to 
`CreateDataSourceTableAsSelectCommand`. It also specifies the schema 
[here](https://github.com/apache/spark/blob/ce3b98bae28af72299722f56e4e4ef831f471ec0/sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala#L259)

I think we are pretty safe now. Please let me know if we still should move 
it? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71826514
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/catalyst/SQLBuilder.scala ---
@@ -193,6 +196,9 @@ class SQLBuilder(logicalPlan: LogicalPlan) extends 
Logging {
 case OneRowRelation =>
   ""
 
+case Hint(_, _, child) =>
--- End diff --

I remember the reason. Originally, I designed `Hint` is removed by some 
rules of `Analyzer` or `Optimizer`. But, now, it's changed. Only rules of 
`Analyzer` handles that and removes all unknown hints. `CheckAnalysis` is the 
correct position.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14314: [SPARK-16678] [SPARK-16677] [SQL] Fix two View-re...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/14314#discussion_r71825974
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -105,6 +105,12 @@ case class CreateViewCommand(
   val tableIdentifier = tableDesc.identifier.copy(database = 
Option(database))
 
   if (sessionState.catalog.tableExists(tableIdentifier)) {
+val tableMetadata = 
sessionState.catalog.getTableMetadata(tableIdentifier)
+if (tableMetadata.tableType != CatalogTableType.VIEW) {
+  throw new AnalysisException(
--- End diff --

If allowing existing, do we still need to throw this excepion?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71825927
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/catalyst/SQLBuilder.scala ---
@@ -193,6 +196,9 @@ class SQLBuilder(logicalPlan: LogicalPlan) extends 
Logging {
 case OneRowRelation =>
   ""
 
+case Hint(_, _, child) =>
--- End diff --

Ur, it's added by previous advice comments.
No problem. I'll move that into `CheckAnalysis`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14259: [SPARK-16622][SQL] Fix NullPointerException when ...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14259#discussion_r71825821
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala ---
@@ -314,4 +343,14 @@ class DatasetAggregatorSuite extends QueryTest with 
SharedSQLContext {
 val ds3 = sql("SELECT 'Some String' AS b, 1279869254 AS a").as[AggData]
 assert(ds3.select(NameAgg.toColumn).schema.head.nullable === true)
   }
+
+  test("Aggregator on empty dataset") {
--- End diff --

we can create a `ObjectExpressionsSuite` under 
org.apache.spark.sql.catalyst.expressions


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71825810
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ---
@@ -339,8 +339,11 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] 
with Logging {
   case SqlBaseParser.SELECT =>
 // Regular select
 
+// Add hints.
+val withHint = relation.optionalMap(ctx.hint)(withHints)
--- End diff --

Yep. I grasp your point. First, the given `relation` is just a logical 
plan, not a table. In the following case, the `relation` is `Join`.
 ```
SELECT /*+ MAPJOIN(parquet_t0) */ * FROM parquet_t0, parquet_t1
```
This means just a context. 

Second, I agree with your point. Generally, `hint` is about tables like 
join order or access path. We can provides some hints on filter conditions, I'm 
not sure about the cases so far. But, even for that cases, we can adapt that in 
`Analyzer` easily. In other words, we can identify `Hint("FILTERHINT")` in the 
same SPJ.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14290: [SPARK-16657] [SQL] Replace children by innerChildren in...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14290
  
The analyzer resolution requires them as children. Although this is 
inconsistent with the other similar nodes, we still can tolerate it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14290: [SPARK-16657] [SQL] Replace children by innerChil...

[gatorsmile]

Github user gatorsmile closed the pull request at:

https://github.com/apache/spark/pull/14290


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14314: [SPARK-16678] [SPARK-16677] Fix two View-related bugs

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14314
  
**[Test build #62712 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62712/consoleFull)**
 for PR 14314 at commit 
[`5c9be14`](https://github.com/apache/spark/commit/5c9be1438e57722dbcea7a77993df636394cd9bf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14259: [SPARK-16622][SQL] Fix NullPointerException when ...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/14259#discussion_r71825205
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala ---
@@ -314,4 +343,14 @@ class DatasetAggregatorSuite extends QueryTest with 
SharedSQLContext {
 val ds3 = sql("SELECT 'Some String' AS b, 1279869254 AS a").as[AggData]
 assert(ds3.select(NameAgg.toColumn).schema.head.nullable === true)
   }
+
+  test("Aggregator on empty dataset") {
--- End diff --

Another problem is this test should not be in `DatasetAggregatorSuite`. I 
don't find a proper file for this test. Which one you would suggest?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14314: [SPARK-16678] [SPARK-16677] Fix two View-related ...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/14314

[SPARK-16678] [SPARK-16677] Fix two View-related bugs

### What changes were proposed in this pull request?
**Issue 1: Disallow Creating a View when the same-name Table Exists**
When we create a view, we check whether the view already exists. In the 
current implementation, if a table with the same name exists, we treat it as a 
view. However, this is not the right behavior. We should follow what Hive does. 
For example,
```
hive> CREATE TABLE tab1 (id int);
OK
Time taken: 0.196 seconds
hive> CREATE OR REPLACE VIEW tab1 AS SELECT * FROM t1;
FAILED: SemanticException [Error 10218]: Existing table is not a view
 The following is an existing table, not a view: default.tab1
```

**Issue 2: Strange Error when Issuing Load Table Against A View**
Users should not be allowed to issue LOAD DATA against a view. Currently, 
when users doing it, we got a very strange runtime error. For example,
```SQL
LOAD DATA LOCAL INPATH "$testData" INTO TABLE $viewName
```
```
java.lang.reflect.InvocationTargetException was thrown.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:680)
```
### How was this patch tested?
Added test cases

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark tableDDLAgainstView

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14314.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14314


commit d92a95cee8572f631813f259620cc6fe2644ed36
Author: gatorsmile 
Date:   2016-07-22T03:48:50Z

fix.

commit 5c9be1438e57722dbcea7a77993df636394cd9bf
Author: gatorsmile 
Date:   2016-07-22T04:13:28Z

changed the test case names.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14292: [SPARK-14131][SQL[STREAMING] Improved fix for avoiding p...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14292
  
**[Test build #3189 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3189/consoleFull)**
 for PR 14292 at commit 
[`0e67e26`](https://github.com/apache/spark/commit/0e67e26c0690dfecb011c872f58baf87e1bde3c0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13373
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62711 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62711/consoleFull)**
 for PR 13373 at commit 
[`1e0a6f2`](https://github.com/apache/spark/commit/1e0a6f296a73822b250ab4d4e9a35d4bec129765).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13373
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62711/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62711 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62711/consoleFull)**
 for PR 13373 at commit 
[`1e0a6f2`](https://github.com/apache/spark/commit/1e0a6f296a73822b250ab4d4e9a35d4bec129765).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14116: [SPARK-16452][SQL] Support basic INFORMATION_SCHE...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14116#discussion_r71823904
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -484,8 +521,11 @@ class SessionCatalog(
 val dbTables =
   externalCatalog.listTables(dbName, pattern).map { t => 
TableIdentifier(t, Some(dbName)) }
 synchronized {
-  val _tempTables = StringUtils.filterPattern(tempTables.keys.toSeq, 
pattern)
+  var _tempTables = StringUtils.filterPattern(tempTables.keys.toSeq, 
pattern)
 .map { t => TableIdentifier(t) }
+  if (db != INFORMATION_SCHEMA_DATABASE) {
--- End diff --

This is the current result.
```
scala> sql("use information_schema")
res11: org.apache.spark.sql.DataFrame = []

scala> sql("show tables").show(false)
++---+
|tableName   |isTemporary|
++---+
|information_schema.columns  |true   |
|information_schema.databases|true   |
|information_schema.schemata |true   |
|information_schema.session_variables|true   |
|information_schema.tables   |true   |
|information_schema.views|true   |
++---+
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14116: [SPARK-16452][SQL] Support basic INFORMATION_SCHE...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14116#discussion_r71823733
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -471,6 +480,34 @@ class SessionCatalog(
   }
 
   /**
+   * Normalize TableIdentifier by consistently ensuring the following two 
rules.
+   * 1. System-side temporary tables should have None as database.
+   * 2. System-side temporary tables should have prefixed table names.
+   * Currently, only INFORMATION_SCHEMA tables are system-side temporary 
tables, and this function
+   * returns TableIdentifier("information_schema.databases", None).
+   */
+  protected def normalizeTableIdentifier(name: TableIdentifier): 
TableIdentifier = synchronized {
+if (name.database.isDefined) {
+  if (name.database.contains(INFORMATION_SCHEMA_DATABASE)) {
+TableIdentifier(s"$INFORMATION_SCHEMA_DATABASE.${name.table}", 
None)
+  } else {
+name
+  }
+} else {
+  val tableName = formatTableName(name.table)
+  if (tableName.startsWith(INFORMATION_SCHEMA_DATABASE + ".")) {
--- End diff --

Backtick-quoted one will not reach here.
```
scala> sql("create table `aaa.bbb`(a int)")
org.apache.spark.sql.AnalysisException: 
org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.ql.metadata.HiveException: [aaa.bbb]: is not a valid 
table name;
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13373
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13373
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62710/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62710 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62710/consoleFull)**
 for PR 13373 at commit 
[`2d9e321`](https://github.com/apache/spark/commit/2d9e32142748ebb92d27f23609607d6395b512b1).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14297: [SPARK-16660][SQL] CreateViewCommand should not take Cat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14297
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62701/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14297: [SPARK-16660][SQL] CreateViewCommand should not take Cat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14297
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71823645
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/catalyst/SQLBuilder.scala ---
@@ -193,6 +196,9 @@ class SQLBuilder(logicalPlan: LogicalPlan) extends 
Logging {
 case OneRowRelation =>
   ""
 
+case Hint(_, _, child) =>
--- End diff --

so why do we have this case? `SQLBuilder` will be applied on analyzed plan 
right?

And I think it's better to push this check in `CheckAnalysis`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71823613
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ---
@@ -339,8 +339,11 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] 
with Logging {
   case SqlBaseParser.SELECT =>
 // Regular select
 
+// Add hints.
+val withHint = relation.optionalMap(ctx.hint)(withHints)
--- End diff --

for now the hint can only affect the relations(broadcast hint), so it's ok 
to just put the `Hint` on top of relation. But generally hint can affect 
anything in the select clause, e.g. maybe the filter condition.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14302: [SPARK-16663][SQL] desc table should be consistent betwe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14302
  
**[Test build #62709 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62709/consoleFull)**
 for PR 14302 at commit 
[`1ffa49d`](https://github.com/apache/spark/commit/1ffa49d6f2b7e7ac4114e215b56002feaf4fde09).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62710 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62710/consoleFull)**
 for PR 13373 at commit 
[`2d9e321`](https://github.com/apache/spark/commit/2d9e32142748ebb92d27f23609607d6395b512b1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14297: [SPARK-16660][SQL] CreateViewCommand should not take Cat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14297
  
**[Test build #62701 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62701/consoleFull)**
 for PR 14297 at commit 
[`5072959`](https://github.com/apache/spark/commit/50729594c1956d185bde4c2b41891ce1567cfd5a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13373
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62707/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62707 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62707/consoleFull)**
 for PR 13373 at commit 
[`f3da998`](https://github.com/apache/spark/commit/f3da9983d79c6bf070cba3ccd863890f1b4c2233).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13373
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14313: [SPARK-16674][SQL] Avoid per-record type dispatch in JDB...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14313
  
**[Test build #62708 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62708/consoleFull)**
 for PR 14313 at commit 
[`5eae0e6`](https://github.com/apache/spark/commit/5eae0e6b7d68a7781b1849441a000cf3ff7fe804).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14307: [SPARK-16672][SQL] SQLBuilder should not raise exception...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/14307
  
Hi, @hvanhovell .
Could you review this PR about `SQLBuilder`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14204: [SPARK-16520] [WEBUI] Link executors to corresponding wo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14204
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71823152
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/catalyst/SQLBuilder.scala ---
@@ -193,6 +196,9 @@ class SQLBuilder(logicalPlan: LogicalPlan) extends 
Logging {
 case OneRowRelation =>
   ""
 
+case Hint(_, _, child) =>
--- End diff --

Also, `SparkStrategies` ensures that here 
https://github.com/apache/spark/pull/14132/files#diff-7253a38df7e111ecf6b1ef71feba383bR347
 , too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62707 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62707/consoleFull)**
 for PR 13373 at commit 
[`f3da998`](https://github.com/apache/spark/commit/f3da9983d79c6bf070cba3ccd863890f1b4c2233).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14204: [SPARK-16520] [WEBUI] Link executors to corresponding wo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14204
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62700/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14204: [SPARK-16520] [WEBUI] Link executors to corresponding wo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14204
  
**[Test build #62700 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62700/consoleFull)**
 for PR 14204 at commit 
[`519c329`](https://github.com/apache/spark/commit/519c32974d3239d5780fb94064126b4a0737c656).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71823079
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/catalyst/SQLBuilder.scala ---
@@ -193,6 +196,9 @@ class SQLBuilder(logicalPlan: LogicalPlan) extends 
Logging {
 case OneRowRelation =>
   ""
 
+case Hint(_, _, child) =>
--- End diff --

After applying the rules of `Analyzer`, there will be no more `Hint` case 
class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71822992
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ---
@@ -339,8 +339,11 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] 
with Logging {
   case SqlBaseParser.SELECT =>
 // Regular select
 
+// Add hints.
+val withHint = relation.optionalMap(ctx.hint)(withHints)
--- End diff --

Ur, could you give some example what you concern?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14298: [SPARK-16283][SQL] Implement `percentile_approx` SQL fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14298
  
**[Test build #62706 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62706/consoleFull)**
 for PR 14298 at commit 
[`da87bfd`](https://github.com/apache/spark/commit/da87bfdb0ab91a265f74df01ec87221651583919).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71822903
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -347,6 +347,16 @@ querySpecification
windows?)
 ;
 
+hint
+: '/*+' hintStatement '*/'
+;
+
+hintStatement
+: hintName=identifier
+| hintName=identifier '(' parameters+=identifier 
parameters+=identifier ')'
--- End diff --

Thank you for review, @cloud-fan .
The first goal of this PR provides a general syntax for hints, not only 
broadcast hints. The `(` and `)` syntax is for `INDEX(t idx_emp)` style. You 
can see the testcase for this in the testcase, too.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14313: [SPARK-16674][SQL] Avoid per-record type dispatch in JDB...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14313
  
**[Test build #62705 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62705/consoleFull)**
 for PR 14313 at commit 
[`ad64483`](https://github.com/apache/spark/commit/ad64483e2263485f843a469310fb8d252824d09e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14313: [SPARK-16674][SQL] Avoid per-record type dispatch in JDB...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/14313
  
Could you please take a look here @cloud-fan and @yhuai ? This is happening 
for writing too. I will open new one for writing as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14313: [SPARK-16674][SQL] Avoid per-record type dispatch...

[HyukjinKwon]

GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/spark/pull/14313

[SPARK-16674][SQL] Avoid per-record type dispatch in JDBC when reading

## What changes were proposed in this pull request?

Currently, `JDBCRDD.compute` is doing type dispatch for each row to read 
appropriate values.
It might not have to be done like this because the schema is already kept 
in `JDBCRDD`.

So, appropriate converters can be created first according to the schema, 
and then apply them to each row.

## How was this patch tested?

Existing tests should cover this.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/spark SPARK-16674

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14313.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14313


commit ad64483e2263485f843a469310fb8d252824d09e
Author: hyukjinkwon 
Date:   2016-07-22T03:32:44Z

Avoid per-record type dispatch in JDBC when reading




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14298: [SPARK-16283][SQL] Implement `percentile_approx` SQL fun...

[lw-lin]

Github user lw-lin commented on the issue:

https://github.com/apache/spark/pull/14298
  
@cloud-fan could you also help review this? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14210: [SPARK-16556] [SPARK-16559] [SQL] Fix Two Bugs in...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14210#discussion_r71822711
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -344,6 +344,11 @@ class SparkSqlAstBuilder(conf: SQLConf) extends 
AstBuilder {
 table, provider, partitionColumnNames, bucketSpec, mode, options, 
query)
 } else {
   val struct = Option(ctx.colTypeList()).map(createStructType)
+  if (struct.isEmpty && bucketSpec.nonEmpty) {
+throw new ParseException(
+  "Expected explicit specification of table schema when using 
CLUSTERED BY clause.", ctx)
--- End diff --

Yeah, let me find a place. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14297: [SPARK-16660][SQL] CreateViewCommand should not t...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14297#discussion_r71822672
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -88,23 +96,23 @@ case class CreateViewCommand(
 qe.assertAnalyzed()
 val analyzedPlan = qe.analyzed
 
-if (tableDesc.schema != Nil && tableDesc.schema.length != 
analyzedPlan.output.length) {
+if (userSpecifiedColumns.nonEmpty &&
--- End diff --

: ) Yeah, we should check it!  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14296: [SPARK-16639][SQL] The query with having condition that ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14296
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14296: [SPARK-16639][SQL] The query with having condition that ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14296
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62702/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14296: [SPARK-16639][SQL] The query with having condition that ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14296
  
**[Test build #62702 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62702/consoleFull)**
 for PR 14296 at commit 
[`4ca4088`](https://github.com/apache/spark/commit/4ca408867c44a3e840c4b7dd91fa4c0408ad7beb).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13373: [SPARK-15616] [SQL] Metastore relation should fallback t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13373
  
**[Test build #62704 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62704/consoleFull)**
 for PR 13373 at commit 
[`4a3e72e`](https://github.com/apache/spark/commit/4a3e72e6321fcbc449eb2fad3c903bd790cea2cb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14297: [SPARK-16660][SQL] CreateViewCommand should not t...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14297#discussion_r71821115
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -88,23 +96,23 @@ case class CreateViewCommand(
 qe.assertAnalyzed()
 val analyzedPlan = qe.analyzed
 
-if (tableDesc.schema != Nil && tableDesc.schema.length != 
analyzedPlan.output.length) {
+if (userSpecifiedColumns.nonEmpty &&
+userSpecifiedColumns.length != analyzedPlan.output.length) {
   throw new AnalysisException(s"The number of columns produced by the 
SELECT clause " +
 s"(num: `${analyzedPlan.output.length}`) does not match the number 
of column names " +
-s"specified by CREATE VIEW (num: `${tableDesc.schema.length}`).")
+s"specified by CREATE VIEW (num: 
`${userSpecifiedColumns.length}`).")
 }
 val sessionState = sparkSession.sessionState
 
 if (isTemporary) {
-  createTemporaryView(tableDesc.identifier, sparkSession, analyzedPlan)
+  createTemporaryView(sparkSession, analyzedPlan)
 } else {
   // Adds default database for permanent table if it doesn't exist, so 
that tableExists()
   // only check permanent tables.
-  val database = tableDesc.identifier.database.getOrElse(
-sessionState.catalog.getCurrentDatabase)
-  val tableIdentifier = tableDesc.identifier.copy(database = 
Option(database))
+  val database = 
name.database.getOrElse(sessionState.catalog.getCurrentDatabase)
+  val qualifiedName = name.copy(database = Option(database))
 
-  if (sessionState.catalog.tableExists(tableIdentifier)) {
+  if (sessionState.catalog.tableExists(qualifiedName)) {
--- End diff --

ah i see, it's an existing issue, feel free to open a JIRA for it :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14285: [SPARK-16649][SQL] Push partition predicates down into m...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14285
  
**[Test build #62703 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62703/consoleFull)**
 for PR 14285 at commit 
[`de3ca68`](https://github.com/apache/spark/commit/de3ca686ce54109dc3564de75b6cb6411ead7d5e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14297: [SPARK-16660][SQL] CreateViewCommand should not t...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14297#discussion_r71820977
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -88,23 +96,23 @@ case class CreateViewCommand(
 qe.assertAnalyzed()
 val analyzedPlan = qe.analyzed
 
-if (tableDesc.schema != Nil && tableDesc.schema.length != 
analyzedPlan.output.length) {
+if (userSpecifiedColumns.nonEmpty &&
--- End diff --

then? if `userSpecifiedColumns.isEmpty`, what should we check? And it's 
wrong to not have this check, users don't need to always specify view columns.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71820467
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/catalyst/SQLBuilder.scala ---
@@ -193,6 +196,9 @@ class SQLBuilder(logicalPlan: LogicalPlan) extends 
Logging {
 case OneRowRelation =>
   ""
 
+case Hint(_, _, child) =>
--- End diff --

will an analyzed plan contain `Hint`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71820260
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ---
@@ -339,8 +339,11 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] 
with Logging {
   case SqlBaseParser.SELECT =>
 // Regular select
 
+// Add hints.
+val withHint = relation.optionalMap(ctx.hint)(withHints)
--- End diff --

according to the grammar: `SELECT /*+ hint... */ ...`, it seems that hint 
should be applied to the whole SELECT clause, not only its relations?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14302: [SPARK-16663][SQL] desc table should be consistent betwe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14302
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14302: [SPARK-16663][SQL] desc table should be consistent betwe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14302
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62699/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14302: [SPARK-16663][SQL] desc table should be consistent betwe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14302
  
**[Test build #62699 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62699/consoleFull)**
 for PR 14302 at commit 
[`6726676`](https://github.com/apache/spark/commit/67266763bac72bee6538603eb05eed5864665079).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14132#discussion_r71819403
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -347,6 +347,16 @@ querySpecification
windows?)
 ;
 
+hint
+: '/*+' hintStatement '*/'
+;
+
+hintStatement
+: hintName=identifier
+| hintName=identifier '(' parameters+=identifier 
parameters+=identifier ')'
--- End diff --

do we need this rule for broadcast hint?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14297: [SPARK-16660][SQL] CreateViewCommand should not t...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14297#discussion_r71819083
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -88,23 +96,23 @@ case class CreateViewCommand(
 qe.assertAnalyzed()
 val analyzedPlan = qe.analyzed
 
-if (tableDesc.schema != Nil && tableDesc.schema.length != 
analyzedPlan.output.length) {
+if (userSpecifiedColumns.nonEmpty &&
--- End diff --

We allow users to create a table containing zero column, right? See my 
previous PR: https://github.com/apache/spark/pull/14007


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14297: [SPARK-16660][SQL] CreateViewCommand should not t...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14297#discussion_r71818729
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -88,23 +96,23 @@ case class CreateViewCommand(
 qe.assertAnalyzed()
 val analyzedPlan = qe.analyzed
 
-if (tableDesc.schema != Nil && tableDesc.schema.length != 
analyzedPlan.output.length) {
+if (userSpecifiedColumns.nonEmpty &&
+userSpecifiedColumns.length != analyzedPlan.output.length) {
   throw new AnalysisException(s"The number of columns produced by the 
SELECT clause " +
 s"(num: `${analyzedPlan.output.length}`) does not match the number 
of column names " +
-s"specified by CREATE VIEW (num: `${tableDesc.schema.length}`).")
+s"specified by CREATE VIEW (num: 
`${userSpecifiedColumns.length}`).")
 }
 val sessionState = sparkSession.sessionState
 
 if (isTemporary) {
-  createTemporaryView(tableDesc.identifier, sparkSession, analyzedPlan)
+  createTemporaryView(sparkSession, analyzedPlan)
 } else {
   // Adds default database for permanent table if it doesn't exist, so 
that tableExists()
   // only check permanent tables.
-  val database = tableDesc.identifier.database.getOrElse(
-sessionState.catalog.getCurrentDatabase)
-  val tableIdentifier = tableDesc.identifier.copy(database = 
Option(database))
+  val database = 
name.database.getOrElse(sessionState.catalog.getCurrentDatabase)
+  val qualifiedName = name.copy(database = Option(database))
 
-  if (sessionState.catalog.tableExists(tableIdentifier)) {
+  if (sessionState.catalog.tableExists(qualifiedName)) {
--- End diff --

Below is what I got in Hive.
```
hive> CREATE TABLE tab1 (id int);
OK
Time taken: 0.196 seconds
hive> CREATE OR REPLACE VIEW tab1 AS SELECT * FROM t1;
FAILED: SemanticException [Error 10218]: Existing table is not a view
 The following is an existing table, not a view: default.tab1
```

I found that `LOAD TABLE` also misses the table type checking. Issue a 
strange runtime error.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14265: [PySpark] add picklable SparseMatrix in pyspark.ml.commo...

[WeichenXu123]

Github user WeichenXu123 commented on the issue:

https://github.com/apache/spark/pull/14265
  
cc @jkbradley Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org