[GitHub] spark issue #13400: [SPARK-15655] [SQL] Fix Wrong Partition Column Order whe...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/13400
  
LGTM, pending jenkins


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13655: [SPARK-15935][PySpark]Enable test for sql/streaming.py a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13655
  
**[Test build #60465 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60465/consoleFull)**
 for PR 13655 at commit 
[`88d5b93`](https://github.com/apache/spark/commit/88d5b93e9b0ea2ebbf5ae62a7c9b32aa6357a647).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13630: Revert "Incorrectly merged AFTAggregator with zero total...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/13630
  
@mengxr @yanboliang Could you please take a look? The build is being failed 
due to this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13655: [SPARK-15935][PySpark]Enable test for sql/streaming.py a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13655
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60465/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13655: [SPARK-15935][PySpark]Enable test for sql/streaming.py a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13655
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13400: [SPARK-15655] [SQL] Fix Wrong Partition Column Order whe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13400
  
**[Test build #60470 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60470/consoleFull)**
 for PR 13400 at commit 
[`351806d`](https://github.com/apache/spark/commit/351806d368277507bce6953ed9b7aef377f44487).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13400: [SPARK-15655] [SQL] Fix Wrong Partition Column Order whe...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/13400
  
@cloud-fan @clockfly Thank you for your review! Let me know if the latest 
code changes look fine. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13620: [SPARK-15590] [WEBUI] Paginate Job Table in Jobs tab

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13620
  
**[Test build #60469 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60469/consoleFull)**
 for PR 13620 at commit 
[`6682e86`](https://github.com/apache/spark/commit/6682e862a3dd36ea22c1b7838dc4355d24e7f82f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13648: [SPARK-15932][SQL][DOC] document the contract of ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13648


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13648: [SPARK-15932][SQL][DOC] document the contract of encoder...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/13648
  
Merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13413


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/13413
  
Thanks. Merging to master and branch 2.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13380: [SPARK-15644] [MLlib] [SQL] Replace SQLContext with Spar...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13380
  
**[Test build #60468 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60468/consoleFull)**
 for PR 13380 at commit 
[`7dcaaa4`](https://github.com/apache/spark/commit/7dcaaa4539324d596c290ca189adc2e65d77bb0c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13593: [SPARK-15864] [SQL] Fix Inconsistent Behaviors when Unca...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13593
  
**[Test build #60467 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60467/consoleFull)**
 for PR 13593 at commit 
[`ef3572c`](https://github.com/apache/spark/commit/ef3572cae7f52c8ded69e767ca704ca74972b600).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13413
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13413
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60464/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13380: [SPARK-15644] [MLlib] [SQL] Replace SQLContext with Spar...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/13380
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13593: [SPARK-15864] [SQL] Fix Inconsistent Behaviors when Unca...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/13593
  
@cloud-fan Thank you for letting me know it. Just updated the PR. Please 
let me know if anything needs a change. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13413
  
**[Test build #60464 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60464/consoleFull)**
 for PR 13413 at commit 
[`9bfd777`](https://github.com/apache/spark/commit/9bfd777298469db49256a52165ba9061dc0f1af3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12938: [WIP][SPARK-15162][SPARK-15164][PySpark][DOCS][ML] updat...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/12938
  
My guess is f7288e166c696da15e790c28fc3ed78531fd362d is the culprit but 
lets see.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12938: [WIP][SPARK-15162][SPARK-15164][PySpark][DOCS][ML] updat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/12938
  
**[Test build #60466 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60466/consoleFull)**
 for PR 12938 at commit 
[`2be8cdf`](https://github.com/apache/spark/commit/2be8cdf15d6431c8d2ef0d7a6c5d136e2b71410e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13653
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60461/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13653
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13653
  
**[Test build #60461 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60461/consoleFull)**
 for PR 13653 at commit 
[`bbfff70`](https://github.com/apache/spark/commit/bbfff70b0bac8b2c33ab1eb736405759bb391633).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13219: [SPARK-15364][ML][PySpark] Implement PySpark picklers fo...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/13219
  
@mengxr Thank you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13655: [SPARK-15935][PySpark]Enable test for sql/streaming.py a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13655
  
**[Test build #60465 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60465/consoleFull)**
 for PR 13655 at commit 
[`88d5b93`](https://github.com/apache/spark/commit/88d5b93e9b0ea2ebbf5ae62a7c9b32aa6357a647).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13636: [SPARK-15637][SPARKR] Remove R version check since maske...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/13636
  
FYI, this is what I see running on R 3.3:
```
1. Error: string operators (@test_sparkSQL.R#1163) 
-
unable to find an inherited method for function ‘startsWith’ for 
signature ‘"character"’
1: expect_true(startsWith("Hello World", "Hello")) at 
/opt/spark-2.0.0-bin-hadoop2.6/R/lib/SparkR/tests/testthat/test_sparkSQL.R:1163
2: expect(identical(as.vector(object), TRUE), sprintf("%s isn't true.", 
lab), info = info)
3: as.expectation(exp, ..., srcref = srcref)
4: identical(as.vector(object), TRUE)
5: as.vector(object)
6: startsWith("Hello World", "Hello")
7: (function (classes, fdef, mtable)
   {
   methods <- .findInheritedMethods(classes, fdef, mtable)
   if (length(methods) == 1L)
   return(methods[[1L]])
   else if (length(methods) == 0L) {
   cnames <- paste0("\"", vapply(classes, as.character, ""), "\"", 
collapse = ", ")
   stop(gettextf("unable to find an inherited method for function 
%s for signature %s",
   sQuote(fdef@generic), sQuote(cnames)), domain = NA)
   }
   else stop("Internal error in finding inherited methods; didn't 
return a unique method",
   domain = NA)
   })(list("character"), structure(function (x, prefix)
   {
   standardGeneric("startsWith")
   }, generic = structure("startsWith", package = "SparkR"), package = 
"SparkR", group = list(), valueClass = character(0), signature = c("x",
   "prefix"), default = `\001NULL\001`, skeleton = (function (x, prefix)
   stop("invalid call in method dispatch to 'startsWith' (no default 
method)", domain = NA))(x,
   prefix), class = structure("nonstandardGenericFunction", package = 
"methods")),
   )
8: stop(gettextf("unable to find an inherited method for function %s for 
signature %s",
   sQuote(fdef@generic), sQuote(cnames)), domain = NA)
```

I will double check if there is indeed a problem with `startsWith("Hello 
World", "Hello")` when SparkR is loaded, or the tests are not up-to-date on 
what's being masked.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13585: [SPARK-15859][SQL] Optimize the partition pruning within...

[lianhuiwang]

Github user lianhuiwang commented on the issue:

https://github.com/apache/spark/pull/13585
  
@liancheng @chenghao-intel I think we can did it like mysql's range 
optimizer:http://dev.mysql.com/doc/refman/5.7/en/range-optimization.html. I 
have implemented it in my 
branch:https://github.com/lianhuiwang/spark/tree/partition_pruning. I did not 
combine many ranges because it will transform partition pruning predicates to 
Hive's filter expression in getPartitionsByFilter() and Hive's metastore 
database can filter partitions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13638
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60459/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13638
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13638
  
**[Test build #60459 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60459/consoleFull)**
 for PR 13638 at commit 
[`0c9ec86`](https://github.com/apache/spark/commit/0c9ec86f7e2646466b91d75b26dc29de31b98d31).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13630: Revert "Incorrectly merged AFTAggregator with zero total...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13630
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60463/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13630: Revert "Incorrectly merged AFTAggregator with zero total...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13630
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13630: Revert "Incorrectly merged AFTAggregator with zero total...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13630
  
**[Test build #60463 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60463/consoleFull)**
 for PR 13630 at commit 
[`4e344cb`](https://github.com/apache/spark/commit/4e344cbbcb916c203917998902bfd4f7da5f7cb6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/13413
  
lgtm pending jenkins


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13219: [SPARK-15364][ML][PySpark] Implement PySpark pick...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13219


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13219: [SPARK-15364][ML][PySpark] Implement PySpark picklers fo...

[mengxr]

Github user mengxr commented on the issue:

https://github.com/apache/spark/pull/13219
  
LGTM. Merged into master and branch-2.0. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13655: [SPARK-15935][PySpark]Enable test for sql/streaming.py a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13655
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13655: [SPARK-15935][PySpark]Enable test for sql/streaming.py a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13655
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60457/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13655: [SPARK-15935][PySpark]Enable test for sql/streaming.py a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13655
  
**[Test build #60457 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60457/consoleFull)**
 for PR 13655 at commit 
[`0b15d88`](https://github.com/apache/spark/commit/0b15d88b0cde790e9aa58eb34aa22c57642b8886).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13638
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60458/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13638
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13638
  
**[Test build #60458 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60458/consoleFull)**
 for PR 13638 at commit 
[`3d27607`](https://github.com/apache/spark/commit/3d27607982a3794c438221153b8e078389da146b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13413
  
**[Test build #60464 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60464/consoleFull)**
 for PR 13413 at commit 
[`9bfd777`](https://github.com/apache/spark/commit/9bfd777298469db49256a52165ba9061dc0f1af3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

[techaddict]

Github user techaddict commented on the issue:

https://github.com/apache/spark/pull/13413
  
@yhuai All comments Addressed 👍 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13654: [SPARK-15868] [Web UI] Executors table in Executors tab ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13654
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13654: [SPARK-15868] [Web UI] Executors table in Executors tab ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13654
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60455/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13654: [SPARK-15868] [Web UI] Executors table in Executors tab ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13654
  
**[Test build #60455 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60455/consoleFull)**
 for PR 13654 at commit 
[`35b280a`](https://github.com/apache/spark/commit/35b280a1882fd3f5ae34c71cb00a12bf12852a90).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13619: [SPARK-15892][ML] Incorrectly merged AFTAggregator with ...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/spark/pull/13619
  
@HyukjinKwon , just see it, thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13619: [SPARK-15892][ML] Incorrectly merged AFTAggregator with ...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/13619
  
@zzcclp I just did here https://github.com/apache/spark/pull/13630


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13619: [SPARK-15892][ML] Incorrectly merged AFTAggregator with ...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/spark/pull/13619
  
@HyukjinKwon ,  could you revert this pr for branch-1.6 ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13546: [SPARK-15808] [SQL] File Format Checking When App...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13546


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13037: [SPARK-1301] [Web UI] Added anchor links to Accumulators...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13037
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60454/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13037: [SPARK-1301] [Web UI] Added anchor links to Accumulators...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13037
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13630: Revert "Incorrectly merged AFTAggregator with zero total...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13630
  
**[Test build #60463 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60463/consoleFull)**
 for PR 13630 at commit 
[`4e344cb`](https://github.com/apache/spark/commit/4e344cbbcb916c203917998902bfd4f7da5f7cb6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13595: [MINOR][SQL] Standardize 'continuous queries' to 'stream...

[lw-lin]

Github user lw-lin commented on the issue:

https://github.com/apache/spark/pull/13595
  
Thanks! 
This patch introduced an compilation error because `DataFrameReader.text`'s 
return type had been changed back to `DataFrame` very recently, and I should 
have noticed this and updated this patch accordingly. Sorry for the 
inconvenience.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13037: [SPARK-1301] [Web UI] Added anchor links to Accumulators...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13037
  
**[Test build #60454 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60454/consoleFull)**
 for PR 13037 at commit 
[`9e62eb3`](https://github.com/apache/spark/commit/9e62eb3344e41814b12a783ec51bec871dc8320b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13630: Revert "Incorrectly merged AFTAggregator with zero total...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/13630
  
@jkbradley I will leave this open just in case. Please feel free to do it 
yourself or merge this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13639: [DOCUMENTATION] fixed typos in python programming guide

[mortada]

Github user mortada commented on the issue:

https://github.com/apache/spark/pull/13639
  
@srowen I went through the docs, found a few more minor fixes 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13546: [SPARK-15808] [SQL] File Format Checking When Appending ...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/13546
  
Thanks. Merging to master and branch 2.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13630: [SPARK-15892][ML] Change spark to sqlContext in t...

[HyukjinKwon]

GitHub user HyukjinKwon reopened a pull request:

https://github.com/apache/spark/pull/13630

[SPARK-15892][ML] Change spark to sqlContext in the test in 
AFTSurvivalRegressionSuite for 1.6

## What changes were proposed in this pull request?

https://github.com/apache/spark/pull/13619 was merged into master, 
branch-2.0 and 1.6 as well but the unit test uses `spark`.

So, this PR change the `spark` to `sqlContext` in the unit test for 1.6.

It seems builds for 1.6 are failing due to this.

## How was this patch tested?

Jenkins tests.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/spark SPARK-15892-1.6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13630.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13630


commit 022d48d52127fb3cab804a78eed9ff253b76
Author: hyukjinkwon 
Date:   2016-06-13T02:04:39Z

Change spark to sqlContext for 1.6




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12938: [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update som...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/12938
  
**[Test build #60462 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60462/consoleFull)**
 for PR 12938 at commit 
[`3981612`](https://github.com/apache/spark/commit/398161285321c976e850f18b486e512efbe0d24e).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12938: [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update som...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/12938
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60462/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12938: [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update som...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/12938
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13561: [SPARK-15824][SQL] Run 'with ... insert ... select' fail...

[hvanhovell]

Github user hvanhovell commented on the issue:

https://github.com/apache/spark/pull/13561
  
Just tried it in Databrick CE Spark 1.6.1 and I can reproduce the bug 
there. The `funny` thing is that the `WITH` clause actually matters; if you 
remove it the query works fine.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12938: [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update som...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/12938
  
**[Test build #60462 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60462/consoleFull)**
 for PR 12938 at commit 
[`3981612`](https://github.com/apache/spark/commit/398161285321c976e850f18b486e512efbe0d24e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13037: [SPARK-1301] [Web UI] Added anchor links to Accumulators...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13037
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60453/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13037: [SPARK-1301] [Web UI] Added anchor links to Accumulators...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13037
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[ueshin]

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/13638
  
@marmbrus It seems this is not a regression from `1.6`.
The test modified for `1.6` also fails with the same error.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13037: [SPARK-1301] [Web UI] Added anchor links to Accumulators...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13037
  
**[Test build #60453 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60453/consoleFull)**
 for PR 13037 at commit 
[`5b12cff`](https://github.com/apache/spark/commit/5b12cff99a7252193da1e555185da50d4b973750).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13653
  
**[Test build #60461 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60461/consoleFull)**
 for PR 13653 at commit 
[`bbfff70`](https://github.com/apache/spark/commit/bbfff70b0bac8b2c33ab1eb736405759bb391633).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13655: [SPARK-15935][PySpark]Enable test for sql/streami...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/13655#discussion_r66898042
  
--- Diff: python/pyspark/sql/streaming.py ---
@@ -209,27 +226,27 @@ def _test():
 import doctest
 import os
 import tempfile
-from pyspark.context import SparkContext
-from pyspark.sql import Row, SQLContext, HiveContext
-import pyspark.sql.readwriter
+from pyspark.sql import Row, SparkSession, HiveContext
+import pyspark.sql.streaming
 
 os.chdir(os.environ["SPARK_HOME"])
 
-globs = pyspark.sql.readwriter.__dict__.copy()
-sc = SparkContext('local[4]', 'PythonTest')
+globs = pyspark.sql.streaming.__dict__.copy()
+try:
+spark = SparkSession.builder.enableHiveSupport().getOrCreate()
+except py4j.protocol.Py4JError:
+spark = SparkSession(sc)
 
 globs['tempfile'] = tempfile
 globs['os'] = os
-globs['sc'] = sc
-globs['sqlContext'] = SQLContext(sc)
-globs['hiveContext'] = HiveContext._createForTesting(sc)
+globs['spark'] = spark
 globs['df'] = \
-
globs['sqlContext'].read.format('text').stream('python/test_support/sql/streaming')
+
globs['spark'].read.format('text').stream('python/test_support/sql/streaming')
--- End diff --

there should be some tests for sqlcontext as well, to make sure something 
fails if it gets removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13653
  
**[Test build #60460 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60460/consoleFull)**
 for PR 13653 at commit 
[`e118631`](https://github.com/apache/spark/commit/e118631fd5bbea962932a812addae21c89f6f4f2).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class DataStreamReader(object):`
  * `class DataStreamWriter(object):`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13653
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60460/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13653
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13655: [SPARK-15935][PySpark]Enable test for sql/streami...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/13655#discussion_r66897860
  
--- Diff: python/pyspark/sql/session.py ---
@@ -549,6 +549,15 @@ def read(self):
 """
 return DataFrameReader(self._wrapped)
 
+@property
+@since(2.0)
+def streams(self):
+"""Returns a :class:`ContinuousQueryManager` that allows managing 
all the
+:class:`ContinuousQuery` ContinuousQueries active on `this` 
context.
--- End diff --

add experimental tag.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13653
  
**[Test build #60460 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60460/consoleFull)**
 for PR 13653 at commit 
[`e118631`](https://github.com/apache/spark/commit/e118631fd5bbea962932a812addae21c89f6f4f2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13546: [SPARK-15808] [SQL] File Format Checking When Appending ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13546
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60448/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13561: [SPARK-15824][SQL] Run 'with ... insert ... select' fail...

[hvanhovell]

Github user hvanhovell commented on the issue:

https://github.com/apache/spark/pull/13561
  
@Sephiroth-Lin thanks for working on this. What I do not understand is, is 
that the current code path and the code path in 1.6 are generally the same. Any 
idea why this is surfacing now?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13546: [SPARK-15808] [SQL] File Format Checking When Appending ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13546
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13546: [SPARK-15808] [SQL] File Format Checking When Appending ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13546
  
**[Test build #60448 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60448/consoleFull)**
 for PR 13546 at commit 
[`9d9d263`](https://github.com/apache/spark/commit/9d9d2632fde85c62b28454f33b24f7ee8fb6f15e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12938: [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update som...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/12938
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12938: [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update som...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/12938
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60456/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12938: [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update som...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/12938
  
**[Test build #60456 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60456/consoleFull)**
 for PR 12938 at commit 
[`d925f38`](https://github.com/apache/spark/commit/d925f3861ac96ddb09682b863160b18adcb4473d).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13638
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60451/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13638
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13561: [SPARK-15824][SQL] Run 'with ... insert ... selec...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/13561#discussion_r66896852
  
--- Diff: 
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
 ---
@@ -225,7 +225,10 @@ private[hive] class SparkExecuteStatementOperation(
 if (useIncrementalCollect) {
   result.toLocalIterator.asScala
 } else {
-  result.collect().iterator
+  
sqlContext.sessionState.executePlan(result.logicalPlan).executedPlan match {
+case command: ExecutedCommandExec => 
command.executeCollectPublic().iterator
--- End diff --

Two things here:
1. I am pretty sure that not every `RunnableCommand` calls 
`withNewExecutionId`. See `SetCommand` for instance. Maybe we should modify 
`InsertIntoHadoopFsRelation` instead.
2. Technically a `Dataset.collect()` can return something else than a 
`Row`, but I think this is OK since this is Thrift.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13637: [SPARK-15914][SQL] Add deprecated method back to SQLCont...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13637
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60447/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13638
  
**[Test build #60451 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60451/consoleFull)**
 for PR 13638 at commit 
[`11dc433`](https://github.com/apache/spark/commit/11dc433975719288a3694746c60a571d3f349b22).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13637: [SPARK-15914][SQL] Add deprecated method back to SQLCont...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13637
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13637: [SPARK-15914][SQL] Add deprecated method back to SQLCont...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13637
  
**[Test build #60447 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60447/consoleFull)**
 for PR 13637 at commit 
[`0cc81b1`](https://github.com/apache/spark/commit/0cc81b1d7dc3c4e05e3adf5f72c25c320a043a0c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13620: [SPARK-15590] [WEBUI] Paginate Job Table in Jobs ...

[nblintao]

Github user nblintao commented on a diff in the pull request:

https://github.com/apache/spark/pull/13620#discussion_r66896475
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -369,3 +361,246 @@ private[ui] class AllJobsPage(parent: JobsTab) 
extends WebUIPage("") {
 }
   }
 }
+
+private[ui] class JobTableRowData(
+val jobData: JobUIData,
+val lastStageName: String,
+val lastStageDescription: String,
+val duration: Long,
+val formattedDuration: String,
+val submissionTime: Long,
+val formattedSubmissionTime: String,
+val jobDescription: NodeSeq,
+val detailUrl: String)
+
+private[ui] class JobDataSource(
+jobs: Seq[JobUIData],
+stageIdToInfo: HashMap[Int, StageInfo],
+stageIdToData: HashMap[(Int, Int), StageUIData],
+basePath: String,
+currentTime: Long,
+pageSize: Int,
+sortColumn: String,
+desc: Boolean) extends PagedDataSource[JobTableRowData](pageSize) {
+
+  // Convert JobUIData to JobTableRowData which contains the final 
contents to show in the table
+  // so that we can avoid creating duplicate contents during sorting the 
data
+  private val data = jobs.map(jobRow).sorted(ordering(sortColumn, desc))
+
+  private var _slicedJobIds: Set[Int] = null
+
+  override def dataSize: Int = data.size
+
+  override def sliceData(from: Int, to: Int): Seq[JobTableRowData] = {
+val r = data.slice(from, to)
+_slicedJobIds = r.map(_.jobData.jobId).toSet
+r
+  }
+
+  def slicedJobIds: Set[Int] = _slicedJobIds
+
+  private def getLastStageNameAndDescription(job: JobUIData): (String, 
String) = {
+val lastStageInfo = Option(job.stageIds)
+  .filter(_.nonEmpty)
+  .flatMap { ids => stageIdToInfo.get(ids.max)}
+val lastStageData = lastStageInfo.flatMap { s =>
+  stageIdToData.get((s.stageId, s.attemptId))
+}
+val name = lastStageInfo.map(_.name).getOrElse("(Unknown Stage Name)")
+val description = lastStageData.flatMap(_.description).getOrElse("")
+(name, description)
+  }
+
+  private def jobRow(jobData: JobUIData): JobTableRowData = {
+val (lastStageName, lastStageDescription) = 
getLastStageNameAndDescription(jobData)
+val duration: Option[Long] = {
+  jobData.submissionTime.map { start =>
+val end = 
jobData.completionTime.getOrElse(System.currentTimeMillis())
+end - start
+  }
+}
+val formattedDuration = duration.map(d => 
UIUtils.formatDuration(d)).getOrElse("Unknown")
+val submissionTime = jobData.submissionTime
+val formattedSubmissionTime = 
submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
+val jobDescription = UIUtils.makeDescription(lastStageDescription, 
basePath, plainText = false)
+
+val detailUrl = "%s/jobs/job?id=%s".format(basePath, jobData.jobId)
+
+new JobTableRowData (
+  jobData,
+  lastStageName,
+  lastStageDescription,
+  duration.getOrElse(-1),
+  formattedDuration,
+  submissionTime.getOrElse(-1),
+  formattedSubmissionTime,
+  jobDescription,
+  detailUrl
+)
+  }
+
+  /**
+   * Return Ordering according to sortColumn and desc
+   */
+  private def ordering(sortColumn: String, desc: Boolean): 
Ordering[JobTableRowData] = {
+val ordering = sortColumn match {
+  case "Job Id" | "Job Id (Job Group)" => new 
Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.Int.compare(x.jobData.jobId, y.jobData.jobId)
+  }
+  case "Description" => new Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.String.compare(x.lastStageDescription, 
y.lastStageDescription)
+  }
+  case "Submitted" => new Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.Long.compare(x.submissionTime, y.submissionTime)
+  }
+  case "Duration" => new Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.Long.compare(x.duration, y.duration)
+  }
+  case "Stages: Succeeded/Total" | "Tasks (for all stages): 
Succeeded/Total" =>
+throw new IllegalArgumentException(s"Unsortable column: 
$sortColumn")
+  case unknownColumn => throw new IllegalArgumentException(s"Unknown 
column: $unknownColumn")
+}
+if (desc) {
+  ordering.reverse
+} else {
+  ordering
+}
+  }
+

[GitHub] spark pull request #13620: [SPARK-15590] [WEBUI] Paginate Job Table in Jobs ...

[nblintao]

Github user nblintao commented on a diff in the pull request:

https://github.com/apache/spark/pull/13620#discussion_r66896310
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -369,3 +361,246 @@ private[ui] class AllJobsPage(parent: JobsTab) 
extends WebUIPage("") {
 }
   }
 }
+
+private[ui] class JobTableRowData(
+val jobData: JobUIData,
+val lastStageName: String,
+val lastStageDescription: String,
+val duration: Long,
+val formattedDuration: String,
+val submissionTime: Long,
+val formattedSubmissionTime: String,
+val jobDescription: NodeSeq,
+val detailUrl: String)
+
+private[ui] class JobDataSource(
+jobs: Seq[JobUIData],
+stageIdToInfo: HashMap[Int, StageInfo],
+stageIdToData: HashMap[(Int, Int), StageUIData],
+basePath: String,
+currentTime: Long,
+pageSize: Int,
+sortColumn: String,
+desc: Boolean) extends PagedDataSource[JobTableRowData](pageSize) {
+
+  // Convert JobUIData to JobTableRowData which contains the final 
contents to show in the table
+  // so that we can avoid creating duplicate contents during sorting the 
data
+  private val data = jobs.map(jobRow).sorted(ordering(sortColumn, desc))
+
+  private var _slicedJobIds: Set[Int] = null
+
+  override def dataSize: Int = data.size
+
+  override def sliceData(from: Int, to: Int): Seq[JobTableRowData] = {
+val r = data.slice(from, to)
+_slicedJobIds = r.map(_.jobData.jobId).toSet
+r
+  }
+
+  def slicedJobIds: Set[Int] = _slicedJobIds
+
+  private def getLastStageNameAndDescription(job: JobUIData): (String, 
String) = {
+val lastStageInfo = Option(job.stageIds)
+  .filter(_.nonEmpty)
+  .flatMap { ids => stageIdToInfo.get(ids.max)}
+val lastStageData = lastStageInfo.flatMap { s =>
+  stageIdToData.get((s.stageId, s.attemptId))
+}
+val name = lastStageInfo.map(_.name).getOrElse("(Unknown Stage Name)")
+val description = lastStageData.flatMap(_.description).getOrElse("")
+(name, description)
+  }
+
+  private def jobRow(jobData: JobUIData): JobTableRowData = {
+val (lastStageName, lastStageDescription) = 
getLastStageNameAndDescription(jobData)
+val duration: Option[Long] = {
+  jobData.submissionTime.map { start =>
+val end = 
jobData.completionTime.getOrElse(System.currentTimeMillis())
+end - start
+  }
+}
+val formattedDuration = duration.map(d => 
UIUtils.formatDuration(d)).getOrElse("Unknown")
+val submissionTime = jobData.submissionTime
+val formattedSubmissionTime = 
submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
+val jobDescription = UIUtils.makeDescription(lastStageDescription, 
basePath, plainText = false)
+
+val detailUrl = "%s/jobs/job?id=%s".format(basePath, jobData.jobId)
+
+new JobTableRowData (
+  jobData,
+  lastStageName,
+  lastStageDescription,
+  duration.getOrElse(-1),
+  formattedDuration,
+  submissionTime.getOrElse(-1),
+  formattedSubmissionTime,
+  jobDescription,
+  detailUrl
+)
+  }
+
+  /**
+   * Return Ordering according to sortColumn and desc
+   */
+  private def ordering(sortColumn: String, desc: Boolean): 
Ordering[JobTableRowData] = {
+val ordering = sortColumn match {
+  case "Job Id" | "Job Id (Job Group)" => new 
Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.Int.compare(x.jobData.jobId, y.jobData.jobId)
+  }
+  case "Description" => new Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.String.compare(x.lastStageDescription, 
y.lastStageDescription)
+  }
+  case "Submitted" => new Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.Long.compare(x.submissionTime, y.submissionTime)
+  }
+  case "Duration" => new Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.Long.compare(x.duration, y.duration)
+  }
+  case "Stages: Succeeded/Total" | "Tasks (for all stages): 
Succeeded/Total" =>
+throw new IllegalArgumentException(s"Unsortable column: 
$sortColumn")
+  case unknownColumn => throw new IllegalArgumentException(s"Unknown 
column: $unknownColumn")
+}
+if (desc) {
+  ordering.reverse
+} else {
+  ordering
+}
+  }
+

[GitHub] spark pull request #13620: [SPARK-15590] [WEBUI] Paginate Job Table in Jobs ...

[nblintao]

Github user nblintao commented on a diff in the pull request:

https://github.com/apache/spark/pull/13620#discussion_r66895790
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -369,3 +361,246 @@ private[ui] class AllJobsPage(parent: JobsTab) 
extends WebUIPage("") {
 }
   }
 }
+
+private[ui] class JobTableRowData(
+val jobData: JobUIData,
+val lastStageName: String,
+val lastStageDescription: String,
+val duration: Long,
+val formattedDuration: String,
+val submissionTime: Long,
+val formattedSubmissionTime: String,
+val jobDescription: NodeSeq,
+val detailUrl: String)
+
+private[ui] class JobDataSource(
+jobs: Seq[JobUIData],
+stageIdToInfo: HashMap[Int, StageInfo],
+stageIdToData: HashMap[(Int, Int), StageUIData],
+basePath: String,
+currentTime: Long,
+pageSize: Int,
+sortColumn: String,
+desc: Boolean) extends PagedDataSource[JobTableRowData](pageSize) {
+
+  // Convert JobUIData to JobTableRowData which contains the final 
contents to show in the table
+  // so that we can avoid creating duplicate contents during sorting the 
data
+  private val data = jobs.map(jobRow).sorted(ordering(sortColumn, desc))
+
+  private var _slicedJobIds: Set[Int] = null
+
+  override def dataSize: Int = data.size
+
+  override def sliceData(from: Int, to: Int): Seq[JobTableRowData] = {
+val r = data.slice(from, to)
+_slicedJobIds = r.map(_.jobData.jobId).toSet
+r
+  }
+
+  def slicedJobIds: Set[Int] = _slicedJobIds
+
+  private def getLastStageNameAndDescription(job: JobUIData): (String, 
String) = {
+val lastStageInfo = Option(job.stageIds)
+  .filter(_.nonEmpty)
+  .flatMap { ids => stageIdToInfo.get(ids.max)}
+val lastStageData = lastStageInfo.flatMap { s =>
+  stageIdToData.get((s.stageId, s.attemptId))
+}
+val name = lastStageInfo.map(_.name).getOrElse("(Unknown Stage Name)")
+val description = lastStageData.flatMap(_.description).getOrElse("")
+(name, description)
+  }
+
+  private def jobRow(jobData: JobUIData): JobTableRowData = {
+val (lastStageName, lastStageDescription) = 
getLastStageNameAndDescription(jobData)
+val duration: Option[Long] = {
+  jobData.submissionTime.map { start =>
+val end = 
jobData.completionTime.getOrElse(System.currentTimeMillis())
+end - start
+  }
+}
+val formattedDuration = duration.map(d => 
UIUtils.formatDuration(d)).getOrElse("Unknown")
+val submissionTime = jobData.submissionTime
+val formattedSubmissionTime = 
submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
+val jobDescription = UIUtils.makeDescription(lastStageDescription, 
basePath, plainText = false)
+
+val detailUrl = "%s/jobs/job?id=%s".format(basePath, jobData.jobId)
+
+new JobTableRowData (
+  jobData,
+  lastStageName,
+  lastStageDescription,
+  duration.getOrElse(-1),
+  formattedDuration,
+  submissionTime.getOrElse(-1),
+  formattedSubmissionTime,
+  jobDescription,
+  detailUrl
+)
+  }
+
+  /**
+   * Return Ordering according to sortColumn and desc
+   */
+  private def ordering(sortColumn: String, desc: Boolean): 
Ordering[JobTableRowData] = {
+val ordering = sortColumn match {
+  case "Job Id" | "Job Id (Job Group)" => new 
Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.Int.compare(x.jobData.jobId, y.jobData.jobId)
+  }
+  case "Description" => new Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.String.compare(x.lastStageDescription, 
y.lastStageDescription)
+  }
+  case "Submitted" => new Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.Long.compare(x.submissionTime, y.submissionTime)
+  }
+  case "Duration" => new Ordering[JobTableRowData] {
+override def compare(x: JobTableRowData, y: JobTableRowData): Int =
+  Ordering.Long.compare(x.duration, y.duration)
+  }
+  case "Stages: Succeeded/Total" | "Tasks (for all stages): 
Succeeded/Total" =>
+throw new IllegalArgumentException(s"Unsortable column: 
$sortColumn")
+  case unknownColumn => throw new IllegalArgumentException(s"Unknown 
column: $unknownColumn")
+}
+if (desc) {
+  ordering.reverse
+} else {
+  ordering
+}
+  }
+

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13638
  
**[Test build #60459 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60459/consoleFull)**
 for PR 13638 at commit 
[`0c9ec86`](https://github.com/apache/spark/commit/0c9ec86f7e2646466b91d75b26dc29de31b98d31).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13655: [SPARK-15935][PySpark]Enable test for sql/streaming.py a...

[brkyvz]

Github user brkyvz commented on the issue:

https://github.com/apache/spark/pull/13655
  
a minor nit, otherwise LGTM!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13655: [SPARK-15935][PySpark]Enable test for sql/streami...

[brkyvz]

Github user brkyvz commented on a diff in the pull request:

https://github.com/apache/spark/pull/13655#discussion_r66895718
  
--- Diff: python/pyspark/sql/streaming.py ---
@@ -106,28 +114,37 @@ def active(self):
 """Returns a list of active queries associated with this SQLContext
 
 >>> cq = 
df.write.format('memory').queryName('this_query').startStream()
->>> cqm = sqlContext.streams
+>>> cqm = spark.streams
 >>> # get the list of active continuous queries
 >>> [q.name for q in cqm.active]
 [u'this_query']
 >>> cq.stop()
 """
 return [ContinuousQuery(jcq) for jcq in self._jcqm.active()]
 
+@ignore_unicode_prefix
 @since(2.0)
-def get(self, name):
+def get(self, id):
 """Returns an active query from this SQLContext or throws 
exception if an active query
 with this name doesn't exist.
 
->>> df.write.format('memory').queryName('this_query').startStream()
->>> cq = sqlContext.streams.get('this_query')
+>>> cq = 
df.write.format('memory').queryName('this_query').startStream()
+>>> cq.name
+u'this_query'
+>>> cq = spark.streams.get(cq.id)
 >>> cq.isActive
 True
 >>> cq.stop()
 """
-if type(name) != str or len(name.strip()) == 0:
-raise ValueError("The name for the query must be a non-empty 
string. Got: %s" % name)
-return ContinuousQuery(self._jcqm.get(name))
+import sys
+if sys.version >= '3':
--- End diff --

can we define this in the top of the file? Maybe it can be re-used?

```python
import sys
if sys.version >= '3':
  intlike = int
else:
  intlike = (int, long)

...

if not isinstance(id, intlike):
  raise Error(...)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13653
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13653
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60452/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13638: [SPARK-15915][SQL] Logical plans should use canonicalize...

[ueshin]

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/13638
  
Updated pr title, description and a test name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org