[GitHub] spark issue #19156: [SPARK-19634][SQL][ML][FOLLOW-UP] Improve interface of d...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19156 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19156: [SPARK-19634][SQL][ML][FOLLOW-UP] Improve interface of d...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19156 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83639/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19156: [SPARK-19634][SQL][ML][FOLLOW-UP] Improve interface of d...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19156 **[Test build #83639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83639/testReport)** for PR 19156 at commit [`525692e`](https://github.com/apache/spark/commit/525692eba56ada030ee34390caa8df0ff5259f22). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19700: [SPARK-22471][SQL] SQLListener consumes much memory caus...
Github user tashoyan commented on the issue: https://github.com/apache/spark/pull/19700 @vanzin would you like to review? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19532: [DOC]update the API doc and modify the stage API descrip...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/19532 @cloud-fan Help merge the code. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19706: [SPARK-22476][R] Add dayofweek function to R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19706 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19706: [SPARK-22476][R] Add dayofweek function to R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19706 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83636/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19706: [SPARK-22476][R] Add dayofweek function to R
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19706 **[Test build #83636 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83636/testReport)** for PR 19706 at commit [`d24a89b`](https://github.com/apache/spark/commit/d24a89b6a756457c651d0c208ccbe59b979e9ecc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19156: [SPARK-19634][SQL][ML][FOLLOW-UP] Improve interface of d...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19156 **[Test build #83640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83640/testReport)** for PR 19156 at commit [`2e4b232`](https://github.com/apache/spark/commit/2e4b232adabe45e9dcafad72ca9c1d3ba5b34dce). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19532: [DOC]update the API doc and modify the stage API descrip...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19532 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83637/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19532: [DOC]update the API doc and modify the stage API descrip...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19532 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19532: [DOC]update the API doc and modify the stage API descrip...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19532 **[Test build #83637 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83637/testReport)** for PR 19532 at commit [`9670d6f`](https://github.com/apache/spark/commit/9670d6f8b3fed58f556b9050814f7c61bc5d65be). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19156: [SPARK-19634][SQL][ML][FOLLOW-UP] Improve interface of d...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19156 **[Test build #83639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83639/testReport)** for PR 19156 at commit [`525692e`](https://github.com/apache/spark/commit/525692eba56ada030ee34390caa8df0ff5259f22). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19156: [SPARK-19634][SQL][ML][FOLLOW-UP] Improve interfa...
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19156#discussion_r149893125
--- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala ---
@@ -94,46 +97,86 @@ object Summarizer extends Logging {
* - min: the minimum for each coefficient.
* - normL2: the Euclidian norm for each coefficient.
* - normL1: the L1 norm of each coefficient (sum of the absolute
values).
- * @param firstMetric the metric being provided
- * @param metrics additional metrics that can be provided.
+ * @param metrics metrics that can be provided.
* @return a builder.
* @throws IllegalArgumentException if one of the metric names is not
understood.
*
* Note: Currently, the performance of this interface is about 2x~3x
slower then using the RDD
* interface.
*/
@Since("2.3.0")
- def metrics(firstMetric: String, metrics: String*): SummaryBuilder = {
-val (typedMetrics, computeMetrics) =
getRelevantMetrics(Seq(firstMetric) ++ metrics)
+ def metrics(metrics: String*): SummaryBuilder = {
--- End diff --
@cloud-fan Do you say about this bug ?
https://issues.apache.org/jira/browse/SPARK-5904
But it is only related to abstract method.
Now I add java testsuite to make sure it works fine.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19657: [SPARK-22344][SPARKR] clean up install dir if running te...
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19657
ouch
```
Error in packageVersion("knitr") : package 'knitr' not found
[00:03:37] Execution halted
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19695: [SPARK-22377][BUILD] Use /usr/sbin/lsof if lsof does not...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19695 **[Test build #83638 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83638/testReport)** for PR 19695 at commit [`a6642fa`](https://github.com/apache/spark/commit/a6642fa41795cff82ec30c38e3c909d8025f358f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19695: [SPARK-22377][BUILD] Use /usr/sbin/lsof if lsof does not...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19695 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19657: [SPARK-22344][SPARKR] clean up install dir if running te...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19657 Build started: [SparkR] `ALL` [](https://ci.appveyor.com/project/spark-test/spark/branch/33DABC16-224C-40D8-8A29-EF9CFA72625E) Diff: https://github.com/apache/spark/compare/master...spark-test:33DABC16-224C-40D8-8A29-EF9CFA72625E --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19532: [DOC]update the API doc and modify the stage API descrip...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19532 **[Test build #83637 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83637/testReport)** for PR 19532 at commit [`9670d6f`](https://github.com/apache/spark/commit/9670d6f8b3fed58f556b9050814f7c61bc5d65be). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19706: [SPARK-22476][R] Add dayofweek function to R
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19706 **[Test build #83636 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83636/testReport)** for PR 19706 at commit [`d24a89b`](https://github.com/apache/spark/commit/d24a89b6a756457c651d0c208ccbe59b979e9ecc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19657: [SPARK-22344][SPARKR] clean up install dir if running te...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19657 Sure, I can! .. but seems something has gone wrong with `knitr` .. Let me trigger it by my account anyway. I can retrigger it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19663 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19706: [SPARK-22476][R] Add dayofweek function to R
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19706 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19706: [SPARK-22476][R] Add dayofweek function to R
GitHub user HyukjinKwon reopened a pull request:
https://github.com/apache/spark/pull/19706
[SPARK-22476][R] Add dayofweek function to R
## What changes were proposed in this pull request?
This PR adds `dayofweek` to R API:
```r
data <- list(list(d = as.Date("2012-12-13")),
list(d = as.Date("2013-12-14")),
list(d = as.Date("2014-12-15")))
df <- createDataFrame(data)
collect(select(df, dayofweek(df$d)))
```
```
dayofweek(d)
15
27
32
```
## How was this patch tested?
Manual tests and unit tests in `R/pkg/tests/fulltests/test_sparkSQL.R`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark add-dayofweek
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19706.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19706
commit d24a89b6a756457c651d0c208ccbe59b979e9ecc
Author: hyukjinkwon
Date: 2017-11-08T11:31:35Z
Add support for dayofweek function in R
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19663 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19706: [SPARK-22476][R] Add dayofweek function to R
Github user HyukjinKwon closed the pull request at: https://github.com/apache/spark/pull/19706 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19532: [DOC]update the API doc and modify the stage API descrip...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19532 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19706: [SPARK-22476][R] Add dayofweek function to R
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19706
Thanks for review @felixcheung. BTW, looks AppVeyor was failed by a
different reason .. ?
```
> packageVersion('knitr'); packageVersion('rmarkdown');
packageVersion('testthat'); packageVersion('e1071'); packageVersion('survival')
Error in packageVersion("knitr") : package 'knitr' not found
Execution halted
Command exited with code 1
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19250: [SPARK-12297] Table timezone correction for Timestamps
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19250 actually I took a look at #16781 , It also proposed a table property, instead of a simple spark config. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19532: [DOC]update the API doc and modify the stage API descrip...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/19532 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19695: [SPARK-22377][BUILD] Use /usr/sbin/lsof if lsof does not...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19695 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19695: [SPARK-22377][BUILD] Use /usr/sbin/lsof if lsof does not...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19695 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83629/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19706: [SPARK-22476][R] Add dayofweek function to R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19706 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83634/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19459 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83635/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19706: [SPARK-22476][R] Add dayofweek function to R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19706 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19706: [SPARK-22476][R] Add dayofweek function to R
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19706 **[Test build #83634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83634/testReport)** for PR 19706 at commit [`d24a89b`](https://github.com/apache/spark/commit/d24a89b6a756457c651d0c208ccbe59b979e9ecc). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19459 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19459 **[Test build #83635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83635/testReport)** for PR 19459 at commit [`421d0be`](https://github.com/apache/spark/commit/421d0beafe0aeff8e689fa05af0505a4c8b1c556). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19695: [SPARK-22377][BUILD] Use /usr/sbin/lsof if lsof does not...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19695 **[Test build #83629 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83629/testReport)** for PR 19695 at commit [`a6642fa`](https://github.com/apache/spark/commit/a6642fa41795cff82ec30c38e3c909d8025f358f). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r149887860 --- Diff: python/pyspark/serializers.py --- @@ -214,6 +214,14 @@ def __repr__(self): def _create_batch(series): +""" +Create an Arrow record batch from the given pandas.Series or list of Series, with optional type. + +:param series: A single pandas.Series, list of Series, or list of (series, arrow_type) +:param copy: Option to make a copy of the series before performing any type casts --- End diff -- We can remove this description. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
