[GitHub] zeppelin issue #2700: [ZEPPELIN-3092] GitHub Integration

2018-01-21 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/2700 @mohamagdy could you check - there are a bunch of errors https://travis-ci.org/mohamagdy/zeppelin/builds/324495310 ---

spark git commit: [SPARK-21293][SS][SPARKR] Add doc example for streaming join, dedup

2018-01-21 Thread felixcheung
lix Cheung <felixcheun...@hotmail.com> Closes #20340 from felixcheung/rstreamdoc. (cherry picked from commit 2239d7a410e906ccd40aa8e84d637e9d06cd7b8a) Signed-off-by: Felix Cheung <felixche...@apache.org> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wi

spark git commit: [SPARK-21293][SS][SPARKR] Add doc example for streaming join, dedup

2018-01-21 Thread felixcheung
lix Cheung <felixcheun...@hotmail.com> Closes #20340 from felixcheung/rstreamdoc. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2239d7a4 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2239d7a4 Diff: http:

[GitHub] zeppelin issue #2700: [ZEPPELIN-3092] GitHub Integration

2018-01-20 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/2700 looks like Jenkins has lost the test run result - could you kick that off again? you can do that by closing and reopening this PR. ---

[GitHub] spark issue #19528: [SPARK-20393][WEBU UI][1.6] Strengthen Spark to prevent ...

2018-01-19 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19528 Jenkins test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20255: [SPARK-23064][DOCS][SS] Added documentation for s...

2018-01-17 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20255#discussion_r162261740 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1089,6 +1098,224 @@ streamingDf.join(staticDf, "type", "right_join&qu

[GitHub] spark pull request #20272: [SPARK-23078] [CORE] allow Spark Thrift Server to...

2018-01-17 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20272#discussion_r161979319 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -328,7 +328,7 @@ object SparkSubmit extends CommandLineUtils

[GitHub] spark issue #20267: [SPARK-23068][BUILD][RELEASE][WIP] doc build error from ...

2018-01-16 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20267 I'm not 100% sure... I injected some error in jekyll but it stopped immediately. let me try to match the report condition more closely

[GitHub] spark pull request #20232: [SPARK-23042][ML] Use OneHotEncoderModel to encod...

2018-01-16 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20232#discussion_r161683935 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala --- @@ -102,36 +102,6 @@ private[classification

[GitHub] spark pull request #20232: [SPARK-23042][ML] Use OneHotEncoderModel to encod...

2018-01-16 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20232#discussion_r161686126 --- Diff: docs/sparkr.md --- @@ -663,3 +663,4 @@ You can inspect the search path in R with [`search()`](https://stat.ethz.ch/R-ma

[GitHub] spark pull request #20204: [SPARK-7721][PYTHON][TESTS] Adds PySpark coverage...

2018-01-15 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20204#discussion_r161677975 --- Diff: python/run-tests-with-coverage --- @@ -0,0 +1,69 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation

[GitHub] spark pull request #20163: [SPARK-22966][PYTHON][SQL] Python UDFs with retur...

2018-01-15 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20163#discussion_r161677327 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala --- @@ -144,6 +145,7 @@ object EvaluatePython

[GitHub] spark pull request #20267: [SPARK-23068][BUILD][RELEASE] doc build error fro...

2018-01-14 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20267#discussion_r161444163 --- Diff: dev/create-release/release-build.sh --- @@ -290,6 +290,8 @@ if [[ "$1" == "docs" ]]; then cd docs # TOD

[GitHub] spark issue #20267: [SPARK-23068][BUILD][RELEASE] doc build error from jekyl...

2018-01-14 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20267 @sameeragarwal FYI. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20267: [SPARK-23068][BUILD][RELEASE] doc build error fro...

2018-01-14 Thread felixcheung
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/20267 [SPARK-23068][BUILD][RELEASE] doc build error from jekyll does not fail ## What changes were proposed in this pull request? check exit code. note that not errors are reported via exit

[GitHub] spark pull request #20264: [SPARK-23070] Bump previousSparkVersion in MimaBu...

2018-01-14 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20264#discussion_r161406899 --- Diff: project/MimaBuild.scala --- @@ -88,7 +88,7 @@ object MimaBuild { def mimaSettings(sparkHome: File, projectRef: ProjectRef

[GitHub] spark issue #20263: [SPARK-23069][DOCS][SPARKR] fix R doc for describe missi...

2018-01-14 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20263 thanks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)

2018-01-13 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20230 merged to master/2.3/2.2 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

spark git commit: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)

2018-01-13 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.2 105ae8680 -> 7022ef800 [SPARK-23038][TEST] Update docker/spark-test (JDK/OS) ## What changes were proposed in this pull request? This PR aims to update the followings in `docker/spark-test`. - JDK7 -> JDK8 Spark 2.2+ supports JDK8

spark git commit: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)

2018-01-13 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.3 1f4a08b15 -> a335a49ce [SPARK-23038][TEST] Update docker/spark-test (JDK/OS) ## What changes were proposed in this pull request? This PR aims to update the followings in `docker/spark-test`. - JDK7 -> JDK8 Spark 2.2+ supports JDK8

spark git commit: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)

2018-01-13 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master c3548d11c -> 7a3d0aad2 [SPARK-23038][TEST] Update docker/spark-test (JDK/OS) ## What changes were proposed in this pull request? This PR aims to update the followings in `docker/spark-test`. - JDK7 -> JDK8 Spark 2.2+ supports JDK8 only.

[GitHub] spark issue #20254: [SPARK-23062][SQL] Improve EXCEPT documentation

2018-01-13 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20254 @henryr could you update this PR to only include EXCEPT DISTINCT without the notes --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #20263: [SPARK-23069] fix R doc for describe missing text

2018-01-13 Thread felixcheung
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/20263 [SPARK-23069] fix R doc for describe missing text ## What changes were proposed in this pull request? fix doc truncated ## How was this patch tested? manually You

spark git commit: [SPARK-23063][K8S] K8s changes for publishing scripts (and a couple of other misses)

2018-01-13 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master afae8f2bc -> c3548d11c [SPARK-23063][K8S] K8s changes for publishing scripts (and a couple of other misses) ## What changes were proposed in this pull request? Including the `-Pkubernetes` flag in a few places it was missed. ## How was

[GitHub] spark issue #20256: [SPARK-23063][K8S] K8s changes for publishing scripts (a...

2018-01-13 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20256 thanks! merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

spark git commit: [SPARK-23063][K8S] K8s changes for publishing scripts (and a couple of other misses)

2018-01-13 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.3 bcd87ae07 -> 1f4a08b15 [SPARK-23063][K8S] K8s changes for publishing scripts (and a couple of other misses) ## What changes were proposed in this pull request? Including the `-Pkubernetes` flag in a few places it was missed. ## How

[GitHub] spark issue #20254: [SPARK-23062][SQL] Improve EXCEPT documentation

2018-01-13 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20254 I see, from reading that PR I think perhaps we should reference migration guide in sql programming guide instead of putting the whole description here

[GitHub] spark issue #20080: [SPARK-22870][CORE] Dynamic allocation should allow 0 id...

2018-01-13 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20080 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK...

2018-01-13 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20230#discussion_r161367060 --- Diff: external/docker/spark-test/base/Dockerfile --- @@ -15,14 +15,14 @@ # limitations under the License. # -FROM ubuntu:precise

[GitHub] spark pull request #20255: [SPARK-23064][DOCS][SS] Added documentation for s...

2018-01-13 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20255#discussion_r161366935 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1089,6 +1098,224 @@ streamingDf.join(staticDf, "type", "right_join&qu

[GitHub] spark pull request #20255: [SPARK-23064][DOCS][SS] Added documentation for s...

2018-01-13 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20255#discussion_r161366948 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1089,6 +1098,224 @@ streamingDf.join(staticDf, "type", "right_join&qu

[GitHub] spark pull request #20256: [SPARK-23063][K8S] K8s changes for publishing scr...

2018-01-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20256#discussion_r161366655 --- Diff: dev/create-release/releaseutils.py --- @@ -185,6 +185,7 @@ def get_commits(tag): "graphx": "GraphX",

[GitHub] spark pull request #20211: [SPARK-23011][PYTHON][SQL] Prepend missing groupi...

2018-01-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20211#discussion_r161366590 --- Diff: python/pyspark/sql/group.py --- @@ -233,6 +233,27 @@ def apply(self, udf): | 2| 1.1094003924504583

[GitHub] spark pull request #20256: [SPARK-23063][K8S] K8s changes for publishing scr...

2018-01-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20256#discussion_r161366220 --- Diff: dev/create-release/releaseutils.py --- @@ -185,6 +185,7 @@ def get_commits(tag): "graphx": "GraphX",

[GitHub] spark pull request #20232: [SPARK-23042][ML] Use OneHotEncoderModel to encod...

2018-01-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20232#discussion_r161365548 --- Diff: R/pkg/tests/fulltests/test_mllib_classification.R --- @@ -382,10 +382,10 @@ test_that("spark.mlp", { trainidxs <- base

[GitHub] spark pull request #20254: [SPARK-23062][SQL] Improve EXCEPT documentation

2018-01-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20254#discussion_r161365422 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1364,7 +1364,9 @@ def subtract(self, other): """ Return a new :c

[GitHub] spark pull request #20254: [SPARK-23062][SQL] Improve EXCEPT documentation

2018-01-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20254#discussion_r161365371 --- Diff: R/pkg/R/DataFrame.R --- @@ -2873,6 +2873,7 @@ setMethod("intersect", #' @rdname except #' @export #' @note except s

[GitHub] spark pull request #20254: [SPARK-23062][SQL] Improve EXCEPT documentation

2018-01-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20254#discussion_r161365416 --- Diff: R/pkg/R/DataFrame.R --- @@ -2873,6 +2873,7 @@ setMethod("intersect", #' @rdname except #' @export #' @note except s

[GitHub] spark pull request #20256: [SPARK-23063][K8S] K8s changes for publishing scr...

2018-01-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20256#discussion_r161365293 --- Diff: dev/create-release/releaseutils.py --- @@ -185,6 +185,7 @@ def get_commits(tag): "graphx": "GraphX",

[GitHub] spark issue #20229: [SPARK-23037][ML] Update RFormula to use VectorSizeHint ...

2018-01-10 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20229 we need to get it to kick off R tests - could you touch one of the files under R/? also please update PR to include [SPARKR

[GitHub] spark pull request #20192: [SPARK-22994][k8s] Use a single image for all Spa...

2018-01-10 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20192#discussion_r160874070 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/executor/Dockerfile --- @@ -1,35 +0,0 @@ -# -# Licensed to the Apache

[GitHub] spark pull request #20192: [SPARK-22994][k8s] Use a single image for all Spa...

2018-01-10 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20192#discussion_r160874300 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -0,0 +1,97 @@ +#!/bin/bash +# +# Licensed

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20222 Wait dev/release-tag.sh does this automatically though. I just want to make sure we are not missing things (like R/DESCRIPTION) I think maybe we should run a subset

[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

2018-01-10 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20151 +1... this is "undocumented" conf, sooo it's an expert one :) --- - To unsubscribe, e-mail: review

[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...

2018-01-09 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20188 merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

spark git commit: [SPARK-22993][ML] Clarify HasCheckpointInterval param doc

2018-01-09 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.3 ecc24ec7f -> 2db523959 [SPARK-22993][ML] Clarify HasCheckpointInterval param doc ## What changes were proposed in this pull request? Add a note to the `HasCheckpointInterval` parameter doc that clarifies that this setting is ignored

spark git commit: [SPARK-22993][ML] Clarify HasCheckpointInterval param doc

2018-01-09 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master eaac60a1e -> 70bcc9d5a [SPARK-22993][ML] Clarify HasCheckpointInterval param doc ## What changes were proposed in this pull request? Add a note to the `HasCheckpointInterval` parameter doc that clarifies that this setting is ignored when

[GitHub] spark issue #19290: [SPARK-22063][R] Fixes lint check failures in R by lates...

2018-01-09 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19290 argh, thanks for the reminder and the fix. I knew calling internal method is going to bite us

[GitHub] spark issue #19290: [SPARK-22063][R] Fixes lint check failures in R by lates...

2018-01-09 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19290 Right we could bump the supported R version for the next release. It should have minimal impact (since we are testing the close to the latest on appveyor... somewhat internally) lintr

[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...

2018-01-09 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20188 Actually in R setCheckpointDir method is not attached to the SparkContext; I’d leave it as “not set” or “not set in the session” https://spark.apache.org/docs/latest/api/R

[GitHub] spark issue #20193: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_C...

2018-01-09 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20193 merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

spark git commit: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors

2018-01-09 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master 0959aa581 -> 6a4206ff0 [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors ## What changes were proposed in this pull request? The environment variable `SPARK_MOUNTED_CLASSPATH` is referenced in the

spark git commit: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors

2018-01-09 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.3 e79480e5d -> 47f975b42 [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors ## What changes were proposed in this pull request? The environment variable `SPARK_MOUNTED_CLASSPATH` is referenced in the

[GitHub] spark issue #20158: [PySpark] Fix typo in comments in PySpark's udf() defini...

2018-01-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20158 let's get them fixed for 2.3? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #20197: [SPARK-21293][SPARKR][DOCS] structured streaming doc upd...

2018-01-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20197 merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

spark git commit: [SPARK-21293][SPARKR][DOCS] structured streaming doc update

2018-01-08 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.3 911a4dbe7 -> a23c07ecb [SPARK-21293][SPARKR][DOCS] structured streaming doc update ## What changes were proposed in this pull request? doc update Author: Felix Cheung <felixcheun...@hotmail.com> Closes #20197 from felixcheu

spark git commit: [SPARK-21293][SPARKR][DOCS] structured streaming doc update

2018-01-08 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master 8486ad419 -> 02214b094 [SPARK-21293][SPARKR][DOCS] structured streaming doc update ## What changes were proposed in this pull request? doc update Author: Felix Cheung <felixcheun...@hotmail.com> Closes #20197 from felixcheu

spark git commit: [SPARK-21292][DOCS] refreshtable example

2018-01-08 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.3 fd46a276c -> 911a4dbe7 [SPARK-21292][DOCS] refreshtable example ## What changes were proposed in this pull request? doc update Author: Felix Cheung <felixcheun...@hotmail.com> Closes #20198 from felixcheung/rrefreshdoc.

[GitHub] spark issue #20198: [SPARK-21292][DOCS] refreshtable example

2018-01-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20198 merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

spark git commit: [SPARK-21292][DOCS] refreshtable example

2018-01-08 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master f20131dd3 -> 8486ad419 [SPARK-21292][DOCS] refreshtable example ## What changes were proposed in this pull request? doc update Author: Felix Cheung <felixcheun...@hotmail.com> Closes #20198 from felixcheung/rrefreshdoc. Proj

[GitHub] spark pull request #20198: [SPARK-21292][DOCS] refreshtable example

2018-01-08 Thread felixcheung
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/20198 [SPARK-21292][DOCS] refreshtable example ## What changes were proposed in this pull request? doc update You can merge this pull request into a Git repository by running: $ git

[GitHub] spark pull request #20197: [SPARK-21293][SPARKR][DOCS] structured streaming ...

2018-01-08 Thread felixcheung
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/20197 [SPARK-21293][SPARKR][DOCS] structured streaming doc update ## What changes were proposed in this pull request? doc update You can merge this pull request into a Git repository

[GitHub] spark issue #19290: [SPARK-22063][R] Fixes lint check failures in R by lates...

2018-01-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19290 Given that we are forking 2.3 and locking down the branch any time now, it might make sense to stay with the "current version running on old centos workers", even though th

[GitHub] spark issue #20193: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_C...

2018-01-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20193 yap I thought about it and it's fine. could you review that PR? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20192: [SPARK-22994][k8s] Use a single image for all Spa...

2018-01-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20192#discussion_r160316799 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -0,0 +1,97 @@ +#!/bin/bash +# +# Licensed

[GitHub] spark pull request #20192: [SPARK-22994][k8s] Use a single image for all Spa...

2018-01-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20192#discussion_r160316427 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/executor/Dockerfile --- @@ -1,35 +0,0 @@ -# -# Licensed to the Apache

[GitHub] spark pull request #20192: [SPARK-22994][k8s] Use a single image for all Spa...

2018-01-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20192#discussion_r160316166 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -0,0 +1,97 @@ +#!/bin/bash +# +# Licensed

[GitHub] spark pull request #20192: [SPARK-22994][k8s] Use a single image for all Spa...

2018-01-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20192#discussion_r160316590 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -29,17 +29,23 @@ private[spark] object

[GitHub] spark pull request #20192: [SPARK-22994][k8s] Use a single image for all Spa...

2018-01-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20192#discussion_r160315955 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile --- @@ -41,7 +41,8 @@ COPY ${spark_jars} /opt/spark/jars COPY

[GitHub] spark issue #19290: [SPARK-22063][R] Fixes lint check failures in R by lates...

2018-01-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19290 Shane is this going to affect one particular branch (eg. 2.3.0), or is it going to be all branches and all test runs? The changes are fairly substantial - if we need to back port

[GitHub] spark issue #20146: [SPARK-11215][ML] Add multiple columns support to String...

2018-01-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20146 ok SGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20146: [SPARK-11215][ML] Add multiple columns support to String...

2018-01-06 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20146 I think all dataset with a string order get indexed, as far as I recall? Pick existing R dataset is just a convenience, we can also make up a few lines of data if that works out better

[GitHub] spark pull request #20164: [SPARK-22971][ML] OneVsRestModel should use tempo...

2018-01-06 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20164#discussion_r160020496 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala --- @@ -170,21 +170,24 @@ final class OneVsRestModel private[ml

[GitHub] spark issue #20167: Allow providing Mesos principal & secret via files (SPAR...

2018-01-05 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20167 is putting secrets as plain text files a good practice..? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #20154: [SPARK-22960][k8s] Make build-push-docker-images.sh more...

2018-01-05 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20154 that's good, I think we should still address the finer point of https://github.com/apache/spark/pull/20154#pullrequestreview-86833216 - if docker hub can't build spark-base then pretty much

[GitHub] zeppelin issue #2704: ZEPPELIN-3061: Updated the SecurityUtils to add Shiro'...

2018-01-04 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/2704 @prabhjyotsingh ---

[GitHub] zeppelin issue #2701: [ZEPPELIN-3098] Livy Interpreter fails if row contains...

2018-01-04 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/2701 I think test passed? it's too long and jenkins no longer has the build @zjffdu ---

[GitHub] zeppelin issue #2689: [ZEPPELIN-3080] Removing duplicate Date header

2018-01-04 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/2689 ping @sjoerdmulder ---

[GitHub] spark issue #20160: [SPARK-22757][K8S] Enable spark.jars and spark.files in ...

2018-01-04 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20160 test passed, merged to master/2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

spark git commit: [SPARK-22757][K8S] Enable spark.jars and spark.files in KUBERNETES mode

2018-01-04 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.3 5b524cc0c -> f9dcdbcef [SPARK-22757][K8S] Enable spark.jars and spark.files in KUBERNETES mode ## What changes were proposed in this pull request? We missed enabling `spark.files` and `spark.jars` in

spark git commit: [SPARK-22757][K8S] Enable spark.jars and spark.files in KUBERNETES mode

2018-01-04 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master cf0aa6557 -> 6cff7d19f [SPARK-22757][K8S] Enable spark.jars and spark.files in KUBERNETES mode ## What changes were proposed in this pull request? We missed enabling `spark.files` and `spark.jars` in

[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...

2018-01-04 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20151#discussion_r159819670 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala --- @@ -34,17 +34,25 @@ private[spark] class PythonWorkerFactory

[GitHub] spark issue #20078: [SPARK-22900] [Spark-Streaming] Remove unnecessary restr...

2018-01-03 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20078 hmm, I didn't know that was changed actually (SPARK-13723) But it seems to me `spark.streaming.dynamicAllocation.minExecutors` is still a valid approach. To match the non-streaming behavior

[GitHub] spark issue #20078: [SPARK-22900] [Spark-Streaming] Remove unnecessary restr...

2018-01-03 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20078 not saying about this change, but I've use streaming dynamic allocation quite a bit back in the day. but in this case I think simply is to set

[GitHub] spark pull request #20146: [SPARK-11215][ML] Add multiple columns support to...

2018-01-03 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20146#discussion_r159584417 --- Diff: R/pkg/tests/fulltests/test_mllib_classification.R --- @@ -348,12 +348,12 @@ test_that("spark.mlp", { # Test r

[GitHub] spark issue #20129: [SPARK-22933][SPARKR] R Structured Streaming API for wit...

2018-01-03 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20129 merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

spark git commit: [SPARK-22933][SPARKR] R Structured Streaming API for withWatermark, trigger, partitionBy

2018-01-03 Thread felixcheung
nBy ## How was this patch tested? manual, unit tests Author: Felix Cheung <felixcheun...@hotmail.com> Closes #20129 from felixcheung/rwater. (cherry picked from commit df95a908baf78800556636a76d58bba9b3dd943f) Signed-off-by: Felix Cheung <felixche...@apache.org> Project:

spark git commit: [SPARK-22933][SPARKR] R Structured Streaming API for withWatermark, trigger, partitionBy

2018-01-03 Thread felixcheung
nBy ## How was this patch tested? manual, unit tests Author: Felix Cheung <felixcheun...@hotmail.com> Closes #20129 from felixcheung/rwater. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/df95a908 Tree: http://git-wip-us.a

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2018-01-03 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16578 We are still merging changes to the 2.3 branch :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #20078: [SPARK-22900] [Spark-Streaming] Remove unnecessary restr...

2018-01-01 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20078 hmm, that sounds like a different problem, why is numReceivers set to > spark.cores.max? --- - To unsubscribe, e-m

[GitHub] spark issue #18714: [SPARK-20236][SQL] runtime partition overwrite

2018-01-01 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/18714 ah yes, please please :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20106: [SPARK-21616][SPARKR][DOCS] update R migration guide and...

2017-12-31 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20106 Maybe. Other test files have that extra empty line though ;) --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20129: [SPARK-22933][SPARKR] R Structured Streaming API ...

2017-12-31 Thread felixcheung
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/20129 [SPARK-22933][SPARKR] R Structured Streaming API for withWatermark, trigger, partitionBy ## What changes were proposed in this pull request? R Structured Streaming API

[GitHub] zeppelin issue #2695: [ZEPPELIN-3089] Create user folders under Trash folder

2017-12-30 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/2695 what's the next step then? ---

[GitHub] spark pull request #20072: [SPARK-22790][SQL] add a configurable factor to d...

2017-12-29 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20072#discussion_r159106981 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -261,6 +261,17 @@ object SQLConf { .booleanConf

[GitHub] spark pull request #20072: [SPARK-22790][SQL] add a configurable factor to d...

2017-12-29 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20072#discussion_r159106860 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -261,6 +261,17 @@ object SQLConf { .booleanConf

[GitHub] spark pull request #20072: [SPARK-22790][SQL] add a configurable factor to d...

2017-12-29 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20072#discussion_r159106914 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -261,6 +261,17 @@ object SQLConf { .booleanConf

[GitHub] spark issue #19758: [SPARK-3162][MLlib] Local Tree Training Pt 1: Refactor R...

2017-12-29 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19758 ping? I'm mostly interested in SPARK-3162 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #20118: [SPARK-22924][SPARKR] R API for sortWithinPartitions

2017-12-29 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20118 @shivaram @HyukjinKwon what do you think about this approach? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20118: [SPARK-22924][SPARKR] R API for sortWithinPartiti...

2017-12-29 Thread felixcheung
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/20118 [SPARK-22924][SPARKR] R API for sortWithinPartitions ## What changes were proposed in this pull request? Add to `arrange` option to sort only within partition ## How

<    7   8   9   10   11   12   13   14   15   16   >