[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-31 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-126750717 Ah I see - so this affects things like the YARN cluster mode where the spark-submit script and the driver are run on different machines ? How does this work out

[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...

2015-07-31 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-126746061 LGTM. Merging this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8742][SPARKR] Improve SparkR error mess...

2015-07-30 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7742#issuecomment-126517480 Merging this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-30 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-126531818 Thanks @brkyvz for the update. I had some minor style comments but I don't quite follow why we had to move the initialization to SparkContext vs. having

[GitHub] spark pull request: [SPARK-9324][SPARK-9322][SPARK-9321][SPARKR] S...

2015-07-30 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7764#issuecomment-126522020 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7139#discussion_r35937520 --- Diff: core/src/main/scala/org/apache/spark/api/r/RBackend.scala --- @@ -28,6 +28,7 @@ import io.netty.channel.socket.SocketChannel import

[GitHub] spark pull request: [SPARK-9324][SPARK-9322][SPARK-9321][SPARKR] S...

2015-07-30 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7764#issuecomment-126524783 @falaki the documentation builds are failing with an error message ``` Error : /home/jenkins/workspace/SparkPullRequestBuilder@2/R/pkg/man/nrow.Rd

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7139#discussion_r35937603 --- Diff: core/src/main/scala/org/apache/spark/api/r/RUtils.scala --- @@ -62,4 +64,7 @@ private[spark] object RUtils

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-31 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-126765707 I see - so the right thing to do is to not build the package on every node which needs this (driver / executor) but to build it once in the spark-submit node

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-31 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-126774354 Well we are unzipping the *source*, building a *binary* and then zipping the *binary*. so its not redundant work :). The script right now only zips the `lib/SparkR

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-31 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-126778965 I think we can have multiple R packages in one zip. If I remember correctly we just unzip the files in every executor (or spark-submit does this) and we include

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-31 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-126801147 Yep that sounds good to me. BTW the way R finds packages is using the `.libPaths` contents. We explicitly set that to include `lib/` which is why it can find `SparkR

[GitHub] spark pull request: [SPARK-9324][SPARK-9322][SPARK-9321][SPARKR] S...

2015-07-31 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7764#discussion_r36016434 --- Diff: R/pkg/R/DataFrame.R --- @@ -1231,6 +1305,22 @@ setMethod(unionAll, dataFrame(unioned) }) +#' @title Union

[GitHub] spark pull request: [SPARK-9324][SPARK-9322][SPARK-9321][SPARKR] S...

2015-07-31 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7764#issuecomment-126812352 Thanks @falaki for the update. LGTM. I had a minor style fix that I'll apply during the merge. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-31 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-126815610 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-31 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-126815591 @brkyvz I'll take another pass later today and try this on YARN etc. We should then be able to get it merged hopefully sometime this weekend. --- If your project

[GitHub] spark pull request: Use vector-friendly comparison for packages ar...

2015-07-28 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7701#issuecomment-125565050 Jenkins, ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Use vector-friendly comparison for packages ar...

2015-07-28 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7701#issuecomment-125565367 Thanks @trestletech -- Could you also add a small unit tests for this in https://github.com/apache/spark/blob/15724fac569258d2a149507d8c767d0de0ae8306/R/pkg/inst/tests

[GitHub] spark pull request: Use vector-friendly comparison for packages ar...

2015-07-28 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7701#issuecomment-125689494 LGTM. Thanks @trestletech -- Merging this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Update docs/README.md to put all prereqs toget...

2015-08-04 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7951#issuecomment-127845415 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-08-04 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-127811415 BTW @brkyvz can you merge the R changes in the https://github.com/databricks/sbt-spark-package as well ? --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-08-04 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-127810784 Yep - I'm out now, but will get back to my computer and merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-8505][SparkR] Add settings to kick `lin...

2015-08-03 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7883#issuecomment-127134483 I don't think we need to do it on every run. If we need a new version of lintr on Jenkins, then we can ask @JoshRosen or @shaneknapp to install a new version

[GitHub] spark pull request: [SPARK-8505][SparkR] Add settings to kick `lin...

2015-08-02 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7883#issuecomment-127132591 Also @yu-iskw I am not sure I understand why we need to install `lintr` in the same library location as SparkR. That will mean that lintr will be installed on every

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-08-02 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-127132992 @brkyvz I just tried this out on a YARN cluster and I actually ran into an error trying to use the CSV reader package. The command I ran was ``` ./bin/sparkR

[GitHub] spark pull request: [SPARK-9562] Change reference to amplab/spark-...

2015-08-03 Thread shivaram
GitHub user shivaram opened a pull request: https://github.com/apache/spark/pull/7899 [SPARK-9562] Change reference to amplab/spark-ec2 from mesos/ cc @srowen @pwendell @nchammas You can merge this pull request into a Git repository by running: $ git pull https://github.com

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-08-03 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-127350331 I'm on it now. BTW is `sparkGLM` the best external package to try out in YARN ? or is there a smaller demo package that I can use ? --- If your project is set up

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-08-03 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-127390781 @brkyvz I tried to run this with YARN and found that the code to create the `zip` might have some problems. It ends up creating a zip file which is multiple gigabytes

[GitHub] spark pull request: [SPARK-9324][SPARK-9322][SPARK-9321][SPARKR] S...

2015-07-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7764#discussion_r35903215 --- Diff: R/pkg/R/DataFrame.R --- @@ -1231,6 +1296,24 @@ setMethod(unionAll, dataFrame(unioned) }) +setGeneric

[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...

2015-07-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7584#discussion_r35895264 --- Diff: R/pkg/R/DataFrame.R --- @@ -1384,7 +1384,7 @@ setMethod(saveAsTable, org.apache.spark.sql.parquet

[GitHub] spark pull request: [SPARK-9248][SparkR] Closing curly-braces shou...

2015-07-30 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7795#issuecomment-126401982 Thanks @yu-iskw -- LGTM. Merging this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8742][SPARKR] Improve SparkR error mess...

2015-07-30 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7742#issuecomment-126517276 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-07-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7139#discussion_r35937631 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala --- @@ -22,6 +22,9 @@ import java.net.URL import java.util.Properties

[GitHub] spark pull request: [SPARK-9324][SPARK-9322][SPARK-9321][SPARKR] S...

2015-07-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7764#discussion_r35896170 --- Diff: R/pkg/R/DataFrame.R --- @@ -534,6 +552,53 @@ setMethod(count, callJMethod(x@sdf, count) }) +#' @rdname

[GitHub] spark pull request: [SPARK-9324][SPARK-9322][SPARK-9321][SPARKR] S...

2015-07-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7764#discussion_r35897285 --- Diff: R/pkg/R/DataFrame.R --- @@ -1231,6 +1296,24 @@ setMethod(unionAll, dataFrame(unioned) }) +setGeneric

[GitHub] spark pull request: [SPARK-8742][SPARKR] Improve SparkR error mess...

2015-07-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7742#discussion_r35897965 --- Diff: core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala --- @@ -148,6 +151,9 @@ private[r] class RBackendHandler(server: RBackend

[GitHub] spark pull request: [SPARK-9324][SPARK-9322][SPARK-9321][SPARKR] S...

2015-07-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7764#discussion_r35896690 --- Diff: R/pkg/R/DataFrame.R --- @@ -1231,6 +1296,24 @@ setMethod(unionAll, dataFrame(unioned) }) +setGeneric

[GitHub] spark pull request: [SPARK-9437][core] avoid overflow in SizeEstim...

2015-07-30 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7750#issuecomment-126414664 LGTM. Merging this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9314] [EC2] add root EBS config options...

2015-07-30 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7647#issuecomment-126415405 Not sure I get it -- Does this mean the increased size will only work if we use the `resizefs` in a script ? --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-07 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-128844893 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9713] [ML] Document SparkR MLlib glm() ...

2015-08-11 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/8085#discussion_r36782862 --- Diff: docs/sparkr.md --- @@ -210,6 +210,43 @@ head(df) {% endhighlight %} /div +### Model Formulae + +SparkR allows

[GitHub] spark pull request: [SPARK-9713] [ML] Document SparkR MLlib glm() ...

2015-08-11 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/8085#discussion_r36782153 --- Diff: docs/sparkr.md --- @@ -210,6 +210,43 @@ head(df) {% endhighlight %} /div +### Model Formulae --- End diff -- +1. I

[GitHub] spark pull request: [SPARK-9713] [ML] Document SparkR MLlib glm() ...

2015-08-11 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8085#issuecomment-130012889 Thanks @ericl - I just left some inline comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-9713] [ML] Document SparkR MLlib glm() ...

2015-08-11 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/8085#discussion_r36782425 --- Diff: docs/sparkr.md --- @@ -210,6 +210,43 @@ head(df) {% endhighlight %} /div +### Model Formulae + --- End diff

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-08-04 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7139#discussion_r36211625 --- Diff: core/src/main/scala/org/apache/spark/api/r/RUtils.scala --- @@ -19,6 +19,8 @@ package org.apache.spark.api.r import java.io.File

[GitHub] spark pull request: [SPARK-8951][SparkR] support Unicode character...

2015-08-04 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7494#issuecomment-127663833 I see - so the problem here is on how to write a unit test that uses UTF-8 and works with `LC_ALL=C` ? One simple thing we might be able to do is to set the locale

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-08-04 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-127664838 @brkyvz I think I figured out the problem with Jenkins -- Since Jenkins uses SBT it doesn't build the SparkR package at the beginning along with the rest

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-08-04 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-127677811 Thanks @brkyvz -- Changes LGTM. Can you check if @andrewor14 wants to take another look at the SparkSubmit changes ? --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-8057][Core]Call TaskAttemptContext.getT...

2015-08-06 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/6599#issuecomment-128448874 @JoshRosen @zsxwing Any thoughts on merging this for 1.5 ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-8951][SparkR] support Unicode character...

2015-08-06 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7494#issuecomment-128447851 Can we still use the `iconv` solution with the `System.setlocale` in the test case ? It doesn't seem right to use `rawToChar` when we are decoding UTF-8 strings

[GitHub] spark pull request: [SPARK-8313] R Spark packages support

2015-08-04 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7139#issuecomment-127740630 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-07 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-128744494 LGTM. Thanks @vanzin for the fix. cc @brkyvz --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-07 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-128744287 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-07 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-128787880 Sigh -- our tests are too flaky. Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-9934] Deprecate NIO ConnectionManager.

2015-08-13 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8162#issuecomment-130769302 Is there any major reason for doing this ? I've still seen some clusters / workloads (as of Spark 1.3) where nio works better than netty and I didn't get a chance

[GitHub] spark pull request: [SPARK-9934] Deprecate NIO ConnectionManager.

2015-08-13 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8162#issuecomment-130978310 Yeah I'll try to produce a small test case for this next time I run into this. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

2015-08-10 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7461#issuecomment-129506547 Thanks @kmadhugit , the test case sounds good to me. Could you try this on a multi-node cluster ? We just set the locality preference for the scheduler and I guess

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-09 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-129223162 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [CORE] [SPARK-9760] Use Option instead of Some...

2015-08-09 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8055#issuecomment-129226373 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-8951][SparkR] support Unicode character...

2015-08-09 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7494#issuecomment-129227462 @CHOIJAEHONG1 This diff is looking good. Can you update the PR with this ? BTW why do we need to clear the encoding bit in `context.R` ? Does the serialization

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-09 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-129260689 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [CORE] [SPARK-9760] Use Option instead of Some...

2015-08-09 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8055#issuecomment-129235074 @brkyvz Removed the test case now. I'm not sure adding a comment is really relevant as the argument to the method is an `Option`. And you are right that we can't quite

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-09 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-129232005 @tdas @rxin I've see the test `SPARK-6222: Do not clear received block data too soon` fail a bunch of times (on Jenkins and on local runs) - Example stack trace at [1

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-10 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-129528087 Yeah I was wondering the same thing. In fact I think the tests have managed to pass piece-wise (i.e. core sometimes, streaming sometimes, sql sometimes etc

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-10 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-129506725 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9871][SparkR] Add expression functions ...

2015-08-14 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/8194#discussion_r37092050 --- Diff: R/pkg/R/functions.R --- @@ -67,6 +67,14 @@ createFunctions - function() { createFunctions() +#' @rdname functions

[GitHub] spark pull request: [SPARK-9871][SparkR] Add expression functions ...

2015-08-14 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/8194#discussion_r37092120 --- Diff: R/pkg/R/generics.R --- @@ -682,6 +682,10 @@ setGeneric(cbrt, function(x) { standardGeneric(cbrt) }) #' @export setGeneric(ceil, function

[GitHub] spark pull request: [SPARK-9871][SparkR] Add expression functions ...

2015-08-14 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8194#issuecomment-131165945 Thanks @yu-iskw. I left some comments inline. One more thing we need to do is to update our `NAMESPACE` file [1] with all the functions we have added so far

[GitHub] spark pull request: [SPARK-9871][SparkR] Add expression functions ...

2015-08-14 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/8194#discussion_r37092216 --- Diff: R/pkg/R/generics.R --- @@ -794,6 +806,10 @@ setGeneric(size, function(x) { standardGeneric(size) }) #' @export setGeneric(soundex

[GitHub] spark pull request: [SPARK-10008] Ensure shuffle locality doesn't ...

2015-08-15 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8220#issuecomment-131306428 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-8844][SPARKR] head/collect is broken in...

2015-08-15 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7419#issuecomment-131412001 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-8844][SPARKR] head/collect is broken in...

2015-08-15 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7419#issuecomment-131412095 Thanks @sun-rui -- this fix is simple and looks good. Couple of minor things 1. There is an empty file `nohup.out` added in the PR which needs to be removed 2

[GitHub] spark pull request: [SPARK-10008] Ensure shuffle locality doesn't ...

2015-08-15 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8220#issuecomment-131412272 LGTM. I think we should merge this into `branch-1.5` as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-10008] Ensure shuffle locality doesn't ...

2015-08-15 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8220#issuecomment-131412441 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9713] [ML] Document SparkR MLlib glm() ...

2015-08-11 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8085#issuecomment-130124028 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-07 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-128884477 Sigh, I guess today is not a good day to get tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-129013022 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-8861][SPARK-8862][SQL] Add basic instru...

2015-08-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7774#issuecomment-129021145 @zsxwing @rxin I just ran into a problem with the annotation thing that was discussed earlier in this PR. My master branch is clean and is at https://github.com

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-129058361 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9263] Added flags to exclude dependenci...

2015-08-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7599#issuecomment-129059640 @brkyvz I ran into a trouble using `sparkR` with the CSV package from the master branch and I think this PR might be the related to the problem. If I run `./bin/sparkR

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-129080680 @JoshRosen @rxin This PR has failed Jenkins 6 (!) times with unrelated failures. Are there any known issues or flaky tests ? I'd like to get this fix into 1.5

[GitHub] spark pull request: [SPARK-8861][SPARK-8862][SQL] Add basic instru...

2015-08-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7774#issuecomment-129081760 I just realized I missed pasting the first two lines of the error ``` error: error while loading SQLListener, Missing dependency 'bad symbolic reference

[GitHub] spark pull request: [CORE] [SPARK-9760] Use Option instead of Some...

2015-08-08 Thread shivaram
GitHub user shivaram opened a pull request: https://github.com/apache/spark/pull/8055 [CORE] [SPARK-9760] Use Option instead of Some for Ivy repos This was introduced in #7599 cc @rxin @brkyvz You can merge this pull request into a Git repository by running: $ git

[GitHub] spark pull request: [CORE] [SPARK-9760] Use Option instead of Some...

2015-08-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8055#issuecomment-129088465 Also I am not sure how to add a test case for this as our existing unit tests use a custom Ivy repository which is always specified in `--repositories`. The bug only

[GitHub] spark pull request: [CORE] [SPARK-9760] Use Option instead of Some...

2015-08-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8055#issuecomment-129100227 Added a test case but I'm not sure how useful it is as a lot of things can throw a RuntimeException ? @rxin let me know if its worth having it vs. a few seconds added

[GitHub] spark pull request: [SPARK-9710] [test] Fix RPackageUtilsSuite whe...

2015-08-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8008#issuecomment-129103841 I give up. I'm going to manually patch this on a clean branch on my laptop and run `dev/run-tests` and report back if it passes --- If your project is set up

[GitHub] spark pull request: Add a prerequisites section for building docs

2015-08-04 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7912#discussion_r36206750 --- Diff: docs/README.md --- @@ -8,6 +8,16 @@ Read on to learn more about viewing documentation in plain text (i.e., markdown) documentation yourself

[GitHub] spark pull request: [SPARK-8505][SparkR] Add settings to kick `lin...

2015-08-06 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7883#issuecomment-128426622 Thanks @yu-iskw for the update. The code changes mostly look good to me. Lets wait for @JoshRosen to take another look at this as there have been some flaky builds

[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...

2015-07-22 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123773927 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...

2015-07-22 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123775036 I think most of the changes are fine. The `!` ones look a little awkward and I want to check the `prev@func` issue to see why lint-r is complaining about

[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...

2015-07-22 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123782306 @shaneknapp Can you manually delete the jar ? The script should auto-download a good version then AFAIK --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...

2015-07-22 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-12319 @shaneknapp @JoshRosen Looks like there is a problem with Jenkins. The builds seem to be failing with ``` Launching sbt from build/sbt-launch-0.13.7.jar

[GitHub] spark pull request: [SPARK-8364][SPARKR] Add crosstab to SparkR Da...

2015-07-22 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7318#issuecomment-123818662 @mengxr I guess we will have a new PR for the documentation update, so this PR LGTM. I will merge this unless you have anything else to add --- If your project is set

[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...

2015-07-22 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7584#discussion_r35237914 --- Diff: R/pkg/R/DataFrame.R --- @@ -1384,7 +1384,7 @@ setMethod(saveAsTable, org.apache.spark.sql.parquet

[GitHub] spark pull request: [SPARK-9253][EC2] Allow to create machines wit...

2015-07-22 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7596#issuecomment-123788290 I am not sure taking in the credentials in plain text is the best solution -- Can we get this from some other environment variable and / or boto's config files etc

[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...

2015-07-22 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7584#discussion_r35237834 --- Diff: R/pkg/R/RDD.R --- @@ -85,7 +85,7 @@ setMethod(initialize, PipelinedRDD, function(.Object, prev, func, jrdd_val) isPipelinable

[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...

2015-07-24 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7584#discussion_r35437342 --- Diff: R/pkg/R/DataFrame.R --- @@ -1384,7 +1384,7 @@ setMethod(saveAsTable, org.apache.spark.sql.parquet

[GitHub] spark pull request: [SPARK-9249][SparkR] local variable assigned b...

2015-07-24 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7640#issuecomment-124569224 LGTM. I had one minor comment -- I'll fix this up during merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-9249][SparkR] local variable assigned b...

2015-07-24 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7640#discussion_r35437569 --- Diff: R/pkg/R/sparkR.R --- @@ -104,16 +104,14 @@ sparkR.init - function( return(get(.sparkRjsc, envir = .sparkREnv

[GitHub] spark pull request: [SPARK-8844][SPARKR] head/collect is broken in...

2015-07-24 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7419#issuecomment-124569785 @sun-rui Any update on this ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

<    4   5   6   7   8   9   10   11   12   13   >