Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165801034
LGTM. Thanks @sun-rui -- Merging this to branch-1.5
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165847552
@sun-rui Could you close this PR ? For branches other than `master` we need
to do this manually.
---
If your project is set up for it, you can reply to this email
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165946896
sure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user sun-rui closed the pull request at:
https://github.com/apache/spark/pull/10372
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165688571
[Test build #47981 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47981/console)
for PR 10372 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165688776
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165688780
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
GitHub user sun-rui opened a pull request:
https://github.com/apache/spark/pull/10372
[SPARK-10500][SPARKR] sparkr.zip cannot be created if /R/lib is unwritable
Backport https://github.com/apache/spark/pull/9390 and
https://github.com/apache/spark/pull/9744 to branch-1.5.
You can
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165661047
Just checking: Did the patches apply cleanly on branch-1.5 or were there
any conflicts ?
---
If your project is set up for it, you can reply to this email and have
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165663611
cc @jerryshao
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165662564
[Test build #47981 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47981/consoleFull)
for PR 10372 at commit
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165663441
clean
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/10372#issuecomment-165664446
Thanks a lot @sun-rui .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-157160063
It looks like the "org.apache.spark.deploy.SparkSubmitSuite.correctly
builds R packages included in a jar with --packages" test is now extremely
flaky in the Maven
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-157164381
Ok I got the error log from log4j and saved it to
https://gist.github.com/shivaram/3a2fecce60768a603dac
From the logs I see
@brkyvz @sun-rui From the logs
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-157163207
@JoshRosen I'm taking a look. If I can't find anything soon I'll send a PR
to disable the test again.
---
If your project is set up for it, you can reply to this
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-157254109
This is probably because when the test is being executed, SparkR library is
not available under $SPARK_HOME/R/lib. Does the build has "-Psparkr" specified?
---
If your
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-157259540
Great catch @sun-rui -- I think it doesn't have `-Psparkr` -- Looking at
the config xml file I see the command being run as
```
build/mvn --force
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156904096
Thanks @sun-rui and @brkyvz -- Sorry for the delay in getting to this.
LGTM. Merging this to master and branch-1.6
---
If your project is set up for it, you can reply
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9390
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156393890
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156393821
**[Test build #45849 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45849/consoleFull)**
for PR 9390 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156362493
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156362464
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156362400
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156363016
**[Test build #45849 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45849/consoleFull)**
for PR 9390 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156617333
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156617332
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156617298
**[Test build #45914 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45914/consoleFull)**
for PR 9390 at commit
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156604715
@brkyvz, the infinite loop if the environment is not cleared is because the
inheritence of the R_PROFILE_USER env var set for launching SparkR, which
causes recursive
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156604724
**[Test build #45914 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45914/consoleFull)**
for PR 9390 at commit
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156627820
don't know what's going error with Jenkins
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156631672
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156660893
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156660722
**[Test build #45921 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45921/consoleFull)**
for PR 9390 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156660894
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156635209
**[Test build #45921 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45921/consoleFull)**
for PR 9390 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156393888
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44724907
--- Diff: R/pkg/inst/profile/general.R ---
@@ -17,6 +17,7 @@
.First <- function() {
packageDir <- Sys.getenv("SPARKR_PACKAGE_DIR")
-
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44725546
--- Diff: core/src/main/scala/org/apache/spark/api/r/RUtils.scala ---
@@ -34,11 +38,15 @@ private[spark] object RUtils {
}
/**
- *
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44724687
--- Diff: R/pkg/inst/worker/daemon.R ---
@@ -18,10 +18,11 @@
# Worker daemon
rLibDir <- Sys.getenv("SPARKR_RLIBDIR")
-script <-
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44724780
--- Diff: R/pkg/inst/worker/daemon.R ---
@@ -18,10 +18,11 @@
# Worker daemon
rLibDir <- Sys.getenv("SPARKR_RLIBDIR")
-script <-
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156329930
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156333984
**[Test build #45836 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45836/consoleFull)**
for PR 9390 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156330945
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156330975
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156356303
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156356304
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156356266
**[Test build #45836 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45836/consoleFull)**
for PR 9390 at commit
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44745947
--- Diff: R/pkg/inst/worker/daemon.R ---
@@ -18,10 +18,11 @@
# Worker daemon
rLibDir <- Sys.getenv("SPARKR_RLIBDIR")
-script <-
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44745978
--- Diff: R/pkg/inst/profile/general.R ---
@@ -17,6 +17,7 @@
.First <- function() {
packageDir <- Sys.getenv("SPARKR_PACKAGE_DIR")
-
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156328944
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156328941
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156119888
@brkyvz, setting env variable like 'R_LIBS' before running "R CMD INSALL"
can fix this. But I will try to be consistent with the existing style that use
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156117336
@felixcheung, "R CMD INSTALL" is used to install R packages. Now if any R
package depends on SparkR package, the installation will fail as the SparkR
package path is
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156328765
**[Test build #45818 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45818/consoleFull)**
for PR 9390 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156315138
**[Test build #45818 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45818/consoleFull)**
for PR 9390 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156314689
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156314710
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156137883
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156140442
**[Test build #45741 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45741/consoleFull)**
for PR 9390 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156137834
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156178460
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156178461
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156178252
**[Test build #45741 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45741/consoleFull)**
for PR 9390 at commit
Github user brkyvz commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-155849546
@sun-rui I tested this package locally. I failed to use an R Spark Package.
The problem is that if that package depends on SparkR, it can't find it during
R CMD INSTALL
Github user brkyvz commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-155865783
@sun-rui With this patch, you should be able to enable this test:
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-155996045
Thanks to @brkyvz for catching this issue. I think a better solution is as
documented at
https://stat.ethz.ch/R-manual/R-devel/library/base/html/libPaths.html:
The
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-156024874
I don't fully understand the issue, but why we have to use `.libPaths` and
not `library(... lib.loc= )`?
---
If your project is set up for it, you can reply to
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-155695077
I tested this PR on yarn-client/yarn-cluster with/without R source
packages. All work fine.
I may investigate later adding test cases for YARN mode.
---
If
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44204368
--- Diff: R/install-dev.bat ---
@@ -25,3 +25,9 @@ set SPARK_HOME=%~dp0..
MKDIR %SPARK_HOME%\R\lib
R.exe CMD INSTALL
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44204429
--- Diff: core/src/main/scala/org/apache/spark/api/r/RUtils.scala ---
@@ -23,6 +23,10 @@ import java.util.Arrays
import org.apache.spark.{SparkEnv,
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44192284
--- Diff: R/install-dev.bat ---
@@ -25,3 +25,9 @@ set SPARK_HOME=%~dp0..
MKDIR %SPARK_HOME%\R\lib
R.exe CMD INSTALL
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9390#discussion_r44192531
--- Diff: core/src/main/scala/org/apache/spark/api/r/RUtils.scala ---
@@ -23,6 +23,10 @@ import java.util.Arrays
import org.apache.spark.{SparkEnv,
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-154023464
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-154023468
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-154023060
**[Test build #45105 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45105/consoleFull)**
for PR 9390 at commit
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-154116173
@sun-rui Do R packages not work in standalone mode right now or is this
something we are breaking as a result of this change ?
Ping @brkyvz to take a look at
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-154252506
I will test this PR on YARN cluster when I am free.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9390#issuecomment-154252457
@shivaram, R packages not work in standalone mode is an existing bug, not a
result of this PR.
---
If your project is set up for it, you can reply to this email and
80 matches
Mail list logo
