[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 Merged to master. Thanks @felixcheung. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98848/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23012 **[Test build #98848 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98848/testReport)** for PR 23012 at commit [`f153413`](https://github.com/apache/spark/commit/f153413357a45b361a83c5eb93a9fcd5762c1499). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23012 **[Test build #98848 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98848/testReport)** for PR 23012 at commit [`f153413`](https://github.com/apache/spark/commit/f153413357a45b361a83c5eb93a9fcd5762c1499). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5038/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 Yup, will address the other comments and update the PR accordingly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/23012 @felixcheung @HyukjinKwon yes: deprecation in this case means we test against R-3.1.1 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 Yea there are some problem with some packages we depend on that are not installable from CRAN (eg too old) so it will be hard to a new version of R and new installation. So to clarify, depreciation as is we still test on R 3.1? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 Ah .. right makes sense to me. Thanks @shaneknapp. +1 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/23012 TL;DR: let's go w/deprecation. still TL;DR: if i never have to install or manage R again, i will be a happy person! @HyukjinKwon upgrading R is easy. getting the right mix of R and all of the associated packages working "as expected" is a nightmare. the biggest problem i foresee is if we upgrade R (and all other packages) on the workers, every version of spark will be tested against this... and there will be bugs, test failures, and other time consuming (and obtuse) problems to debug. multiply this by every branch, and you can see the rabbit hole you've just entered. for example, a month ago when i finally had time to dive back in to the ubuntu port, after finally figuring out how to install R+friends on ubuntu in an identical way to the centos workers, i STILL was finding problems w/lintr (see: https://github.com/apache/spark/pull/22896). anyways: i'm more than happy to upgrade R and all the packages to something much more recent, but i will definitely appreciate some help in the game of test-failure whack-a-mole. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 @shaneknapp, do you roughly know how difficult it is (and do you have some time shortly) to upgrade R from 3.1 to 3.4? I am asking this because I had some difficulties when I tried to manually upgrade from a certain low version to another non-latest version. If it's expected to take a while, let's go deprecation step. If that's expected to be less difficult, let's go saying unsupporting way. Does this sound okay to you @felixcheung? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 Hey shane I donât think we are saying to test multiple R version at all. In fact quite the opposite, just the new(er) version at some point in the future. (We donât have a better solution for packages though. Thereâs another PR for R arrow package for example) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/23012 howdy howdy! unless we dockerize spark builds (someday!), we're going to be stuck w/testing against one version of R on the jenkins workers... i've been looking in to packrat to help manage packages, but having more than one version of R will require me manually building and disting it out. and i really, truly, don't want to do that. let me know how you think i should proceed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 I think it's easier to say unsupported if we are not testing it in jenkins or appveyer. I don't know if we any coverage at release for older R version anyway, so it's better to unsupported then deprecate. but agree maybe the way to do this is deprecate without updating R in jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 In this way, we could postpone R upgrade after Spark 3.0.0 release in Jenkins, and could still test the deprecated R version 3.1. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 Nice. Thanks!. BTW Felix, are you maybe worrying about that we happen to upgrade R version in Jenkins to 3.4 and .. we could break lower deprecated R version support in Spark 3.0 I guess? If so, let me put the version check into both places `generic.R` and `shell.R`. In this way, both shell and submit still show the errors but the tests will pass. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 FYI This is unused code Iâm going to remove it https://github.com/apache/spark/blob/master/R/pkg/src-native/string_hash_code.c --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23012 **[Test build #98736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98736/testReport)** for PR 23012 at commit [`3ec34f1`](https://github.com/apache/spark/commit/3ec34f12eea0a3463234d8acae3526b7ed11a4ec). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98736/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23012 **[Test build #98736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98736/testReport)** for PR 23012 at commit [`3ec34f1`](https://github.com/apache/spark/commit/3ec34f12eea0a3463234d8acae3526b7ed11a4ec). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4956/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 Yea will take a look to address. But about documenting unsupported, if we explicitly are going to say it's unsupported and dropped, for instance, we should remove the compatibility change (https://github.com/apache/spark/blob/master/R/pkg/src-native/string_hash_code.c) and I assume previous versions don't work. Deprecation step might be more concervative and consistent with dropping steps of other language APIs. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 Also I think the warning should be in .First in general.R --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 I think this should say unsupported (ie could still work) instead of deprecated Also the compareVersion should check both major and minor ie 3.4.0 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98718/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23012 **[Test build #98718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98718/testReport)** for PR 23012 at commit [`dc2dbd9`](https://github.com/apache/spark/commit/dc2dbd923a1396ca5a7a950df35da57cc70c2ab8). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23012 **[Test build #98718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98718/testReport)** for PR 23012 at commit [`dc2dbd9`](https://github.com/apache/spark/commit/dc2dbd923a1396ca5a7a950df35da57cc70c2ab8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23012 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4944/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 adding @srowen too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 Tests probably will fail since it produces warnings. cc @felixcheung. @shaneknapp, @viirya, @shivaram, @falaki, @mengxr, @yanboliang FYI. This PR is made per http://apache-spark-developers-list.1001551.n3.nabble.com/discuss-SparkR-CRAN-feasibility-check-server-problem-td25605.html --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
