[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21278 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187514621 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,48 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + javaReqs <- utils::packageDescription(utils::packageName(), fields=c("SystemRequirements")) + sparkJavaVersion <- as.numeric(tail(strsplit(javaReqs, "[(=)]")[[1]], n = 1L)) + if (javaHome != "") { +javaBin <- file.path(javaHome, "bin", javaBin) --- End diff -- ah ok --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187470922 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,48 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + javaReqs <- utils::packageDescription(utils::packageName(), fields=c("SystemRequirements")) + sparkJavaVersion <- as.numeric(tail(strsplit(javaReqs, "[(=)]")[[1]], n = 1L)) + if (javaHome != "") { +javaBin <- file.path(javaHome, "bin", javaBin) --- End diff -- Yes. If you look at https://github.com/apache/spark/blob/d3c426a5b02abdec49ff45df12a8f11f9e473a88/bin/spark-class2.cmd#L54 we append `bin` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187469392 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,48 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + javaReqs <- utils::packageDescription(utils::packageName(), fields=c("SystemRequirements")) + sparkJavaVersion <- as.numeric(tail(strsplit(javaReqs, "[(=)]")[[1]], n = 1L)) + if (javaHome != "") { +javaBin <- file.path(javaHome, "bin", javaBin) --- End diff -- this `bin` - is it applicable to Windows? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187469216 --- Diff: R/pkg/DESCRIPTION --- @@ -13,6 +13,7 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"), License: Apache License (== 2.0) URL: http://www.apache.org/ http://spark.apache.org/ BugReports: http://spark.apache.org/contributing.html +SystemRequirements: Java (== 8) Depends: R (>= 3.0), --- End diff -- yap let's separate this out --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187407193 --- Diff: R/pkg/DESCRIPTION --- @@ -13,6 +13,7 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"), License: Apache License (== 2.0) URL: http://www.apache.org/ http://spark.apache.org/ BugReports: http://spark.apache.org/contributing.html +SystemRequirements: Java (== 8) Depends: R (>= 3.0), --- End diff -- Thats a valid point. Lets discuss this in JIRA / mailing list ? I think if it still works for 3.0 we should keep it because R might throw an error / warning for users on 3.0. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187405804 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,48 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + javaReqs <- packageDescription("SparkR", fields=c("SystemRequirements")) + sparkJavaVersion <- as.numeric(tail(strsplit(javaReqs, "[(=)]")[[1]], n = 1L)) + if (javaHome != "") { +javaBin <- file.path(javaHome, javaBin) + } + + # If java is missing from PATH, we get an error in Unix and a warning in Windows + javaVersionOut <- tryCatch( + launchScript(javaBin, "-version", wait = TRUE, stdout = TRUE, stderr = TRUE), + error = function(e) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }, + warning = function(w) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }) + javaVersionFilter <- Filter( + function(x) { +grepl("java version", x) + }, javaVersionOut) + + javaVersionStr <- strsplit(javaVersionFilter[[1]], "[\"]")[[1L]][2] + # javaVersionStr is of the form 1.8.0_92. + # Extract 8 from it to compare to sparkJavaVersion + javaVersionNum <- as.numeric(paste0(strsplit(javaVersionStr, "[.]")[[1L]][2], collapse = ".")) --- End diff -- Fixed - I was initially trying to get `1.8` but given that the requirements says Java 8 I think this is fine for now. As you said the format keeps changing though. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187405812 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,48 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + javaReqs <- packageDescription("SparkR", fields=c("SystemRequirements")) --- End diff -- Fixed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187238820 --- Diff: R/pkg/DESCRIPTION --- @@ -13,6 +13,7 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"), License: Apache License (== 2.0) URL: http://www.apache.org/ http://spark.apache.org/ BugReports: http://spark.apache.org/contributing.html +SystemRequirements: Java (== 8) Depends: R (>= 3.0), --- End diff -- btw, I saw this the other day, and thought we should update this to `>= 3.11` to reflect what we test with? what do you think? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187237615 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,48 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + javaReqs <- packageDescription("SparkR", fields=c("SystemRequirements")) + sparkJavaVersion <- as.numeric(tail(strsplit(javaReqs, "[(=)]")[[1]], n = 1L)) + if (javaHome != "") { +javaBin <- file.path(javaHome, javaBin) + } + + # If java is missing from PATH, we get an error in Unix and a warning in Windows + javaVersionOut <- tryCatch( + launchScript(javaBin, "-version", wait = TRUE, stdout = TRUE, stderr = TRUE), + error = function(e) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }, + warning = function(w) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }) + javaVersionFilter <- Filter( + function(x) { +grepl("java version", x) + }, javaVersionOut) + + javaVersionStr <- strsplit(javaVersionFilter[[1]], "[\"]")[[1L]][2] + # javaVersionStr is of the form 1.8.0_92. + # Extract 8 from it to compare to sparkJavaVersion + javaVersionNum <- as.numeric(paste0(strsplit(javaVersionStr, "[.]")[[1L]][2], collapse = ".")) --- End diff -- isn't `as.numeric(strsplit(javaVersionStr, "[.]")[[1L]][2])` sufficient? or `as.integer(strsplit(javaVersionStr, "[.]")[[1L]][2])` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187237024 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,48 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + javaReqs <- packageDescription("SparkR", fields=c("SystemRequirements")) --- End diff -- nit: use `packageName()`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187169358 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,39 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + if (javaHome != "") { +javaBin <- file.path(javaHome, javaBin) + } + + # If java is missing from PATH, we get an error in Unix and a warning in Windows + javaVersionOut <- tryCatch( + launchScript(javaBin, "-version", wait = TRUE, stdout = TRUE, stderr = TRUE), + error = function(e) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }, + warning = function(w) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }) + javaVersionStr <- strsplit(javaVersionOut[[1]], "[\"]")[[1L]][2] + javaVersionNum <- as.numeric(paste0(strsplit(javaVersionStr, "[.]")[[1L]][1:2], collapse = ".")) + if(javaVersionNum < 1.8) { +stop(paste("Java 8, or greater, is required for this package; found version:", javaVersionNum)) --- End diff -- I changed this to javaVersionStr. In addition I parsed the requirements field -- more details on this below --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187169209 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,39 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + if (javaHome != "") { +javaBin <- file.path(javaHome, javaBin) + } + + # If java is missing from PATH, we get an error in Unix and a warning in Windows + javaVersionOut <- tryCatch( + launchScript(javaBin, "-version", wait = TRUE, stdout = TRUE, stderr = TRUE), + error = function(e) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }, + warning = function(w) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }) + javaVersionStr <- strsplit(javaVersionOut[[1]], "[\"]")[[1L]][2] + javaVersionNum <- as.numeric(paste0(strsplit(javaVersionStr, "[.]")[[1L]][1:2], collapse = ".")) + if(javaVersionNum < 1.8) { --- End diff -- Fixed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187169097 --- Diff: R/pkg/R/sparkR.R --- @@ -163,6 +163,10 @@ sparkR.sparkContext <- function( submitOps <- getClientModeSparkSubmitOpts( Sys.getenv("SPARKR_SUBMIT_ARGS", "sparkr-shell"), sparkEnvirMap) +tryCatch(checkJavaVersion(), + error = function(e) { + stop("Java version check failed") --- End diff -- Right- Initially I had thought of returning an error code. But I think stop within the function is good enough. Removed this --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r187168952 --- Diff: R/pkg/R/utils.R --- @@ -756,7 +756,7 @@ launchScript <- function(script, combinedArgs, wait = FALSE) { # stdout = F means discard output # stdout = "" means to its console (default) # Note that the console of this child process might not be the same as the running R process. -system2(script, combinedArgs, stdout = "", wait = wait) +system2(script, combinedArgs, stdout = stdout, wait = wait, stder = stderr) --- End diff -- Fixed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r186940485 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,39 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + if (javaHome != "") { +javaBin <- file.path(javaHome, javaBin) + } + + # If java is missing from PATH, we get an error in Unix and a warning in Windows + javaVersionOut <- tryCatch( + launchScript(javaBin, "-version", wait = TRUE, stdout = TRUE, stderr = TRUE), + error = function(e) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }, + warning = function(w) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }) + javaVersionStr <- strsplit(javaVersionOut[[1]], "[\"]")[[1L]][2] + javaVersionNum <- as.numeric(paste0(strsplit(javaVersionStr, "[.]")[[1L]][1:2], collapse = ".")) + if(javaVersionNum < 1.8) { +stop(paste("Java 8, or greater, is required for this package; found version:", javaVersionNum)) --- End diff -- print javaVersionStr instead? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r186940268 --- Diff: R/pkg/R/sparkR.R --- @@ -163,6 +163,10 @@ sparkR.sparkContext <- function( submitOps <- getClientModeSparkSubmitOpts( Sys.getenv("SPARKR_SUBMIT_ARGS", "sparkr-shell"), sparkEnvirMap) +tryCatch(checkJavaVersion(), + error = function(e) { + stop("Java version check failed") --- End diff -- ah ok, I see it. but then why tryCatch here? aren't we calling stop() multiple times? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r186939722 --- Diff: R/pkg/R/utils.R --- @@ -756,7 +756,7 @@ launchScript <- function(script, combinedArgs, wait = FALSE) { # stdout = F means discard output # stdout = "" means to its console (default) # Note that the console of this child process might not be the same as the running R process. -system2(script, combinedArgs, stdout = "", wait = wait) +system2(script, combinedArgs, stdout = stdout, wait = wait, stder = stderr) --- End diff -- nit: `stder` -> `stderr` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r186940416 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,39 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + if (javaHome != "") { +javaBin <- file.path(javaHome, javaBin) + } + + # If java is missing from PATH, we get an error in Unix and a warning in Windows + javaVersionOut <- tryCatch( + launchScript(javaBin, "-version", wait = TRUE, stdout = TRUE, stderr = TRUE), + error = function(e) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }, + warning = function(w) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }) + javaVersionStr <- strsplit(javaVersionOut[[1]], "[\"]")[[1L]][2] + javaVersionNum <- as.numeric(paste0(strsplit(javaVersionStr, "[.]")[[1L]][1:2], collapse = ".")) + if(javaVersionNum < 1.8) { +stop(paste("Java 8, or greater, is required for this package; found version:", javaVersionNum)) --- End diff -- re:the part about java 9 discussion... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r186939913 --- Diff: R/pkg/R/sparkR.R --- @@ -163,6 +163,10 @@ sparkR.sparkContext <- function( submitOps <- getClientModeSparkSubmitOpts( Sys.getenv("SPARKR_SUBMIT_ARGS", "sparkr-shell"), sparkEnvirMap) +tryCatch(checkJavaVersion(), + error = function(e) { + stop("Java version check failed") --- End diff -- perhaps add more here, eg. "only Java 8 is supported" (though maybe we shouldn't encode the version here so...) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21278#discussion_r186940332 --- Diff: R/pkg/R/client.R --- @@ -60,13 +60,39 @@ generateSparkSubmitArgs <- function(args, sparkHome, jars, sparkSubmitOpts, pack combinedArgs } +checkJavaVersion <- function() { + javaBin <- "java" + javaHome <- Sys.getenv("JAVA_HOME") + if (javaHome != "") { +javaBin <- file.path(javaHome, javaBin) + } + + # If java is missing from PATH, we get an error in Unix and a warning in Windows + javaVersionOut <- tryCatch( + launchScript(javaBin, "-version", wait = TRUE, stdout = TRUE, stderr = TRUE), + error = function(e) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }, + warning = function(w) { + stop("Java version check failed. Please make sure Java is installed", + " and set JAVA_HOME to point to the installation directory.") + }) + javaVersionStr <- strsplit(javaVersionOut[[1]], "[\"]")[[1L]][2] + javaVersionNum <- as.numeric(paste0(strsplit(javaVersionStr, "[.]")[[1L]][1:2], collapse = ".")) + if(javaVersionNum < 1.8) { --- End diff -- nit style: space after `if` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21278: [SPARKR] Require Java 8 for SparkR
GitHub user shivaram opened a pull request: https://github.com/apache/spark/pull/21278 [SPARKR] Require Java 8 for SparkR This change updates the SystemRequirements and also includes a runtime check if the JVM is being launched by R. The runtime check is done by querying `java -version` ## How was this patch tested? Tested on a Mac and Windows machine You can merge this pull request into a Git repository by running: $ git pull https://github.com/shivaram/spark-1 sparkr-skip-solaris Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21278.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21278 commit b4e10397d890a90a9314dbe4ceef3986c7e0f698 Author: Shivaram Venkataraman Date: 2018-05-09T04:59:12Z Require Java 8 for SparkR This change updates the SystemRequirements and also includes a runtime check if the JVM is being launched by R. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org