Repository: spark Updated Branches: refs/heads/master c03b6d115 -> 94e2064fa
[SPARK-11244][SPARKR] sparkR.stop() should remove SQLContext SparkR should remove `.sparkRSQLsc` and `.sparkRHivesc` when `sparkR.stop()` is called. Otherwise even when SparkContext is reinitialized, `sparkRSQL.init` returns the stale copy of the object and complains: ```r sc <- sparkR.init("local") sqlContext <- sparkRSQL.init(sc) sparkR.stop() sc <- sparkR.init("local") sqlContext <- sparkRSQL.init(sc) sqlContext ``` producing ```r Error in callJMethod(x, "getClass") : Invalid jobj 1. If SparkR was restarted, Spark operations need to be re-executed. ``` I have added the check and removal only when SparkContext itself is initialized. I have also added corresponding test for this fix. Let me know if you want me to move the test to SQL test suite instead. p.s. I tried lint-r but ended up a lots of errors on existing code. Author: Forest Fang <forest.f...@outlook.com> Closes #9205 from saurfang/sparkR.stop. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/94e2064f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/94e2064f Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/94e2064f Branch: refs/heads/master Commit: 94e2064fa1b04c05c805d9175c7c78bf583db5c6 Parents: c03b6d1 Author: Forest Fang <forest.f...@outlook.com> Authored: Thu Oct 22 09:34:07 2015 -0700 Committer: Shivaram Venkataraman <shiva...@cs.berkeley.edu> Committed: Thu Oct 22 09:34:07 2015 -0700 ---------------------------------------------------------------------- R/pkg/R/sparkR.R | 8 ++++++++ R/pkg/inst/tests/test_context.R | 10 ++++++++++ 2 files changed, 18 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/94e2064f/R/pkg/R/sparkR.R ---------------------------------------------------------------------- diff --git a/R/pkg/R/sparkR.R b/R/pkg/R/sparkR.R index 9cf2f1a..043b005 100644 --- a/R/pkg/R/sparkR.R +++ b/R/pkg/R/sparkR.R @@ -39,6 +39,14 @@ sparkR.stop <- function() { sc <- get(".sparkRjsc", envir = env) callJMethod(sc, "stop") rm(".sparkRjsc", envir = env) + + if (exists(".sparkRSQLsc", envir = env)) { + rm(".sparkRSQLsc", envir = env) + } + + if (exists(".sparkRHivesc", envir = env)) { + rm(".sparkRHivesc", envir = env) + } } if (exists(".backendLaunched", envir = env)) { http://git-wip-us.apache.org/repos/asf/spark/blob/94e2064f/R/pkg/inst/tests/test_context.R ---------------------------------------------------------------------- diff --git a/R/pkg/inst/tests/test_context.R b/R/pkg/inst/tests/test_context.R index 513bbc8..e99815e 100644 --- a/R/pkg/inst/tests/test_context.R +++ b/R/pkg/inst/tests/test_context.R @@ -26,6 +26,16 @@ test_that("repeatedly starting and stopping SparkR", { } }) +test_that("repeatedly starting and stopping SparkR SQL", { + for (i in 1:4) { + sc <- sparkR.init() + sqlContext <- sparkRSQL.init(sc) + df <- createDataFrame(sqlContext, data.frame(a = 1:20)) + expect_equal(count(df), 20) + sparkR.stop() + } +}) + test_that("rdd GC across sparkR.stop", { sparkR.stop() sc <- sparkR.init() # sc should get id 0 --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org