[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20681


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/20681#discussion_r171910649
  
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -67,6 +67,8 @@ sparkSession <- if (windows_with_hadoop()) {
 sparkR.session(master = sparkRTestMaster, enableHiveSupport = FALSE)
   }
 sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", 
"getJavaSparkContext", sparkSession)
+# materialize the catalog implementation
+listTables()
--- End diff --

ok


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...

[liufengdb]

Github user liufengdb commented on a diff in the pull request:

https://github.com/apache/spark/pull/20681#discussion_r171770982
  
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -67,6 +67,8 @@ sparkSession <- if (windows_with_hadoop()) {
 sparkR.session(master = sparkRTestMaster, enableHiveSupport = FALSE)
   }
 sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", 
"getJavaSparkContext", sparkSession)
+# materialize the catalog implementation
+listTables()
--- End diff --

`test_sparkSQL.R` is the only one uses 
`newJObject("org.apache.spark.sql.hive.test.TestHiveContext", ssc, FALSE)` on 
the `ssc`, so the catalog impl spark conf is changed. So ``test_sparkSQL.R` is 
the only one broken.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/20681#discussion_r171750940
  
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -67,6 +67,8 @@ sparkSession <- if (windows_with_hadoop()) {
 sparkR.session(master = sparkRTestMaster, enableHiveSupport = FALSE)
   }
 sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", 
"getJavaSparkContext", sparkSession)
+# materialize the catalog implementation
+listTables()
--- End diff --

we are calling `sparkR.session(master = sparkRTestMaster, enableHiveSupport 
= FALSE)` is almost every other test files - does the same apply in those other 
places?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/20681#discussion_r170862425
  
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -885,24 +894,24 @@ test_that("collect() and take() on a DataFrame return 
the same number of rows an
 })
 
 test_that("collect() support Unicode characters", {
-  lines <- c("{\"name\":\"안녕하세요\"}",
- "{\"name\":\"您好\", \"age\":30}",
- "{\"name\":\"こんにちは\", \"age\":19}",
- "{\"name\":\"Xin chào\"}")
+  lines <- c("{\"name\":\"?\"}",
+ "{\"name\":\"??\", \"age\":30}",
+ "{\"name\":\"?\", \"age\":19}",
+ "{\"name\":\"Xin ch?o\"}")
--- End diff --

no, we should not change these


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/20681#discussion_r170840835
  
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -885,24 +894,24 @@ test_that("collect() and take() on a DataFrame return 
the same number of rows an
 })
 
 test_that("collect() support Unicode characters", {
-  lines <- c("{\"name\":\"안녕하세요\"}",
- "{\"name\":\"您好\", \"age\":30}",
- "{\"name\":\"こんにちは\", \"age\":19}",
- "{\"name\":\"Xin chào\"}")
+  lines <- c("{\"name\":\"?\"}",
+ "{\"name\":\"??\", \"age\":30}",
+ "{\"name\":\"?\", \"age\":19}",
+ "{\"name\":\"Xin ch?o\"}")
--- End diff --

Is it okay to replace these unicode characters?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/20681#discussion_r170787801
  
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -46,16 +49,19 @@ setHiveContext <- function(sc) {
 })
 hiveSession <- callJMethod(hiveCtx, "sparkSession")
   }
-  previousSession <- get(".sparkRsession", envir = .sparkREnv)
   assign(".sparkRsession", hiveSession, envir = .sparkREnv)
   assign(".prevSparkRsession", previousSession, envir = .sparkREnv)
+  assign(".prevSessionConf", previousConf, envir = .sparkREnv)
   hiveSession
 }
 
 unsetHiveContext <- function() {
   previousSession <- get(".prevSparkRsession", envir = .sparkREnv)
+  previousConf <- get(".prevSessionConf", envir = .sparkREnv)
+  callJStatic("org.apache.spark.sql.api.r.SQLUtils", 
"setSparkContextSessionConf", previousSessioni, previousConf)
--- End diff --

this should be `previousSession`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/20681#discussion_r170787695
  
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -46,16 +49,19 @@ setHiveContext <- function(sc) {
 })
 hiveSession <- callJMethod(hiveCtx, "sparkSession")
   }
-  previousSession <- get(".sparkRsession", envir = .sparkREnv)
   assign(".sparkRsession", hiveSession, envir = .sparkREnv)
   assign(".prevSparkRsession", previousSession, envir = .sparkREnv)
+  assign(".prevSessionConf", previousConf, envir = .sparkREnv)
   hiveSession
 }
 
 unsetHiveContext <- function() {
   previousSession <- get(".prevSparkRsession", envir = .sparkREnv)
+  previousConf <- get(".prevSessionConf", envir = .sparkREnv)
+  callJStatic("org.apache.spark.sql.api.r.SQLUtils", 
"setSparkContextSessionConf", previousSessioni, previousConf)
   assign(".sparkRsession", previousSession, envir = .sparkREnv)
   remove(".prevSparkRsession", envir = .sparkREnv)
+  remove(".prevSparkConf", envir = .sparkREnv)
--- End diff --

this should be `. prevSessionConf`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org