[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user yaooqinn commented on the issue: https://github.com/apache/spark/pull/18666 @liufengdb yes you are right. what's more, the Hive's `SessionState` is too overstaffed for spark to create an hive client, we may only need an `IMetaStoreClient` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user liufengdb commented on the issue: https://github.com/apache/spark/pull/18666 Maybe I missed something, but it seems Spark has its own class loader right now, which can load the class from the given URL: https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala#L211. I doubt it needs to download the jar to the hive configured temp directory. In other words, this line of code is not necessary: https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L837. This can be verified very easily: Remove https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L837 and run `add jar` and see whether it works. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user yaooqinn commented on the issue: https://github.com/apache/spark/pull/18666 @liufengdb its necessary to create these for `add jar` cmd --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user liufengdb commented on the issue: https://github.com/apache/spark/pull/18666 I asked the following question in https://github.com/apache/spark/pull/20864: is it necessary to create these temp directories when the hive thrift server starts? It sounds some legacy from Hive and we can skip creating them in the first place. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18666 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1633/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user yaooqinn commented on the issue: https://github.com/apache/spark/pull/18666 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18666 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1625/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/18666
@gatorsmile would you plz take a look at this.
this pr mainly want to close HiveSessionState explicitly to delete
`hive.downloaded.resources.dir` which points to `"${system:java.io.tmpdir}" +
File.separator + "${hive.session.id}_resources"` by default
`hive.exec.local.scratchdir` which points to `"${system:java.io.tmpdir}" +
File.separator + "${system:user.name}"` by default and some other dirs which
used only for hive but without deleting hook on shutdown.
the below code is how HiveSessionState create
`hive.downloaded.resources.dir`, `isCleanUp` is set to `false`.
```scala
// 3. Download resources dir
path = new Path(HiveConf.getVar(conf,
HiveConf.ConfVars.DOWNLOADED_RESOURCES_DIR));
createPath(conf, path, scratchDirPermission, true, **isCleanUp** = false);
Plenty of unused dirs left after submit a lot of Hive supported spark
applications.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18666 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1624/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user yaooqinn commented on the issue: https://github.com/apache/spark/pull/18666 @samartinucci thanks for reminding of this, i have fixed the conflicts. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user samartinucci commented on the issue: https://github.com/apache/spark/pull/18666 Do we have a workaround for this? Will this PR get merged? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18666: [SPARK-21449][SQL][Hive]Close HiveClient's SessionState ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18666 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
