[GitHub] spark pull request #21613: [SPARK-24629][SQL]thrift server memory leaks when...
GitHub user ChenjunZou opened a pull request: https://github.com/apache/spark/pull/21613 [SPARK-24629][SQL]thrift server memory leaks when Beeline session is closed ## What changes were proposed in this pull request? Maintaining a running statement map. When the session is closed, the relative statements should receive a cancel event. ## How was this patch tested? the attached UI show statements is successfully cancelled in UI. https://user-images.githubusercontent.com/2654887/41772001-94117874-7649-11e8-9701-6edf13593b9d.png;> Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ChenjunZou/spark mem-leak Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21613.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21613 commit bc647cc135edb7f05a47a1eccbd33a8faa410253 Author: zouchenjun Date: 2018-06-15T08:53:45Z add cancel status commit 31a3d3fb22c6231622489c3fdbd6b282775c6b93 Author: zouchenjun Date: 2018-06-22T08:35:30Z fix memory leak when beeline close connection to thrift server commit 4a6d2e9d52e097f2ecd9c27a5b85253451341874 Author: zouchenjun Date: 2018-06-22T09:51:03Z bug fix --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19961: [SPARK-22496][SQL] thrift server adds operation l...
Github user ChenjunZou commented on a diff in the pull request:
https://github.com/apache/spark/pull/19961#discussion_r157117909
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
---
@@ -67,4 +68,16 @@ protected void setConfOverlay(Map<String, String>
confOverlay) {
this.confOverlay = confOverlay;
}
}
+
+ protected void registerCurrentOperationLog() {
+if (isOperationLogEnabled) {
+ if (operationLog == null) {
+LOG.warn("Failed to get current OperationLog object of Operation:
" +
+getHandle().getHandleIdentifier());
--- End diff --
thanks @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19961: [SPARK-22496][SQL] thrift server adds operation l...
GitHub user ChenjunZou opened a pull request: https://github.com/apache/spark/pull/19961 [SPARK-22496][SQL] thrift server adds operation logs ## What changes were proposed in this pull request? since hive 2.0+ upgrades log4j to log4j2ï¼a lot of [changes](https://issues.apache.org/jira/browse/HIVE-11304) are made working on it. as spark is not to ready to update its inner hive version(1.2.1) , so I manage to make little changes. the function registerCurrentOperationLog is moved from SQLOperstion to its parent class ExecuteStatementOperation so spark can use it. ## How was this patch tested? manual test Author: zouchenjun <zouchen...@youzan.com> Closes #19721 from ChenjunZou/operation-log. ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ChenjunZou/spark spark-22496 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19961.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19961 commit b39d36fba5ffbbcac6ba9c15b4062b7ade7b6f1e Author: zouchenjun <zouchen...@youzan.com> Date: 2017-12-11T04:36:14Z [SPARK-22496][SQL] thrift server adds operation logs ## What changes were proposed in this pull request? since hive 2.0+ upgrades log4j to log4j2ï¼a lot of [changes](https://issues.apache.org/jira/browse/HIVE-11304) are made working on it. as spark is not to ready to update its inner hive version(1.2.1) , so I manage to make little changes. the function registerCurrentOperationLog is moved from SQLOperstion to its parent class ExecuteStatementOperation so spark can use it. ## How was this patch tested? manual test Author: zouchenjun <zouchen...@youzan.com> Closes #19721 from ChenjunZou/operation-log. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19721: [SPARK-22496][SQL]thrift server adds operation lo...
GitHub user ChenjunZou opened a pull request: https://github.com/apache/spark/pull/19721 [SPARK-22496][SQL]thrift server adds operation logs ## What changes were proposed in this pull request? since hive 2.0+ uses upgrade log4j to log4j2ï¼a lot of [changes](https://issues.apache.org/jira/browse/HIVE-11304) are made working on it. As spark is not to ready to update its inner hive version(1.2.1) , so I manages to make little changes. the function registerCurrentOperationLog is moved from SQLOperstion to its parent class ExecuteStatementOperation so spark can use it. ## How was this patch tested? manual test You can merge this pull request into a Git repository by running: $ git pull https://github.com/ChenjunZou/spark operation-log Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19721.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19721 commit 0ee967c658da40b3f5f5a79276df340b66208b3f Author: zouchenjun <zouchen...@youzan.com> Date: 2017-11-11T09:36:58Z spark thrift server adds operation logs --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19637: [SPARK-22243][DStream]spark.yarn.jars should reload from...
Github user ChenjunZou commented on the issue: https://github.com/apache/spark/pull/19637 add spark.yarn.jars to the checkpoint reload configs. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19637: [SPARK-22243][DStream]spark.yarn.jars should relo...
GitHub user ChenjunZou opened a pull request: https://github.com/apache/spark/pull/19637 [SPARK-22243][DStream]spark.yarn.jars should reload from config when checkpoint recovery the previous pr branch is deleted by mistake You can merge this pull request into a Git repository by running: $ git pull https://github.com/ChenjunZou/spark checkpoint-yarn-jars Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19637.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19637 commit 4a7d3d80dff14ba7bd9c71be1307f261051bed12 Author: ZouChenjun <zouchen...@youzan.com> Date: 2017-10-10T12:34:07Z set spark.yarn.jars reload from config --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19469: [SPARK-22243][DStreams]spark.yarn.jars reload fro...
Github user ChenjunZou closed the pull request at: https://github.com/apache/spark/pull/19469 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19469: [SPARK-22243][DStreams]spark.yarn.jars reload fro...
GitHub user ChenjunZou opened a pull request: https://github.com/apache/spark/pull/19469 [SPARK-22243][DStreams]spark.yarn.jars reload from config when Checkpoint recovery add spark.yarn.jars to the checkpoint reload configs. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ChenjunZou/spark checkpoint-yarn-jars Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19469.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19469 commit 7a03073d0cdca3e65ada1e710a96cfadfa332d05 Author: ZouChenjun <zouchen...@youzan.com> Date: 2017-10-10T12:34:07Z set spark.yarn.jars reload from config --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
