This is an automated email from the ASF dual-hosted git repository.
panbingkun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 77055b812559 [SPARK-50385][CORE] Use class name prefix for REST
Submission API thread pool
77055b812559 is described below
commit 77055b8125591abb6bff9cb029bb725ad7d47d2e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 22 15:54:13 2024 +0800
[SPARK-50385][CORE] Use class name prefix for REST Submission API thread
pool
### What changes were proposed in this pull request?
This PR aims to use a meaningful class name prefix for REST Submission API
thread pool instead of the default value of Jetty QueuedThreadPool,
`"qtp"+super.hashCode()`.
https://github.com/dekellum/jetty/blob/3dc0120d573816de7d6a83e2d6a97035288bdd4a/jetty-util/src/main/java/org/eclipse/jetty/util/thread/QueuedThreadPool.java#L64
### Why are the changes needed?
This is helpful during JVM investigation.
**BEFORE (4.0.0-preview2)**
```
$ SPARK_MASTER_OPTS='-Dspark.master.rest.enabled=true' sbin/start-master.sh
$ jstack 28217 | grep qtp
"qtp1925630411-52" #52 daemon prio=5 os_prio=31 cpu=0.07ms elapsed=19.06s
tid=0x0000000134906c10 nid=0xde03 runnable [0x0000000314592000]
"qtp1925630411-53" #53 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=19.06s
tid=0x0000000134ac6810 nid=0xc603 runnable [0x000000031479e000]
"qtp1925630411-54" #54 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=19.06s
tid=0x000000013491ae10 nid=0xdc03 runnable [0x00000003149aa000]
"qtp1925630411-55" #55 daemon prio=5 os_prio=31 cpu=0.08ms elapsed=19.06s
tid=0x0000000134ac9810 nid=0xc803 runnable [0x0000000314bb6000]
"qtp1925630411-56" #56 daemon prio=5 os_prio=31 cpu=0.04ms elapsed=19.06s
tid=0x0000000134ac9e10 nid=0xda03 runnable [0x0000000314dc2000]
"qtp1925630411-57" #57 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=19.06s
tid=0x0000000134aca410 nid=0xca03 runnable [0x0000000314fce000]
"qtp1925630411-58" #58 daemon prio=5 os_prio=31 cpu=0.04ms elapsed=19.06s
tid=0x0000000134acaa10 nid=0xcb03 runnable [0x00000003151da000]
"qtp1925630411-59" #59 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=19.06s
tid=0x0000000134acb010 nid=0xcc03 runnable [0x00000003153e6000]
"qtp1925630411-60-acceptor-0108e9815-ServerConnector1e497474{HTTP/1.1,
(http/1.1)}{M3-Max.local:6066}" #60 daemon prio=3 os_prio=31 cpu=0.11ms
elapsed=19.06s tid=0x00000001317ffa10 nid=0xcd03 runnable [0x00000003155f2000]
"qtp1925630411-61-acceptor-11d90f2aa-ServerConnector1e497474{HTTP/1.1,
(http/1.1)}{M3-Max.local:6066}" #61 daemon prio=3 os_prio=31 cpu=0.10ms
elapsed=19.06s tid=0x00000001314ed610 nid=0xcf03 waiting on condition
[0x00000003157fe000]
```
**AFTER**
```
$ SPARK_MASTER_OPTS='-Dspark.master.rest.enabled=true' sbin/start-master.sh
$ jstack 28317 | grep StandaloneRestServer
"StandaloneRestServer-52" #52 daemon prio=5 os_prio=31 cpu=0.09ms
elapsed=60.06s tid=0x00000001284a8e10 nid=0xdb03 runnable [0x000000032cfce000]
"StandaloneRestServer-53" #53 daemon prio=5 os_prio=31 cpu=0.06ms
elapsed=60.06s tid=0x00000001284acc10 nid=0xda03 runnable [0x000000032d1da000]
"StandaloneRestServer-54" #54 daemon prio=5 os_prio=31 cpu=0.05ms
elapsed=60.06s tid=0x00000001284ae610 nid=0xd803 runnable [0x000000032d3e6000]
"StandaloneRestServer-55" #55 daemon prio=5 os_prio=31 cpu=0.09ms
elapsed=60.06s tid=0x00000001284aec10 nid=0xd703 runnable [0x000000032d5f2000]
"StandaloneRestServer-56" #56 daemon prio=5 os_prio=31 cpu=0.06ms
elapsed=60.06s tid=0x00000001284af210 nid=0xc803 runnable [0x000000032d7fe000]
"StandaloneRestServer-57" #57 daemon prio=5 os_prio=31 cpu=0.05ms
elapsed=60.06s tid=0x00000001284af810 nid=0xc903 runnable [0x000000032da0a000]
"StandaloneRestServer-58" #58 daemon prio=5 os_prio=31 cpu=0.06ms
elapsed=60.06s tid=0x00000001284afe10 nid=0xcb03 runnable [0x000000032dc16000]
"StandaloneRestServer-59" #59 daemon prio=5 os_prio=31 cpu=0.05ms
elapsed=60.06s tid=0x00000001284b0410 nid=0xcc03 runnable [0x000000032de22000]
"StandaloneRestServer-60-acceptor-04aefbaa8-ServerConnector44284d85{HTTP/1.1,
(http/1.1)}{M3-Max.local:6066}" #60 daemon prio=3 os_prio=31 cpu=0.13ms
elapsed=60.05s tid=0x000000015cda1a10 nid=0xcd03 runnable [0x000000032e02e000]
"StandaloneRestServer-61-acceptor-148976251-ServerConnector44284d85{HTTP/1.1,
(http/1.1)}{M3-Max.local:6066}" #61 daemon prio=3 os_prio=31 cpu=0.12ms
elapsed=60.05s tid=0x000000015cd1c810 nid=0xce03 waiting on condition
[0x000000032e23a000]
```
### Does this PR introduce _any_ user-facing change?
No, the thread names are accessed during the debugging.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48924 from dongjoon-hyun/SPARK-50385.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: panbingkun <[email protected]>
---
.../main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala | 1 +
1 file changed, 1 insertion(+)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
index 877349da18dd..9e3aab125689 100644
---
a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
@@ -94,6 +94,7 @@ private[spark] abstract class RestSubmissionServer(
*/
private def doStart(startPort: Int): (Server, Int) = {
val threadPool = new
QueuedThreadPool(masterConf.get(MASTER_REST_SERVER_MAX_THREADS))
+ threadPool.setName(getClass().getSimpleName())
if (Utils.isJavaVersionAtLeast21 &&
masterConf.get(MASTER_REST_SERVER_VIRTUAL_THREADS)) {
val newVirtualThreadPerTaskExecutor =
classOf[Executors].getMethod("newVirtualThreadPerTaskExecutor")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
