[
https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448272#comment-17448272
]
Yongjun Zhang edited comment on SPARK-25642 at 11/23/21, 10:46 PM:
-------------------------------------------------------------------
Thanks [~pgandhi] .
What I observed is most of the time numRegisteredConnections is smaller than
numActiveConnections.
For example, in one case the gap can be as large as beyond 3k. while
numRegisteredConnections range between 0 - 2.5k, and numActiveConnections range
between 0 - 3.6k.
Wonder if anyone else who look at these metrics have different observations.
was (Author: yzhangal):
Thanks [~pgandhi] .
What I observed is most of the time numRegisteredConnections is smaller than
numActiveConnections.
For example, in one case the gap can be as large as beyond 3k. while
numRegisteredConnections range between 0 - 2.5k, and
numActiveRegisteredConnections range between 0 - 3.6k.
Wonder if anyone else who look at these metrics have different observations.
> Add new Metrics in External Shuffle Service to help determine Network
> performance and Connection Handling capabilities of the Shuffle Service
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-25642
> URL: https://issues.apache.org/jira/browse/SPARK-25642
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Affects Versions: 2.4.0
> Reporter: Parth Gandhi
> Assignee: Parth Gandhi
> Priority: Minor
> Fix For: 3.0.0
>
>
> Recently, the ability to expose the metrics for YARN Shuffle Service was
> added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]].
> We need to add some metrics to be able to determine the number of active
> connections as well as open connections to the external shuffle service to
> benchmark network and connection issues on large cluster environments.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
