[
https://issues.apache.org/jira/browse/SPARK-31646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418366#comment-17418366
]
Yongjun Zhang commented on SPARK-31646:
---------------------------------------
HI [~mauzhang],
Thanks a lot for your answers and sorry for late reply. I think I understand it
better now why you are doing this change: the registeredConnections metrics
added in ExternalShuffleBlockHandler was not used.
However, the one added to TransportContext is used, see in
YarnShuffleService.java:
{code:java}
// register metrics on the block handler into the Node Manager's metrics
system.
blockHandler.getAllMetrics().getMetrics().put("numRegisteredConnections",
shuffleServer.getRegisteredConnections());
YarnShuffleServiceMetrics serviceMetrics =
new YarnShuffleServiceMetrics(blockHandler.getAllMetrics());
MetricsSystemImpl metricsSystem = (MetricsSystemImpl)
DefaultMetricsSystem.instance();
metricsSystem.register(
"sparkShuffleService", "Metrics on the Spark Shuffle Service",
serviceMetrics);
logger.info("Registered metrics with Hadoop's DefaultMetricsSystem");
{code}
The TransportContext version of registeredConnections is retrieved by
"shuffleServer.getRegisteredConnections())" in the above code. That means both
the activeConnections and registeredConnections are still available with your
change. Is that your expectation?
If my understanding is correct, we can either derive "registeredConnections -
activeConnections" as the backlogged connections, or we can add a new metrics
as backloggedConnection to have the value of "registeredConnections -
activeConnections" .
What do you think?
Thanks!
> Remove unused registeredConnections counter from ShuffleMetrics
> ---------------------------------------------------------------
>
> Key: SPARK-31646
> URL: https://issues.apache.org/jira/browse/SPARK-31646
> Project: Spark
> Issue Type: Improvement
> Components: Deploy, Shuffle, Spark Core
> Affects Versions: 3.0.0
> Reporter: Manu Zhang
> Assignee: Manu Zhang
> Priority: Minor
> Fix For: 3.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
