[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448272#comment-17448272 ] Yongjun Zhang commented on SPARK-25642: --- Thanks [~pgandhi] . What I observed is most of the time numRegisteredConnections is smaller than numActiveConnections. For example, in one case the gap can be as large as beyond 3k. while numRegisteredConnections range between 0 - 2.5k, and numActiveRegisteredConnections range between 0 - 3.6k. Wonder if anyone else who look at these metrics have different observations. > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Assignee: Parth Gandhi >Priority: Minor > Fix For: 3.0.0 > > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]]. > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17446592#comment-17446592 ] Parth Gandhi commented on SPARK-25642: -- Hello [~yzhangal] , ideally you are correct, numActiveConnections should be a subset of numRegisteredConnections. So if you are seeing a different observation, it may need to be investigated. Please let me know if you have any more questions regarding this PR. Thank you. > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Assignee: Parth Gandhi >Priority: Minor > Fix For: 3.0.0 > > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]]. > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442924#comment-17442924 ] Yongjun Zhang commented on SPARK-25642: --- Hi [~pgandhi] , thanks for your work here. I thought numActiveConnections is a subset of numRegisteredConnections thus the former should be smaller than the latter, somehow we are observing the former is larger than the latter. Wonder if my understanding is correct or if you had similar observation? thanks. > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Assignee: Parth Gandhi >Priority: Minor > Fix For: 3.0.0 > > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]]. > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[
https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726999#comment-16726999
]
ASF GitHub Bot commented on SPARK-25642:
asfgit closed pull request #22498: [SPARK-25642] : Adding two new metrics to
record the number of registered connections as well as the number of active
connections to YARN Shuffle Service
URL: https://github.com/apache/spark/pull/22498
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
b/common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
index 480b52652de53..1a3f3f2a6f249 100644
---
a/common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
+++
b/common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
@@ -20,6 +20,7 @@
import java.util.ArrayList;
import java.util.List;
+import com.codahale.metrics.Counter;
import io.netty.channel.Channel;
import io.netty.channel.ChannelPipeline;
import io.netty.channel.EventLoopGroup;
@@ -66,6 +67,8 @@
private final RpcHandler rpcHandler;
private final boolean closeIdleConnections;
private final boolean isClientOnly;
+ // Number of registered connections to the shuffle service
+ private Counter registeredConnections = new Counter();
/**
* Force to create MessageEncoder and MessageDecoder so that we can make
sure they will be created
@@ -221,7 +224,7 @@ private TransportChannelHandler
createChannelHandler(Channel channel, RpcHandler
TransportRequestHandler requestHandler = new
TransportRequestHandler(channel, client,
rpcHandler, conf.maxChunksBeingTransferred());
return new TransportChannelHandler(client, responseHandler, requestHandler,
- conf.connectionTimeoutMs(), closeIdleConnections);
+ conf.connectionTimeoutMs(), closeIdleConnections, this);
}
/**
@@ -234,4 +237,8 @@ private ChunkFetchRequestHandler
createChunkFetchHandler(TransportChannelHandler
}
public TransportConf getConf() { return conf; }
+
+ public Counter getRegisteredConnections() {
+return registeredConnections;
+ }
}
diff --git
a/common/network-common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java
b/common/network-common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java
index c824a7b0d4740..ca81099c4d5cb 100644
---
a/common/network-common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java
+++
b/common/network-common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java
@@ -21,6 +21,7 @@
import io.netty.channel.SimpleChannelInboundHandler;
import io.netty.handler.timeout.IdleState;
import io.netty.handler.timeout.IdleStateEvent;
+import org.apache.spark.network.TransportContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -57,18 +58,21 @@
private final TransportRequestHandler requestHandler;
private final long requestTimeoutNs;
private final boolean closeIdleConnections;
+ private final TransportContext transportContext;
public TransportChannelHandler(
TransportClient client,
TransportResponseHandler responseHandler,
TransportRequestHandler requestHandler,
long requestTimeoutMs,
- boolean closeIdleConnections) {
+ boolean closeIdleConnections,
+ TransportContext transportContext) {
this.client = client;
this.responseHandler = responseHandler;
this.requestHandler = requestHandler;
this.requestTimeoutNs = requestTimeoutMs * 1000L * 1000;
this.closeIdleConnections = closeIdleConnections;
+this.transportContext = transportContext;
}
public TransportClient getClient() {
@@ -176,4 +180,16 @@ public TransportResponseHandler getResponseHandler() {
return responseHandler;
}
+ @Override
+ public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
+transportContext.getRegisteredConnections().inc();
+super.channelRegistered(ctx);
+ }
+
+ @Override
+ public void channelUnregistered(ChannelHandlerContext ctx) throws Exception {
+transportContext.getRegisteredConnections().dec();
+super.channelUnregistered(ctx);
+ }
+
}
diff --git
a/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java
b/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java
index 9c85ab2f5f06f..eb5f10a5c1a1e 100644
---
a/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java
+++
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718222#comment-16718222 ] ASF GitHub Bot commented on SPARK-25642: AmplabJenkins removed a comment on issue #22498: [SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service URL: https://github.com/apache/spark/pull/22498#issuecomment-446407716 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Priority: Minor > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]]. > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[
https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718216#comment-16718216
]
ASF GitHub Bot commented on SPARK-25642:
vanzin commented on a change in pull request #22498: [SPARK-25642] : Adding two
new metrics to record the number of registered connections as well as the
number of active connections to YARN Shuffle Service
URL: https://github.com/apache/spark/pull/22498#discussion_r240835732
##
File path:
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
##
@@ -199,6 +191,18 @@ protected void serviceInit(Configuration conf) throws
Exception {
port = shuffleServer.getPort();
boundPort = port;
String authEnabledString = authEnabled ? "enabled" : "not enabled";
+
+ // register metrics on the block handler into the Node Manager's metrics
system.
+ blockHandler.getAllMetrics().getMetrics().put("numRegisteredConnections",
+ shuffleServer.getRegisteredConnections());
Review comment:
nit: indented too far, here and in others.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add new Metrics in External Shuffle Service to help determine Network
> performance and Connection Handling capabilities of the Shuffle Service
> -
>
> Key: SPARK-25642
> URL: https://issues.apache.org/jira/browse/SPARK-25642
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
>Affects Versions: 2.4.0
>Reporter: Parth Gandhi
>Priority: Minor
>
> Recently, the ability to expose the metrics for YARN Shuffle Service was
> added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]].
> We need to add some metrics to be able to determine the number of active
> connections as well as open connections to the external shuffle service to
> benchmark network and connection issues on large cluster environments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718217#comment-16718217 ] ASF GitHub Bot commented on SPARK-25642: vanzin commented on issue #22498: [SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service URL: https://github.com/apache/spark/pull/22498#issuecomment-446406987 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Priority: Minor > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]]. > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718223#comment-16718223 ] ASF GitHub Bot commented on SPARK-25642: AmplabJenkins removed a comment on issue #22498: [SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service URL: https://github.com/apache/spark/pull/22498#issuecomment-446407720 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5992/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Priority: Minor > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]]. > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718220#comment-16718220 ] ASF GitHub Bot commented on SPARK-25642: AmplabJenkins commented on issue #22498: [SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service URL: https://github.com/apache/spark/pull/22498#issuecomment-446407720 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5992/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Priority: Minor > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]]. > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718221#comment-16718221 ] ASF GitHub Bot commented on SPARK-25642: SparkQA commented on issue #22498: [SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service URL: https://github.com/apache/spark/pull/22498#issuecomment-446407732 **[Test build #6 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/6/testReport)** for PR 22498 at commit [`70472a2`](https://github.com/apache/spark/commit/70472a255e5da3ea4522959e26f5c403641e1ce6). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Priority: Minor > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]]. > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718219#comment-16718219 ] ASF GitHub Bot commented on SPARK-25642: AmplabJenkins commented on issue #22498: [SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service URL: https://github.com/apache/spark/pull/22498#issuecomment-446407716 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Priority: Minor > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485]]. > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638722#comment-16638722 ] Apache Spark commented on SPARK-25642: -- User 'pgandhi999' has created a pull request for this issue: https://github.com/apache/spark/pull/22498 > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Priority: Minor > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485].] > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638723#comment-16638723 ] Apache Spark commented on SPARK-25642: -- User 'pgandhi999' has created a pull request for this issue: https://github.com/apache/spark/pull/22498 > Add new Metrics in External Shuffle Service to help determine Network > performance and Connection Handling capabilities of the Shuffle Service > - > > Key: SPARK-25642 > URL: https://issues.apache.org/jira/browse/SPARK-25642 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 2.4.0 >Reporter: Parth Gandhi >Priority: Minor > > Recently, the ability to expose the metrics for YARN Shuffle Service was > added as part of [SPARK-18364|[https://github.com/apache/spark/pull/22485].] > We need to add some metrics to be able to determine the number of active > connections as well as open connections to the external shuffle service to > benchmark network and connection issues on large cluster environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
