xumanbu commented on code in PR #1838:
URL:
https://github.com/apache/incubator-uniffle/pull/1838#discussion_r1689291659
##########
client-spark/common/src/main/java/org/apache/uniffle/shuffle/manager/RssShuffleManagerBase.java:
##########
@@ -606,25 +609,28 @@ protected synchronized MutableShuffleHandleInfo
getRemoteShuffleHandleInfoWithBl
RssPartitionToShuffleServerRequest rssPartitionToShuffleServerRequest =
new RssPartitionToShuffleServerRequest(shuffleId);
RssReassignOnBlockSendFailureResponse rpcPartitionToShufflerServer =
- getOrCreateShuffleManagerClient()
+ getOrCreateShuffleManagerClientWrapper()
+ .get()
.getPartitionToShufflerServerWithBlockRetry(rssPartitionToShuffleServerRequest);
MutableShuffleHandleInfo shuffleHandleInfo =
MutableShuffleHandleInfo.fromProto(rpcPartitionToShufflerServer.getHandle());
return shuffleHandleInfo;
}
- // todo: automatic close client when the client is idle to avoid too much
connections for spark
- // driver.
- protected ShuffleManagerClient getOrCreateShuffleManagerClient() {
- if (shuffleManagerClient == null) {
+ protected synchronized ExpireCloseableSupplier<ShuffleManagerClient>
+ getOrCreateShuffleManagerClientWrapper() {
+ if (managerClientSupplier == null) {
RssConf rssConf = RssSparkConfig.toRssConf(sparkConf);
String driver = rssConf.getString("driver.host", "");
int port = rssConf.get(RssClientConf.SHUFFLE_MANAGER_GRPC_PORT);
- this.shuffleManagerClient =
- ShuffleManagerClientFactory.getInstance()
- .createShuffleManagerClient(ClientType.GRPC, driver, port);
+ long rpcTimeout =
rssConf.getLong(RssBaseConf.RSS_CLIENT_TYPE_GRPC_TIMEOUT_MS);
Review Comment:
It's not delay interval config. delay interval not added config at now.
It's another one config for GRPC called timeout. to fix the issue
https://github.com/apache/incubator-uniffle/issues/1774 of @rickyma proposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
