This is an automated email from the ASF dual-hosted git repository.
mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 79b4dea [SPARK-32663][CORE] Avoid individual closing of pooled
TransportClients (which must be closed through the pool)
79b4dea is described below
commit 79b4dea1b08adc9d4b545a1af29ac50fa603936a
Author: “attilapiros” <[email protected]>
AuthorDate: Fri Aug 21 01:02:33 2020 -0500
[SPARK-32663][CORE] Avoid individual closing of pooled TransportClients
(which must be closed through the pool)
### What changes were proposed in this pull request?
Removing the individual `close` method calls on the pooled
`TransportClient` instances.
The pooled clients should be only closed via
`TransportClientFactory#close()`.
### Why are the changes needed?
Reusing a closed `TransportClient` leads to the exception
`java.nio.channels.ClosedChannelException`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This is a trivial case which is not tested by specific test.
Closes #29492 from attilapiros/SPARK-32663.
Authored-by: “attilapiros” <[email protected]>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
.../org/apache/spark/network/shuffle/ExternalBlockStoreClient.java | 6 ------
1 file changed, 6 deletions(-)
diff --git
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java
index 51dc333..9fdb632 100644
---
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java
+++
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java
@@ -175,8 +175,6 @@ public class ExternalBlockStoreClient extends
BlockStoreClient {
logger.warn("Error trying to remove RDD blocks " +
Arrays.toString(blockIds) +
" via external shuffle service from executor: " + execId, t);
numRemovedBlocksFuture.complete(0);
- } finally {
- client.close();
}
}
@@ -185,7 +183,6 @@ public class ExternalBlockStoreClient extends
BlockStoreClient {
logger.warn("Error trying to remove RDD blocks " +
Arrays.toString(blockIds) +
" via external shuffle service from executor: " + execId, e);
numRemovedBlocksFuture.complete(0);
- client.close();
}
});
return numRemovedBlocksFuture;
@@ -212,8 +209,6 @@ public class ExternalBlockStoreClient extends
BlockStoreClient {
Arrays.toString(getLocalDirsMessage.execIds) + " via external
shuffle service",
t.getCause());
hostLocalDirsCompletable.completeExceptionally(t);
- } finally {
- client.close();
}
}
@@ -223,7 +218,6 @@ public class ExternalBlockStoreClient extends
BlockStoreClient {
Arrays.toString(getLocalDirsMessage.execIds) + " via external
shuffle service",
t.getCause());
hostLocalDirsCompletable.completeExceptionally(t);
- client.close();
}
});
} catch (IOException | InterruptedException e) {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
