This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 19487cb [SPARK-28261][CORE] Fix client reuse test
19487cb is described below
commit 19487cb55cc82515b020060851f470312111b5a2
Author: Gabor Somogyi <[email protected]>
AuthorDate: Mon Jul 8 11:10:03 2019 -0700
[SPARK-28261][CORE] Fix client reuse test
There is the following code in
[TransportClientFactory#createClient](https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java#L150)
```
int clientIndex = rand.nextInt(numConnectionsPerPeer);
TransportClient cachedClient = clientPool.clients[clientIndex];
```
which choose a client from its pool randomly. If we are unlucky we might
not get the max number of connections out, but less than that.
To prove that I've tried out the following test:
```java
Test
public void testRandom() {
Random rand = new Random();
Set<Integer> clients = Collections.synchronizedSet(new HashSet<>());
long iterCounter = 0;
while (true) {
iterCounter++;
int maxConnections = 4;
clients.clear();
for (int i = 0; i < maxConnections * 10; i++) {
int clientIndex = rand.nextInt(maxConnections);
clients.add(clientIndex);
}
if (clients.size() != maxConnections) {
System.err.println("Unexpected clients size (iterCounter=" +
iterCounter + "): " + clients.size() + ", maxConnections: " + maxConnections);
}
if (iterCounter % 100000 == 0) {
System.out.println("IterCounter: " + iterCounter);
}
}
}
```
Result:
```
Unexpected clients size (iterCounter=22388): 3, maxConnections: 4
Unexpected clients size (iterCounter=36244): 3, maxConnections: 4
Unexpected clients size (iterCounter=85798): 3, maxConnections: 4
IterCounter: 100000
Unexpected clients size (iterCounter=97108): 3, maxConnections: 4
Unexpected clients size (iterCounter=119121): 3, maxConnections: 4
Unexpected clients size (iterCounter=129948): 3, maxConnections: 4
Unexpected clients size (iterCounter=173736): 3, maxConnections: 4
Unexpected clients size (iterCounter=178138): 3, maxConnections: 4
Unexpected clients size (iterCounter=195108): 3, maxConnections: 4
IterCounter: 200000
Unexpected clients size (iterCounter=209006): 3, maxConnections: 4
Unexpected clients size (iterCounter=217105): 3, maxConnections: 4
Unexpected clients size (iterCounter=222456): 3, maxConnections: 4
Unexpected clients size (iterCounter=226899): 3, maxConnections: 4
Unexpected clients size (iterCounter=229101): 3, maxConnections: 4
Unexpected clients size (iterCounter=253549): 3, maxConnections: 4
Unexpected clients size (iterCounter=277550): 3, maxConnections: 4
Unexpected clients size (iterCounter=289637): 3, maxConnections: 4
...
```
In this PR I've adapted the test code not to have this flakyness.
Additional (not committed test) + existing unit tests in a loop.
Closes #25075 from gaborgsomogyi/SPARK-28261.
Authored-by: Gabor Somogyi <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit e11a55827e7475aab77e8a4ea0baed7c14059908)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../test/java/org/apache/spark/network/TransportClientFactorySuite.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java
b/common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java
index e95d25f..86482f1 100644
---
a/common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java
+++
b/common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java
@@ -116,7 +116,7 @@ public class TransportClientFactorySuite {
}
Assert.assertEquals(0, failed.get());
- Assert.assertEquals(clients.size(), maxConnections);
+ Assert.assertTrue(clients.size() <= maxConnections);
for (TransportClient client : clients) {
client.close();
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
