This is an automated email from the ASF dual-hosted git repository.
mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0ebfa691fc8e [SPARK-45730][CORE] Make ReloadingX509TrustManagerSuite
less flaky
0ebfa691fc8e is described below
commit 0ebfa691fc8e88d2910bf41900718252edea1ee3
Author: Hasnain Lakhani <hasnain.lakh...@databricks.com>
AuthorDate: Fri Nov 3 00:09:21 2023 -0500
[SPARK-45730][CORE] Make ReloadingX509TrustManagerSuite less flaky
### What changes were proposed in this pull request?
Improve a few timing related constraints:
* Wait 10s instead of 5 for a reload to happen when under high load. This
should not delay the test in the average case as it checks every 100ms for an
event to happen.
* In certain cases we run *too fast* so the new file we create has the same
timestamp as the old file, and thus we never reload. Add a sleep there so the
modification times are different. This was accidentally reverted in
https://github.com/apache/spark/pull/43249/commits/b7dac1f96ec45a300963e4da1dc4fc1173470da7#diff-de96db6b61e9f48fb9bd8b781f4367e60a48b3886dfe03d4cf16b47ef6c26d0a
### Why are the changes needed?
These changes are needed to make the test more reliable and less flaky
under load.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Ran this test in parallel on a machine under high load. Previously under
those conditions I would repeatedly get high rates of failure (80%+) and now it
does not fail.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43596 from hasnain-db/fix-flaky-test.
Authored-by: Hasnain Lakhani <hasnain.lakh...@databricks.com>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
.../spark/network/ssl/ReloadingX509TrustManagerSuite.java | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git
a/common/network-common/src/test/java/org/apache/spark/network/ssl/ReloadingX509TrustManagerSuite.java
b/common/network-common/src/test/java/org/apache/spark/network/ssl/ReloadingX509TrustManagerSuite.java
index 0526fcb11bea..5bb47ff38867 100644
---
a/common/network-common/src/test/java/org/apache/spark/network/ssl/ReloadingX509TrustManagerSuite.java
+++
b/common/network-common/src/test/java/org/apache/spark/network/ssl/ReloadingX509TrustManagerSuite.java
@@ -161,14 +161,17 @@ public class ReloadingX509TrustManagerSuite {
// At this point we haven't reloaded, just the initial load
assertEquals(0, tm.reloadCount);
+ // Wait so that the file modification time is different
+ Thread.sleep((tm.getReloadInterval() + 1000));
+
// Add another cert
Map<String, X509Certificate> certs = new HashMap<String,
X509Certificate>();
certs.put("cert1", cert1);
certs.put("cert2", cert2);
createTrustStore(trustStore, "password", certs);
- // Wait up to 5s until we reload
- waitForReloadCount(tm, 1, 50);
+ // Wait up to 10s until we reload
+ waitForReloadCount(tm, 1, 100);
assertEquals(2, tm.getAcceptedIssuers().length);
} finally {
@@ -286,8 +289,8 @@ public class ReloadingX509TrustManagerSuite {
trustStoreSymlink.delete();
Files.createSymbolicLink(trustStoreSymlink.toPath(),
trustStore2.toPath());
- // Wait up to 5s until we reload
- waitForReloadCount(tm, 1, 50);
+ // Wait up to 10s until we reload
+ waitForReloadCount(tm, 1, 100);
assertEquals(2, tm.getAcceptedIssuers().length);
@@ -295,8 +298,8 @@ public class ReloadingX509TrustManagerSuite {
certs.put("cert3", cert3);
createTrustStore(trustStore2, "password", certs);
- // Wait up to 5s until we reload
- waitForReloadCount(tm, 2, 50);
+ // Wait up to 10s until we reload
+ waitForReloadCount(tm, 2, 100);
assertEquals(3, tm.getAcceptedIssuers().length);
} finally {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
