This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 97976d4172d3 [SPARK-58028][INFRA][TEST] Retry Docker image pull in
DockerJDBCIntegrationSuite to avoid transient 502 aborts
97976d4172d3 is described below
commit 97976d4172d39a60536b129b7837c52ddab2615a
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Jul 8 13:21:04 2026 +0900
[SPARK-58028][INFRA][TEST] Retry Docker image pull in
DockerJDBCIntegrationSuite to avoid transient 502 aborts
### What changes were proposed in this pull request?
Docker-backed JDBC integration suites (e.g. `PostgresKrbIntegrationSuite`)
pull their Docker image in `beforeAll` with a single attempt. A transient
registry/proxy error (observed on GitHub Actions: `HTTP 502 Bad Gateway`) or a
one-off pull timeout aborts the entire suite even though the rest of the run
had zero test failures.
This wraps the image pull in a bounded retry loop
(`spark.test.docker.imagePullAttempts`, default `3`) with linear backoff, so a
transient failure no longer aborts the suite. The final failure is rethrown
once all attempts are exhausted.
### Why are the changes needed?
A single transient registry hiccup should not abort a whole Docker
integration suite that otherwise passes. This was seen aborting the scheduled
`build_java17` workflow on `branch-4.1` (`PostgresKrbIntegrationSuite` aborted
in `beforeAll` on a 502). Targeting `master` so it can be backported to the
maintenance branches where the flake was observed.
### Does this PR introduce _any_ user-facing change?
No. Test-infrastructure change only.
### How was this patch tested?
Fork CI (`build_java17`, `Run Docker integration tests`): ✅ **PASS** —
https://github.com/HyukjinKwon/spark/actions/runs/28906195126
`Run completed ... Tests: succeeded 606, failed 0, canceled 0, ignored 151`
— "All tests passed", with `PostgresKrbIntegrationSuite` (the suite that had
aborted on the 502) running clean. (Verification ran on `branch-4.1`; the
changed pull block is identical on `master`.)
JIRA: https://issues.apache.org/jira/browse/SPARK-58028
Closes #57111 from HyukjinKwon/DO-NOT-MERGE/deflake-docker-master.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 1f8715707d25cafd43d60065786ae6a4a9780e80)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../sql/jdbc/DockerJDBCIntegrationSuite.scala | 62 +++++++++++++++-------
1 file changed, 44 insertions(+), 18 deletions(-)
diff --git
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
index 27b9debeb1e0..3a199b6639cb 100644
---
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
+++
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
@@ -119,6 +119,11 @@ abstract class DockerJDBCIntegrationSuite
sys.props.getOrElse("spark.test.docker.removePulledImage",
"true").toBoolean
protected val imagePullTimeout: Long =
timeStringAsSeconds(sys.props.getOrElse("spark.test.docker.imagePullTimeout",
"5min"))
+ // Number of attempts to pull the Docker image before giving up. Image pulls
occasionally fail
+ // with transient registry/proxy errors (e.g. HTTP 502 Bad Gateway) that
abort the whole suite
+ // in beforeAll even though a retry would succeed, so retry a few times with
backoff.
+ protected val imagePullAttempts: Int =
+ sys.props.getOrElse("spark.test.docker.imagePullAttempts", "3").toInt
protected val startContainerTimeout: Long =
timeStringAsSeconds(sys.props.getOrElse("spark.test.docker.startContainerTimeout",
"5min"))
// `lazy` so that `db` (abstract, defined by concrete suites) is initialized
before it is read.
@@ -159,27 +164,48 @@ abstract class DockerJDBCIntegrationSuite
} catch {
case e: NotFoundException =>
log.warn(s"Docker image ${db.imageName} not found; pulling image
from registry")
- val callback = new PullImageResultCallback {
- override def onNext(item: PullResponseItem): Unit = {
- super.onNext(item)
- val status = item.getStatus
- if (status != null && status != "Downloading" && status !=
"Extracting") {
- logInfo(s"$status ${item.getId}")
+ // Retry the pull a few times: it can fail with transient
registry/proxy errors
+ // (e.g. HTTP 502 Bad Gateway) or a timeout that would otherwise
abort the suite.
+ var attempt = 0
+ var lastError: Throwable = null
+ while (!pulled && attempt < imagePullAttempts) {
+ attempt += 1
+ val callback = new PullImageResultCallback {
+ override def onNext(item: PullResponseItem): Unit = {
+ super.onNext(item)
+ val status = item.getStatus
+ if (status != null && status != "Downloading" && status !=
"Extracting") {
+ logInfo(s"$status ${item.getId}")
+ }
}
}
+ try {
+ val (success, time) = Utils.timeTakenMs(
+ docker.pullImageCmd(db.imageName)
+ .exec(callback)
+ .awaitCompletion(imagePullTimeout, TimeUnit.SECONDS))
+
+ if (success) {
+ pulled = true
+ logInfo(s"Successfully pulled image ${db.imageName} in $time
ms")
+ } else {
+ lastError = new TimeoutException(
+ s"Timeout('$imagePullTimeout secs') waiting for image
${db.imageName} " +
+ "to be pulled")
+ }
+ } catch {
+ case NonFatal(e) =>
+ lastError = e
+ }
+ if (!pulled && attempt < imagePullAttempts) {
+ log.warn(s"Failed to pull image ${db.imageName} " +
+ s"(attempt $attempt/$imagePullAttempts); retrying", lastError)
+ // Linear backoff between attempts to let a transient registry
issue recover.
+ Thread.sleep(TimeUnit.SECONDS.toMillis(5L * attempt))
+ }
}
-
- val (success, time) = Utils.timeTakenMs(
- docker.pullImageCmd(db.imageName)
- .exec(callback)
- .awaitCompletion(imagePullTimeout, TimeUnit.SECONDS))
-
- if (success) {
- pulled = success
- logInfo(s"Successfully pulled image ${db.imageName} in $time ms")
- } else {
- throw new TimeoutException(
- s"Timeout('$imagePullTimeout secs') waiting for image
${db.imageName} to be pulled")
+ if (!pulled) {
+ throw lastError
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
