This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new b8d527166e0 [Flink 34569][e2e] fail fast if AWS cli container fails to
start (#24491)
b8d527166e0 is described below
commit b8d527166e095653ae3ff5c0431bf27297efe229
Author: Robert Young <[email protected]>
AuthorDate: Thu Jun 6 00:10:20 2024 +1200
[Flink 34569][e2e] fail fast if AWS cli container fails to start (#24491)
* [FLINK-34569][e2e] Fail fast if aws cli container fails to run
Why:
An end-to-end test run failed and in the test logs you could see that the
AWS cli container failed to start. Because of the way it's organised the
failure in the subshell did not cause a failure and AWSCLI_CONTAINER_ID was
empty. This lead to a loop trying to docker exec a command in a container
named "" and the test taking 15 minutes to time out. This change speeds up
the failure.
Note that we use 'return' to prevent an immediate failure of the script so
that we have the potential to implement a simple retry.
Signed-off-by: Robert Young <[email protected]>
* [FLINK-34569][e2e] Add naive retry when creating aws cli container
Why:
An end-to-end test run failed with what looked like a transient network
exception when pulling the aws cli image. This retries once.
Signed-off-by: Robert Young <[email protected]>
* [FLINK-34569][e2e] Remove jq containers after user
Why:
A large pile of exited jq containers were left in docker after
an operation was retried repeatedly.
Signed-off-by: Robert Young <[email protected]>
* [FLINK-34569][e2e] Clean up after failed awscli container run
Why:
If for some reason the command can return a non-zero exit code and also
create a container, this will remove it so we don't have an orphan sitting
stranded.
Signed-off-by: Robert Young <[email protected]>
---------
Signed-off-by: Robert Young <[email protected]>
---
.../test-scripts/common_s3_operations.sh | 21 ++++++++++++++++++---
1 file changed, 18 insertions(+), 3 deletions(-)
diff --git a/flink-end-to-end-tests/test-scripts/common_s3_operations.sh
b/flink-end-to-end-tests/test-scripts/common_s3_operations.sh
index f784c7d58b4..965099b6527 100644
--- a/flink-end-to-end-tests/test-scripts/common_s3_operations.sh
+++ b/flink-end-to-end-tests/test-scripts/common_s3_operations.sh
@@ -29,12 +29,23 @@
# AWSCLI_CONTAINER_ID
###################################
function aws_cli_start() {
- export AWSCLI_CONTAINER_ID=$(docker run -d \
+ local CONTAINER_ID
+ CONTAINER_ID=$(docker run -d \
--network host \
--mount type=bind,source="$TEST_INFRA_DIR",target=/hostdir \
-e AWS_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
--entrypoint python \
-it banst/awscli)
+ if [ $? -ne 0 ]; then
+ echo "running aws cli container failed"
+ if [ -n "$CONTAINER_ID" ]
+ then
+ docker kill "$CONTAINER_ID"
+ docker rm "$CONTAINER_ID"
+ fi
+ return 1
+ fi
+ export AWSCLI_CONTAINER_ID="$CONTAINER_ID"
while [[ "$(docker inspect -f {{.State.Running}} "$AWSCLI_CONTAINER_ID")"
-ne "true" ]]; do
sleep 0.1
@@ -58,7 +69,11 @@ function aws_cli_stop() {
if [[ $AWSCLI_CONTAINER_ID ]]; then
aws_cli_stop
fi
-aws_cli_start
+aws_cli_start || aws_cli_start
+if [ $? -ne 0 ]; then
+ echo "running the aws cli container failed"
+ exit 1
+fi
###################################
# Runs an aws command on the previously started container.
@@ -135,7 +150,7 @@ function s3_get_number_of_lines_by_prefix() {
# find all files that have the given prefix
parts=$(aws_cli s3api list-objects --bucket "$IT_CASE_S3_BUCKET" --prefix
"$1" |
- docker run -i stedolan/jq -r '[.Contents[].Key] | join(" ")')
+ docker run -i --rm stedolan/jq -r '[.Contents[].Key] | join(" ")')
# in parallel (N tasks), query the number of lines, store result in a file
named lines
N=10
