github-actions[bot] commented on code in PR #41119:
URL: https://github.com/apache/beam/pull/41119#discussion_r4515089161
##########
.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml:
##########
@@ -111,6 +112,22 @@ spec:
- image: gcr.io/apache-beam-testing/kubernetes-hadoop:3.3.4
imagePullPolicy: Always
name: namenode
+ command: ["/bin/bash", "-c"]
+ args:
+ - |
+
DOMAIN_SUFFIX="hadoop-datanodes.${NODE_NAMESPACE:-default}.svc.cluster.local"
+ for ((i=0; i<${DATANODE_COUNT:-3}; i++)); do
+ HOST="datanode-${i}.${DOMAIN_SUFFIX}"
+ echo "Waiting for SSH on ${HOST}..."
+ for attempt in $(seq 1 60); do
+ if ssh -o StrictHostKeyChecking=no -o ConnectTimeout=2 "${HOST}"
true 2>/dev/null; then
+ echo "${HOST} is ready."
+ break
+ fi
+ sleep 2
+ done
Review Comment:
**🟡 Medium**: If all 60 attempts fail (for example, if GKE node autoscaling
or `imagePullPolicy: Always` causes a datanode pod to take longer than ~2
minutes to start), this loop finishes without error and executes
`/opt/bootstrap.sh -d`, which will fail to start the datanode daemon over SSH
and then remain running indefinitely in daemon mode with missing datanodes.
Exiting non-zero on timeout allows kubelet (`restartPolicy: Always`) to
restart the `namenode` container and retry until all datanodes are reachable.
```suggestion
if [ "${attempt}" -eq 60 ]; then echo "Timed out waiting for SSH
on ${HOST}" >&2; exit 1; fi
done
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
