damccorm opened a new pull request, #41063: URL: https://github.com/apache/beam/pull/41063
## Summary Fixes flaky failures in `PerformanceTests SparkReceiver IO` (`:sdks:java:io:sparkreceiver:3:integrationTest` `SparkReceiverIOIT.testSparkReceiverIOReadsInStreamingWithOffset`, e.g. https://github.com/apache/beam/actions/runs/42310450785, https://github.com/apache/beam/actions/runs/40188354039, https://github.com/apache/beam/actions/runs/39692762153): 1. **Wait for RabbitMQ StatefulSet rollout before starting tests**: Previously, `Install rabbitmq` in `beam_PerformanceTests_SparkReceiver_IO.yml` only waited for `svc/rabbitmq` to receive a LoadBalancer IP (`kubectl wait svc/rabbitmq --for=jsonpath='{.status.loadBalancer.ingress[0].ip}'`) without waiting for the `statefulset/rabbitmq` pod (`rabbitmq-0`) to become Ready, causing `SocketTimeoutException: Connect timed out` when the LoadBalancer IP was assigned before the pod finished starting. 2. **Relax `rabbitmqctl status` probe timeouts and increase PVC size**: During `SparkReceiverIOIT.writeToRabbitMq`, the test publishes 5,000,000 persistent messages in a tight loop. Running `rabbitmqctl status` every 10s with a 10s liveness/readiness timeout on a 3Gi standard PD caused `livenessProbe` timeouts under heavy write load, causing kubelet to kill and restart `rabbitmq-0` mid-publish (`SocketException: Broken pipe` / `Connection reset`). Increasing the probe timeout/period and PVC size prevents premature container restarts during bulk publishing. ## Verification - Triggered `PerformanceTests SparkReceiver IO` on this branch: https://github.com/apache/beam/actions/runs/42316569422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
