lostluck commented on issue #25223: URL: https://github.com/apache/beam/issues/25223#issuecomment-1409526339
First, title is bad: Doesn't specify which Direct Runner. Go pipelines can't run on the Java Direct runner. Vacuously, nothing but python can run on the Python Direct runner, since that's kinda what distinguishes direct runners from portable runners. Cross language doesn't execute on the Go direct runner, so I assume that's not the runner in question either. Go pipelines *can* run on the Python Portable runner however, so I'll assume that's the case. ------- In this case, it looks like however the Java Expansion Service is being spun up, it's not yetpermitting external connections. "transport: Error while dialing dial tcp 127.0.0.1:41189: connect: connection refused" That likely means it's not yet ready. Looking back in history, it looks like this does pass sometimes, (3666 is an example), so why? ----- Given that the tests fail in ~5-6 seconds when it does fail, it feels like the connection/request stage isn't retrying/backing off since the container may not yet be ready. So that means this line (or the one below): https://github.com/apache/beam/blob/bdfd27e62885bd736c5600d0c6496bf96fe66a77/sdks/go/pkg/beam/core/runtime/xlangx/expand.go#L168 is what's failing, and we don't do a retry/backoff. Reading error, it's probably the connection isn't ready/waiting yet, which is expected/default behavior. The connection happens asynchronously before the first request, which is why we don't see it fail here: https://github.com/apache/beam/blob/bdfd27e62885bd736c5600d0c6496bf96fe66a77/sdks/go/pkg/beam/core/runtime/xlangx/expand.go#L157 So the fix is a simple re-try + backoff to some timeout in that area, which aligns with the Bruno Conjecture that we don't have resilient enough tests/infra/frameworks implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
