ahmedabu98 commented on PR #27495: URL: https://github.com/apache/beam/pull/27495#issuecomment-1635813204
I see, should the sink retry with a new stream name? Or can the same stream name exist across workers? Looking at logs for an individual stream name, I see the following pattern of logs repeated a bunch of times for 3 different workers: Start connecting to stream.. Finish connecting to stream.. Received done callback FAILED_PRECONDITION error Start connecting to stream.. etc. Each cycle takes ~30sec and this goes on for ~4.5 hours. After this it reaches a steady state where the stream is only accessed by one worker. Then we see a more healthy pattern for only that one worker (and across multiple work items): Start connecting to stream.. Finish connecting to stream.. Start connecting to stream.. Finish connecting to stream.. etc. Each start-finish cycle is comparatively very short (~0.4sec tops). There's one moment where the other two workers attempt to connect to the same stream again. As soon as this happens, we they each trigger a `OffsetAlreadyExists: ALREADY_EXISTS` and `FAILED_PRECONDITION` error. But then for the rest of the pipeline the stream stays with one worker and is generally healthy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
