reuvenlax commented on PR #27495: URL: https://github.com/apache/beam/pull/27495#issuecomment-1636107237
Yes - on FAILED_PRECONDITION error the sink will immediately create a new stream name before retrying. There should be a log that says "Will retry with a new stream" We should try to dive deeper into this pipeline that got stuck, as it sounds like we don't understand what happened and there's no reason to believe that retrying the work item will be any different than simply retrying the call. On Fri, Jul 14, 2023 at 1:46 PM Ahmed Abualsaud ***@***.***> wrote: > I see, should the sink retry with a new stream name? Or can the same > stream name exist across workers? Looking at logs for an individual stream > name, I see the following pattern of logs repeated a bunch of times for 3 > different workers: > > Start connecting to stream.. > Finish connecting to stream.. > Received done callback > FAILED_PRECONDITION error > Start connecting to stream.. > etc. > > Each cycle takes ~30sec and this goes on for ~4.5 hours. After this it > reaches a steady state where the stream is only accessed by one worker. > Then we see a more healthy pattern for only that one worker (and across > multiple work items): > > Start connecting to stream.. > Finish connecting to stream.. > Start connecting to stream.. > Finish connecting to stream.. > etc. > > Each start-finish cycle is comparatively very short (~0.4sec tops). > There's one moment where the other two workers attempt to connect to the > same stream again. As soon as this happens, we they each trigger a OffsetAlreadyExists: > ALREADY_EXISTS and FAILED_PRECONDITION error. But then for the rest of > the pipeline the stream stays with one worker and is generally healthy > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/beam/pull/27495#issuecomment-1635813204>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFAYJVPGFAMIXVBWVU7ZXSDXQE5QRANCNFSM6AAAAAA2JSTZ3A> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
