Could you please check the log of how long for each pipeline stage takes? Also, did you set expiry for workflows? Are they piled up for long time? How long for each workflow completes?
best, Junkai On Wed, Sep 26, 2018 at 8:52 AM DImuthu Upeksha <dimuthu.upeks...@gmail.com> wrote: > Hi Junkai, > > Average load is like 10 - 20 workflows per minutes. In some cases it's less > than that However based on the observations, I feel like it does not depend > on the load and it is sporadic. Is there a particular log lines that I can > filter in controller and participant to capture the timeline of workflow so > that I can figure out which which component is malfunctioning? We use helix > v 0.8.1. > > Thanks > Dimuthu > > On Tue, Sep 25, 2018 at 5:19 PM Xue Junkai <junkai....@gmail.com> wrote: > > > Hi Dimuthu, > > > > At which rate, you are keep submitting workflows? Usually, Workflow > > scheduling is very fast. And which version of Helix you are using? > > > > Best, > > > > Junkai > > > > On Tue, Sep 25, 2018 at 8:58 AM DImuthu Upeksha < > > dimuthu.upeks...@gmail.com> > > wrote: > > > > > Hi Folks, > > > > > > We have noticed some delays between workflow submission and actual > > picking > > > up by participants and seems like that delay is somewhat constant > around > > 2- > > > 3 minutes. We used to continuously submit workflows and after 2 -3 > > minutes, > > > a bulk of workflows are picked by participant and execute them. Then it > > > remain silent for next 2 -3 minutes event we submit more workflows. > It's > > > like participant picking up workflows in discrete time intervals. I'm > not > > > sure whether this is an issue of controller or the participant. Do you > > have > > > any experience with this sort of behavior? > > > > > > Thanks > > > Dimuthu > > > > > > > > > -- > > Junkai Xue > > > -- Junkai Xue
