Hi Dimuthu, What's the version of Helix you are using?
Best, Junkai On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha <dimuthu.upeks...@gmail.com> wrote: > Hi Helix Dev, > > We are again seeing this delay in task execution. Please have a look at the > screencast [1] of logs printed in participant (top shell) and controller > (bottom shell). When I record this, there were about 90 - 100 workflows > pending to be executed. As you can see some tasks were suddenly executed > and then participant freezed for about 30 seconds before executing next set > of tasks. I can see some WARN logs on controller log. I feel like this 30 > second delay is some sort of a pattern. What do you think as the reason for > this? I can provide you more information by turning on verbose logs on > controller if you want. > > [1] https://youtu.be/3EUdSxnIxVw > > Thanks > Dimuthu > > On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha <dimuthu.upeks...@gmail.com > > > wrote: > > > Hi Junkai, > > > > I'm CCing Airavata dev list as this is directly related to the project. > > > > I just went through the zookeeper path like /<Cluster Name>/EXTERNALVIEW, > > /<Cluster Name>/CONFIGS/RESOURCE as I have noticed that helix controller > is > > periodically monitoring for the children of those paths even though all > the > > Workflows have moved into a saturated state like COMPLETED and STOPPED. > In > > our case, we have a lot of completed workflows piled up in those paths. I > > believe that helix is clearing up those resources after some TTL. What I > > did was writing an external spectator [1] that continuously monitors for > > saturated workflows and clearing up resources before controller does that > > after a TTL. After that, we didn't see such delays in workflow execution > > and everything seems to be running smoothly. However we are continuously > > monitoring our deployments for any form of adverse effect introduced by > > that improvement. > > > > Please let us know if we are doing something wrong in this improvement or > > is there any better way to achieve this directly through helix task > > framework. > > > > [1] > > > https://github.com/apache/airavata/blob/staging/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/controller/WorkflowCleanupAgent.java > > > > Thanks > > Dimuthu > > > > On Tue, Oct 2, 2018 at 1:12 PM Xue Junkai <junkai....@gmail.com> wrote: > > > >> Could you please check the log of how long for each pipeline stage > takes? > >> > >> Also, did you set expiry for workflows? Are they piled up for long time? > >> How long for each workflow completes? > >> > >> best, > >> > >> Junkai > >> > >> On Wed, Sep 26, 2018 at 8:52 AM DImuthu Upeksha < > >> dimuthu.upeks...@gmail.com> > >> wrote: > >> > >> > Hi Junkai, > >> > > >> > Average load is like 10 - 20 workflows per minutes. In some cases it's > >> less > >> > than that However based on the observations, I feel like it does not > >> depend > >> > on the load and it is sporadic. Is there a particular log lines that I > >> can > >> > filter in controller and participant to capture the timeline of > >> workflow so > >> > that I can figure out which which component is malfunctioning? We use > >> helix > >> > v 0.8.1. > >> > > >> > Thanks > >> > Dimuthu > >> > > >> > On Tue, Sep 25, 2018 at 5:19 PM Xue Junkai <junkai....@gmail.com> > >> wrote: > >> > > >> > > Hi Dimuthu, > >> > > > >> > > At which rate, you are keep submitting workflows? Usually, Workflow > >> > > scheduling is very fast. And which version of Helix you are using? > >> > > > >> > > Best, > >> > > > >> > > Junkai > >> > > > >> > > On Tue, Sep 25, 2018 at 8:58 AM DImuthu Upeksha < > >> > > dimuthu.upeks...@gmail.com> > >> > > wrote: > >> > > > >> > > > Hi Folks, > >> > > > > >> > > > We have noticed some delays between workflow submission and actual > >> > > picking > >> > > > up by participants and seems like that delay is somewhat constant > >> > around > >> > > 2- > >> > > > 3 minutes. We used to continuously submit workflows and after 2 -3 > >> > > minutes, > >> > > > a bulk of workflows are picked by participant and execute them. > >> Then it > >> > > > remain silent for next 2 -3 minutes event we submit more > workflows. > >> > It's > >> > > > like participant picking up workflows in discrete time intervals. > >> I'm > >> > not > >> > > > sure whether this is an issue of controller or the participant. Do > >> you > >> > > have > >> > > > any experience with this sort of behavior? > >> > > > > >> > > > Thanks > >> > > > Dimuthu > >> > > > > >> > > > >> > > > >> > > -- > >> > > Junkai Xue > >> > > > >> > > >> > >> > >> -- > >> Junkai Xue > >> > > > -- Junkai Xue
