Hi Junkai, Thanks a lot. I'll try with expiry time then. Is this[1] the place where Helix has implemented this logic? If that so, default expiry time should be 24 hours. Am I right?
[1] https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java#L711 Thanks Dimuthu On Mon, Nov 12, 2018 at 10:17 PM Xue Junkai <junkai....@gmail.com> wrote: > 1 and 2 are correct. 3 is wrong. The expiry time start counting only when > the workflow is completed. If it is not scheduled ( dont have enought > resource) or still running, Helix never deletes it. > > > > On Sun, Nov 11, 2018 at 8:01 PM DImuthu Upeksha < > dimuthu.upeks...@gmail.com> wrote: > >> Hi Junkai, >> >> Thanks for the clarification. That helped a lot. >> >> I our case, each of the task of the workflow are depending on the previous >> task. So there is no parallel execution. And we are not using Job Queues. >> >> Regarding the expiry time, what are the rules that you are imposing on >> that? For example let's say I setup an expiry time to 2 hours, I assume >> following situations are covered in Helix, >> >> 1. Even though the workflow is completed before 2 hours, resources related >> to that workflow will not be cleared until 2 hours are elapsed and exactly >> after 2 hours, all the resources will be cleared by the framework. >> 2. If the workflow failed, resources will not be cleared even after 2 >> hours >> 3. If the workflow wasn't scheduled within 2 hours in a participant, it >> will be deleted >> >> Is my understanding correct? >> >> Thanks >> Dimuthu >> >> >> On Sat, Nov 10, 2018 at 4:26 PM Xue Junkai <junkai....@gmail.com> wrote: >> >> > Hi Dimuthu, >> > >> > Couple things here: >> > 1. Only JobQueue in Helix is single branch DAG and 1 job running at a >> time >> > with defining parallel job number to be 1. Otherwise, you may see many >> jobs >> > running at same time as you set parallel job number to be a different >> > number. For generic workflow, all jobs without dependencies could be >> > dispatched together. >> > 2. Helix only cleans up the completed generic workflows by deleting all >> > the related znode, not for JobQueue. For JobQueue you have to set up >> > periodical purge time. As Helix defined, JobQueue never finishes and >> only >> > can be terminated by manual kill and it can keep accepting dynamic jobs. >> > Thus you have to understand your workflow is generic workflow or >> JobQueue. >> > For failed generic workflow, even if you setup the expiry time, Helix >> will >> > not clean it up as Helix would like to keep it for user further >> > investigation. >> > 3. For Helix controller, if Helix failed to clean up workflows, the only >> > thing you can see is the having workflows with context but no resource >> > config and idealstate there. This is because of ZK write fail to clean >> last >> > piece, context node. And there is no ideal state can trigger clean up >> again >> > for this workflow. >> > >> > Please take a look for this task framework tutorial for detailed >> > configurations: >> > https://helix.apache.org/0.8.2-docs/tutorial_task_framework.html >> > >> > Best, >> > >> > Junkai >> > >> > On Sat, Nov 10, 2018 at 8:29 AM DImuthu Upeksha < >> > dimuthu.upeks...@gmail.com> wrote: >> > >> >> Hi Junkai, >> >> >> >> Thanks for the clarification. There are few special properties in our >> >> workflows. All the workflows are single branch DAGs so there will be >> only >> >> one job running at a time. By looking at the log, I could see that only >> >> the >> >> task with this error has been failed. Cleanup agent deleted this >> workflow >> >> after this task is failed so it is clear that no other task is >> triggering >> >> this issue (I checked the timestamp). >> >> >> >> However for the instance, I disabled the cleanup agent for a while. >> Reason >> >> for adding this agent is because Helix became slow to schedule pending >> >> jobs >> >> when the load is high and participant was waiting without running >> anything >> >> for few minutes. We discussed this on thread "Sporadic delays in task >> >> execution". Before implementing this agent, I noticed that, there were >> >> lots >> >> of uncleared znodes related to Completed and Failed workflows and I >> though >> >> that was the reason to slow down controller / participant. After >> >> implementing this agent, things went smoothly until this point. >> >> >> >> Now I understand that you have your own workflow cleanup logic >> implemented >> >> in Helix but we might need to tune it to our case. Can you point me >> into >> >> code / documentation where I can have an idea about that? >> >> >> >> And this for my understanding, let's say that for some reason Helix >> failed >> >> to clean up completed workflows and related resources in zk. Will that >> >> affect to the performance of controller / participant? My understanding >> >> was >> >> that Helix was registering zk watchers for all the paths irrespective >> of >> >> the status of the workflow/ job/ task. Please correct me if I'm wrong. >> >> >> >> Thanks >> >> Dimuthu >> >> >> >> On Sat, Nov 10, 2018 at 1:49 AM Xue Junkai <junkai....@gmail.com> >> wrote: >> >> >> >> > It is possible. For example, if other jobs caused the workflow >> failed, >> >> it >> >> > will trigger the monitoring to clean up the workflow. Then if this >> job >> >> is >> >> > still running, you may see the problem. That's what I am trying to >> ask >> >> for, >> >> > extra thread deleting/cleaning workflows. >> >> > >> >> > I can understand it clean up the failed workflow. But I am wondering >> why >> >> > not just set expiry and let Helix controller does the clean up for >> >> > completed workflows. >> >> > >> >> > On Sat, Nov 10, 2018 at 1:30 PM DImuthu Upeksha < >> >> > dimuthu.upeks...@gmail.com> wrote: >> >> > >> >> >> Hi Junkai, >> >> >> >> >> >> There is a cleanup agent [1] who is monitoring currently available >> >> >> workflows and deleting completed and failed workflows to clear up >> >> >> zookeeper >> >> >> storage. Do you think that this will be causing this issue? >> >> >> >> >> >> [1] >> >> >> >> >> >> >> >> >> https://github.com/apache/airavata/blob/staging/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/controller/WorkflowCleanupAgent.java >> >> >> >> >> >> Thanks >> >> >> Dimuthu >> >> >> >> >> >> On Fri, Nov 9, 2018 at 11:14 PM DImuthu Upeksha < >> >> >> dimuthu.upeks...@gmail.com> >> >> >> wrote: >> >> >> >> >> >> > Hi Junkai, >> >> >> > >> >> >> > There is no manual workflow killing logic implemented but as you >> have >> >> >> > suggested, I need to verify that. Unfortunately all the helix log >> >> >> levels in >> >> >> > our servers were set to WARN as helix is printing a whole lot of >> >> logs in >> >> >> > INFO level so there is no much valuable information in logs. Can >> you >> >> >> > specify which class is printing logs associated for workflow >> >> termination >> >> >> > and I'll enable DEBUG level for that class and observe further. >> >> >> > >> >> >> > Thanks >> >> >> > Dimuthu >> >> >> > >> >> >> > On Fri, Nov 9, 2018 at 9:18 PM Xue Junkai <junkai....@gmail.com> >> >> wrote: >> >> >> > >> >> >> >> Hmm, that's very strange. The user content store znode only has >> been >> >> >> >> deleted when the workflow is gone. From the log, it shows the >> znode >> >> is >> >> >> >> gone. Could you please try to dig the log to find whether the >> >> workflow >> >> >> has >> >> >> >> been manually killed? If that's the case, then it is possible you >> >> have >> >> >> the >> >> >> >> problem. >> >> >> >> >> >> >> >> On Fri, Nov 9, 2018 at 12:13 PM DImuthu Upeksha < >> >> >> >> dimuthu.upeks...@gmail.com> >> >> >> >> wrote: >> >> >> >> >> >> >> >> > Hi Junkai, >> >> >> >> > >> >> >> >> > Thanks for your suggestion. You have captured most of the parts >> >> >> >> correctly. >> >> >> >> > There are two jobs as job1 and job2. And there is a dependency >> >> that >> >> >> job2 >> >> >> >> > depends on job1. Until job1 is completed job2 should not be >> >> >> scheduled. >> >> >> >> And >> >> >> >> > task 1 in job 1 is calling that method and it is not updating >> >> >> anyone's >> >> >> >> > content. It's just putting and value in workflow level. What do >> >> you >> >> >> >> mean my >> >> >> >> > keeping a key-value store in workflow level? I already use that >> >> key >> >> >> >> value >> >> >> >> > store given by helix by calling putUserContent method. >> >> >> >> > >> >> >> >> > public void sendNextJob(String jobId) { >> >> >> >> > putUserContent(WORKFLOW_STARTED, "TRUE", Scope.WORKFLOW); >> >> >> >> > if (jobId != null) { >> >> >> >> > putUserContent(NEXT_JOB, jobId, Scope.WORKFLOW); >> >> >> >> > } >> >> >> >> > } >> >> >> >> > >> >> >> >> > Dimuthu >> >> >> >> > >> >> >> >> > >> >> >> >> > On Fri, Nov 9, 2018 at 2:48 PM Xue Junkai < >> junkai....@gmail.com> >> >> >> wrote: >> >> >> >> > >> >> >> >> > > In my understanding, it could be you have job1 and job2. The >> >> task >> >> >> >> running >> >> >> >> > > in job1 tries to update content for job2. Then, there could >> be a >> >> >> race >> >> >> >> > > condition happening here that job2 is not scheduled. >> >> >> >> > > >> >> >> >> > > If that's the case, I suggest you can put key-value store at >> >> >> workflow >> >> >> >> > level >> >> >> >> > > since this is cross-job operation. >> >> >> >> > > >> >> >> >> > > Best, >> >> >> >> > > >> >> >> >> > > Junkai >> >> >> >> > > >> >> >> >> > > On Fri, Nov 9, 2018 at 11:45 AM DImuthu Upeksha < >> >> >> >> > > dimuthu.upeks...@gmail.com> >> >> >> >> > > wrote: >> >> >> >> > > >> >> >> >> > > > Hi Junkai, >> >> >> >> > > > >> >> >> >> > > > This method is being called inside a running task. And it >> is >> >> >> working >> >> >> >> > for >> >> >> >> > > > most of the time. I only saw this in 2 occasions for last >> few >> >> >> months >> >> >> >> > and >> >> >> >> > > > both of them happened today and yesterday. >> >> >> >> > > > >> >> >> >> > > > Thanks >> >> >> >> > > > Dimuthu >> >> >> >> > > > >> >> >> >> > > > On Fri, Nov 9, 2018 at 2:40 PM Xue Junkai < >> >> junkai....@gmail.com> >> >> >> >> > wrote: >> >> >> >> > > > >> >> >> >> > > > > User content store node will be created one the job has >> been >> >> >> >> > scheduled. >> >> >> >> > > > In >> >> >> >> > > > > your case, I think the job is not scheduled. This method >> >> >> usually >> >> >> >> has >> >> >> >> > > been >> >> >> >> > > > > utilized in running task. >> >> >> >> > > > > >> >> >> >> > > > > Best, >> >> >> >> > > > > >> >> >> >> > > > > Junkai >> >> >> >> > > > > >> >> >> >> > > > > On Fri, Nov 9, 2018 at 8:19 AM DImuthu Upeksha < >> >> >> >> > > > dimuthu.upeks...@gmail.com >> >> >> >> > > > > > >> >> >> >> > > > > wrote: >> >> >> >> > > > > >> >> >> >> > > > > > Hi Helix Folks, >> >> >> >> > > > > > >> >> >> >> > > > > > I'm having this sporadic issue in some tasks of our >> >> workflows >> >> >> >> when >> >> >> >> > we >> >> >> >> > > > try >> >> >> >> > > > > > to store a value in the workflow context and I have >> added >> >> >> both >> >> >> >> code >> >> >> >> > > > > section >> >> >> >> > > > > > and error message below. Do you have an idea what's >> >> causing >> >> >> >> this? >> >> >> >> > > > Please >> >> >> >> > > > > > let me know if you need further information. We are >> using >> >> >> Helix >> >> >> >> > 0.8.2 >> >> >> >> > > > > > >> >> >> >> > > > > > public void sendNextJob(String jobId) { >> >> >> >> > > > > > putUserContent(WORKFLOW_STARTED, "TRUE", >> >> Scope.WORKFLOW); >> >> >> >> > > > > > if (jobId != null) { >> >> >> >> > > > > > putUserContent(NEXT_JOB, jobId, >> Scope.WORKFLOW); >> >> >> >> > > > > > } >> >> >> >> > > > > > } >> >> >> >> > > > > > >> >> >> >> > > > > > Failed to setup environment of task >> >> >> >> > > > > > TASK_55096de4-2cb6-4b09-84fd-7fdddba93435 >> >> >> >> > > > > > java.lang.NullPointerException: null >> >> >> >> > > > > > at >> >> >> >> > org.apache.helix.task.TaskUtil$1.update(TaskUtil.java:358) >> >> >> >> > > > > > at >> >> >> >> > org.apache.helix.task.TaskUtil$1.update(TaskUtil.java:356) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.helix.manager.zk.HelixGroupCommit.commit(HelixGroupCommit.java:126) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.helix.manager.zk.ZkCacheBaseDataAccessor.update(ZkCacheBaseDataAccessor.java:306) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.helix.store.zk.AutoFallbackPropertyStore.update(AutoFallbackPropertyStore.java:61) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.helix.task.TaskUtil.addWorkflowJobUserContent(TaskUtil.java:356) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.helix.task.UserContentStore.putUserContent(UserContentStore.java:78) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.airavata.helix.core.AbstractTask.sendNextJob(AbstractTask.java:136) >> >> >> >> > > > > > at >> >> >> >> > > > >> org.apache.airavata.helix.core.OutPort.invoke(OutPort.java:42) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.airavata.helix.core.AbstractTask.onSuccess(AbstractTask.java:123) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.airavata.helix.impl.task.AiravataTask.onSuccess(AiravataTask.java:97) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.airavata.helix.impl.task.env.EnvSetupTask.onRun(EnvSetupTask.java:52) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:349) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > >> >> org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:92) >> >> >> >> > > > > > at >> >> >> >> org.apache.helix.task.TaskRunner.run(TaskRunner.java:71) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > >> >> >> >> >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> >> >> > > > > > at >> >> >> >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> >> >> > > > > > at >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> >> >> > > > > > at java.lang.Thread.run(Thread.java:748) >> >> >> >> > > > > > >> >> >> >> > > > > > Thanks >> >> >> >> > > > > > Dimuthu >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > > >> >> >> >> > > > > -- >> >> >> >> > > > > Junkai Xue >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > -- >> >> >> >> > > Junkai Xue >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Junkai Xue >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> > >> >> > -- >> >> > Junkai Xue >> >> > >> >> >> > >> > >> > -- >> > Junkai Xue >> > >> > > > -- > Junkai Xue >
