Hi Dlmuthu and Kishore, Yes. Task framework does support tagged instance at job level. It can be set in JobConfig for InstanceGroupTag. Also, do not forget to tag the instances.
Best, Junkai On Thu, Nov 16, 2017 at 8:50 PM, kishore g <g.kish...@gmail.com> wrote: > Hi Dlmuthu, > > You can achieve this by using the tag feature in Helix. For example, to > Participants P1, P2 should only handle DataCollecting tasks, you need to do > the following > > - Tag Participant P1 and P2 as "DataCollecting" > - When you create a job resource that represents DataCollecting task, > you have to tag the resource also as DataCollecting. > > Helix will then assign all tasks in this job to nodes that are tagged as > DataCollecting. > > Note this is available only in FULL_AUTO rebalance mode. I am not sure if > the rebalancer in task framework supports this feature. > > Lei/Junkai, do you know if this is supported in Task Framework? > > thanks > > > On Thu, Nov 16, 2017 at 8:31 PM, Ajinkya Dhamnaskar <adham...@umail.iu.edu > > > wrote: > > > Dlmuthu, > > > > This explains your need. Could you please point me to the code? I was > > wondering, where and how are we registering callbacks for the > participants? > > > > On Thu, Nov 16, 2017 at 7:16 PM, DImuthu Upeksha < > > dimuthu.upeks...@gmail.com > > > wrote: > > > > > Adding Helix Dev > > > > > > Hi Ajinkya > > > > > > Thank you for the explanation. > > > > > > Let me explain my requirement. [1] is the correct task registry > > > configuration in a working participant code. All the transition call > > backs > > > are registered in this code. The problem here is, we have to bundle > > > binaries of all the tasks into a single Participant. If we want to > change > > > one task, we need to rebuild the participant with other tasks as well. > > What > > > I thought is, why can't we build Participants that only do specific set > > of > > > tasks. For example, in the cluster there are 3 participants, > Participant > > 1 > > > [2], Participant 2 [3] and Participant 3 [4]. If we want to change the > > > DataCollecting task, we only need to re build Participant 2. I got > above > > > error, once I run 3 participants in above configuration. I can > understand > > > this is due to the missing transition callback of CommandTask in > > > Participant 1 as I have purposely commented out it. What I need to know > > is > > > that, is this configuration allowed in Helix architecture or do we have > > to > > > implement Participants that contain all the task implementations as in > > [1]. > > > Workflow configuration for both scenarios is here [5]. > > > > > > [1] https://gist.github.com/DImuthuUpe/41c0db579e7d86d101d112f07ed6ea > 00 > > > [2] https://gist.github.com/DImuthuUpe/ec72df1ec3207ce2dce88ff7f1756d > a4 > > > [3] https://gist.github.com/DImuthuUpe/4d193a3dff3008315efa2e31f6209c > ac > > > [4] https://gist.github.com/DImuthuUpe/872c432935b8d33944dd571b3ac420 > 7b > > > [5] https://gist.github.com/DImuthuUpe/f61851b68b685b8d6744689dc130ba > bd > > > > > > Thanks > > > Dimuthu > > > > > > On Fri, Nov 17, 2017 at 3:21 AM, Ajinkya Dhamnaskar < > > adham...@umail.iu.edu > > > > wrote: > > > > > >> Hey Dlmuthu, > > >> > > >> Not an expert in Helix, but from exceptions it seems, system is > entering > > >> in a state not expected by reflection. I feel > https://github.com/apache > > >> /helix/blob/master/helix-core/src/main/java/org/apache/helix > > >> /messaging/handling/HelixStateTransitionHandler.java#L295 is > triggering > > >> this exception. > > >> As mentioned in the later part of the stack trace and from Helix > Apache > > >> Docs <http://helix.apache.org/0.7.1-docs/tutorial_agent.html> > *("Helix > > >> is built on the following assumption: if your distributed resource is > > >> modeled by a finite state machine, then Helix can tell participants > when > > >> they should transition between states. In the Java API, this means > > >> implementing transition callbacks. In the Helix agent API, this means > > >> providing commands than can run for each transition"),* did you > > >> implement *transition callback* for these tasks? > > >> > > >> On Thu, Nov 16, 2017 at 10:01 AM, DImuthu Upeksha < > > >> dimuthu.upeks...@gmail.com> wrote: > > >> > > >>> Hi Devs, > > >>> > > >>> I'm working on the technology evaluation to re architecture Apache > > >>> Airavata task execution framework and Helix seems like a good > > candidate for > > >>> that as it has an in built distributed generic workflow execution > > >>> capability. After going through several tutorials, I tried to > > implement a > > >>> simple workflow on Helix to demonstrate following transition > > >>> > > >>> Data Collecting Job -> Command Executing Job -> Data Uploading Job > > >>> > > >>> I managed to implement this using a Participant node that includes > all > > >>> the tasks required for above workflow. However my goal is to > implement > > >>> specialized Participants for each Job type. For example, Participant > 1 > > >>> knows only about the tasks to perform Data Collecting Job and > > Participant > > >>> knows only about the task to perform Command Executing Job. When I > > tried to > > >>> implement such Participants, I got following error from Participant > 1. > > I > > >>> can share the code samples that I have tried but before that I need > to > > know > > >>> whether my approach is compatible with Helix's design? Does Helix > > require > > >>> all the Participants to be homogeneous? > > >>> > > >>> Executing data collecting > > >>> 675892 [pool-2-thread-8] ERROR org.apache.helix.messaging.han > > >>> dling.HelixStateTransitionHandler - Exception while executing a > state > > >>> transition task Airavata_Workflow_commandExecuteJob_0 > > >>> java.lang.reflect.InvocationTargetException > > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce > > >>> ssorImpl.java:62) > > >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > > >>> thodAccessorImpl.java:43) > > >>> at java.lang.reflect.Method.invoke(Method.java:498) > > >>> at org.apache.helix.messaging.handling.HelixStateTransitionHand > > >>> ler.invoke(HelixStateTransitionHandler.java:344) > > >>> at org.apache.helix.messaging.handling.HelixStateTransitionHand > > >>> ler.handleMessage(HelixStateTransitionHandler.java:290) > > >>> at org.apache.helix.messaging.handling.HelixTask.call( > > HelixTask.java:84) > > >>> at org.apache.helix.messaging.handling.HelixTask.call( > > HelixTask.java:41) > > >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > > >>> Executor.java:1149) > > >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > > >>> lExecutor.java:624) > > >>> at java.lang.Thread.run(Thread.java:748) > > >>> Caused by: java.lang.IllegalStateException: No callback implemented > > for > > >>> task COMMAND > > >>> at org.apache.helix.task.TaskStateModel.startTask(TaskStateMode > > >>> l.java:302) > > >>> at org.apache.helix.task.TaskStateModel.onBecomeRunningFromInit > > >>> (TaskStateModel.java:69) > > >>> ... 12 more > > >>> 675900 [pool-2-thread-8] ERROR org.apache.helix.participant. > > statemachine.StateModel > > >>> - Default rollback method invoked on error. Error Code: ERROR > > >>> 675908 [pool-2-thread-8] ERROR org.apache.helix.messaging. > > handling.HelixTask > > >>> - Message execution failed. msgId: 8d9a08c9-0ae7-419a-8940- > > 6b8e6e7c1ecb, > > >>> errorMsg: java.lang.reflect.InvocationTargetException > > >>> 675922 [pool-2-thread-8] ERROR org.apache.helix.messaging.han > > >>> dling.HelixStateTransitionHandler - Skip internal error. errCode: > > >>> ERROR, errMsg: null > > >>> > > >>> Thanks > > >>> Dimuthu > > >>> > > >> > > >> > > >> > > >> -- > > >> Thanks and regards, > > >> > > >> Ajinkya Dhamnaskar > > >> Student ID : 0003469679 > > >> Masters (CS) > > >> +1 (812) 369- 5416 <(812)%20369-5416> > > >> > > > > > > > > > > > > -- > > Thanks and regards, > > > > Ajinkya Dhamnaskar > > Student ID : 0003469679 > > Masters (CS) > > +1 (812) 369- 5416 > > >
