Hi Digant, Helix Task Framework has automatic task-reassign built in. However, I've recently discovered an edge case where a task could get stuck in case there is a Participant connection issue/Zookeeper session reset. I am currently working on the fix, but just to verify this, could you tell me what you see under currentStates for the instance?
The currentState of the task should be located in /<YOUR_CLUSTER_NAME>/INSTANCES/<YOUR_INSTANCE>/<session string>/<YOUR_JOB_NAME> and it should contain fields like ASSIGNED_PARTICIPANT, INFO, STATE, etc. Thanks, Hunter On Mon, Oct 22, 2018 at 11:36 AM Modha, Digant < [email protected]> wrote: > Hi Hunter, > > > > I am currently evaluating the Helix Job functionality. The number of > live instances varies between 1 and 3. I set the > numConcurrentTasksPerInstance > to 1 since I don’t expect the workers to do more than one task > concurrently. Right now I am only testing it with 1 task per job, but I > expect to have tens or hundreds of tasks when fully functional. > > > > > > ZKHelixManager.*ALLOW_PARTICIPANT_AUTO_JOIN **= true *; > > > > > > JobConfig.Builder jobBuilder = *new * > JobConfig.Builder().addTaskConfigs(helixTaskConfigs) > .setInstanceGroupTag(instanceTag) > .setNumConcurrentTasksPerInstance(numConcurrentTasksPerInstance) > .setIgnoreDependentJobFailure(*true*) > .setMaxAttemptsPerTask(10) > .setFailureThreshold(Integer.*MAX_VALUE*) > ; > > Thanks, > > > > Digant > > > > *From:* Hunter Lee [mailto:[email protected]] > *Sent:* Monday, October 22, 2018 2:20 PM > *To:* [email protected] > *Subject:* Re: Helix 0.8.2 Task Reassignment > > > > Digant, > > I might have a clue - but before we jump to a conclusion, do you mind > sharing your workflow/job configs? Also, the number of jobs and tasks per > job, and the number of live instances. > > More specifically, what are you setting NumConcurrentTasksPerInstance in > the jobs' configs? > > Thanks, > > Hunter > > Sent from BlueMail > <https://clicktime.symantec.com/a/1/Aox4D8ZPi6QeSixwvmmcRl7KP66IhKEG9zr_FPlhc5Q=?d=fo8FV_9pq_NjX4B52Yb9IQYC9iXsFs9tyjfKDyiAoa8BxTcK1JJm05YzzI0q86U_-O09Yj7_Pq72APcLoIeYwzohUBT7yzao85v6r1LO39Ym4pljTzW9Afiy8ng8FvQjEK5y8g3nZVBuSt_ICHDRI_Gi5z6VhJmVdk1mi7dNnhdWWU_4Ie4IiCv4CnVBHNRv5DpBEMlkrmdGpfOSpPbPxrqckWTOngpoRqUAVwIhucE48Ki6z27Kb4zHJlzra4Jsz1YEVUj7EeFPj_i3lFypFErVZQL_bg6qFbaPYVhztYM6Zcvrm0teF3heNBvDXCJNvnzMGnq1muLQYhQjQdUQaoN7GPB1uEQ5pb2jrL_i5o2hZmEF5n7TAhYDtE8NewgSrjfHxh_p0nDWUY7Qfo7dGDa1P9AQTgXCR8PiziXvP7gXZg%3D%3D&u=http%3A%2F%2Fwww.bluemail.me%2Fr%3Fb%3D13688> > > On Oct 22, 2018, at 11:08, "Modha, Digant" <[email protected]> > wrote: > > Hi, > > > > When an instance a task is assigned to dies after task is assigned but > before task is completed, task does not seem to be reassigned to another > instance. Even if the original instance comes back up, it does not seem to > process the task. > > > > > > I am using helix 0.8.2 and submitting task to a job queue. > > > > Should the job context partition state be set to TASK_ERROR in > AssignableInstanceManager when the instance a task is assigned to is no > longer a LiveInstance? > > > > jobContext.setPartitionState(taskIndex, TaskPartitionState.*TASK_ERROR*); > > > > Digant > > > > > > TD Securities disclaims any liability or losses either direct or > consequential caused by the use of this information. This communication is > for informational purposes only and is not intended as an offer or > solicitation for the purchase or sale of any financial instrument or as an > official confirmation of any transaction. TD Securities is neither making > any investment recommendation nor providing any professional or advisory > services relating to the activities described herein. All market prices, > data and other information are not warranted as to completeness or accuracy > and are subject to change without notice Any products described herein are > (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or > guaranteed by, an insured depository institution and (iii) subject to > investment risks, including possible loss of the principal amount invested. > The information shall not be further distributed or duplicated in whole or > in part by any means without the prior written consent of TD Securities. TD > Securities is a trademark of The Toronto-Dominion Bank and represents TD > Securities (USA) LLC and certain investment banking activities of The > Toronto-Dominion Bank and its subsidiaries. > > > TD Securities disclaims any liability or losses either direct or > consequential caused by the use of this information. This communication is > for informational purposes only and is not intended as an offer or > solicitation for the purchase or sale of any financial instrument or as an > official confirmation of any transaction. TD Securities is neither making > any investment recommendation nor providing any professional or advisory > services relating to the activities described herein. All market prices, > data and other information are not warranted as to completeness or accuracy > and are subject to change without notice Any products described herein are > (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or > guaranteed by, an insured depository institution and (iii) subject to > investment risks, including possible loss of the principal amount invested. > The information shall not be further distributed or duplicated in whole or > in part by any means without the prior written consent of TD Securities. TD > Securities is a trademark of The Toronto-Dominion Bank and represents TD > Securities (USA) LLC and certain investment banking activities of The > Toronto-Dominion Bank and its subsidiaries. >
