Hi Hunter, Thanks for the info. I am glad you’re able to reproduce the issue. Using zooInspector, I do not see any session string under instance.
Also, is there a limit on the number of participants for task processing? Thanks, Digant From: Hunter Lee [mailto:[email protected]] Sent: Monday, October 22, 2018 10:47 PM To: [email protected] Subject: Re: Helix 0.8.2 Task Reassignment Digant, As for the edge case I spoke of earlier: there is an already-identified bug in Task Framework where if task A gets assigned to an instance, and if the same instance goes dead and comes right back alive, task A will just be stuck on that instance and won't be fully executed even after the instance has come back alive, and the job will hang. I am able to reproduce this issue. Note that Task Framework does re-assign tasks in case of Participant loss. For example, suppose you have 5 instances, and if one goes down, the remaining tasks will be assigned onto the remaining 4 instances. As far as I know, this only affects 0.8.2. A fix has been identified and I suspect the fix will be included in the next Apache release, which should be coming fairly soon. Thanks, Hunter On Mon, Oct 22, 2018 at 3:11 PM Hunter Lee <[email protected]<mailto:[email protected]>> wrote: Digant, Sorry I was being unclear on this. Are you able to look at the ZNodes via Zooinspector? I don't think Helix REST api lets you take a look at the currentstates. Hunter On Mon, Oct 22, 2018 at 3:07 PM Modha, Digant <[email protected]<mailto:[email protected]>> wrote: Hi, I can’t find anything w/ session id after worker is restarted. http://localhost:8100/admin/v2/clusters/taskTestCluster/instances/BOB-2/1669dbeb1fa0003 I get error 404 http://localhost:8100/admin/v2/clusters/taskTestCluster/instances/BOB-2/1669dbeb1fa0003/Task_REST-GET_1540245606312 also gets error 404 http://localhost:8100/admin/v2/clusters/taskTestCluster/instances/BOB-2 liveInstance id "BOB-2" simpleFields HELIX_VERSION "0.8.2" LIVE_INSTANCE "35436@XXXXXXXX" SESSION_ID "1669dbeb1fa0003" mapFields {} listFields {} id "BOB-2" config id "BOB-2" simpleFields HELIX_ENABLED "true" HELIX_ENABLED_TIMESTAMP "1539205545920" HELIX_HOST "BOB-2" HELIX_PORT "" mapFields {} listFields TAG_LIST 0 "minion_untagged" Digant From: Hunter Lee [mailto:[email protected]<mailto:[email protected]>] Sent: Monday, October 22, 2018 5:20 PM To: [email protected]<mailto:[email protected]> Subject: Re: Helix 0.8.2 Task Reassignment Hi Digant, Helix Task Framework has automatic task-reassign built in. However, I've recently discovered an edge case where a task could get stuck in case there is a Participant connection issue/Zookeeper session reset. I am currently working on the fix, but just to verify this, could you tell me what you see under currentStates for the instance? The currentState of the task should be located in /<YOUR_CLUSTER_NAME>/INSTANCES/<YOUR_INSTANCE>/<session string>/<YOUR_JOB_NAME> and it should contain fields like ASSIGNED_PARTICIPANT, INFO, STATE, etc. Thanks, Hunter On Mon, Oct 22, 2018 at 11:36 AM Modha, Digant <[email protected]<mailto:[email protected]>> wrote: Hi Hunter, I am currently evaluating the Helix Job functionality. The number of live instances varies between 1 and 3. I set the numConcurrentTasksPerInstance to 1 since I don’t expect the workers to do more than one task concurrently. Right now I am only testing it with 1 task per job, but I expect to have tens or hundreds of tasks when fully functional. ZKHelixManager.ALLOW_PARTICIPANT_AUTO_JOIN = true ; JobConfig.Builder jobBuilder = new JobConfig.Builder().addTaskConfigs(helixTaskConfigs) .setInstanceGroupTag(instanceTag) .setNumConcurrentTasksPerInstance(numConcurrentTasksPerInstance) .setIgnoreDependentJobFailure(true) .setMaxAttemptsPerTask(10) .setFailureThreshold(Integer.MAX_VALUE) ; Thanks, Digant From: Hunter Lee [mailto:[email protected]<mailto:[email protected]>] Sent: Monday, October 22, 2018 2:20 PM To: [email protected]<mailto:[email protected]> Subject: Re: Helix 0.8.2 Task Reassignment Digant, I might have a clue - but before we jump to a conclusion, do you mind sharing your workflow/job configs? Also, the number of jobs and tasks per job, and the number of live instances. More specifically, what are you setting NumConcurrentTasksPerInstance in the jobs' configs? Thanks, Hunter Sent from BlueMail<https://clicktime.symantec.com/a/1/Aox4D8ZPi6QeSixwvmmcRl7KP66IhKEG9zr_FPlhc5Q=?d=fo8FV_9pq_NjX4B52Yb9IQYC9iXsFs9tyjfKDyiAoa8BxTcK1JJm05YzzI0q86U_-O09Yj7_Pq72APcLoIeYwzohUBT7yzao85v6r1LO39Ym4pljTzW9Afiy8ng8FvQjEK5y8g3nZVBuSt_ICHDRI_Gi5z6VhJmVdk1mi7dNnhdWWU_4Ie4IiCv4CnVBHNRv5DpBEMlkrmdGpfOSpPbPxrqckWTOngpoRqUAVwIhucE48Ki6z27Kb4zHJlzra4Jsz1YEVUj7EeFPj_i3lFypFErVZQL_bg6qFbaPYVhztYM6Zcvrm0teF3heNBvDXCJNvnzMGnq1muLQYhQjQdUQaoN7GPB1uEQ5pb2jrL_i5o2hZmEF5n7TAhYDtE8NewgSrjfHxh_p0nDWUY7Qfo7dGDa1P9AQTgXCR8PiziXvP7gXZg%3D%3D&u=http%3A%2F%2Fwww.bluemail.me%2Fr%3Fb%3D13688> On Oct 22, 2018, at 11:08, "Modha, Digant" <[email protected]<mailto:[email protected]>> wrote: Hi, When an instance a task is assigned to dies after task is assigned but before task is completed, task does not seem to be reassigned to another instance. Even if the original instance comes back up, it does not seem to process the task. I am using helix 0.8.2 and submitting task to a job queue. Should the job context partition state be set to TASK_ERROR in AssignableInstanceManager when the instance a task is assigned to is no longer a LiveInstance? jobContext.setPartitionState(taskIndex, TaskPartitionState.TASK_ERROR); Digant TD Securities disclaims any liability or losses either direct or consequential caused by the use of this information. This communication is for informational purposes only and is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. TD Securities is neither making any investment recommendation nor providing any professional or advisory services relating to the activities described herein. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice Any products described herein are (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or guaranteed by, an insured depository institution and (iii) subject to investment risks, including possible loss of the principal amount invested. The information shall not be further distributed or duplicated in whole or in part by any means without the prior written consent of TD Securities. TD Securities is a trademark of The Toronto-Dominion Bank and represents TD Securities (USA) LLC and certain investment banking activities of The Toronto-Dominion Bank and its subsidiaries. TD Securities disclaims any liability or losses either direct or consequential caused by the use of this information. This communication is for informational purposes only and is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. TD Securities is neither making any investment recommendation nor providing any professional or advisory services relating to the activities described herein. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice Any products described herein are (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or guaranteed by, an insured depository institution and (iii) subject to investment risks, including possible loss of the principal amount invested. The information shall not be further distributed or duplicated in whole or in part by any means without the prior written consent of TD Securities. TD Securities is a trademark of The Toronto-Dominion Bank and represents TD Securities (USA) LLC and certain investment banking activities of The Toronto-Dominion Bank and its subsidiaries. TD Securities disclaims any liability or losses either direct or consequential caused by the use of this information. This communication is for informational purposes only and is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. TD Securities is neither making any investment recommendation nor providing any professional or advisory services relating to the activities described herein. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice Any products described herein are (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or guaranteed by, an insured depository institution and (iii) subject to investment risks, including possible loss of the principal amount invested. The information shall not be further distributed or duplicated in whole or in part by any means without the prior written consent of TD Securities. TD Securities is a trademark of The Toronto-Dominion Bank and represents TD Securities (USA) LLC and certain investment banking activities of The Toronto-Dominion Bank and its subsidiaries. TD Securities disclaims any liability or losses either direct or consequential caused by the use of this information. This communication is for informational purposes only and is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. TD Securities is neither making any investment recommendation nor providing any professional or advisory services relating to the activities described herein. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice Any products described herein are (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or guaranteed by, an insured depository institution and (iii) subject to investment risks, including possible loss of the principal amount invested. The information shall not be further distributed or duplicated in whole or in part by any means without the prior written consent of TD Securities. TD Securities is a trademark of The Toronto-Dominion Bank and represents TD Securities (USA) LLC and certain investment banking activities of The Toronto-Dominion Bank and its subsidiaries.
