Digant, As for the edge case I spoke of earlier: there is an already-identified bug in Task Framework where if task A gets assigned to an instance, and if the same instance goes dead and comes right back alive, task A will just be stuck on that instance and won't be fully executed even after the instance has come back alive, and the job will hang. I am able to reproduce this issue.
Note that Task Framework does re-assign tasks in case of Participant loss. For example, suppose you have 5 instances, and if one goes down, the remaining tasks will be assigned onto the remaining 4 instances. As far as I know, this only affects 0.8.2. A fix has been identified and I suspect the fix will be included in the next Apache release, which should be coming fairly soon. Thanks, Hunter On Mon, Oct 22, 2018 at 3:11 PM Hunter Lee <[email protected]> wrote: > Digant, > > Sorry I was being unclear on this. > Are you able to look at the ZNodes via Zooinspector? I don't think Helix > REST api lets you take a look at the currentstates. > > Hunter > > On Mon, Oct 22, 2018 at 3:07 PM Modha, Digant < > [email protected]> wrote: > >> Hi, >> >> >> >> I can’t find anything w/ session id after worker is restarted. >> >> >> >> >> http://localhost:8100/admin/v2/clusters/taskTestCluster/instances/BOB-2/1669dbeb1fa0003 >> I get error 404 >> >> http://localhost:8100/admin/v2/clusters/taskTestCluster/instances/BOB-2/1669dbeb1fa0003/Task_REST-GET_1540245606312 >> also gets error 404 >> >> >> >> http://localhost:8100/admin/v2/clusters/taskTestCluster/instances/BOB-2 >> >> >> >> liveInstance >> >> id >> >> "BOB-2" >> >> simpleFields >> >> HELIX_VERSION >> >> "0.8.2" >> >> LIVE_INSTANCE >> >> "35436@XXXXXXXX" >> >> SESSION_ID >> >> "1669dbeb1fa0003" >> >> mapFields >> >> {} >> >> listFields >> >> {} >> >> id >> >> "BOB-2" >> >> config >> >> id >> >> "BOB-2" >> >> simpleFields >> >> HELIX_ENABLED >> >> "true" >> >> HELIX_ENABLED_TIMESTAMP >> >> "1539205545920" >> >> HELIX_HOST >> >> "BOB-2" >> >> HELIX_PORT >> >> "" >> >> mapFields >> >> {} >> >> listFields >> >> TAG_LIST >> >> 0 >> >> "minion_untagged" >> >> >> >> >> >> Digant >> >> >> >> *From:* Hunter Lee [mailto:[email protected]] >> *Sent:* Monday, October 22, 2018 5:20 PM >> *To:* [email protected] >> *Subject:* Re: Helix 0.8.2 Task Reassignment >> >> >> >> Hi Digant, >> >> >> >> Helix Task Framework has automatic task-reassign built in. However, I've >> recently discovered an edge case where a task could get stuck in case there >> is a Participant connection issue/Zookeeper session reset. I am currently >> working on the fix, but just to verify this, could you tell me what you see >> under currentStates for the instance? >> >> >> >> The currentState of the task should be located in >> /<YOUR_CLUSTER_NAME>/INSTANCES/<YOUR_INSTANCE>/<session >> string>/<YOUR_JOB_NAME> >> >> and it should contain fields like ASSIGNED_PARTICIPANT, INFO, STATE, etc. >> >> >> >> Thanks, >> >> Hunter >> >> >> >> On Mon, Oct 22, 2018 at 11:36 AM Modha, Digant < >> [email protected]> wrote: >> >> Hi Hunter, >> >> >> >> I am currently evaluating the Helix Job functionality. The number of >> live instances varies between 1 and 3. I set the >> numConcurrentTasksPerInstance to 1 since I don’t expect the workers to do >> more than one task concurrently. Right now I am only testing it with 1 >> task per job, but I expect to have tens or hundreds of tasks when fully >> functional. >> >> >> >> >> >> ZKHelixManager.*ALLOW_PARTICIPANT_AUTO_JOIN **= true *; >> >> >> >> >> >> JobConfig.Builder jobBuilder = *new * >> JobConfig.Builder().addTaskConfigs(helixTaskConfigs) >> .setInstanceGroupTag(instanceTag) >> .setNumConcurrentTasksPerInstance(numConcurrentTasksPerInstance) >> .setIgnoreDependentJobFailure(*true*) >> .setMaxAttemptsPerTask(10) >> .setFailureThreshold(Integer.*MAX_VALUE*) >> ; >> >> Thanks, >> >> >> >> Digant >> >> >> >> *From:* Hunter Lee [mailto:[email protected]] >> *Sent:* Monday, October 22, 2018 2:20 PM >> *To:* [email protected] >> *Subject:* Re: Helix 0.8.2 Task Reassignment >> >> >> >> Digant, >> >> I might have a clue - but before we jump to a conclusion, do you mind >> sharing your workflow/job configs? Also, the number of jobs and tasks per >> job, and the number of live instances. >> >> More specifically, what are you setting NumConcurrentTasksPerInstance in >> the jobs' configs? >> >> Thanks, >> >> Hunter >> >> Sent from BlueMail >> <https://clicktime.symantec.com/a/1/Aox4D8ZPi6QeSixwvmmcRl7KP66IhKEG9zr_FPlhc5Q=?d=fo8FV_9pq_NjX4B52Yb9IQYC9iXsFs9tyjfKDyiAoa8BxTcK1JJm05YzzI0q86U_-O09Yj7_Pq72APcLoIeYwzohUBT7yzao85v6r1LO39Ym4pljTzW9Afiy8ng8FvQjEK5y8g3nZVBuSt_ICHDRI_Gi5z6VhJmVdk1mi7dNnhdWWU_4Ie4IiCv4CnVBHNRv5DpBEMlkrmdGpfOSpPbPxrqckWTOngpoRqUAVwIhucE48Ki6z27Kb4zHJlzra4Jsz1YEVUj7EeFPj_i3lFypFErVZQL_bg6qFbaPYVhztYM6Zcvrm0teF3heNBvDXCJNvnzMGnq1muLQYhQjQdUQaoN7GPB1uEQ5pb2jrL_i5o2hZmEF5n7TAhYDtE8NewgSrjfHxh_p0nDWUY7Qfo7dGDa1P9AQTgXCR8PiziXvP7gXZg%3D%3D&u=http%3A%2F%2Fwww.bluemail.me%2Fr%3Fb%3D13688> >> >> On Oct 22, 2018, at 11:08, "Modha, Digant" <[email protected]> >> wrote: >> >> Hi, >> >> >> >> When an instance a task is assigned to dies after task is assigned but >> before task is completed, task does not seem to be reassigned to another >> instance. Even if the original instance comes back up, it does not seem to >> process the task. >> >> >> >> >> >> I am using helix 0.8.2 and submitting task to a job queue. >> >> >> >> Should the job context partition state be set to TASK_ERROR in >> AssignableInstanceManager when the instance a task is assigned to is no >> longer a LiveInstance? >> >> >> >> jobContext.setPartitionState(taskIndex, TaskPartitionState.*TASK_ERROR*); >> >> >> >> Digant >> >> >> >> >> >> TD Securities disclaims any liability or losses either direct or >> consequential caused by the use of this information. This communication is >> for informational purposes only and is not intended as an offer or >> solicitation for the purchase or sale of any financial instrument or as an >> official confirmation of any transaction. TD Securities is neither making >> any investment recommendation nor providing any professional or advisory >> services relating to the activities described herein. All market prices, >> data and other information are not warranted as to completeness or accuracy >> and are subject to change without notice Any products described herein are >> (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or >> guaranteed by, an insured depository institution and (iii) subject to >> investment risks, including possible loss of the principal amount invested. >> The information shall not be further distributed or duplicated in whole or >> in part by any means without the prior written consent of TD Securities. TD >> Securities is a trademark of The Toronto-Dominion Bank and represents TD >> Securities (USA) LLC and certain investment banking activities of The >> Toronto-Dominion Bank and its subsidiaries. >> >> >> >> TD Securities disclaims any liability or losses either direct or >> consequential caused by the use of this information. This communication is >> for informational purposes only and is not intended as an offer or >> solicitation for the purchase or sale of any financial instrument or as an >> official confirmation of any transaction. TD Securities is neither making >> any investment recommendation nor providing any professional or advisory >> services relating to the activities described herein. All market prices, >> data and other information are not warranted as to completeness or accuracy >> and are subject to change without notice Any products described herein are >> (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or >> guaranteed by, an insured depository institution and (iii) subject to >> investment risks, including possible loss of the principal amount invested. >> The information shall not be further distributed or duplicated in whole or >> in part by any means without the prior written consent of TD Securities. TD >> Securities is a trademark of The Toronto-Dominion Bank and represents TD >> Securities (USA) LLC and certain investment banking activities of The >> Toronto-Dominion Bank and its subsidiaries. >> >> >> >> TD Securities disclaims any liability or losses either direct or >> consequential caused by the use of this information. This communication is >> for informational purposes only and is not intended as an offer or >> solicitation for the purchase or sale of any financial instrument or as an >> official confirmation of any transaction. TD Securities is neither making >> any investment recommendation nor providing any professional or advisory >> services relating to the activities described herein. All market prices, >> data and other information are not warranted as to completeness or accuracy >> and are subject to change without notice Any products described herein are >> (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or >> guaranteed by, an insured depository institution and (iii) subject to >> investment risks, including possible loss of the principal amount invested. >> The information shall not be further distributed or duplicated in whole or >> in part by any means without the prior written consent of TD Securities. TD >> Securities is a trademark of The Toronto-Dominion Bank and represents TD >> Securities (USA) LLC and certain investment banking activities of The >> Toronto-Dominion Bank and its subsidiaries. >> >
