Hi Digant, I'm not sure what you're exactly asking, so I'll go out on a limb and try answering what I *think* you are asking :)
1. Is there a limit on the number of Helix Participants (instances) for tasks? -> As far as I am aware, there is no upper limit on the number of Participants in a Helix cluster. I am aware of use cases that use up to several hundred instances in a single Helix cluster. 2. Is there a limit on the number of concurrent tasks on a Helix Participant? -> For Task Framework, the size of the thread pool (assume each task occupies one thread) is set at 40. Now, 40 is an arbitrary number that has worked reasonably well in a lot of cases, but there is a ticket to make this configurable. This means that a Participant will be able to run up to 40 tasks concurrently. The idea is (in the future) that the user should be able to reason about & come up with the appropriate size of the thread pool given their task characteristics. This of course assumes that there are no job-level configs set that restrict the number of concurrent tasks (NumConcurrentTasksPerInstance, for example). Hope that helps, Hunter On Tue, Oct 23, 2018 at 6:58 AM Modha, Digant <[email protected]> wrote: > Hi Hunter, > > > Thanks for the info. I am glad you’re able to reproduce the issue. Using > zooInspector, I do not see any session string under instance. > > > > Also, is there a limit on the number of participants for task processing? > > > > Thanks, > > > > Digant > > > > > > *From:* Hunter Lee [mailto:[email protected]] > *Sent:* Monday, October 22, 2018 10:47 PM > *To:* [email protected] > *Subject:* Re: Helix 0.8.2 Task Reassignment > > > > Digant, > > > > As for the edge case I spoke of earlier: there is an already-identified > bug in Task Framework where if task A gets assigned to an instance, and if > the same instance goes dead and comes right back alive, task A will just be > stuck on that instance and won't be fully executed even after the instance > has come back alive, and the job will hang. I am able to reproduce this > issue. > > > > Note that Task Framework does re-assign tasks in case of Participant loss. > For example, suppose you have 5 instances, and if one goes down, the > remaining tasks will be assigned onto the remaining 4 instances. > > > > As far as I know, this only affects 0.8.2. A fix has been identified and I > suspect the fix will be included in the next Apache release, which should > be coming fairly soon. > > > > Thanks, > > Hunter > > > > On Mon, Oct 22, 2018 at 3:11 PM Hunter Lee <[email protected]> wrote: > > Digant, > > > > Sorry I was being unclear on this. > > Are you able to look at the ZNodes via Zooinspector? I don't think Helix > REST api lets you take a look at the currentstates. > > > > Hunter > > > > On Mon, Oct 22, 2018 at 3:07 PM Modha, Digant < > [email protected]> wrote: > > Hi, > > > > I can’t find anything w/ session id after worker is restarted. > > > > > http://localhost:8100/admin/v2/clusters/taskTestCluster/instances/BOB-2/1669dbeb1fa0003 > I get error 404 > > http://localhost:8100/admin/v2/clusters/taskTestCluster/instances/BOB-2/1669dbeb1fa0003/Task_REST-GET_1540245606312 > also gets error 404 > > > > http://localhost:8100/admin/v2/clusters/taskTestCluster/instances/BOB-2 > > > > liveInstance > > id > > "BOB-2" > > simpleFields > > HELIX_VERSION > > "0.8.2" > > LIVE_INSTANCE > > "35436@XXXXXXXX" > > SESSION_ID > > "1669dbeb1fa0003" > > mapFields > > {} > > listFields > > {} > > id > > "BOB-2" > > config > > id > > "BOB-2" > > simpleFields > > HELIX_ENABLED > > "true" > > HELIX_ENABLED_TIMESTAMP > > "1539205545920" > > HELIX_HOST > > "BOB-2" > > HELIX_PORT > > "" > > mapFields > > {} > > listFields > > TAG_LIST > > 0 > > "minion_untagged" > > > > > > Digant > > > > *From:* Hunter Lee [mailto:[email protected]] > *Sent:* Monday, October 22, 2018 5:20 PM > *To:* [email protected] > *Subject:* Re: Helix 0.8.2 Task Reassignment > > > > Hi Digant, > > > > Helix Task Framework has automatic task-reassign built in. However, I've > recently discovered an edge case where a task could get stuck in case there > is a Participant connection issue/Zookeeper session reset. I am currently > working on the fix, but just to verify this, could you tell me what you see > under currentStates for the instance? > > > > The currentState of the task should be located in > /<YOUR_CLUSTER_NAME>/INSTANCES/<YOUR_INSTANCE>/<session > string>/<YOUR_JOB_NAME> > > and it should contain fields like ASSIGNED_PARTICIPANT, INFO, STATE, etc. > > > > Thanks, > > Hunter > > > > On Mon, Oct 22, 2018 at 11:36 AM Modha, Digant < > [email protected]> wrote: > > Hi Hunter, > > > > I am currently evaluating the Helix Job functionality. The number of > live instances varies between 1 and 3. I set the > numConcurrentTasksPerInstance to 1 since I don’t expect the workers to do > more than one task concurrently. Right now I am only testing it with 1 > task per job, but I expect to have tens or hundreds of tasks when fully > functional. > > > > > > ZKHelixManager.*ALLOW_PARTICIPANT_AUTO_JOIN **= true *; > > > > > > JobConfig.Builder jobBuilder = *new * > JobConfig.Builder().addTaskConfigs(helixTaskConfigs) > .setInstanceGroupTag(instanceTag) > .setNumConcurrentTasksPerInstance(numConcurrentTasksPerInstance) > .setIgnoreDependentJobFailure(*true*) > .setMaxAttemptsPerTask(10) > .setFailureThreshold(Integer.*MAX_VALUE*) > ; > > Thanks, > > > > Digant > > > > *From:* Hunter Lee [mailto:[email protected]] > *Sent:* Monday, October 22, 2018 2:20 PM > *To:* [email protected] > *Subject:* Re: Helix 0.8.2 Task Reassignment > > > > Digant, > > I might have a clue - but before we jump to a conclusion, do you mind > sharing your workflow/job configs? Also, the number of jobs and tasks per > job, and the number of live instances. > > More specifically, what are you setting NumConcurrentTasksPerInstance in > the jobs' configs? > > Thanks, > > Hunter > > Sent from BlueMail > <https://clicktime.symantec.com/a/1/Aox4D8ZPi6QeSixwvmmcRl7KP66IhKEG9zr_FPlhc5Q=?d=fo8FV_9pq_NjX4B52Yb9IQYC9iXsFs9tyjfKDyiAoa8BxTcK1JJm05YzzI0q86U_-O09Yj7_Pq72APcLoIeYwzohUBT7yzao85v6r1LO39Ym4pljTzW9Afiy8ng8FvQjEK5y8g3nZVBuSt_ICHDRI_Gi5z6VhJmVdk1mi7dNnhdWWU_4Ie4IiCv4CnVBHNRv5DpBEMlkrmdGpfOSpPbPxrqckWTOngpoRqUAVwIhucE48Ki6z27Kb4zHJlzra4Jsz1YEVUj7EeFPj_i3lFypFErVZQL_bg6qFbaPYVhztYM6Zcvrm0teF3heNBvDXCJNvnzMGnq1muLQYhQjQdUQaoN7GPB1uEQ5pb2jrL_i5o2hZmEF5n7TAhYDtE8NewgSrjfHxh_p0nDWUY7Qfo7dGDa1P9AQTgXCR8PiziXvP7gXZg%3D%3D&u=http%3A%2F%2Fwww.bluemail.me%2Fr%3Fb%3D13688> > > On Oct 22, 2018, at 11:08, "Modha, Digant" <[email protected]> > wrote: > > Hi, > > > > When an instance a task is assigned to dies after task is assigned but > before task is completed, task does not seem to be reassigned to another > instance. Even if the original instance comes back up, it does not seem to > process the task. > > > > > > I am using helix 0.8.2 and submitting task to a job queue. > > > > Should the job context partition state be set to TASK_ERROR in > AssignableInstanceManager when the instance a task is assigned to is no > longer a LiveInstance? > > > > jobContext.setPartitionState(taskIndex, TaskPartitionState.*TASK_ERROR*); > > > > Digant > > > > > > TD Securities disclaims any liability or losses either direct or > consequential caused by the use of this information. This communication is > for informational purposes only and is not intended as an offer or > solicitation for the purchase or sale of any financial instrument or as an > official confirmation of any transaction. TD Securities is neither making > any investment recommendation nor providing any professional or advisory > services relating to the activities described herein. All market prices, > data and other information are not warranted as to completeness or accuracy > and are subject to change without notice Any products described herein are > (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or > guaranteed by, an insured depository institution and (iii) subject to > investment risks, including possible loss of the principal amount invested. > The information shall not be further distributed or duplicated in whole or > in part by any means without the prior written consent of TD Securities. TD > Securities is a trademark of The Toronto-Dominion Bank and represents TD > Securities (USA) LLC and certain investment banking activities of The > Toronto-Dominion Bank and its subsidiaries. > > > > TD Securities disclaims any liability or losses either direct or > consequential caused by the use of this information. This communication is > for informational purposes only and is not intended as an offer or > solicitation for the purchase or sale of any financial instrument or as an > official confirmation of any transaction. TD Securities is neither making > any investment recommendation nor providing any professional or advisory > services relating to the activities described herein. All market prices, > data and other information are not warranted as to completeness or accuracy > and are subject to change without notice Any products described herein are > (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or > guaranteed by, an insured depository institution and (iii) subject to > investment risks, including possible loss of the principal amount invested. > The information shall not be further distributed or duplicated in whole or > in part by any means without the prior written consent of TD Securities. TD > Securities is a trademark of The Toronto-Dominion Bank and represents TD > Securities (USA) LLC and certain investment banking activities of The > Toronto-Dominion Bank and its subsidiaries. > > > > TD Securities disclaims any liability or losses either direct or > consequential caused by the use of this information. This communication is > for informational purposes only and is not intended as an offer or > solicitation for the purchase or sale of any financial instrument or as an > official confirmation of any transaction. TD Securities is neither making > any investment recommendation nor providing any professional or advisory > services relating to the activities described herein. All market prices, > data and other information are not warranted as to completeness or accuracy > and are subject to change without notice Any products described herein are > (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or > guaranteed by, an insured depository institution and (iii) subject to > investment risks, including possible loss of the principal amount invested. > The information shall not be further distributed or duplicated in whole or > in part by any means without the prior written consent of TD Securities. TD > Securities is a trademark of The Toronto-Dominion Bank and represents TD > Securities (USA) LLC and certain investment banking activities of The > Toronto-Dominion Bank and its subsidiaries. > > > > TD Securities disclaims any liability or losses either direct or > consequential caused by the use of this information. This communication is > for informational purposes only and is not intended as an offer or > solicitation for the purchase or sale of any financial instrument or as an > official confirmation of any transaction. TD Securities is neither making > any investment recommendation nor providing any professional or advisory > services relating to the activities described herein. All market prices, > data and other information are not warranted as to completeness or accuracy > and are subject to change without notice Any products described herein are > (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or > guaranteed by, an insured depository institution and (iii) subject to > investment risks, including possible loss of the principal amount invested. > The information shall not be further distributed or duplicated in whole or > in part by any means without the prior written consent of TD Securities. TD > Securities is a trademark of The Toronto-Dominion Bank and represents TD > Securities (USA) LLC and certain investment banking activities of The > Toronto-Dominion Bank and its subsidiaries. >
