This issue we was solved by ResumeTimeout option. 2015-01-15 16:05 GMT+02:00 Anatoliy Kovalenko <[email protected]>:
> yes, my script does work exactly as you say, first it updates !P for > node through scontrol command and only after IP has been updated, script > start the slurm, but we still got an error above from slurm-controller > (slurmctld) :( > Could you please explain in more details what the error means? :) > > > 2015-01-14 19:52 GMT+02:00 Robert Zeigler <[email protected]>: > >> Although the job is allocated to the node, slum won't try to actually >> send the job to the node until the node's slurmd is responsive. So there is >> window of opportunity to update the ip address before slurm starts trying >> to send jobs to the node. The difficult part is that it's simplest to set >> the ip on the head node, but you have to set out after it's been assigned, >> before the slurmd comes online. Typically, this will involve some means of >> synchronization between head and new compute node so that the compute node >> initialization, which should boot the slurmd, waits to do so until >> receiving some "all clear" signal from the head node, indicating that the >> updated up has been set. Assuming you are using AWS, you could transmit >> this information via, e.g., their SQS service. >> >> Robert >> >> GATAATGCTATTTCTTTAATTTTCGAA >> >> On Jan 14, 2015, at 11:03 AM, Anatoliy Kovalenko < >> [email protected]> wrote: >> >> how can we do it? I mean, that as soon as slurm receives a job, along >> with launching of the "ResumeProgram", it already knows an old IP address >> of the node (the one the node had last time). Thus, while node is loading >> and update information about it, slurm returns a message that an error has >> occurred >> >> 2015-01-14 17:03 GMT+02:00 Robert Zeigler <[email protected]>: >> >>> The trick is having your resume program operation update the ip address >>> of the newly booted node before it returns. >>> >>> Robert >>> >>> GATAATGCTATTTCTTTAATTTTCGAA >>> >>> On Jan 14, 2015, at 4:51 AM, Anatoliy Kovalenko < >>> [email protected]> wrote: >>> >>> Our script executes the suggested operation every time when >>> RedsumeProgram is starting. This is proposed in the 4th step of >>> documentation http://slurm.schedmd.com/elastic_computing.html >>> Also, if every time we must do this operation "by hand", then dynamic >>> allocation of nodes for its jobs by slurm doesn't make much sense. >>> Is there an alternative way to solve the issue? >>> >>> 2015-01-14 0:02 GMT+02:00 <[email protected]>: >>> >>>> >>>> >>>> Quoting Anatoliy Kovalenko <[email protected]>: >>>> >>>> We are using slurm_elastics ( >>>>> http://schedmd.com/slurmdocs/elastic_computing.html) and we have got >>>>> an >>>>> error error: agent waited too long for nodes to respond, sending batch >>>>> request anyway..., >>>>> We noticed that slurmctld remembers IP address of each node, but in our >>>>> case when a node is shutdown/launched by slurm, its IP may change, even >>>>> though the list of node names is constant >>>>> Is the error above associated with different IP addresses? >>>>> >>>> >>>> Possibly >>>> >>>> How can we fix it? >>>>> >>>> >>>> Use "scontrol update NodeName=... NodeAddr=..." to reset the IP address. >>>> -- >>>> Morris "Moe" Jette >>>> CTO, SchedMD LLC >>>> Commercial Slurm Development and Support >>>> >>> >>> >> >
