Although the job is allocated to the node, slum won't try to actually send the job to the node until the node's slurmd is responsive. So there is window of opportunity to update the ip address before slurm starts trying to send jobs to the node. The difficult part is that it's simplest to set the ip on the head node, but you have to set out after it's been assigned, before the slurmd comes online. Typically, this will involve some means of synchronization between head and new compute node so that the compute node initialization, which should boot the slurmd, waits to do so until receiving some "all clear" signal from the head node, indicating that the updated up has been set. Assuming you are using AWS, you could transmit this information via, e.g., their SQS service.
Robert GATAATGCTATTTCTTTAATTTTCGAA > On Jan 14, 2015, at 11:03 AM, Anatoliy Kovalenko <[email protected]> > wrote: > > how can we do it? I mean, that as soon as slurm receives a job, along with > launching of the "ResumeProgram", it already knows an old IP address of the > node (the one the node had last time). Thus, while node is loading and update > information about it, slurm returns a message that an error has occurred > > 2015-01-14 17:03 GMT+02:00 Robert Zeigler <[email protected]>: >> The trick is having your resume program operation update the ip address of >> the newly booted node before it returns. >> >> Robert >> >> GATAATGCTATTTCTTTAATTTTCGAA >> >>> On Jan 14, 2015, at 4:51 AM, Anatoliy Kovalenko <[email protected]> >>> wrote: >>> >>> Our script executes the suggested operation every time when RedsumeProgram >>> is starting. This is proposed in the 4th step of documentation >>> http://slurm.schedmd.com/elastic_computing.html >>> Also, if every time we must do this operation "by hand", then dynamic >>> allocation of nodes for its jobs by slurm doesn't make much sense. >>> Is there an alternative way to solve the issue? >>> >>> 2015-01-14 0:02 GMT+02:00 <[email protected]>: >>>> >>>> >>>> Quoting Anatoliy Kovalenko <[email protected]>: >>>> >>>>> We are using slurm_elastics ( >>>>> http://schedmd.com/slurmdocs/elastic_computing.html) and we have got an >>>>> error error: agent waited too long for nodes to respond, sending batch >>>>> request anyway..., >>>>> We noticed that slurmctld remembers IP address of each node, but in our >>>>> case when a node is shutdown/launched by slurm, its IP may change, even >>>>> though the list of node names is constant >>>>> Is the error above associated with different IP addresses? >>>> >>>> Possibly >>>> >>>>> How can we fix it? >>>> >>>> Use "scontrol update NodeName=... NodeAddr=..." to reset the IP address. >>>> -- >>>> Morris "Moe" Jette >>>> CTO, SchedMD LLC >>>> Commercial Slurm Development and Support >>> >> >
