how can we do it? I mean, that as soon as slurm receives a job, along with launching of the "ResumeProgram", it already knows an old IP address of the node (the one the node had last time). Thus, while node is loading and update information about it, slurm returns a message that an error has occurred
2015-01-14 17:03 GMT+02:00 Robert Zeigler <[email protected]>: > The trick is having your resume program operation update the ip address of > the newly booted node before it returns. > > Robert > > GATAATGCTATTTCTTTAATTTTCGAA > > On Jan 14, 2015, at 4:51 AM, Anatoliy Kovalenko <[email protected]> > wrote: > > Our script executes the suggested operation every time when RedsumeProgram > is starting. This is proposed in the 4th step of documentation > http://slurm.schedmd.com/elastic_computing.html > Also, if every time we must do this operation "by hand", then dynamic > allocation of nodes for its jobs by slurm doesn't make much sense. > Is there an alternative way to solve the issue? > > 2015-01-14 0:02 GMT+02:00 <[email protected]>: > >> >> >> Quoting Anatoliy Kovalenko <[email protected]>: >> >> We are using slurm_elastics ( >>> http://schedmd.com/slurmdocs/elastic_computing.html) and we have got an >>> error error: agent waited too long for nodes to respond, sending batch >>> request anyway..., >>> We noticed that slurmctld remembers IP address of each node, but in our >>> case when a node is shutdown/launched by slurm, its IP may change, even >>> though the list of node names is constant >>> Is the error above associated with different IP addresses? >>> >> >> Possibly >> >> How can we fix it? >>> >> >> Use "scontrol update NodeName=... NodeAddr=..." to reset the IP address. >> -- >> Morris "Moe" Jette >> CTO, SchedMD LLC >> Commercial Slurm Development and Support >> > >
