That's likely to cause you problems, so it's worth using NTP to keep all your nodes in sync with each other. Authentication systems are very sensitive to clockskew (including MUNGE), so if they drift too far out of sync, things stop working. It can happen gradually as nodes drift at different rates.
On 25 August 2015 at 13:33, Fahad Ibrahim Alzannan <[email protected]> wrote: > Hi, > > Actually some working nodes are delayed by around 5 mins also the down > nodes ! > ------------------------------ > *From:* Nathan Harper [[email protected]] > *Sent:* Tuesday, August 25, 2015 2:04 PM > *To:* slurm-dev > *Subject:* [slurm-dev] Re: Nodes are getting DOWN state > > Hi - can you check that your clocks are in sync between your compute nodes > and controllers? > > > -- > *Nathan Harper* > > On 25 August 2015 at 11:51, Fahad Ibrahim Alzannan <[email protected] > > wrote: > >> Hi, >> >> >> We have a cluster and some nodes are down we tried to set them idle using >> "scontrol update NodeName=xx State=idle" but they get back to down also we >> tried to troubleshoot them using SLURM Troubleshooting Guide but >> unfortunately nodes are still not working. >> >> >> Here are commands outputs >> >> >> Sinfo output is :sinfo: error: slurm_receive_msg: Zero Bytes were >> transmitted or received >> slurm_load_partitions: Zero Bytes were transmitted or received. >> >> >> scontrol ping output is: scontrol: error: slurm_receive_msg: Zero Bytes >> were transmitted or received >> Slurmctld(primary/backup) at master/(NULL) are DOWN/DOWN >> ************************************************************** >> ** RESTORE SLURMCTLD DAEMON TO SERVICE ** >> ************************************************************** >> >> >> Regrads >> Fahad Alzannan >> CTAM >> KACST >> >> >> >> *Warning: *This message and its attachment, if any, are confidential and >> may contain information protected by law. If you are not the intended >> recipient, please contact the sender immediately and delete the message and >> its attachment, if any. You should not copy the message and its attachment, >> if any, or disclose its contents to any other person or use it for any >> purpose. Statements and opinions expressed in this e-mail and its >> attachment, if any, are those of the sender, and do not necessarily reflect >> those of King Abdulaziz city for Science and Technology (KACST) in the >> Kingdom of Saudi Arabia. KACST accepts no liability for any damage caused >> by this email. >> >> *تحذير:* هذه الرسالة وما تحويه من مرفقات (إن وجدت) تمثل وثيقة سرية قد >> تحتوي على معلومات محمية بموجب القانون. إذا لم تكن الشخص المعني بهذه الرسالة >> فيجب عليك تنبيه المُرسل بخطأ وصولها إليك، وحذف الرسالة ومرفقاتها (إن وجدت)، >> ولا يجوز لك نسخ أو توزيع هذه الرسالة أو مرفقاتها (إن وجدت) أو أي جزء منها، >> أو البوح بمحتوياتها للغير أو استعمالها لأي غرض. علماً بأن فحوى هذه الرسالة >> ومرفقاتها (ان وجدت) تعبر عن رأي المُرسل وليس بالضرورة رأي مدينة الملك >> عبدالعزيز للعلوم والتقنية بالمملكة العربية السعودية، ولا تتحمل المدينة أي >> مسئولية عن الأضرار الناتجة عن ما قد يحتويه هذا البريد. >> > > *Warning: *This message and its attachment, if any, are confidential and > may contain information protected by law. If you are not the intended > recipient, please contact the sender immediately and delete the message and > its attachment, if any. You should not copy the message and its attachment, > if any, or disclose its contents to any other person or use it for any > purpose. Statements and opinions expressed in this e-mail and its > attachment, if any, are those of the sender, and do not necessarily reflect > those of King Abdulaziz city for Science and Technology (KACST) in the > Kingdom of Saudi Arabia. KACST accepts no liability for any damage caused > by this email. > > *تحذير:* هذه الرسالة وما تحويه من مرفقات (إن وجدت) تمثل وثيقة سرية قد > تحتوي على معلومات محمية بموجب القانون. إذا لم تكن الشخص المعني بهذه الرسالة > فيجب عليك تنبيه المُرسل بخطأ وصولها إليك، وحذف الرسالة ومرفقاتها (إن وجدت)، > ولا يجوز لك نسخ أو توزيع هذه الرسالة أو مرفقاتها (إن وجدت) أو أي جزء منها، > أو البوح بمحتوياتها للغير أو استعمالها لأي غرض. علماً بأن فحوى هذه الرسالة > ومرفقاتها (ان وجدت) تعبر عن رأي المُرسل وليس بالضرورة رأي مدينة الملك > عبدالعزيز للعلوم والتقنية بالمملكة العربية السعودية، ولا تتحمل المدينة أي > مسئولية عن الأضرار الناتجة عن ما قد يحتويه هذا البريد. >
