Hello Bai, On further debug, I've found the following entries in the logs:
cluster.log: cluster.log:Nov 1 16:22:34 xcat xcat[13231]: xCAT: Allowing syncfiles from <server> commands.log: commands.log:<server>: rsync: connection unexpectedly closed (0 bytes received so far) [sender] When I run "openssl s_client -no_ssl3 -no_ssl2 -connect <xcat-mn>:3001", system connects to the MN without any problems. I actually see the SSL session progress. As a side note, when I run xcatd with "xcatd -f" after stopping all xcat services, installation waits in the syncfiles state for a very long time. As I said the network is transparent, xCAT is the latest version. I'm trying my best to debug, but I'm a little lost. Thanks for all the help, Regards, Hakan On 11/01/2017 10:16 AM, Hakan Bayındır wrote: > Hello Bai, > > Sorry for the late reply. I was a bit busy and had no time to work on > the issue. I'm probably using xCAT 2.13.7 (Output of xcatconfig -v). > > xCAT MN is CentOS 7.4. Since the cluster is isolated from the internet, > MN is not running a firewall, so all ports are open. The OS we are > installing is CentOS 7.3 with packages from 1611 iso. > > I will watch and debug the process with "xcatd -f" and will provide more > info (and the solution if I can find the problem). Also will check > whether the port 3001 is accessible during install / postscript stage. > > Thanks for all the help, > > Best regards, > > Hakan > > On 10/16/2017 03:54 PM, Yuan Y Bai wrote: >> Hi Hakan, >> >> I found startsyncfiles.awk in syncfiles is return 1 in xcat.log; And >> which xcat version do you use? which OS do you use? >> >> syncfiles call startsyncfiles.awk script; you can find these 2 scripts >> in xcat MN /install/postscripts dir; >> >> syncfiles pass `USEOPENSSLFORXCAT=1 XCATSERVER=<ip of MN>:3001` to >> startsyncfiles.awk script, so I do not know if 3001 port exists in your >> xcat MN or not during the postscript stage, you can use `netstat >> -an|grep 3001` during postscript stage. Another tips, before provision >> CN, you can open one shell terminal tab to open xcatd using command >> `xcatd -f`, then you can provision CN in another shell terminal tab, >> during postscript stage, you may get some useful information from `xcatd >> -f` page. >> >> After you finish use `xcatd -f`, you can close it, and start xcat >> service backgroud using `service xcatd start` command; >> >> >> Best Regards >> -------------------------------------------------- >> Yuan Bai (白媛) >> >> CSTL HPC System Management Development >> Tel:86-10-82451401 >> E-mail: by...@cn.ibm.com >> Address: IBM ZGC Campus. Ring Building 28, >> ZhongGuanCun Software Park,No.8 Dong Bei Wang West Road, Haidian District, >> Beijing P.R.China 100193 >> >> IBM环宇大厦 >> 北京市海淀区东北旺西路8号，中关村软件园28号楼 >> 邮编:100193 >> >> >> >> ----- Original message ----- >> From: Hakan Bayındır <hakan.bayin...@tubitak.gov.tr> >> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>, Yuan >> Y Bai <by...@cn.ibm.com> >> Cc: >> Subject: Re: [xcat-user] Problem with syncfiles after install. >> Date: Fri, Oct 13, 2017 9:07 PM >> >> Hello Bai, >> >> I've enabled the debugmode and made a system provision. I'm reading the >> log right now, but I also attached relevant parts of the log to this >> e-mail. >> >> Thanks for all the help, >> >> Regards, >> >> Hakan >> >> On 10/12/2017 01:02 PM, Yuan Y Bai wrote: >> > Hi Hakan, >> > >> > Run updatenode to synchronize the files : "updatenode <nodename> >> -F -V" . >> > Other tips: `remoteshell` is running before `syncfiles` in >> postscripts, >> > `remoteshell` will start up the sshd for syncfiles postscript to >> do the >> > sync work. you can look if there is error during running remoteshell, >> > "updatenode <nodename> remoteshell" >> > >> > If you got nothing from updatenode, after open xcatdebugmode, you can >> > re-provision the node, and look into the xcat.log for all postscript >> > remoteshell/syncfiles may help. >> > >> > In `syncfiles`, it calls `startsyncfiles.awk` using openssl. >> > >> > >> > Best Regards >> > -------------------------------------------------- >> > Yuan Bai (白媛) >> > >> > CSTL HPC System Management Development >> > Tel:86-10-82451401 >> > E-mail: by...@cn.ibm.com >> > Address: IBM ZGC Campus. Ring Building 28, >> > ZhongGuanCun Software Park,No.8 Dong Bei Wang West Road, Haidian >> District, >> > Beijing P.R.China 100193 >> > >> > IBM环宇大厦 >> > 北京市海淀区东北旺西路8号，中关村软件园28号楼 >> > 邮编:100193 >> > >> > >> > >> > ----- Original message ----- >> > From: Hakan Bayındır <hakan.bayin...@tubitak.gov.tr> >> > To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>, >> Xiao >> > Peng Wang <w...@cn.ibm.com> >> > Cc: >> > Subject: Re: [xcat-user] Problem with syncfiles after install. >> > Date: Thu, Oct 12, 2017 5:32 PM >> > >> > Hello Again, >> > >> > After enabling debuglevel with the suggestion of Yuan, I >> noticed that >> > syncfiles also doesn't work after update, but with a different >> error. >> > The complete log is below. >> > >> > server1: Thu Oct 12 12:29:27 +03 2017 Running postscript: >> syncfiles >> > server1: + '[' -d /.statelite ']' >> > server1: + '[' -f /etc/os-release ']' >> > server1: + cat /etc/os-release >> > server1: + grep -i -e '^NAME=[ "'\'']*Cumulus Linux[ "'\'']*$' >> > server1: + '[' -n 1 ']' >> > server1: + '[' 1 -eq 1 ']' >> > server1: + logger -t xcat -p local4.err './syncfiles: Did not >> sync any >> > files. Use updatenode -F to sync the files.' >> > server1: + exit 0 >> > server1: postscript: syncfiles exited with code 0 >> > server1: Running of postscripts has completed. >> > >> > Regards, >> > >> > Hakan >> > >> > On 10/12/2017 11:34 AM, Hakan Bayındır wrote: >> > > Hello Wang, >> > > >> > > When I run 'updatenode <node> -P syncfiles' after automated >> > install and >> > > reboot, the process runs without any problems. The log is below. >> > > >> > > server1: xcatdsklspost: downloaded postscripts successfully >> > > server1: Thu Oct 12 11:30:16 +03 2017 Running postscript: >> syncfiles >> > > server1: postscript: syncfiles exited with code 0 >> > > server1: Running of postscripts has completed. >> > > >> > > Regards, >> > > >> > > Hakan >> > > >> > > On 10/11/2017 01:34 PM, Xiao Peng Wang wrote: >> > >> xdcp -F calls the 'rsync' to sync files to CN directly from >> MN. But >> > >> during the OS deployment, a postscript named 'syncfiles' is >> > executed in >> > >> compute node side. >> > >> >> > >> You may try 'updatenode <node> -P syncfiles'. And could you >> paste >> > the log? >> > >> >> > >> Best Regards >> > >> >> > >> ---------------------------------------------------------------------- >> > >> Wang Xiaopeng (王晓朋) >> > >> >> > >> Manager for HPC SW Dev: xCAT, ESSL, SMI, Test >> > >> IBM China Systems Laboratory (CSL) >> > >> >> > >> Tel: 86-10-82453455 >> > >> Email: w...@cn.ibm.com >> > >> >> > >> >> > >> >> > >> ----- Original message ----- >> > >> From: Hakan Bayındır <hakan.bayin...@tubitak.gov.tr> >> > >> To: xcat-user@lists.sourceforge.net >> > >> Cc: >> > >> Subject: Re: [xcat-user] Problem with syncfiles after >> install. >> > >> Date: Wed, Oct 11, 2017 2:13 PM >> > >> >> > >> Hello Wang, >> > >> >> > >> I guessed that they're different, but don't know the >> details >> > abut the >> > >> differences. I tested with xdcp anyway to be sure that my >> > syncfile is >> > >> correct. >> > >> >> > >> Is there any way to debug syncfiles further? >> > >> >> > >> Thanks in advance, >> > >> >> > >> Hakan >> > >> >> > >> On 10/10/2017 05:57 PM, Xiao Peng Wang wrote: >> > >> > One thing to be aware that using xdcp -F is different >> with the >> > >> syncfile >> > >> > process during OS deployment. >> > >> > >> > >> > Best Regards >> > >> > >> > >> ---------------------------------------------------------------------- >> > >> > Wang Xiaopeng (王晓朋) >> > >> > >> > >> > Manager for HPC SW Dev: xCAT, ESSL, SMI, Test >> > >> > IBM China Systems Laboratory (CSL) >> > >> > >> > >> > Tel: 86-10-82453455 >> > >> > Email: w...@cn.ibm.com >> > >> > >> > >> > >> > >> > >> > >> > ----- Original message ----- >> > >> > From: Russ Auld <russa...@comcast.net> >> > >> > To: xCAT Users Mailing list >> > <xcat-user@lists.sourceforge.net> >> > >> > Cc: >> > >> > Subject: Re: [xcat-user] Problem with syncfiles after >> > install. >> > >> > Date: Tue, Oct 10, 2017 7:17 PM >> > >> > >> > >> > Typically this is indicative of a problem with ssh >> > >> connectivity between >> > >> > the compute node and the master/service node. >> > >> > Ensure that DNS is also working correctly. >> > >> > >> > >> > On Mon, 2017-10-09 at 19:00 +0300, Hakan Bayındır >> wrote: >> > >> > > Hello All, >> > >> > > >> > >> > > I'm having a problem with the default syncfiles >> > script, which is >> > >> > > fired by default as a "postscript" after an >> > installation. The >> > >> > > installation completes as it should be and the >> system >> > reboots, >> > >> > > however I get an error in the xcat.log which states >> > that the >> > >> > > syncfiles have returned 1, hence failed. The >> strange >> > thing >> > >> is, when I >> > >> > > fire the same syncfile with xdcp, everything >> works as it >> > >> should. Did >> > >> > > anyone had this error and nudge me in the right >> > direction? >> > >> > > >> > >> > > Best regards, >> > >> > > >> > >> > > Hakan >> > >> > > >> > >> > > Hakan BAYINDIR >> > >> > > Uzman Araştırmacı >> > >> > > Ağ Teknolojileri Birimi >> > >> > > YÖK Binası B-5 Blok Kat:4 >> > >> > > 06539 Bilkent ANKARA >> > >> > > T +90 312 298 9373 >> > >> > > F +90 312 266 5181 >> > >> > > [ >> > >> > >> > >> >> > >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ulakbim.gov.tr_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=J6K2ffHm0Jm29fFeOyQp-hggkr_YHQfohHuRVUtCy3Q&e= >> | >> > >> > www.ulakbim.gov.tr ] >> > >> > > hakan.bayin...@tubitak.gov.tr >> > >> > > >> > >> >> > >> ..................................................................... >> > >> > > >> > ........................................................... >> > >> > > >> > >> > > [ >> > >> > >> > >> >> > >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ulakbim.gov.tr_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=J6K2ffHm0Jm29fFeOyQp-hggkr_YHQfohHuRVUtCy3Q&e= >> ] >> > >> > > [ >> > >> > >> > >> >> > >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tubitak.gov.tr_sorumlulukreddi&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=X1PvO5UA5EIZma9wQQ-7Daf9LWCzEzRNp3VLqIMBqc4&e= >> | >> > >> > > Sorumluluk Reddi ] >> > >> > > >> > >> > > >> > >> >> > >> ------------------------------------------------------------------- >> > >> > > ----------- >> > >> > > Check out the vibrant tech community on one of the >> > world's most >> > >> > > engaging tech sites, Slashdot.org! >> > >> > >> > >> >> > >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__sdm.link_slashdot&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=GdwB33kriDHKDlBuKHx051_0AI_Rd4Ifm89G-ou6FUU&e= >> > >> > > _______________________________________________ >> > >> > > xCAT-user mailing list >> > >> > > xCAT-user@lists.sourceforge.net >> > >> > > >> > >> > >> > >> >> > >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_xcat-2Duser&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=3xM66Ed2EAjR4sHpz4DtDUanrCtY2mygM6_P130WwPY&e= >> > >> > >> > >> > >> > >> >> > >> >> ------------------------------------------------------------------------------ >> > >> > Check out the vibrant tech community on one of the >> > world's most >> > >> > engaging tech sites, Slashdot.org! >> > >> > >> > >> >> > >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__sdm.link_slashdot&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=GdwB33kriDHKDlBuKHx051_0AI_Rd4Ifm89G-ou6FUU&e= >> > >> > _______________________________________________ >> > >> > xCAT-user mailing list >> > >> > xCAT-user@lists.sourceforge.net >> > >> > >> > >> >> > >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_xcat-2Duser&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=3xM66Ed2EAjR4sHpz4DtDUanrCtY2mygM6_P130WwPY&e= >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >> > >> >> ------------------------------------------------------------------------------ >> > >> > Check out the vibrant tech community on one of the >> world's most >> > >> > engaging tech sites, Slashdot.org! >> http://sdm.link/slashdot >> > >> > >> > >> > >> > >> > >> > >> > _______________________________________________ >> > >> > xCAT-user mailing list >> > >> > xCAT-user@lists.sourceforge.net >> > >> > https://lists.sourceforge.net/lists/listinfo/xcat-user >> > >> > >> > >> >> > >> -- >> > >> *Hakan BAYINDIR* >> > >> Başuzman Araştırmacı >> > >> Ağ Teknolojileri Birimi >> > >> TÜBİTAK ULAKBİM >> > >> T.C. Bilim, Sanayi ve Teknoloji Bakanlığı (Eski Bina) >> > >> Mustafa Kemal Mahallesi Dumlupınar Bulvarı >> > >> (Eskişehir Yolu 7.Km) 2151.Cadde No:154 >> > >> ODTÜ Karşısı >> > >> 06510 Çankaya, ANKARA >> > >> T +90 312 298 9373 >> > >> F +90 312 266 5181 >> > >> www.ulakbim.gov.tr <http://www.ulakbim.gov.tr> >> > >> hakan.bayin...@tubitak.gov.tr >> > >> >> > >> >> ................................................................................................................................ >> > >> >> > >> <http://www.tubitak.gov.tr> >> > >> >> > >> Sorumluluk Reddi >> <http://www.tubitak.gov.tr/sorumlulukreddi> >> > >> >> > >> >> > >> >> > >> >> ------------------------------------------------------------------------------ >> > >> Check out the vibrant tech community on one of the >> world's most >> > >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> > >> _______________________________________________ >> > >> xCAT-user mailing list >> > >> xCAT-user@lists.sourceforge.net >> > >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> ------------------------------------------------------------------------------ >> > >> Check out the vibrant tech community on one of the world's most >> > >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> > >> >> > >> >> > >> >> > >> _______________________________________________ >> > >> xCAT-user mailing list >> > >> xCAT-user@lists.sourceforge.net >> > >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> > >> >> > > >> > > >> > > >> > > >> > >> >> ------------------------------------------------------------------------------ >> > > Check out the vibrant tech community on one of the world's most >> > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> > > >> > > >> > > >> > > _______________________________________________ >> > > xCAT-user mailing list >> > > xCAT-user@lists.sourceforge.net >> > > https://lists.sourceforge.net/lists/listinfo/xcat-user >> > > >> > >> > -- >> > *Hakan BAYINDIR* >> > Başuzman Araştırmacı >> > Ağ Teknolojileri Birimi >> > TÜBİTAK ULAKBİM >> > T.C. Bilim, Sanayi ve Teknoloji Bakanlığı (Eski Bina) >> > Mustafa Kemal Mahallesi Dumlupınar Bulvarı >> > (Eskişehir Yolu 7.Km) 2151.Cadde No:154 >> > ODTÜ Karşısı >> > 06510 Çankaya, ANKARA >> > T +90 312 298 9373 >> > F +90 312 266 5181 >> > www.ulakbim.gov.tr <http://www.ulakbim.gov.tr> >> > hakan.bayin...@tubitak.gov.tr >> > >> >> ................................................................................................................................ >> > >> > <http://www.tubitak.gov.tr> >> > >> > Sorumluluk Reddi <http://www.tubitak.gov.tr/sorumlulukreddi> >> > >> > >> > >> >> ------------------------------------------------------------------------------ >> > Check out the vibrant tech community on one of the world's most >> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> > _______________________________________________ >> > xCAT-user mailing list >> > xCAT-user@lists.sourceforge.net >> > https://lists.sourceforge.net/lists/listinfo/xcat-user >> > >> > >> > >> > >> > >> > >> >> ------------------------------------------------------------------------------ >> > Check out the vibrant tech community on one of the world's most >> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> > >> > >> > >> > _______________________________________________ >> > xCAT-user mailing list >> > xCAT-user@lists.sourceforge.net >> > https://lists.sourceforge.net/lists/listinfo/xcat-user >> > >> >> -- >> *Hakan BAYINDIR* >> Başuzman Araştırmacı >> Ağ Teknolojileri Birimi >> TÜBİTAK ULAKBİM >> T.C. Bilim, Sanayi ve Teknoloji Bakanlığı (Eski Bina) >> Mustafa Kemal Mahallesi Dumlupınar Bulvarı >> (Eskişehir Yolu 7.Km) 2151.Cadde No:154 >> ODTÜ Karşısı >> 06510 Çankaya, ANKARA >> T +90 312 298 9373 >> F +90 312 266 5181 >> www.ulakbim.gov.tr <http://www.ulakbim.gov.tr> >> hakan.bayin...@tubitak.gov.tr >> >> ................................................................................................................................ >> >> <http://www.tubitak.gov.tr> >> >> Sorumluluk Reddi <http://www.tubitak.gov.tr/sorumlulukreddi> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> >> >> >> _______________________________________________ >> xCAT-user mailing list >> xCAT-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > _______________________________________________ > xCAT-user mailing list > xCAT-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xcat-user > -- *Hakan BAYINDIR* Başuzman Araştırmacı Ağ Teknolojileri Birimi TÜBİTAK ULAKBİM T.C. Bilim, Sanayi ve Teknoloji Bakanlığı (Eski Bina) Mustafa Kemal Mahallesi Dumlupınar Bulvarı (Eskişehir Yolu 7.Km) 2151.Cadde No:154 ODTÜ Karşısı 06510 Çankaya, ANKARA T +90 312 298 9373 F +90 312 266 5181 www.ulakbim.gov.tr <http://www.ulakbim.gov.tr> hakan.bayin...@tubitak.gov.tr ................................................................................................................................ <http://www.tubitak.gov.tr> Sorumluluk Reddi <http://www.tubitak.gov.tr/sorumlulukreddi>
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
