Hello All, I've understood the problem and created a workaround for it. It seems like xCAT tries to rsync into the host during the postinstall stage. If it needs to connect with another method, it doesn't do because of reasons I don't know, but since during installation the OS is "locked down" with "nologin", rsync cannot connect to the host during post-install stage.
To solve the problem, I've moved syncfiles as the first script for "postbootscripts" stage. The script is working now. We're currently in a big update stage and will re-install nearly all of our nodes, and I have some time for further debugging the problem, if anyone is interested. Regards, Hakan On 11/01/2017 04:44 PM, Hakan Bayındır wrote: > Hello Bai, > > On further debug, I've found the following entries in the logs: > > cluster.log: cluster.log:Nov 1 16:22:34 xcat xcat[13231]: xCAT: > Allowing syncfiles from <server> > > commands.log: commands.log:<server>: rsync: connection unexpectedly > closed (0 bytes received so far) [sender] > > When I run "openssl s_client -no_ssl3 -no_ssl2 -connect <xcat-mn>:3001", > system connects to the MN without any problems. I actually see the SSL > session progress. > > As a side note, when I run xcatd with "xcatd -f" after stopping all xcat > services, installation waits in the syncfiles state for a very long time. > > As I said the network is transparent, xCAT is the latest version. I'm > trying my best to debug, but I'm a little lost. > > Thanks for all the help, > > Regards, > > Hakan > > On 11/01/2017 10:16 AM, Hakan Bayındır wrote: >> Hello Bai, >> >> Sorry for the late reply. I was a bit busy and had no time to work on >> the issue. I'm probably using xCAT 2.13.7 (Output of xcatconfig -v). >> >> xCAT MN is CentOS 7.4. Since the cluster is isolated from the internet, >> MN is not running a firewall, so all ports are open. The OS we are >> installing is CentOS 7.3 with packages from 1611 iso. >> >> I will watch and debug the process with "xcatd -f" and will provide more >> info (and the solution if I can find the problem). Also will check >> whether the port 3001 is accessible during install / postscript stage. >> >> Thanks for all the help, >> >> Best regards, >> >> Hakan >> >> On 10/16/2017 03:54 PM, Yuan Y Bai wrote: >>> Hi Hakan, >>> >>> I found startsyncfiles.awk in syncfiles is return 1 in xcat.log; And >>> which xcat version do you use? which OS do you use? >>> >>> syncfiles call startsyncfiles.awk script; you can find these 2 scripts >>> in xcat MN /install/postscripts dir; >>> >>> syncfiles pass `USEOPENSSLFORXCAT=1 XCATSERVER=<ip of MN>:3001` to >>> startsyncfiles.awk script, so I do not know if 3001 port exists in your >>> xcat MN or not during the postscript stage, you can use `netstat >>> -an|grep 3001` during postscript stage. Another tips, before provision >>> CN, you can open one shell terminal tab to open xcatd using command >>> `xcatd -f`, then you can provision CN in another shell terminal tab, >>> during postscript stage, you may get some useful information from `xcatd >>> -f` page. >>> >>> After you finish use `xcatd -f`, you can close it, and start xcat >>> service backgroud using `service xcatd start` command; >>> >>> >>> Best Regards >>> -------------------------------------------------- >>> Yuan Bai (白媛) >>> >>> CSTL HPC System Management Development >>> Tel:86-10-82451401 >>> E-mail: by...@cn.ibm.com >>> Address: IBM ZGC Campus. Ring Building 28, >>> ZhongGuanCun Software Park,No.8 Dong Bei Wang West Road, Haidian District, >>> Beijing P.R.China 100193 >>> >>> IBM环宇大厦 >>> 北京市海淀区东北旺西路8号，中关村软件园28号楼 >>> 邮编:100193 >>> >>> >>> >>> ----- Original message ----- >>> From: Hakan Bayındır <hakan.bayin...@tubitak.gov.tr> >>> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>, Yuan >>> Y Bai <by...@cn.ibm.com> >>> Cc: >>> Subject: Re: [xcat-user] Problem with syncfiles after install. >>> Date: Fri, Oct 13, 2017 9:07 PM >>> >>> Hello Bai, >>> >>> I've enabled the debugmode and made a system provision. I'm reading the >>> log right now, but I also attached relevant parts of the log to this >>> e-mail. >>> >>> Thanks for all the help, >>> >>> Regards, >>> >>> Hakan >>> >>> On 10/12/2017 01:02 PM, Yuan Y Bai wrote: >>> > Hi Hakan, >>> > >>> > Run updatenode to synchronize the files : "updatenode <nodename> >>> -F -V" . >>> > Other tips: `remoteshell` is running before `syncfiles` in >>> postscripts, >>> > `remoteshell` will start up the sshd for syncfiles postscript to >>> do the >>> > sync work. you can look if there is error during running remoteshell, >>> > "updatenode <nodename> remoteshell" >>> > >>> > If you got nothing from updatenode, after open xcatdebugmode, you can >>> > re-provision the node, and look into the xcat.log for all postscript >>> > remoteshell/syncfiles may help. >>> > >>> > In `syncfiles`, it calls `startsyncfiles.awk` using openssl. >>> > >>> > >>> > Best Regards >>> > -------------------------------------------------- >>> > Yuan Bai (白媛) >>> > >>> > CSTL HPC System Management Development >>> > Tel:86-10-82451401 >>> > E-mail: by...@cn.ibm.com >>> > Address: IBM ZGC Campus. Ring Building 28, >>> > ZhongGuanCun Software Park,No.8 Dong Bei Wang West Road, Haidian >>> District, >>> > Beijing P.R.China 100193 >>> > >>> > IBM环宇大厦 >>> > 北京市海淀区东北旺西路8号，中关村软件园28号楼 >>> > 邮编:100193 >>> > >>> > >>> > >>> > ----- Original message ----- >>> > From: Hakan Bayındır <hakan.bayin...@tubitak.gov.tr> >>> > To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>, >>> Xiao >>> > Peng Wang <w...@cn.ibm.com> >>> > Cc: >>> > Subject: Re: [xcat-user] Problem with syncfiles after install. >>> > Date: Thu, Oct 12, 2017 5:32 PM >>> > >>> > Hello Again, >>> > >>> > After enabling debuglevel with the suggestion of Yuan, I >>> noticed that >>> > syncfiles also doesn't work after update, but with a different >>> error. >>> > The complete log is below. >>> > >>> > server1: Thu Oct 12 12:29:27 +03 2017 Running postscript: >>> syncfiles >>> > server1: + '[' -d /.statelite ']' >>> > server1: + '[' -f /etc/os-release ']' >>> > server1: + cat /etc/os-release >>> > server1: + grep -i -e '^NAME=[ "'\'']*Cumulus Linux[ "'\'']*$' >>> > server1: + '[' -n 1 ']' >>> > server1: + '[' 1 -eq 1 ']' >>> > server1: + logger -t xcat -p local4.err './syncfiles: Did not >>> sync any >>> > files. Use updatenode -F to sync the files.' >>> > server1: + exit 0 >>> > server1: postscript: syncfiles exited with code 0 >>> > server1: Running of postscripts has completed. >>> > >>> > Regards, >>> > >>> > Hakan >>> > >>> > On 10/12/2017 11:34 AM, Hakan Bayındır wrote: >>> > > Hello Wang, >>> > > >>> > > When I run 'updatenode <node> -P syncfiles' after automated >>> > install and >>> > > reboot, the process runs without any problems. The log is below. >>> > > >>> > > server1: xcatdsklspost: downloaded postscripts successfully >>> > > server1: Thu Oct 12 11:30:16 +03 2017 Running postscript: >>> syncfiles >>> > > server1: postscript: syncfiles exited with code 0 >>> > > server1: Running of postscripts has completed. >>> > > >>> > > Regards, >>> > > >>> > > Hakan >>> > > >>> > > On 10/11/2017 01:34 PM, Xiao Peng Wang wrote: >>> > >> xdcp -F calls the 'rsync' to sync files to CN directly from >>> MN. But >>> > >> during the OS deployment, a postscript named 'syncfiles' is >>> > executed in >>> > >> compute node side. >>> > >> >>> > >> You may try 'updatenode <node> -P syncfiles'. And could you >>> paste >>> > the log? >>> > >> >>> > >> Best Regards >>> > >> >>> > >>> ---------------------------------------------------------------------- >>> > >> Wang Xiaopeng (王晓朋) >>> > >> >>> > >> Manager for HPC SW Dev: xCAT, ESSL, SMI, Test >>> > >> IBM China Systems Laboratory (CSL) >>> > >> >>> > >> Tel: 86-10-82453455 >>> > >> Email: w...@cn.ibm.com >>> > >> >>> > >> >>> > >> >>> > >> ----- Original message ----- >>> > >> From: Hakan Bayındır <hakan.bayin...@tubitak.gov.tr> >>> > >> To: xcat-user@lists.sourceforge.net >>> > >> Cc: >>> > >> Subject: Re: [xcat-user] Problem with syncfiles after >>> install. >>> > >> Date: Wed, Oct 11, 2017 2:13 PM >>> > >> >>> > >> Hello Wang, >>> > >> >>> > >> I guessed that they're different, but don't know the >>> details >>> > abut the >>> > >> differences. I tested with xdcp anyway to be sure that my >>> > syncfile is >>> > >> correct. >>> > >> >>> > >> Is there any way to debug syncfiles further? >>> > >> >>> > >> Thanks in advance, >>> > >> >>> > >> Hakan >>> > >> >>> > >> On 10/10/2017 05:57 PM, Xiao Peng Wang wrote: >>> > >> > One thing to be aware that using xdcp -F is different >>> with the >>> > >> syncfile >>> > >> > process during OS deployment. >>> > >> > >>> > >> > Best Regards >>> > >> > >>> > >>> ---------------------------------------------------------------------- >>> > >> > Wang Xiaopeng (王晓朋) >>> > >> > >>> > >> > Manager for HPC SW Dev: xCAT, ESSL, SMI, Test >>> > >> > IBM China Systems Laboratory (CSL) >>> > >> > >>> > >> > Tel: 86-10-82453455 >>> > >> > Email: w...@cn.ibm.com >>> > >> > >>> > >> > >>> > >> > >>> > >> > ----- Original message ----- >>> > >> > From: Russ Auld <russa...@comcast.net> >>> > >> > To: xCAT Users Mailing list >>> > <xcat-user@lists.sourceforge.net> >>> > >> > Cc: >>> > >> > Subject: Re: [xcat-user] Problem with syncfiles after >>> > install. >>> > >> > Date: Tue, Oct 10, 2017 7:17 PM >>> > >> > >>> > >> > Typically this is indicative of a problem with ssh >>> > >> connectivity between >>> > >> > the compute node and the master/service node. >>> > >> > Ensure that DNS is also working correctly. >>> > >> > >>> > >> > On Mon, 2017-10-09 at 19:00 +0300, Hakan Bayındır >>> wrote: >>> > >> > > Hello All, >>> > >> > > >>> > >> > > I'm having a problem with the default syncfiles >>> > script, which is >>> > >> > > fired by default as a "postscript" after an >>> > installation. The >>> > >> > > installation completes as it should be and the >>> system >>> > reboots, >>> > >> > > however I get an error in the xcat.log which states >>> > that the >>> > >> > > syncfiles have returned 1, hence failed. The >>> strange >>> > thing >>> > >> is, when I >>> > >> > > fire the same syncfile with xdcp, everything >>> works as it >>> > >> should. Did >>> > >> > > anyone had this error and nudge me in the right >>> > direction? >>> > >> > > >>> > >> > > Best regards, >>> > >> > > >>> > >> > > Hakan >>> > >> > > >>> > >> > > Hakan BAYINDIR >>> > >> > > Uzman Araştırmacı >>> > >> > > Ağ Teknolojileri Birimi >>> > >> > > YÖK Binası B-5 Blok Kat:4 >>> > >> > > 06539 Bilkent ANKARA >>> > >> > > T +90 312 298 9373 >>> > >> > > F +90 312 266 5181 >>> > >> > > [ >>> > >> > >>> > >> >>> > >>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ulakbim.gov.tr_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=J6K2ffHm0Jm29fFeOyQp-hggkr_YHQfohHuRVUtCy3Q&e= >>> | >>> > >> > www.ulakbim.gov.tr ] >>> > >> > > hakan.bayin...@tubitak.gov.tr >>> > >> > > >>> > >> >>> > >>> ..................................................................... >>> > >> > > >>> > ........................................................... >>> > >> > > >>> > >> > > [ >>> > >> > >>> > >> >>> > >>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ulakbim.gov.tr_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=J6K2ffHm0Jm29fFeOyQp-hggkr_YHQfohHuRVUtCy3Q&e= >>> ] >>> > >> > > [ >>> > >> > >>> > >> >>> > >>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tubitak.gov.tr_sorumlulukreddi&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=X1PvO5UA5EIZma9wQQ-7Daf9LWCzEzRNp3VLqIMBqc4&e= >>> | >>> > >> > > Sorumluluk Reddi ] >>> > >> > > >>> > >> > > >>> > >> >>> > >>> ------------------------------------------------------------------- >>> > >> > > ----------- >>> > >> > > Check out the vibrant tech community on one of the >>> > world's most >>> > >> > > engaging tech sites, Slashdot.org! >>> > >> > >>> > >> >>> > >>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__sdm.link_slashdot&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=GdwB33kriDHKDlBuKHx051_0AI_Rd4Ifm89G-ou6FUU&e= >>> > >> > > _______________________________________________ >>> > >> > > xCAT-user mailing list >>> > >> > > xCAT-user@lists.sourceforge.net >>> > >> > > >>> > >> > >>> > >> >>> > >>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_xcat-2Duser&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=3xM66Ed2EAjR4sHpz4DtDUanrCtY2mygM6_P130WwPY&e= >>> > >> > >>> > >> > >>> > >> >>> > >>> >>> ------------------------------------------------------------------------------ >>> > >> > Check out the vibrant tech community on one of the >>> > world's most >>> > >> > engaging tech sites, Slashdot.org! >>> > >> > >>> > >> >>> > >>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__sdm.link_slashdot&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=GdwB33kriDHKDlBuKHx051_0AI_Rd4Ifm89G-ou6FUU&e= >>> > >> > _______________________________________________ >>> > >> > xCAT-user mailing list >>> > >> > xCAT-user@lists.sourceforge.net >>> > >> > >>> > >> >>> > >>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_xcat-2Duser&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=178VMrolfAlMh59RUGWQ4w&m=l7Ul6dv91Ff5_6PZBSWmfgvE6bVWX6QQzOXqenGV4jE&s=3xM66Ed2EAjR4sHpz4DtDUanrCtY2mygM6_P130WwPY&e= >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> >>> > >>> >>> ------------------------------------------------------------------------------ >>> > >> > Check out the vibrant tech community on one of the >>> world's most >>> > >> > engaging tech sites, Slashdot.org! >>> http://sdm.link/slashdot >>> > >> > >>> > >> > >>> > >> > >>> > >> > _______________________________________________ >>> > >> > xCAT-user mailing list >>> > >> > xCAT-user@lists.sourceforge.net >>> > >> > https://lists.sourceforge.net/lists/listinfo/xcat-user >>> > >> > >>> > >> >>> > >> -- >>> > >> *Hakan BAYINDIR* >>> > >> Başuzman Araştırmacı >>> > >> Ağ Teknolojileri Birimi >>> > >> TÜBİTAK ULAKBİM >>> > >> T.C. Bilim, Sanayi ve Teknoloji Bakanlığı (Eski Bina) >>> > >> Mustafa Kemal Mahallesi Dumlupınar Bulvarı >>> > >> (Eskişehir Yolu 7.Km) 2151.Cadde No:154 >>> > >> ODTÜ Karşısı >>> > >> 06510 Çankaya, ANKARA >>> > >> T +90 312 298 9373 >>> > >> F +90 312 266 5181 >>> > >> www.ulakbim.gov.tr <http://www.ulakbim.gov.tr> >>> > >> hakan.bayin...@tubitak.gov.tr >>> > >> >>> > >>> >>> ................................................................................................................................ >>> > >> >>> > >> <http://www.tubitak.gov.tr> >>> > >> >>> > >> Sorumluluk Reddi >>> <http://www.tubitak.gov.tr/sorumlulukreddi> >>> > >> >>> > >> >>> > >> >>> > >>> >>> ------------------------------------------------------------------------------ >>> > >> Check out the vibrant tech community on one of the >>> world's most >>> > >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>> > >> _______________________________________________ >>> > >> xCAT-user mailing list >>> > >> xCAT-user@lists.sourceforge.net >>> > >> https://lists.sourceforge.net/lists/listinfo/xcat-user >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >>> >>> ------------------------------------------------------------------------------ >>> > >> Check out the vibrant tech community on one of the world's most >>> > >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>> > >> >>> > >> >>> > >> >>> > >> _______________________________________________ >>> > >> xCAT-user mailing list >>> > >> xCAT-user@lists.sourceforge.net >>> > >> https://lists.sourceforge.net/lists/listinfo/xcat-user >>> > >> >>> > > >>> > > >>> > > >>> > > >>> > >>> >>> ------------------------------------------------------------------------------ >>> > > Check out the vibrant tech community on one of the world's most >>> > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>> > > >>> > > >>> > > >>> > > _______________________________________________ >>> > > xCAT-user mailing list >>> > > xCAT-user@lists.sourceforge.net >>> > > https://lists.sourceforge.net/lists/listinfo/xcat-user >>> > > >>> > >>> > -- >>> > *Hakan BAYINDIR* >>> > Başuzman Araştırmacı >>> > Ağ Teknolojileri Birimi >>> > TÜBİTAK ULAKBİM >>> > T.C. Bilim, Sanayi ve Teknoloji Bakanlığı (Eski Bina) >>> > Mustafa Kemal Mahallesi Dumlupınar Bulvarı >>> > (Eskişehir Yolu 7.Km) 2151.Cadde No:154 >>> > ODTÜ Karşısı >>> > 06510 Çankaya, ANKARA >>> > T +90 312 298 9373 >>> > F +90 312 266 5181 >>> > www.ulakbim.gov.tr <http://www.ulakbim.gov.tr> >>> > hakan.bayin...@tubitak.gov.tr >>> > >>> >>> ................................................................................................................................ >>> > >>> > <http://www.tubitak.gov.tr> >>> > >>> > Sorumluluk Reddi <http://www.tubitak.gov.tr/sorumlulukreddi> >>> > >>> > >>> > >>> >>> ------------------------------------------------------------------------------ >>> > Check out the vibrant tech community on one of the world's most >>> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>> > _______________________________________________ >>> > xCAT-user mailing list >>> > xCAT-user@lists.sourceforge.net >>> > https://lists.sourceforge.net/lists/listinfo/xcat-user >>> > >>> > >>> > >>> > >>> > >>> > >>> >>> ------------------------------------------------------------------------------ >>> > Check out the vibrant tech community on one of the world's most >>> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>> > >>> > >>> > >>> > _______________________________________________ >>> > xCAT-user mailing list >>> > xCAT-user@lists.sourceforge.net >>> > https://lists.sourceforge.net/lists/listinfo/xcat-user >>> > >>> >>> -- >>> *Hakan BAYINDIR* >>> Başuzman Araştırmacı >>> Ağ Teknolojileri Birimi >>> TÜBİTAK ULAKBİM >>> T.C. Bilim, Sanayi ve Teknoloji Bakanlığı (Eski Bina) >>> Mustafa Kemal Mahallesi Dumlupınar Bulvarı >>> (Eskişehir Yolu 7.Km) 2151.Cadde No:154 >>> ODTÜ Karşısı >>> 06510 Çankaya, ANKARA >>> T +90 312 298 9373 >>> F +90 312 266 5181 >>> www.ulakbim.gov.tr <http://www.ulakbim.gov.tr> >>> hakan.bayin...@tubitak.gov.tr >>> >>> ................................................................................................................................ >>> >>> <http://www.tubitak.gov.tr> >>> >>> Sorumluluk Reddi <http://www.tubitak.gov.tr/sorumlulukreddi> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Check out the vibrant tech community on one of the world's most >>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>> >>> >>> >>> _______________________________________________ >>> xCAT-user mailing list >>> xCAT-user@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>> >> >> >> >> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> >> >> >> _______________________________________________ >> xCAT-user mailing list >> xCAT-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > _______________________________________________ > xCAT-user mailing list > xCAT-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xcat-user >
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
