Well, I should have looked in the logs first. There were more detailed
messages in /var/log/messages on the MN:
Jun 16 14:10:14 xcat-master xcat[30550]: Error dispatching request to
xcat-serv1:3001, trying other service nodes: Connection failure: SSL
connect attempt failed because of handshake problems error:14094418:SSL
routines:SSL3_READ_BYTES:tlsv1 alert unknown ca at
/opt/xcat/lib/perl/xCAT/Client.pm line 265.
Jun 16 14:10:15 xcat-master xcat[30550]: Error dispatching request to
xcat-serv2:3001, trying other service nodes: Connection failure: SSL
connect attempt failed because of handshake problems error:14094418:SSL
routines:SSL3_READ_BYTES:tlsv1 alert unknown ca at
/opt/xcat/lib/perl/xCAT/Client.pm line 265.
Which SSL cert or key is involved in this connection? Although I copied
over the rsa keys in /root/.ssh from the old MN to the new one I did not do
the same for either /etc/xcat/cert/ or /etc/ssh/. Might a missing key or
cert from either of those directories be responsible for that error?
Thanks,
Josh
On Thu, Jun 16, 2016 at 2:23 PM, Josh Nielsen <jniel...@hudsonalpha.org>
wrote:
> Xiao,
>
> Okay, so I followed those four steps with some modifications. I did 1 & 4
> as instructed with no issues. The service nodes are getting their database
> access from the new MN now, and I updated the SN object definitions to
> point xcatmaster, tftpserver, and other relevant parameters to the new MN.
>
> I avoided step #3 because I just copied the old /root/.ssh/id_rsa and
> corresponding .pub file to the new MN and passwordless logon works fine. I
> also tested this from the two service nodes to make sure they could fetch
> the host keys: "USEOPENSSLFORXCAT=yes XCATSERVER=<MN_IP>:3001
> /xcatpost/getcredentials.awk ssh_rsa_hostkey. Is that sufficient for the
> key step?
>
> And lastly for #3 I only selectively updated certain packages on the SNs
> like syslog and NTP, because I didn't want to run all of the packages and
> in particular the servicenode postscript.
>
> So, I was able to use updatenode with no issues from the new MN to update
> the SNs, however when I try to update any cluster client nodes it is having
> problems dispatching to the service nodes in the hierarchy:
>
>
> # updatenode node0010 -P addsiteyum
> Error: Failed to dispatch command to any of the following service nodes:
> xcat-serv1,xcat-serv2
>
> What is most likely causing that issue?
>
> Thanks,
> Josh
>
> On Fri, Jun 3, 2016 at 7:01 AM, Xiao Peng Wang <w...@cn.ibm.com> wrote:
>
>> I think we should talk it as opposite way that how to make the MN to use
>> the new SN.
>>
>> Following steps are necessary to switch a SN:
>> 1. rerun 'mysqlsetup -f' to assign the access permission for SN to access
>> DB on MN
>> 2. run 'updatenode -k <sn>' to set up the ssh key
>> 3. run 'updatenode -P' to update the SN
>> 4. change the 'servicenode' attribute for compute node accordingly.
>>
>>
>> Thanks
>> Best Regards
>> ----------------------------------------------------------------------
>> Wang Xiaopeng (王晓朋)
>> IBM China System Technology Laboratory
>> Tel: 86-10-82453455
>> Email: w...@cn.ibm.com
>> Address: 28,ZhongGuanCun Software Park,No.8 Dong Bei Wang West Road,
>> Haidian District Beijing P.R.China 100193
>>
>>
>>
>> ----- Original message -----
>> From: Josh Nielsen <jniel...@hudsonalpha.org>
>> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
>> Cc:
>> Subject: Re: [xcat-user] How can I migrate to a new xCAT MN in a
>> hierarchical environment?
>> Date: Thu, Jun 2, 2016 3:49 AM
>>
>> Can anyone verify if simply updating cfgloc should be all I need to for
>> the SNs to start using the new MN? By pointing it to the new MN's MySQL
>> instance, which has a site table with the new MN specified as the
>> xcatmaster, it should even update the content the the xcatmaster value
>> shown in an 'lsdef' of the service nodes automatically, right?
>>
>> Thanks,
>> Josh
>>
>> On Tue, May 17, 2016 at 3:42 PM, Josh Nielsen <jniel...@hudsonalpha.org>
>> wrote:
>>
>> A correction below for something I wrote previously.
>>
>> "...and the SNs then shouldn't need newly generated keys (right?)..."
>>
>> On Tue, May 17, 2016 at 3:36 PM, Josh Nielsen <jniel...@hudsonalpha.org>
>> wrote:
>>
>> I looked at the 'servicenode' postscript and it does _way_ too much for
>> what I want to accomplish. I don't think the script was written with
>> changes or upgrades in mind. It looks like it freshly copies everything to
>> the SNs' $installdir/postscripts and /etc/xcat on the service node and
>> generates (new?) keys. The SNs don't need those updates/changes in my case.
>> From looking at the following comment in the 'servicenode' postscript and
>> the code I'm wondering if all I need to do is manually
>> modify /etc/xcat/cfgloc to update the IP for the new MN database location
>> and if everything else will be fine. They keys should already be in place
>> because I am copying the same keys from the old MN onto the new MN server,
>> and the SNs then shouldn't need to keys (right?). Please let me know if you
>> see any problems with this.
>>
>> The comment in the code:
>>
>> For Linux:
>> It calls xcatserver and xcatclient script to get the ssh keys, ssl
>> redentials and cfgloc file and transfer from the MN to the SN
>> to be able to access the
>> database, setup ssh keys on the nodes and have daemon to daemon
>> commmunication between the SN and MN and have the SN access the DB.
>>
>>
>> P.S. Also would just giving the new MN the same IP and hostname (even as
>> an alias to a different primary hostname) more or less prevent any changes
>> from needing to be made on the SNs at all (no postscripts run nor manual
>> modifications of files)?
>>
>> Thanks,
>> Josh
>>
>> On Thu, May 5, 2016 at 11:42 AM, Josh Nielsen <jniel...@hudsonalpha.org>
>> wrote:
>>
>> Hi Christian,
>>
>> Thanks for the response. So do I actually have to reinstall the SNs
>> and/or rerun the service node postscript? If reruning the SN post script
>> just makes some minor adjustments but doesn't clear the dhcpd.leases and
>> the .conf files for named and dhcp, as I have them configured, then that
>> would be fine, but if it blows all that away and starts over that would
>> qualify as disruptive for my environment since the cluster depends on slave
>> DNS services and dhcp on the SN. I would ideally like minimal changes on
>> the SNs except to point them to the new MN.
>>
>> As far as the postscripts, my question was what common (if not default in
>> most installs) postscripts that come with xCAT have code in them that would
>> result in the hardcoding of the MN's IP in some configuration file. I
>> actually thought of one possible example along those lines, and that is
>> whatever configures the client compute nodes to send all their syslog
>> messages to the /var/log/messages log on the headnode instead of locally
>> will need to be rerun/updated. What will need to be run to change that to
>> make the clients log to the new MN server?
>>
>> Regarding the server identity (even though it will have a new IP address
>> and hostname) can we just copy the keys in /etc/ssh/ to the new MN so that
>> the SSH fingerprint doesn't change?
>>
>> Lastly, as regards running updatenode -k I definitely (in this case) do
>> not want to replace the root rsa_id private and public keys on the cluster,
>> the MN, or the SNs since other critical services like GPFS require the
>> current keys to remain in place. Why is rerunning the key deploy necessary
>> and is there not a way to make it work with the current keys?
>>
>> I just need to be very careful with my current setup so that I don't
>> knock out critical services while changing the MN, which is why I was
>> wondering how disruptive doing this might be. I appreciate the help!
>>
>> Thanks,
>> Josh
>>
>> On Tue, May 3, 2016 at 10:05 AM, Christian Caruthers <
>> ccaruth...@lenovo.com> wrote:
>>
>> I would begin by looking at the servicenode postscript. It sets up the
>> daemon and database communications between SN & MN. Beyond that, the
>> default postscripts are listed in the "xcatdefaults" entry of the
>> postscripts table. You will probably want to run updatenode -k once you
>> have xCAT configured on the new MN. After that, you probably want to rerun
>> the remoteshell and syslog postscripts on the cluster members (updatenode
>> -P) at the very least.
>>
>>
>>
>> Second, you can dump the xCAT DB using dumpxCATdb command. After that,
>> grep out the management node (hostname and/or IP) to see where changes need
>> to be made for the DB on the new MN.
>>
>>
>>
>> If the SNs are handling DHCP, it only needs to be enabled on the MN if
>> you plan in reinstaling a SN.
>>
>>
>>
>> Anything that resolves DNS through the MN will need an updated
>> resolv.conf.
>>
>>
>>
>> Depending on how you're maintaining your /install directory on the SNs,
>> that mechanism will need to be updated.
>>
>>
>>
>> If your MN is routing for any nodes, that will need to be addressed. You
>> might want to check the network configuration on the IMMs. On discovery, if
>> you have a gateway defined on your management network (I believe it
>> defaults to <xcatmaster>), they might be pointing to the old MN. Shouldn't
>> be an issue, but it's something to think about. If you're not routing on
>> that network, I would use pasu to set the IMM gateway to 0.0.0.0 and be
>> done with it.
>>
>>
>>
>> The only other concern I can think of would be the installation repos
>> configured on the cluster nodes and SNs. If any point to the MN, they will
>> need to be changed.
>>
>>
>>
>> Aside from all of that, it really depends on the particulars your cluster.
>>
>>
>>
>> Regards,
>> *Christian Caruthers*
>> Lenovo xESS IT Consultant
>>
>> Mobile: 757-289-9872
>>
>>
>>
>>
>>
>> *From:* Josh Nielsen [mailto:jniel...@hudsonalpha.org]
>> *Sent:* Monday, May 02, 2016 8:32 PM
>> *To:* xCAT Users Mailing list
>> *Subject:* [xcat-user] How can I migrate to a new xCAT MN in a
>> hierarchical environment?
>>
>>
>>
>> Hello all,
>>
>> My team is trying to move the xCAT MN role off of an old server and get
>> it over onto new virtual infrastructure, but I am a little unsure about
>> whether it is possible to do while leaving everything else in its place as
>> we currently have it in our environment. We have an MN with two SNs for our
>> xCAT environment, and I would need to make the SNs recognize that the new
>> MN (with a new IP and hostname) is now their xcatmaster, and they would
>> need to take hierarchical command updates from the new MN, look to the new
>> MN for the xCAT database (which is a MySQL database in our environment),
>> etc.
>>
>> So a few questions along those lines.
>>
>> 1. Which/how many xCAT database fields would I need to update that use
>> the MN's IP (other than "master" in the site table), and would I have to
>> reinstall or otherwise update anything on the SNs (I imagine restarting the
>> daemons is necessary at a minimum) in case they have anything statically
>> configured for the current MN's IP?
>>
>> 2. Do any default postscripts for deployed clients ever place the MN's
>> hostname or IP in any config files that would require manual alteration if
>> the MN is changed? Our client nodes should, however, have one of the two
>> SNs as their designated xcatmaster, instead of the MN, as shown by an
>> 'lsdef'.
>>
>> 3. And as far as DHCP, the MN does not even need DHCP running if the SNs
>> are handling DHCP, correct? Would I have to change any of my 'networks'
>> table entries and DHCP IP pool config in any case, or should simply dumping
>> and importing the current DB settings in to the new MN instance be seamless?
>>
>> DNS I think (hope) should be an easier matter, since we already have an
>> external DNS server configured that the MN pushes entries to with a
>> 'makedns -e', so no DNS dependency lies on the present MN itself. I imagine
>> I'd have to copy the /etc/hosts from the current MN over to the new though
>> for the makedns (and other things) to continue working.
>>
>> I have attached an image with a simplified sketch of what our xCAT
>> environment looks like. Overall I'm just wondering what changes would I
>> need to make for this to be possible.
>>
>> Thanks for your input.
>>
>> Josh Nielsen
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Find and fix application performance issues faster with Applications
>> Manager
>> Applications Manager provides deep performance insights into multiple
>> tiers of
>> your business applications. It resolves application problems quickly and
>> reduces your MTTR. Get your free trial!
>> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
>> _______________________________________________
>> xCAT-user mailing list
>> xCAT-user@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>
>>
>>
>> ------------------------------------------------------------------------------
>> What NetFlow Analyzer can do for you? Monitors network bandwidth and
>> traffic
>> patterns at an interface-level. Reveals which users, apps, and protocols
>> are
>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>> J-Flow, sFlow and other flows. Make informed decisions using capacity
>> planning reports.
>> https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
>> _______________________________________________
>> xCAT-user mailing list
>> xCAT-user@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> What NetFlow Analyzer can do for you? Monitors network bandwidth and
>> traffic
>> patterns at an interface-level. Reveals which users, apps, and protocols
>> are
>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>> J-Flow, sFlow and other flows. Make informed decisions using capacity
>> planning reports.
>> https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
>> _______________________________________________
>> xCAT-user mailing list
>> xCAT-user@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>
>>
>
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://sdm.link/zohomanageengine
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
