So to set expectations, my team cannot guarantee when/if things we do will land in xcat.org distribution (we work on lenovobuild2.14 for hpc.lenovo.com). Also I personally love a lot of questions, certainly given the lack of detailed documentation.
If we are talking about authentication of non-system users, then the short of it is that either kerberos or user keys would do the job. xCAT actually doesn't do anything at all for user authentication. Now to write more than anyone probably wanted to hear about the matter from my understanding... There are four key things that I think are worth thinking about individually. Normal user authentication within a cluster strategies (Today xCAT does nothing in particular for or against this): ====================================================================== -Kerberos: works fine for user authentication, if so equipped and configured. This is generally well covered everywhere so we don't talk much about it specifically in xCAT. Additionally it's a bit more different site to site so we tend to avoid getting too wrapped up in how a kerberos realm is administered and used. -User keys without passphrase protection: This works, but the same reasons for unprotected private keys being considered risky in a private device are exacerbated in network/cluster filesystem which may not be as locked down with respect to reads (often lax authentication, even more usually not encrypted on the wire) as one would like when dealing with private key material. -Protected keys unlocked through SSH agent forwarding: not particularly scheduler friendly. Additionally rightfully considered giving too much open ended access to a private ssh agent through less trusted devices. A good mitigation when it can work though compared to having the key unencrypted on a network filesystem -SSH remote command: Widely suggested as an alternative for SSH agent forwarding, but impractical to manage in an open ended scenario like a cluster. -SSH CA for user management: Not particularly valuable for intra-cluster, but helpful for a person with numerous client devices to provide a single authorized_hosts lines to cover them and manage multiple keys without updating authorized_hosts on services. So not pertinent to this particular discussion but worth a look if you find juggling your personal authorized_keys a tad annoying. -Munge: I honestly don't know enough about how it works to have an opinion one way or another -Host based authentication: Generally a pain as you have to have a centralized or synchronized known_hosts file with shifting contents, so historically less interesting, however.... -Host based authentication with SSH CA: This changes the above so that the known_hosts problem is much more tractable. A single line in a single known_hosts can cover arbitrarily many names and keys, so long as your onboarding solution manages it. It has the side effect of sidestepping known_hosts problems in user home directories. This is where I currently want to go with 'secureshell' postscript/confluent. The user auth portion of it is a useful side effect that might as well be enabled along the way, the main reasoning will be below. Root able to login to managed nodes without password: ========================================================================= Kerberos: A bad idea for this use case. If for no other reason, we want root to be able to login even if kerberos is not viable (e.g. some config mistake left clocks messed up). We want the least fragile possible strategy so that we may be able to repair a kerberos problem, for example /root/.ssh authorized_keys: fine and will continue to be what my team would be targeting, albeit perhaps with higher quality authenticity guarantees Known_hosts ============================================================================ Use kerberos principles instead of ssh keys: possible but not too popular, and again kerberos is a bit fragile which contributes to this not being too popular Having a cluster filesystem host a global known hosts file: This is workable and in fact what xCAT did way back in the day. However some fundamentally disagreed with how prescriptive that was about cluster mounts, so we moved to strategies that didn't dictate any particular mount relationship between manager and managed nodes. Persisting host keys across reinstall/stateless boot: This is what xCAT does in remoteshell and it is in general a bad idea security wise. Any time you are moving a private key across a network it would be better if there were a strategy to avoid this. It is convenient though in sidestepping having to curate known_hosts at all. SSH host key CA: Same as above, this takes care of known hosts problems and by extension enables host based authentication easily. Now we would have only one line in /etc/ssh/known_hosts to worry about and that line would not need to change to approve key changes. This is what I want 'secureshell' to do. There is more to how I want to harden than the ultimate strategy, but this is the key part for the discussion. Root able to ssh *between* nodes without password =============================================================================== ~/.ssh/id_ecdsa without a passphrase: this is optionally something remoteshell will enable but it is a riskier idea than even host key persistence, as a more significant private key is moved on the wire -Host based auth with ~/.shosts: Again going back to having known_hosts well managed, this would enable node to node ssh as root without as much fuss or muss -Skipping that capability: You may be able to determine that only root from select management nodes need to have this ability. This is at least loss of convenience for security. In some cluster system software, not having this breaks the software but this is less prevalent of an issue than it used to be as software has grown more aware of better ways of managing itself. ________________________________ From: Vinícius Ferrão <fer...@versatushpc.com.br> Sent: Tuesday, October 8, 2019 5:52 PM To: Jarrod Johnson Cc: xCAT Users Mailing list Subject: Re: [External] [xcat-user] What remoteshell exactly does? Jarrod, thank you. If I understood correctly, remoteshell does the following: * Copies the exactly same host key to all the nodes. So they share the same keys. * Enable SSH passwordless using RSA keys only available to root. That’s it? Regarding the next release, it will be do something similar but with Hostbased Authentication by default. Do you guys have any ETA for this updated version? Actually I’m trying to achieve HBA between the nodes, but I’m trying to find a use case for this instead of other methods: publickey authentication or SSO with Kerberos, since I’m already running a FreeIPA instance on the headnode. Sorry for asking a lot of questions but I was able to find anything related to remoteshell and it’s features on http://xcat-docs.readthedocs.io and I’m trying to put a lot of functionality with xCAT without breaking it. Thanks. On 8 Oct 2019, at 17:46, Jarrod Johnson <jjohns...@lenovo.com<mailto:jjohns...@lenovo.com>> wrote: You may remove it from postscripts tabedit postscripts You may remove remoteshell. Our team was actually going to provide a new 'secureshell' postscript as an alternative to supersede with a more sane security strategy, but if you have another strategy that takes care of the authentication, neither would be required. remoteshell tries to avoid perturbing known_hosts by persisting host key across install. Which is a tricky proposition. It also does some potentially undesirable changes to ssh/sshd configuration. For the lazy, it tends to still create a passwordless experience for root with mitigated known_hosts perturbance for all. secureshell is going to replace the key bootstrap with a more hardened mechanism, replace host key persistence with a managed SSH CA, and enable host-based authentication within a cluster if requested to enable non-root ssh enablement without per-user action. It would also allow enabling ssh between nodes as root without user key sharing (using .shosts with host authentication). This is intended as a more modern and manageable 'works regardless of other configuration' ssh bootstrap scheme. Neither should be required if you otherwise enable ssh to your likeing. ________________________________ From: Vinícius Ferrão via xCAT-user <xcat-user@lists.sourceforge.net<mailto:xcat-user@lists.sourceforge.net>> Sent: Tuesday, October 8, 2019 4:19 PM To: xCAT Users Mailing list Cc: Vinícius Ferrão Subject: [External] [xcat-user] What remoteshell exactly does? Hello, I’m trying to add some features to an xCAT HPC Cluster but I’m with some issues with SSH Host Keys. My clients are running ipa-client-install on bootime since I’ve already deployed an FreeIPA Server on the headnode and I want the nodes to authenticate on it. After some time trying to debug issues I’ve came across to this file. This file appears to copy some fixed host keys to compute images. But I’m not sure about it. The point is: what this script really do? Is it really necessary since it’s added by default by xCAT. Can it be replaced? Thanks, _______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net<mailto:xCAT-user@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/xcat-user xcat-user List Signup and Options<https://lists.sourceforge.net/lists/listinfo/xcat-user> lists.sourceforge.net<http://lists.sourceforge.net/> An extreme cluster/cloud administration toolkit
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
