subject:"FAI \+ SaltStack anybody\?"

On Fri, 2023-10-06 at 20:02 +0200, Henning Glawe wrote:
> p.s.: call me biased, as I implemented ``softupdate`` almost 20 years ago
> and use it since then as a configuration manager for a few 1k hosts in
> various contexts

softupdate is very handy. We used to use it at work (and I still do at home)
for building Linux VServers, and I have happily used it to bootstrap cloud
based instances.

The difficulty I run into, and which others might as well, is there is broad
knowledge of Puppet, Salt, Ansible, Chef, etc. as well as there being many
many modules/recipes/whatever for these tools. Neither of these so much for
FAI[0], so the tool used for on-going configuration management tends to be
one of these other tools.

At work we have a similar model as other people in this thread. FAI to
handle the hardware and get a server into a state that it is ready to be
taken over by configuration management.

For operating system upgrades we do reinstalls rather than dist-upgrades, so
FAI also needs to be able to rebuild a server - hence having various secrets
able to be deployed during a build.

Cheers,
Andrew

[0] Hmm, is there scope for sharing classes for doing common useful tasks? I
have some I can most probably share. Could FAI have a "plugins" directory
that mimics the top level of a profile and allow for contained plugins to be
installed?

-- 
Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz |
Catalyst Cloud:   | This space intentionally left blank
 https://catalystcloud.nz |

Re: FAI + SaltStack anybody?

2023-10-06 Diskussionsfäden Henning Glawe

Moin,

as I mentioned: check ``fai softupdate``, this feature of 
FAI makes it a configuration manager.

Your running system gets updated to the state you define
in your FAI config without a downtime. No reinstall required.

p.s.: call me biased, as I implemented ``softupdate`` almost 20 years ago
and use it since then as a configuration manager for a few 1k hosts in
various contexts

On Fri, Oct 06, 2023 at 04:24:48PM +, Diego Zuccato wrote:
> Yes. They're different tools with different objectives. FAI excels at
> reinstalling a system, but is not a configuration manager: say you have a
> webserver (actually 3: dev, test and prod) and you need to change the PHP
> version in use. Sure, you can reinstall from scratch with FAI, but why? Way
> faster to just update packages. Or are you reinstalling every time there are
> system updates instead of using apt full-upgrade? 

-- 
Mit freundlichen Grüßen
Henning Glawe

Dr. Henning Glawe
Max-Planck-Institut für Struktur und Dynamik der Materie
Geb. 99 (CFEL), Luruper Chaussee 149, 22761 Hamburg, Germany
http://www.mpsd.mpg.de/, Email: henning.gl...@mpsd.mpg.de
Building/Room: 99/O2.100, Phone: +49-40-8998-88392

Re: FAI + SaltStack anybody?

2023-10-06 Diskussionsfäden Matthew Pounsett

On Thu, Oct 5, 2023 at 9:00 AM Diego Zuccato  wrote:
>
> My current idea is to use Salt to orchestrate the install, but maybe
> it's better left to FAI? How can I "pass around" minion key so I don't
> have to manually re-approve the new key every time?

This is how we manage it.  FAI knows what our "base server" should
look like in terms of, how we partition disks, and what network
interfaces get used for what.  The only package beyond the base OS
that it installs is salt-minion, and it puts in place our
/etc/salt/minion.d/* files.   On first boot, the minion tries to join
the master, and we approve the new key there manually.  The first
highstate takes care of adding our site standard base packages,
configuration, etc.

In our case manually approving the minion key on the master is a small
extra step, but I can see how if you're doing dozens of servers a day,
or if you have a strong motive for completely unattended reinstalls
(fire and forget) that having to approve the minion's key would be a
problem.

Someone has suggested something like this up-thread, but I think the
only way you're going to eliminate that step is if you push a keypair
to the minion from FAI, and then have FAI share the public key with
the master.  You're probably not going to be able to get the minion to
start up and do its thing properly until the system boots, and by that
time I think you've lost any opportunity to transfer its public key
securely without a manual approval step.   You could store the public
keys that FAI generates in a repository on the FAI server, and have it
trigger a Salt webhook to tell the master when it needs to retrieve
and install new ones.

Re: FAI + SaltStack anybody?

Yes. They're different tools with different objectives. FAI excels at 
reinstalling a system, but is not a configuration manager: say you have a 
webserver (actually 3: dev, test and prod) and you need to change the PHP 
version in use. Sure, you can reinstall from scratch with FAI, but why? Way 
faster to just update packages. Or are you reinstalling every time there are 
system updates instead of using apt full-upgrade?

Diego

Da: linux-fai  per conto di Henning Glawe 

Inviato: venerdì 6 ottobre 2023 17:21
A: fully automatic installation for Linux 
Oggetto: Re: FAI + SaltStack anybody?

Moin,

On Thu, Oct 05, 2023 at 02:59:40PM +0200, Diego Zuccato wrote:
> Does someone use FAI to install the base system that will be managed by
> Salt?

Do you have a concrete reason for introducing Salt on top of FAI?
FAI can be used to do most of your configuration management via
``fai softupdate``

--
Mit freundlichen Grüßen
Henning Glawe

Dr. Henning Glawe
Max-Planck-Institut für Struktur und Dynamik der Materie
Geb. 99 (CFEL), Luruper Chaussee 149, 22761 Hamburg, Germany
http://www.mpsd.mpg.de/, Email: henning.gl...@mpsd.mpg.de
Building/Room: 99/O2.100, Phone: +49-40-8998-88392

Re: FAI + SaltStack anybody?

2023-10-06 Diskussionsfäden Holger Levsen

On Fri, Oct 06, 2023 at 05:21:30PM +0200, Henning Glawe wrote:
> Do you have a concrete reason for introducing Salt on top of FAI?

I don't wanna speak for the original poster, but your question sounds a bit
like "Do you have a concrete reason for introducing LibreOffice on top of
this Unix system which comes with emacs and LaTeX?" ;p

> FAI can be used to do most of your configuration management via
> ``fai softupdate``

yes, but Salt has features which FAI doesn't have, also the design is
very different and then there are tons of existing Salt receipts etc.

that said, two things about me, which are besides the point but will
probably entertain nonetheless: a.) I've looked at Salt some years
ago and didn't like it (too hard to debug was my main concern).
b.) since more than a decade (maybe more like 15y) I think that FAI
should mean "Fully Automatic Infrastructure" because indeed FAI can
do infrastructure management very well, thanks to softupdates! :-)

-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

No more excuses. Small actions can make a big difference. We recently switched
to paper straws on both of my private jets. (@lcamtuf)

signature.asc
Description: PGP signature

Re: FAI + SaltStack anybody?

2023-10-06 Diskussionsfäden Henning Glawe

Moin,

On Thu, Oct 05, 2023 at 02:59:40PM +0200, Diego Zuccato wrote:
> Does someone use FAI to install the base system that will be managed by
> Salt?

Do you have a concrete reason for introducing Salt on top of FAI?
FAI can be used to do most of your configuration management via
``fai softupdate``

-- 
Mit freundlichen Grüßen
Henning Glawe

Dr. Henning Glawe
Max-Planck-Institut für Struktur und Dynamik der Materie
Geb. 99 (CFEL), Luruper Chaussee 149, 22761 Hamburg, Germany
http://www.mpsd.mpg.de/, Email: henning.gl...@mpsd.mpg.de
Building/Room: 99/O2.100, Phone: +49-40-8998-88392

Re: FAI + SaltStack anybody?


Il 06/10/2023 15:15, Johan Beisser ha scritto:


With that, on the salt-master, either autoaccept, or find a way to place the minion's 
public key in `/etc/salt/pki/master/minions/` and that will bypass 
the key acceptance entirely. Keys, inside of salt, are just managing where the file 
sits under the various minion directories in `/etc/salt/pki/master/` after all.


Yup. that's exactly where my problem lies: that "find a way" is what I'm 
looking for :)



Don't have to do it if you set the master's public key, and minion keys, before 
the minion is started though.


Well, for the minion it's not a problem, as long as it finds the correct 
pubkey: if its key is missing, a keypair can be generated. But the 
master doesn't know this new key (yet).



Then it's just having a single job starting after FAI's reboot, and doing 
`salt-call state.highstate` on first boot.


It's not a Salt problem, it's just a "timing issue" that I have to 
understand well. Once Salt knows a minion is being reinstalled (ideally 
I triggered it applying a given state), it should sync with FAI to wait 
the moment the minion is rebooting (or, even better, it receives the 
minion key before the reboot) and knows it can trust that key.


--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Re: FAI + SaltStack anybody?

2023-10-06 Diskussionsfäden Johan Beisser

> On Oct 6, 2023, at 10:59, Diego Zuccato  wrote:
> 
> Il 06/10/2023 10:36, Sinh Lam ha scritto:
>> With the above said, I do not see what you mean there is a chicken and the 
>> egg problem.
> 
> To approve a minion key, Salt does have to trust the request is coming from 
> the right minion, but it can't know till the key is approved.

Embed the Salt Master's public key in the FAI environment before the minion is 
started. While the master won't have the minion key before it contacts it, you 
can ensure the initial communication with the master is secured and the master 
is properly identified. The keys themselves are generated by `salt-key 
--gen-keys` on first run of the minion, if they don't exist.

So, during the install, pre-generate the keys with `salt-key`. Just ensure the 
target directory for the private key ends up in 
`/etc/salt/pki/minion/minion.pem` and the public key in 
`/etc/salt/pki/minion/minion.pub`. The master's key needs to be in 
`/etc/salt/pki/minion/minion_master.pub`. 

With that, on the salt-master, either autoaccept, or find a way to place the 
minion's public key in `/etc/salt/pki/master/minions/` and that will 
bypass the key acceptance entirely. Keys, inside of salt, are just managing 
where the file sits under the various minion directories in 
`/etc/salt/pki/master/` after all.

>>   Your concern was how to move the minion around servers that are getting 
>> provisioned/re-provisioned so you don’t have to approve the minion each time 
>> and I’m sure there’s a couple of ways to do this but right now I see two :
>> 1) turn on auto-accept - you don’t have to worry about approving any minions 
>> because they’ll be auto-approved
> 
> Can't do that on public networks. [*]

Don't have to do it if you set the master's public key, and minion keys, before 
the minion is started though. Then it's just having a single job starting after 
FAI's reboot, and doing `salt-call state.highstate` on first boot.

Re: FAI + SaltStack anybody?

On Fri, 2023-10-06 at 11:18 +0200, Thomas Lange wrote:
> > > > > > On Fri, 06 Oct 2023 21:57:28 +1300, Andrew Ruthven
> > > > > >  said:
> 
>     > This isn't ideal as the secrets are still present in the NFSROOT for
> a short
>     > period of time, but does solve the chicken and egg issue others
> mentioned
> This reminds me of a solution I once saw.
> Put some info into a fifo (named pipe), so only one receiver can read
> it. After that the fifo is empty.
> 
> What about having a daemon on the FAI server which serves some secrect
> using:
> echo secrect | nc -p 12345 -l
> 
> So only one FAI client can read the secrect from port 12345 once.
> This may help a little bit.

This could help. It could also do some level of validation of the IP/MAC
that the request is coming from, especially if you've used fai-chboot. Again
not ideal, but better.

The thing I like about my solution is that fcopy just works. :)

Cheers,
Andrew


-- 
Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz |
Catalyst Cloud: | This space intentionally left blank
https://catalystcloud.nz |

Re: FAI + SaltStack anybody?

On Fri, 2023-10-06 at 11:36 +0200, Diego Zuccato wrote:
> I really like it a lot!
> Not bulletproof but more secure than a file.
> 
> Still no way to have "hooks" run on FAI server?

We kind of do this, we call it Semi Automatic Installer (SAI). But the
problem is that you still need to have some credentials in the NFSROOT to
talk to the server.

It is primarily used for fetching and storing answers to questions on the
server. This allows us to have interactive builds that can save the answers
to the server so the next build can be automatic. Hence the "Semi".

-- 
Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz |
Catalyst Cloud: | This space intentionally left blank
https://catalystcloud.nz |

Re: FAI + SaltStack anybody?


I really like it a lot!
Not bulletproof but more secure than a file.

Still no way to have "hooks" run on FAI server?

Diego

Il 06/10/2023 11:18, Thomas Lange ha scritto:

On Fri, 06 Oct 2023 21:57:28 +1300, Andrew Ruthven  said:


 > This isn't ideal as the secrets are still present in the NFSROOT for a 
short
 > period of time, but does solve the chicken and egg issue others mentioned
This reminds me of a solution I once saw.
Put some info into a fifo (named pipe), so only one receiver can read
it. After that the fifo is empty.

What about having a daemon on the FAI server which serves some secrect
using:
echo secrect | nc -p 12345 -l

So only one FAI client can read the secrect from port 12345 once.
This may help a little bit.


--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Re: FAI + SaltStack anybody?

2023-10-06 Diskussionsfäden Thomas Lange

> On Fri, 06 Oct 2023 21:57:28 +1300, Andrew Ruthven  
> said:

> This isn't ideal as the secrets are still present in the NFSROOT for a 
short
> period of time, but does solve the chicken and egg issue others mentioned
This reminds me of a solution I once saw.
Put some info into a fifo (named pipe), so only one receiver can read
it. After that the fifo is empty.

What about having a daemon on the FAI server which serves some secrect
using:
echo secrect | nc -p 12345 -l

So only one FAI client can read the secrect from port 12345 once.
This may help a little bit.
-- 
regards Thomas

Re: FAI + SaltStack anybody?


Il 06/10/2023 10:36, Sinh Lam ha scritto:
Reading through your original post - I think there might be some 
confusion as to what SaltStack does and what FAI does (if not, I 
apologize).  SaltStack is a configuration management tool that is 
normally used to ensure the target minion's configuration is exactly as 
it should, while FAI is a provisioning tool that essentially builds the 
server that SaltStack is used to manage.


No need to apologize even if I already knew the difference between FAI 
and Salt  :)


With the above said, I do not see what you mean there is a chicken and 
the egg problem.


To approve a minion key, Salt does have to trust the request is coming 
from the right minion, but it can't know till the key is approved.


I do not believe that salt can do the actual installation of the 
server’s OS because there’s no minion running to begin with.  You should 
really leave that to FAI.


Yes, sure. Other tools (like xCat) try to do both and are IMVHO way less 
versatile.


  Your concern was how to move the minion 
around servers that are getting provisioned/re-provisioned so you don’t 
have to approve the minion each time and I’m sure there’s a couple of 
ways to do this but right now I see two :


1) turn on auto-accept - you don’t have to worry about approving any 
minions because they’ll be auto-approved


Can't do that on public networks. [*]

2) issue a call to the salt master to accept the new minion when it is 
registered during fai.  This involves you knowing the minion id/name of 
the key.


That's what I'd like to make FAI do. If only there was a 'hook' system 
on FAI server, triggering scripts at the different stages... The nearest 
thing I could think of is a custom fai-monitor but it seems quite unsafe :(



This is how I’m planning on using SaltStack with FAI -
I have a dedicated network that is tightly controlled so to that point I 
know what connects to it and I know why those servers are connected to 
that particular network.  In essence, I trust this particular network.  
Because of this, I have auto-accept turned on my salt master.


Can't do that: my networks are often exposed. The alternative would mean 
having to dynamically reconfigure switches to move ports to different 
VLANs... Quite a big can of worms on its own :(


I have FAI install the base os on the server, toward the end of the 
process, a couple things will happen:

* a script will be used to auto-register this server with our CMDB
* a script will be used to enroll this minion with the salt-master and 
set the minion_id (if needed).


How does your script authenticate the requests? Or are you just relying 
on the "secured network" to bypass authentication?


[...]
That’s it.  FAI does the OS (and handles the registering of the server 
with our CMDB and the minion with the master), and salt stack takes care 
of the configuration of the server.


That's the desiderable outcome :)

The glue I believe you’re talking about is a source of truth to populate 
pillar data and grains so your salt states can actually do something 
useful.


No, that's already taken care of :)

And MAC addresses can be spoofed quite easily, so you really shouldn’t 
rely on that as your ‘root of trust’.  I deal with a lot of VMs and each 
one of those VMs I can easily specify whatever MAC address I want (you 
really shouldn’t).  But spoofing a MAC while it’s in the early parts of 
pxe/net boot process is harder (if not impossible), you still shouldn’t 
use it as the ‘root of trust’.


Yup, I know. Already did it in DOS :)
But stronger authentication either requires TPM or interaction.

--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Re: FAI + SaltStack anybody?

On Fri, 2023-10-06 at 06:47 +0200, Diego Zuccato wrote:
> Il 05/10/2023 15:54, Laura Smith via linux-fai ha scritto:
> > Its been a while since I worked with Salt, but IIRC it sounds like what
> > is not "clicking" is that you need to fix the TOFU problem.
> 
> Actually there are 2 distinct problems:
> - pass the pubkey from the minion to FAI during the install (possibly in 
> an authenticated way)
> - authorize that key in Salt from FAI

Not related to Salt, but possibly an approach that can be used here.

I have a script that we run on the FAI server for managing secrets. It will
copy secrets, generating them as required, into the NFSROOT and then remove
them after a period of time.

I have this handling ssh hostkeys so we can get the same keys on a rebuild.
It can handle Puppet keys, including signing them, although we no longer use
it for Puppet.

This isn't ideal as the secrets are still present in the NFSROOT for a short
period of time, but does solve the chicken and egg issue others mentioned
and removes the need for a generic "sign any request that comes in" that
others have suggested.

Cheers,
Andrew

-- 
Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz |
Catalyst Cloud:   | This space intentionally left blank
 https://catalystcloud.nz |

Reading through your original post - I think there might be some confusion
as to what SaltStack does and what FAI does (if not, I apologize).
SaltStack is a configuration management tool that is normally used to
ensure the target minion's configuration is exactly as it should, while FAI
is a provisioning tool that essentially builds the server that SaltStack is
used to manage.

With the above said, I do not see what you mean there is a chicken and the
egg problem.

I do not believe that salt can do the actual installation of the server’s
OS because there’s no minion running to begin with. You should really
leave that to FAI. Your concern was how to move the minion around servers
that are getting provisioned/re-provisioned so you don’t have to approve
the minion each time and I’m sure there’s a couple of ways to do this but
right now I see two :

1) turn on auto-accept - you don’t have to worry about approving any
minions because they’ll be auto-approved
2) issue a call to the salt master to accept the new minion when it is
registered during fai. This involves you knowing the minion id/name of the
key.

This is how I’m planning on using SaltStack with FAI -

I have a dedicated network that is tightly controlled so to that point I
know what connects to it and I know why those servers are connected to that
particular network. In essence, I trust this particular network. Because
of this, I have auto-accept turned on my salt master.

I have FAI install the base os on the server, toward the end of the
process, a couple things will happen:
* a script will be used to auto-register this server with our CMDB
* a script will be used to enroll this minion with the salt-master and set
the minion_id (if needed).

Once the above two is done, FAI will issue a reboot of the server. Once
the server is back up, the salt-minion will reconnect with the salt master
and because the key is already accepted (because I have auto-accept turned
on), a high state will run. Regarding pillar data, I have an external
pillar that pulls information about the minion from the CMDB and generates
all the relevant pillar data the states need to use.

That’s it. FAI does the OS (and handles the registering of the server with
our CMDB and the minion with the master), and salt stack takes care of the
configuration of the server.

I hope this provides some clarity as to how to use FAI with SaltStack.

The glue I believe you’re talking about is a source of truth to populate
pillar data and grains so your salt states can actually do something
useful. The above scenario is what I’m about to implement. I have a CMDB
that contains information about the very server FAI is provisioning. This
information from the CMDB will populate the pillars which will feed into
states to dynamically do whatever it is that server is provisioned for.
You can do some manual work and pre-populate the CMDB or use FAI to
auto-register this information with the CMDB so on the next boot, when the
salt-minion is started up again and connects to the master (because the key
is already accepted) a high state will get ran against that particular
minion.

And MAC addresses can be spoofed quite easily, so you really shouldn’t rely
on that as your ‘root of trust’. I deal with a lot of VMs and each one of
those VMs I can easily specify whatever MAC address I want (you really
shouldn’t). But spoofing a MAC while it’s in the early parts of pxe/net
boot process is harder (if not impossible), you still shouldn’t use it as
the ‘root of trust’.

https://www.tutorialspoint.com/how-to-modify-mac-address-in-windows-10-both-wired-and-wireless-adapter
https://wiki.archlinux.org/title/MAC_address_spoofing

The above are two examples of how to do just that but I’m not sure what you
mean by protected connections. FAI is ran after the server is pxe/net
booted. You can pull the root down using a squashfs image or something via
https but otherwise I’m not entirely sure what you mean.

On October 5, 2023 at 9:59:01 PM, Diego Zuccato (diego.zucc...@unibo.it)
wrote:

Il 05/10/2023 16:58, Sinh Lam ha scritto:
> You can essentially establish a ’trust’ to auto-accept keys. Then you
> wouldn’t really have to worry about moving the minion keys around. Once
> your bootstrap/installation is done, have it run a state to remove the
> key or auto-purge it somehow.

Uh? If the minion is not known to the master, it doesn't receive
pillars. And can't interact with the master. Chicken and egg.

> Honestly I would just leave the base install and anything else that
> needs to be set up to FAI and run salt against the booted up server
> after FAI is done and the server has been rebooted.
That's what I was planning to do. But without extra "glue" I'm losing
context. In particular if FAI tells Salt "I'm having *this* machine
reinstalled and its key is this" then Salt can auto-accept that key. But
if the machine is not being reinstalled by FAI, there's no reason to
auto accept a new key: it could be anybody!

Re: FAI + SaltStack anybody?


Il 05/10/2023 16:58, Sinh Lam ha scritto:
You can essentially establish a ’trust’ to auto-accept keys.  Then you 
wouldn’t really have to worry about moving the minion keys around.  Once 
your bootstrap/installation is done, have it run a state to remove the 
key or auto-purge it somehow.


Uh? If the minion is not known to the master, it doesn't receive 
pillars. And can't interact with the master. Chicken and egg.


Honestly I would just leave the base install and anything else that 
needs to be set up to FAI and run salt against the booted up server 
after FAI is done and the server has been rebooted.
That's what I was planning to do. But without extra "glue" I'm losing 
context. In particular if FAI tells Salt "I'm having *this* machine 
reinstalled and its key is this" then Salt can auto-accept that key. But 
if the machine is not being reinstalled by FAI, there's no reason to 
auto accept a new key: it could be anybody!


Does FAI use protected connections (given that usually there's no 
available "root of trust" stronger than the MAC address...) to the 
machine being installed?


--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Re: FAI + SaltStack anybody?


Il 05/10/2023 15:54, Laura Smith via linux-fai ha scritto:

Its been a while since I worked with Salt, but IIRC it sounds like what is not 
"clicking" is that you need to fix the TOFU problem.


Actually there are 2 distinct problems:
- pass the pubkey from the minion to FAI during the install (possibly in 
an authenticated way)

- authorize that key in Salt from FAI


Looking back through my notes, it 
seemshttps://docs.saltproject.io/en/latest/topics/tutorials/multimaster_pki.html
  might be worth a read.


I don't understand. In my scenario, FAI is not a Salt master. And I 
don't see how making it one could help. It would only double the burden.



In particular, maybe "master_sign_pubkey: True" on the Salt master, 
"verify_master_pubkey_sign: True" on the minion, and the master pubkeys put in 
"/etc/salt/pki/minion/" on the minions.
Then on Salt master all you have to do is approve the new connections as they 
come online.


I'd have to approve on *both* masters. :(

--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Re: FAI + SaltStack anybody?

2023-10-05 Diskussionsfäden Sinh Lam

You can essentially establish a ’trust’ to auto-accept keys.  Then you
wouldn’t really have to worry about moving the minion keys around.  Once
your bootstrap/installation is done, have it run a state to remove the key
or auto-purge it somehow.

Honestly I would just leave the base install and anything else that needs
to be set up to FAI and run salt against the booted up server after FAI is
done and the server has been rebooted.

On October 5, 2023 at 6:54:51 AM, Laura Smith via linux-fai (
linux-fai@uni-koeln.de) wrote:

Diese Nachricht wurde eingewickelt um DMARC-kompatibel zu sein. Die
eigentliche Nachricht steht dadurch in einem Anhang.

This message was wrapped to be DMARC compliant. The actual message
text is therefore in an attachment.
Hi Diego

Its been a while since I worked with Salt, but IIRC it sounds like what is
not "clicking" is that you need to fix the TOFU problem.

Looking back through my notes, it seems
https://docs.saltproject.io/en/latest/topics/tutorials/multimaster_pki.html
might be worth a read.

In particular, maybe "master_sign_pubkey: True" on the Salt master,
"verify_master_pubkey_sign: True" on the minion, and the master pubkeys put
in "/etc/salt/pki/minion/" on the minions.

Then on Salt master all you have to do is approve the new connections as
they come online.

--- Original Message ---
On Thursday, October 5th, 2023 at 13:59, Diego Zuccato <
diego.zucc...@unibo.it> wrote:

> Hello all.
>
> Does someone use FAI to install the base system that will be managed by
> Salt?
> I'm trying to integrate 'em but there's still something that doesn't
> "click"...
>
> My current idea is to use Salt to orchestrate the install, but maybe
> it's better left to FAI? How can I "pass around" minion key so I don't
> have to manually re-approve the new key every time?
> The ideal scenario would be: target generates its keypair, sends the
> pubkey to FAI that "certifies" it's from the system being installed and
> passes it to Salt. Should I write a custom fai-monitor (that would be
> needed anyway to disable netboot once system is reinstalled)?
>
> TIA.
>
> --
> Diego Zuccato
> DIFA - Dip. di Fisica e Astronomia
> Servizi Informatici
> Alma Mater Studiorum - Università di Bologna
> V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
> tel.: +39 051 20 95786

Re: FAI + SaltStack anybody?

2023-10-05 Diskussionsfäden Laura Smith via linux-fai

Diese Nachricht wurde eingewickelt um DMARC-kompatibel zu sein. Die
eigentliche Nachricht steht dadurch in einem Anhang.

This message was wrapped to be DMARC compliant. The actual message
text is therefore in an attachment.--- Begin Message ---
Hi Diego

Its been a while since I worked with Salt, but IIRC it sounds like what is not 
"clicking" is that you need to fix the TOFU problem.

Looking back through my notes, it seems 
https://docs.saltproject.io/en/latest/topics/tutorials/multimaster_pki.html 
might be worth a read.

In particular, maybe "master_sign_pubkey: True" on the Salt master, 
"verify_master_pubkey_sign: True" on the minion, and the master pubkeys put in 
"/etc/salt/pki/minion/" on the minions.

Then on Salt master all you have to do is approve the new connections as they 
come online.

--- Original Message ---
On Thursday, October 5th, 2023 at 13:59, Diego Zuccato  
wrote:

> Hello all.
> 
> Does someone use FAI to install the base system that will be managed by
> Salt?
> I'm trying to integrate 'em but there's still something that doesn't
> "click"...
> 
> My current idea is to use Salt to orchestrate the install, but maybe
> it's better left to FAI? How can I "pass around" minion key so I don't
> have to manually re-approve the new key every time?
> The ideal scenario would be: target generates its keypair, sends the
> pubkey to FAI that "certifies" it's from the system being installed and
> passes it to Salt. Should I write a custom fai-monitor (that would be
> needed anyway to disable netboot once system is reinstalled)?
> 
> TIA.
> 
> --
> Diego Zuccato
> DIFA - Dip. di Fisica e Astronomia
> Servizi Informatici
> Alma Mater Studiorum - Università di Bologna
> V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
> tel.: +39 051 20 95786
--- End Message ---

Re: FAI + SaltStack anybody?


Il 05/10/2023 15:17, Carsten Aulbert ha scritto:

we usually try with the hardware level configuration being the "border", 
i.e. everything related to partitioning, initial OS install, at least 
initial networking set-up is done with FAI (well, and salt is installed 
configured as well).

Ok, that's good.

Then FAI reboots the server and upon service start, the server starts a 
highstate and performs the remaining configuration.

Ok, no problem here.

To set-up salt, we wrote our own script around fai-chboot which ssh into 
the salt-master, creates a keypair and copies the files to the 
appropriate places.
Uhm... I don't really like that ssh step. But probably can be 
straightened out making salt get the pubkey from FAI's state.


FAI will install the private key during the 
installation and the public key is already known on the master, no need 
to accept the keys anymore.
I like even less that the private key is passed from FAI to the target, 
I'd prefer to only pass back the pubkey.



Does that help a bit?

Yes, tks.

--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Re: FAI + SaltStack anybody?

2023-10-05 Diskussionsfäden Carsten Aulbert


Hi Diego,

On 10/5/23 14:59, Diego Zuccato wrote:
Does someone use FAI to install the base system that will be managed by 
Salt?
I'm trying to integrate 'em but there's still something that doesn't 
"click"...


My current idea is to use Salt to orchestrate the install, but maybe 
it's better left to FAI? How can I "pass around" minion key so I don't 
have to manually re-approve the new key every time?


The ideal scenario would be: target generates its keypair, sends the 
pubkey to FAI that "certifies" it's from the system being installed and 
passes it to Salt. Should I write a custom fai-monitor (that would be 
needed anyway to disable netboot once system is reinstalled)?


we usually try with the hardware level configuration being the "border", 
i.e. everything related to partitioning, initial OS install, at least 
initial networking set-up is done with FAI (well, and salt is installed 
configured as well).


Then FAI reboots the server and upon service start, the server starts a 
highstate and performs the remaining configuration.


To set-up salt, we wrote our own script around fai-chboot which ssh into 
the salt-master, creates a keypair and copies the files to the 
appropriate places. FAI will install the private key during the 
installation and the public key is already known on the master, no need 
to accept the keys anymore.


Does that help a bit?

Cheers

Carsten

--
Dr. Carsten Aulbert, Max Planck Institute for Gravitational Physics,
Callinstraße 38, 30167 Hannover, Germany, Phone +49 511 762 17185


smime.p7s
Description: S/MIME Cryptographic Signature

FAI + SaltStack anybody?