Re: Transient secrets

[Markus Köberl via linux-fai]

Diese Nachricht wurde eingewickelt um DMARC-kompatibel zu sein. Die
eigentliche Nachricht steht dadurch in einem Anhang.

This message was wrapped to be DMARC compliant. The actual message
text is therefore in an attachment.--- Begin Message ---
On Thursday, 7 July 2022 08:12:54 CEST Diego Zuccato wrote:
> Hi all.
> 
> Is there a preferred way to pass a (different) secret to every host
> being installed?
> 
> Something to implement a workflow like:
> - admin asks Salt to (re)install a host
> - salt handles shutdown and switch reconfiguration (OT)
> - salt tells FAIserver to enable install of given host
> - FAI generates the secret and passes it back to Salt (or Salt generates
> the secret and passes it to FAI, as long there's a shared secret)
> - the host boots via network and installs as usual, saving/using the
> given secret
> - FAI (or the reinstalled host) tells Salt reinstall is complete and
> Salt "cleans up" (reconfig switches & so on) (OT)
> 
> The only "solution" I could find is to save the secret in
> /srv/tftp/fai/pxelinux.cfg/C0A8xxyy in append line, like FAI_FLAGS,
> FAI_CONFIG_SRC and FAI_ACTION, but since append line can be at most 255
> chars there's not much space... I's good just for very small "secrets"
> (that gets transferred in the clear, hence the need to reconfigure the
> switches).

I am asking at the beginning (with a script in `class` using dialog) for 
username and password for the salt api and save a cookie which I later use in 
a script to get the salt key for the host.

The relevant part in the cookie contains of 97 chars and base64 encoded it 
gets 134 chars therefore it might still be too long. Maybe encrypt the cookie 
file and pass the password for decryption which could be short enough.  Or just 
make the time the cookie is valid very short.


regards
Markus Köberl
-- 
Markus Koeberl
Graz University of Technology
Signal Processing and Speech Communication Laboratory
E-mail: markus.koeb...@tugraz.at

smime.p7s
Description: S/MIME cryptographic signature
--- End Message ---

Re: Transient secrets

[Robert Markula]

We distribute secrets via configuration management (in our case, via 
Cfengine).


During the first reboot after FAI the Cfengine client registers itself 
to the Cfengine server and pulls its credentials from a dedicated part 
of the repository. In Cfengine, it is possible to restrict the allowed 
ip addresses for self registration as well as for repository access. 
Each host is assigned a dedicated host-specific "secure repository" 
where all credentials are stored, to which other clients do not have access.


So when a host is about to be installed, the hosts ip address is 
manually enabled and immediately disabled after self registration is 
complete.


This system is far from bullet proof, with the highest risk being a race 
condition during the self-registration phase of the client. All further 
repository access is authenticated and encrypted.


With a clever combination of timing and faking the ip address of the 
legitimate client, one could take advantage of this race condition. But 
even then, this would attract attention, because the legitimate client 
couldn't self-register anymore.


Another downside is that this system doesn't scale well. Credentials 
have to be placed manually for all hosts and hosts have to be enabled 
manually as well. We could write some kind of management interface to 
further automate this, but with about 70 hosts that are not reinstalled 
on a daily basis, there is simply no necessity for this.


Cheers,


Robert



Am 07.07.22 um 08:12 schrieb Diego Zuccato:

Hi all.

Is there a preferred way to pass a (different) secret to every host 
being installed?


Something to implement a workflow like:
- admin asks Salt to (re)install a host
- salt handles shutdown and switch reconfiguration (OT)
- salt tells FAIserver to enable install of given host
- FAI generates the secret and passes it back to Salt (or Salt 
generates the secret and passes it to FAI, as long there's a shared 
secret)
- the host boots via network and installs as usual, saving/using the 
given secret
- FAI (or the reinstalled host) tells Salt reinstall is complete and 
Salt "cleans up" (reconfig switches & so on) (OT)


The only "solution" I could find is to save the secret in 
/srv/tftp/fai/pxelinux.cfg/C0A8xxyy in append line, like FAI_FLAGS, 
FAI_CONFIG_SRC and FAI_ACTION, but since append line can be at most 
255 chars there's not much space... I's good just for very small 
"secrets" (that gets transferred in the clear, hence the need to 
reconfigure the switches).

Re: Transient secrets

[Steffen Grunewald]

On Thu, 2022-07-07 at 11:16:33 +0200, Diego Zuccato wrote:
> Now I only have to figure out how to reliably detect its presence during
> install, then it's just matter of copying files.

Give each USB key a (filesystem) label, and check for its presence in 
/dev/disk/by-partlabel - you could even make the label match the hostname,
this would allow you to rename the machine (in DHCP) to ignore the (then
invalid) setup on the key drive.

Of course, there are always two more ways to do it ;)

- S

-- 
Steffen Grunewald, Cluster Administrator
Max Planck Institute for Gravitational Physics (Albert Einstein Institute)
Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany
~~~
Fon: +49-331-567 7274
Mail: steffen.grunewald(at)aei.mpg.de
~~~

Re: Transient secrets

[Diego Zuccato]

The more I think about it, the more I convince myself that an USB key 
(preferably connected to the internal USB connector) could be quite a 
good compromise: cannot be stolen too easily (requires opening the 
chassis), can be installed w/o requiring special skills, is cheap, and 
stores "more than enough" :)
Now I only have to figure out how to reliably detect its presence during 
install, then it's just matter of copying files.
Even better could be a "virtual key" connected via IPMI storage when 
needed (even if data gets transferred in cleartext, it's over a "secured 
network": only a fool would expose IPMI interface to unsecure networks!)...


Il 07/07/2022 08:55, Andrew Ruthven ha scritto:


Hey,

Yes, agreed, depends on the use case. For the gear I'm dealing with 
they're on physically very secure networks and NFS is firewalled off.


You could potentially have a kernel token as you suggest and then go to 
fetch the secrets from a HashiCorp Vault with an approval needing to be 
issued.


I know of someone who used to store GPG encrypted tar files in their FAI 
profile and you had to enter a GPG key during the build to decrypt them.


We do in some cases generate passwords (root and encrypted filesystems) 
during build and have those emailled with GPG encryption to the relevant 
parties.


Cheers,
Andrew

On Thu, 2022-07-07 at 08:35 +0200, Diego Zuccato wrote:

Hi Andrew.

That's an option, but is seems less secure: while PXE net have to be
quite "locked down", NFS could potentially be exposed on a "public"
network (say to handle reinstalls on many networks with a single server).
If only machines had an "attestation key" by default... Maybe an USB key
to insert in the machine being reinstalled... Possibly in an internal
slot... Uhm... Still brainstorming...

Tks,
  Diego


Il 07/07/2022 08:22, Andrew Ruthven ha scritto:

Hey,

I'm not sure if this is preferred or not, but the approach I take is to
have a command we run first, that copies any required secrets (and will
generate SSH host keys and puppet certs if required first) into the NFS
root. A cron job runs every 15 minutes and cleans up any of those
secrets which are older than 2 hours (this could be much shorter).

Cheers,
Andrew

On Thu, 2022-07-07 at 08:12 +0200, Diego Zuccato wrote:

Hi all.

Is there a preferred way to pass a (different) secret to every host
being installed?

Something to implement a workflow like:
- admin asks Salt to (re)install a host
- salt handles shutdown and switch reconfiguration (OT)
- salt tells FAIserver to enable install of given host
- FAI generates the secret and passes it back to Salt (or Salt generates
the secret and passes it to FAI, as long there's a shared secret)
- the host boots via network and installs as usual, saving/using the
given secret
- FAI (or the reinstalled host) tells Salt reinstall is complete and
Salt "cleans up" (reconfig switches & so on) (OT)

The only "solution" I could find is to save the secret in
/srv/tftp/fai/pxelinux.cfg/C0A8xxyy in append line, like FAI_FLAGS,
FAI_CONFIG_SRC and FAI_ACTION, but since append line can be at most 255
chars there's not much space... I's good just for very small "secrets"
(that gets transferred in the clear, hence the need to reconfigure the
switches).



--

Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz  |
Catalyst Cloud:   | This space intentionally left blank
https://catalystcloud.nz  |





--

Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz |
Catalyst Cloud:   | This space intentionally left blank
  https://catalystcloud.nz |



--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Re: Transient secrets

[Andrew Ruthven]

Hey,

Yes, agreed, depends on the use case. For the gear I'm dealing with
they're on physically very secure networks and NFS is firewalled off.

You could potentially have a kernel token as you suggest and then go to
fetch the secrets from a HashiCorp Vault with an approval needing to be
issued.

I know of someone who used to store GPG encrypted tar files in their
FAI profile and you had to enter a GPG key during the build to decrypt
them.

We do in some cases generate passwords (root and encrypted filesystems)
during build and have those emailled with GPG encryption to the
relevant parties.

Cheers,
Andrew

On Thu, 2022-07-07 at 08:35 +0200, Diego Zuccato wrote:
> Hi Andrew.
> 
> That's an option, but is seems less secure: while PXE net have to be 
> quite "locked down", NFS could potentially be exposed on a "public" 
> network (say to handle reinstalls on many networks with a single
> server).
> If only machines had an "attestation key" by default... Maybe an USB
> key 
> to insert in the machine being reinstalled... Possibly in an internal
> slot... Uhm... Still brainstorming...
> 
> Tks,
>   Diego
> 
> 
> Il 07/07/2022 08:22, Andrew Ruthven ha scritto:
> > Hey,
> > 
> > I'm not sure if this is preferred or not, but the approach I take
> > is to 
> > have a command we run first, that copies any required secrets (and
> > will 
> > generate SSH host keys and puppet certs if required first) into the
> > NFS 
> > root. A cron job runs every 15 minutes and cleans up any of those 
> > secrets which are older than 2 hours (this could be much shorter).
> > 
> > Cheers,
> > Andrew
> > 
> > On Thu, 2022-07-07 at 08:12 +0200, Diego Zuccato wrote:
> > > Hi all.
> > > 
> > > Is there a preferred way to pass a (different) secret to every
> > > host
> > > being installed?
> > > 
> > > Something to implement a workflow like:
> > > - admin asks Salt to (re)install a host
> > > - salt handles shutdown and switch reconfiguration (OT)
> > > - salt tells FAIserver to enable install of given host
> > > - FAI generates the secret and passes it back to Salt (or Salt
> > > generates
> > > the secret and passes it to FAI, as long there's a shared secret)
> > > - the host boots via network and installs as usual, saving/using
> > > the
> > > given secret
> > > - FAI (or the reinstalled host) tells Salt reinstall is complete
> > > and
> > > Salt "cleans up" (reconfig switches & so on) (OT)
> > > 
> > > The only "solution" I could find is to save the secret in
> > > /srv/tftp/fai/pxelinux.cfg/C0A8xxyy in append line, like
> > > FAI_FLAGS,
> > > FAI_CONFIG_SRC and FAI_ACTION, but since append line can be at
> > > most 255
> > > chars there's not much space... I's good just for very small
> > > "secrets"
> > > (that gets transferred in the clear, hence the need to
> > > reconfigure the
> > > switches).
> > > 
> > 
> > -- 
> > 
> > Andrew Ruthven, Wellington, New Zealand
> > and...@etc.gen.nz |
> > Catalyst Cloud:   | This space intentionally left blank
> >   https://catalystcloud.nz |
> > 
> 

-- 
Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz |
Catalyst Cloud: | This space intentionally left blank
https://catalystcloud.nz |

Re: Transient secrets

[Diego Zuccato]

Hi Andrew.

That's an option, but is seems less secure: while PXE net have to be 
quite "locked down", NFS could potentially be exposed on a "public" 
network (say to handle reinstalls on many networks with a single server).
If only machines had an "attestation key" by default... Maybe an USB key 
to insert in the machine being reinstalled... Possibly in an internal 
slot... Uhm... Still brainstorming...


Tks,
 Diego


Il 07/07/2022 08:22, Andrew Ruthven ha scritto:

Hey,

I'm not sure if this is preferred or not, but the approach I take is to 
have a command we run first, that copies any required secrets (and will 
generate SSH host keys and puppet certs if required first) into the NFS 
root. A cron job runs every 15 minutes and cleans up any of those 
secrets which are older than 2 hours (this could be much shorter).


Cheers,
Andrew

On Thu, 2022-07-07 at 08:12 +0200, Diego Zuccato wrote:

Hi all.

Is there a preferred way to pass a (different) secret to every host
being installed?

Something to implement a workflow like:
- admin asks Salt to (re)install a host
- salt handles shutdown and switch reconfiguration (OT)
- salt tells FAIserver to enable install of given host
- FAI generates the secret and passes it back to Salt (or Salt generates
the secret and passes it to FAI, as long there's a shared secret)
- the host boots via network and installs as usual, saving/using the
given secret
- FAI (or the reinstalled host) tells Salt reinstall is complete and
Salt "cleans up" (reconfig switches & so on) (OT)

The only "solution" I could find is to save the secret in
/srv/tftp/fai/pxelinux.cfg/C0A8xxyy in append line, like FAI_FLAGS,
FAI_CONFIG_SRC and FAI_ACTION, but since append line can be at most 255
chars there's not much space... I's good just for very small "secrets"
(that gets transferred in the clear, hence the need to reconfigure the
switches).



--

Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz |
Catalyst Cloud:   | This space intentionally left blank
  https://catalystcloud.nz |



--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Re: Transient secrets

[Andrew Ruthven]

Hey,

I'm not sure if this is preferred or not, but the approach I take is to
have a command we run first, that copies any required secrets (and will
generate SSH host keys and puppet certs if required first) into the NFS
root. A cron job runs every 15 minutes and cleans up any of those
secrets which are older than 2 hours (this could be much shorter).

Cheers,
Andrew

On Thu, 2022-07-07 at 08:12 +0200, Diego Zuccato wrote:
> Hi all.
> 
> Is there a preferred way to pass a (different) secret to every host 
> being installed?
> 
> Something to implement a workflow like:
> - admin asks Salt to (re)install a host
> - salt handles shutdown and switch reconfiguration (OT)
> - salt tells FAIserver to enable install of given host
> - FAI generates the secret and passes it back to Salt (or Salt
> generates 
> the secret and passes it to FAI, as long there's a shared secret)
> - the host boots via network and installs as usual, saving/using the 
> given secret
> - FAI (or the reinstalled host) tells Salt reinstall is complete and 
> Salt "cleans up" (reconfig switches & so on) (OT)
> 
> The only "solution" I could find is to save the secret in 
> /srv/tftp/fai/pxelinux.cfg/C0A8xxyy in append line, like FAI_FLAGS, 
> FAI_CONFIG_SRC and FAI_ACTION, but since append line can be at most
> 255 
> chars there's not much space... I's good just for very small
> "secrets" 
> (that gets transferred in the clear, hence the need to reconfigure
> the 
> switches).
> 

-- 
Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz |
Catalyst Cloud: | This space intentionally left blank
https://catalystcloud.nz |