Re: sysupdate: Limit update to at most one major version

2024-01-02 Thread Nils Kattenbeck
Continuing in https://github.com/systemd/systemd/issues/30695

On Tue, Jan 2, 2024 at 2:06 PM Lennart Poettering
 wrote:
>
> On Di, 02.01.24 13:49, Nils Kattenbeck (nilskem...@gmail.com) wrote:
>
> > > I'd be fine with adding MaxVersion=. Happy to review a patch, merge
> > > something like this (at least file an RFE issue)
> >
> > Should that be inclusive or exclusive? Naming it MaxVersion would
> > imply it to be inclusive though an exclusive bound would likely be
> > more useful most of the time. One could then specify MaxVersion=1.3.0
> > in their 1.2.x images and once they have an upgrade path they would
> > explicitly raise the max version in e.g. 1.2.15. Otherwise they would
> > have to specify 1.99.99.
> > In retrospect a VersionBound= property with syntax similar to
> > ConditionKernelVersion= would have been better though I guess that
> > ship has sailed - or is it? Is sd-sysupdate still considered
> > experimental? Not sure if this warrants such a change though :shrug:
>
> We do not allow "=" nor "<" in version strings, as per
> https://uapi-group.org/specifications/specs/version_format_specification/.
>
> Hence we could use that fact and say: "MaxVersion= <=47.11",
> "MaxVersion= <47.11" could be used to make the type of version
> comparison explicit. This would implement a tiny subset of the
> ConditionKernelVersion= logic, and simply default to imply <= if the
> comparison is not specified explicitly.
>
> Of course, a similar logic should then be implemented for MinVersion,
> i.e. >= and >
>
> > Should we continue this discussion on the mailing list or an issue?
>
> Issue is better.
>
> Lennart
>
> --
> Lennart Poettering, Berlin


Re: Masking swap.target to disable swaps (on old systemd versions)

2024-01-02 Thread Andrei Borzenkov

On 02.01.2024 18:47, Max Gautier wrote:

Hi,

Is masking `swap.target` a reliable/supported way to disable all swaps
(partition, file, whatever) ?

I used that approach in Kubespray[1] (a k8s installer) while refactoring
our "disable swap" steps, but it looks likes it does not work on Centos
7 [2] (more specifically, systemd 219) ; and in fact while it seemed
obvious to me at the time, I didn't find it documented in
systemd.special for instance.

It does work on more recent version on systemd, my problem is that I
don't know how recent that is. 255 is fine, 219 is not.

Apparently, the .swap unit generated by systemd-fstab-generator does
shows as "inactive" in `systemctl status` (see [2] last comment), but
the swap **is activated** regardless.

I've searched issues, PRs and commits grepping for swap.target, but I
have not found anything relevant (the only thing was
https://github.com/systemd/systemd/commit/90060fa6605446bef7078867423b691e4effa575,
but it's way before 219...)


● dev-mapper-centos\x2dswap.swap - /dev/mapper/centos-swap
Loaded: loaded (/etc/fstab; bad; vendor preset: disabled)
Active: inactive (dead) since Tue 2024-01-02 02:15:31 EST; 2h 6min ago
  What: /dev/mapper/centos-swap

The `bad` part in the Loaded line makes me think maybe something else is
creating/activating that swap than systemd-fstab-generator ? From the
systemctl man page this conveys that "the unit file is invalid or some
other error occurred", but the generated unit looks fine to me...


If anyone has had some success trying similar things on old systemd
versions, please let me know if you can.


[1]: https://github.com/kubernetes-sigs/kubespray
[2]: https://github.com/kubernetes-sigs/kubespray/issues/10741



As clearly visible in "systemctl show" output

WantedBy=dev-mapper-centos\x2dswap.device

So swap is activated when systemd gets notification that device appeared.



Thanks a lot for your time





Masking swap.target to disable swaps (on old systemd versions)

2024-01-02 Thread Max Gautier
Hi,

Is masking `swap.target` a reliable/supported way to disable all swaps
(partition, file, whatever) ?

I used that approach in Kubespray[1] (a k8s installer) while refactoring
our "disable swap" steps, but it looks likes it does not work on Centos
7 [2] (more specifically, systemd 219) ; and in fact while it seemed
obvious to me at the time, I didn't find it documented in
systemd.special for instance.

It does work on more recent version on systemd, my problem is that I
don't know how recent that is. 255 is fine, 219 is not.

Apparently, the .swap unit generated by systemd-fstab-generator does
shows as "inactive" in `systemctl status` (see [2] last comment), but
the swap **is activated** regardless.

I've searched issues, PRs and commits grepping for swap.target, but I
have not found anything relevant (the only thing was
https://github.com/systemd/systemd/commit/90060fa6605446bef7078867423b691e4effa575,
but it's way before 219...)


● dev-mapper-centos\x2dswap.swap - /dev/mapper/centos-swap
   Loaded: loaded (/etc/fstab; bad; vendor preset: disabled)
   Active: inactive (dead) since Tue 2024-01-02 02:15:31 EST; 2h 6min ago
 What: /dev/mapper/centos-swap

The `bad` part in the Loaded line makes me think maybe something else is
creating/activating that swap than systemd-fstab-generator ? From the
systemctl man page this conveys that "the unit file is invalid or some
other error occurred", but the generated unit looks fine to me...


If anyone has had some success trying similar things on old systemd
versions, please let me know if you can.


[1]: https://github.com/kubernetes-sigs/kubespray
[2]: https://github.com/kubernetes-sigs/kubespray/issues/10741


Thanks a lot for your time

-- 
Max Gautier


Re: Can mkosi replace Kickkstart / Calamares?

2024-01-02 Thread Lennart Poettering
On Mo, 25.12.23 02:39, Patrick Schleizer (patrick-mailingli...@whonix.org) 
wrote:

> Hi,
>
> I am maintaining a systemd, Debian-based Linux distribution (Kicksecure) and
> am considering moving to mkosi as the "base image creation tool".
>
> It seems mkosi is a fine OS image builder. With systemd-repart, you even
> solved the resizing of partitions at the first boot, which is magic.
>
> Suppose a Linux distribution is providing an OS image that can be written to
> USB. Maybe soon, even to a CD/DVD. [1]
>
> Suppose that OS image is supposed to be able to act as an installer, so the
> user can use it to install it on an internal hard drive.
>
> Is something like Kickstart or Calamares still required? It seems (at least
> Calamares, whose code I am reading) is kind of "yet another OS image
> builder". It doesn't build an image but instead writes to a hard drive.
> However, I find it problematic that a lot of code (creating partition
> tables, creating file systems, making bootable) is duplicated. [2]

I don't really know what Kickstart/Calamares really do.

But it's certainly our intention to allow systemd-repart to operate
like an installer, in the sense that you boot from a USB stick and you
can use systemd-repart to copy the relevant partitions you just booted
from to a target disk very efficiently, which will then be basically
the same OS, just with maybe differently sized data/home partitions,
new uuids, different crypto keys and such.

More specifically, systemd-repart + bootctl install +
systemd-firstboot is supposed to be enough to do what a classic
installer disk can do on traditional OSes.

Note that currently there are still some gaps, but people are workng
on this in various places.

> Do you have any suggestions?
>
> Did you envision replacing installers, or do you already have tools for
> that?

Well, depends on what you mean by "installers". We certainly have no
interest to replace a package-based installer. But we certainly do
want to provide you with basic tools which you can combine into an A/B
image-based OS installer

> [2] But what about installer questions, customization such as time zone,
> keyboard layout? I think the crucial question for an installer is the target
> drive, and that's it. Perhaps partitioning and file system choices, but that
> is more for geeks. How about time zone, keyboard layout? Valid points. But I
> think those would be better handled through a first-boot GUI wizard.

systemd-firstboot is supposed to be just that – but it only covers the
offline and console cases. It's also supposed to be useful as a blueprint
to implement something similar in a graphical tool.

systemd-firstboot can be used in two modes. In "offline" mode, where
you call it from the cmdline and specify --root= or --image= to let it
operate directly on an OS tree you mounted somewhere or on a block
device/image file you have accessible. Or in "online" mode where it is
run at first boot, and asks the user interactively.

systemd-firstboot covers hostname, locale, keyboard, timezone, root pw
currently. In systemd git main you also fine a "homectl firstboot"
command which can prompt the user interactively for a user to create
at boot.

Regarding partitioning: my thinking was that installers would ship
multiple alternative sets of repart .conf files, of which the first
that can be applied is applied or of which the user can pick one
explicitly, depending on the use case. The focus is clearly on
automatic partitioning here though, if people want to manually and
precisely set the sizes of each partition in a UI, then repart is not
the tool they should use.

Lennart

--
Lennart Poettering, Berlin


Re: systemd-sysupdate support for slow rollout (aka A/B testing)

2024-01-02 Thread Lennart Poettering
On Di, 02.01.24 14:40, Nils Kattenbeck (nilskem...@gmail.com) wrote:

> > > does sysupdate currently support any way to slowly roll out updates
> > > where the server providing the files can be in control? [...]
> >
> > This is currently not available, no.
> >
> > The idea so far was always that the server is dumb, and the client
> > picks the release it wants.
>
> I feel like it would be more flexible to have the client mostly
> handling transferring and applying the data and any additional logic
> should be handled by either the server or secondary applications which
> call into sysupdate (or its future dbus API).

Well, our idea was really that you can use a bog standard static http
server to serve this stuff and get as much of the feature set as you
could possibly get.

> > I have thought about this usecase a while back, and my thinking was
> > that such a staged update logic should be driven by the machine
> > ID. i.e. we should teach sysupdate a simple logic that allows pattern
> > matching of new versions based on some arithmetic of the machine
> > ID. More specifically, include some value in the URL pattern that
> > indicates the percentage of hosts that shall update to this
> > release. Then, each client takes its machine ID, treats it as an
> > integer and calculates modulo 100 of it or so, and then checks if the
> > resulting value is below the intended percentage, and if so it
> > updates, otherwise it doesn't.
> >
> > (or something like that, the above is probably not ideal, since it
> > would mean it's always the same hosts that try a new release first,
> > and it probably should be evened out across the set of clients).
>
> Any logic based on the machine ID would also have the problem I
> mentioned below that the ratios would be skewed for stateless devices
> which cannot persist their machine id to disk.
>
> One would at least be able to override it with something persistent
> like a MAC address though this could be exposed as some argument or
> environment variable which a secondary application could set before
> calling sysupdate.

Yeah, we have something similar with the seed logic of repart already
(by default repart derives partition uuids to create from the machine
ID but you can also specify a seed explicitly), I see no problem with
adding the same to sysupdate. That should be trivial and just adopt a
scheme we already introduced at one more place. I am fully on board
with that.

I'd also be fine with a kernel cmdline option or so which allows
fine-tuning where PID 1 takes the machine ID from if it generates a
new one. Right now it takes the SMBIOS ID when running in a VM, and
has a similar mechanism for containers. We could probably add
something to optionally tell it to pull the ID to use from smbios/dt
even on physical systems, or even from the TPM.

(I am not sure how far the MAC thing would work. AFAIK on a lot of
embedded systems the MAC is expected to be randomly generated by
software, hence would not be useful as an identifier here. Also, it's
directly publically visible, which makes it too easily guessable. And
there's the raciness issue: usually device drivers for such auxiliary
hardware are loaded relatively late, and we want the machine ID relatively
early.)

> > > I also remember there being a discussion about plugging in different
> > > sd-pull like implementations/backends[1] to support delta updates,
> > > other transports, or TLS client authentication. This could at least be
> > > adapted to support my idea to send the machine-id as an HTTP header
> > > (e.g. X-MACHINE-ID).
> >
> > If we can avoid it, I'd always adopt a logic whether identifying info
> > doesn't have to be sent to the server. After all the logic should be
> > generic and applicable in scenarios where the client should get
> > anonymity as much as it wants.
>
> If the client automatically applies updates the server could always
> deliver an image which exposes information by e.g. simply updating the
> Path= to include %m somewhere in it.
> Though I agree that always sending such information in headers would
> not be optimal.
>
> I also found out that sd-import drops query parameters from the URL.

I guess we could change this. if a query parameter is explicitly
specified I see no reason to unconditionally drop it.

> If this were not the case my use case would already be possible by
> embedding the machine ID as part of the query.
> This would also make it possible to opt in to sending the information.
>
> The problem I think is that there are two user groups of sysupdate
> with different requirements.
> On one hand we have end user distributions with A/B style updates
> where the distribution only has limited to no interest over precise
> control of updates and user devices and the users wish for anonymity.
> On the other hand though are enterprises which deploy sysupdate for
> (I)IoT devices. In these case devices commonly have to be registered
> anyhow, and the enterprise controls how updates are 

Re: systemd-sysupdate support for slow rollout (aka A/B testing)

2024-01-02 Thread Nils Kattenbeck
> > does sysupdate currently support any way to slowly roll out updates
> > where the server providing the files can be in control? [...]
>
> This is currently not available, no.
>
> The idea so far was always that the server is dumb, and the client
> picks the release it wants.

I feel like it would be more flexible to have the client mostly
handling transferring and applying the data and any additional logic
should be handled by either the server or secondary applications which
call into sysupdate (or its future dbus API).

> I have thought about this usecase a while back, and my thinking was
> that such a staged update logic should be driven by the machine
> ID. i.e. we should teach sysupdate a simple logic that allows pattern
> matching of new versions based on some arithmetic of the machine
> ID. More specifically, include some value in the URL pattern that
> indicates the percentage of hosts that shall update to this
> release. Then, each client takes its machine ID, treats it as an
> integer and calculates modulo 100 of it or so, and then checks if the
> resulting value is below the intended percentage, and if so it
> updates, otherwise it doesn't.
>
> (or something like that, the above is probably not ideal, since it
> would mean it's always the same hosts that try a new release first,
> and it probably should be evened out across the set of clients).

Any logic based on the machine ID would also have the problem I
mentioned below that the ratios would be skewed for stateless devices
which cannot persist their machine id to disk.
One would at least be able to override it with something persistent
like a MAC address though this could be exposed as some argument or
environment variable which a secondary application could set before
calling sysupdate.

> This would then mean for the server that it would first serve
> foobar_47.11_3.raw which would be version 47.11 of the OS, and 3% of
> the hosts would update to it. And then, once you collected enough
> feedback you'd rename the file to foobar_47.11_25.raw and 25% of the
> hosts would switch over. Finally you'd set the value to 100 (or maybe
> just drop it, which should be considered equivalent to 100), and then
> all remaining hosts would update.
>
> The effect of this is that client's could still explicitly upgrade if
> they want, and the updates would be entirely driven by the clients,
> but simply via naming the download images the server can control that
> "by default" only the chosen number of clients update.

The explicit update by clients is definitely a nice bonus though this
can also be achieved by a secondary set of definitions looking for
files under s3.domain.com/rc/.

> > Currently it seems like I would have to implement a different service
> > which calls the sysupdate binary (or uses dbus once #28134 has landed)
> > and then decides based on some other information.
> >
> > One idea I had would be that systemd-pull could send the machine-id
> > based on which the server could then decide to provide the newer file
> > (e.g. last two chars == "00" would roll it out to ~1/255). Though I am
> > not sure if sd-pull is supposed to be "anonymous", i.e. do not provide
> > this identifying information. Another drawback of this would be that
> > stateless systems which reboot often get a new machine-id each boot,
> > thus having an increased chance to get the newer version.
>
> So this idea is not entirely different from my idea, I was just
> thinking about pushing this into sysupdate rather than pull.
>
> > Does anything like this already exist or is planned? Or should that be
> > done by different applications on the client side?
>
> I think it makes a ton of sense to add this to sysupdate. Would love
> to review/merge a patch for that.
>
> > I also remember there being a discussion about plugging in different
> > sd-pull like implementations/backends[1] to support delta updates,
> > other transports, or TLS client authentication. This could at least be
> > adapted to support my idea to send the machine-id as an HTTP header
> > (e.g. X-MACHINE-ID).
>
> If we can avoid it, I'd always adopt a logic whether identifying info
> doesn't have to be sent to the server. After all the logic should be
> generic and applicable in scenarios where the client should get
> anonymity as much as it wants.

If the client automatically applies updates the server could always
deliver an image which exposes information by e.g. simply updating the
Path= to include %m somewhere in it.
Though I agree that always sending such information in headers would
not be optimal.

I also found out that sd-import drops query parameters from the URL.
If this were not the case my use case would already be possible by
embedding the machine ID as part of the query.
This would also make it possible to opt in to sending the information.

The problem I think is that there are two user groups of sysupdate
with different requirements.
On one hand we have end user distributions with A/B 

Re: systemd-sysupdate support for slow rollout (aka A/B testing)

2024-01-02 Thread Lennart Poettering
On Di, 02.01.24 13:11, Simon McVittie (s...@collabora.com) wrote:

> Prior art: Debian/Ubuntu apt does slow rollout for packages like
> this, with simple filesystem-based http mirrors combined with "smart"
> clients. It works by adding a Phased-Update-Percentage field to the
> metadata of each package. The client calculates some sort of ID for itself
> (I don't know precisely how), and then takes the upgrade if it finds that
> its ID is in the first x% of the available range.
>
> If I understand correctly, Ubuntu is using this mechanism in production
> but Debian is not.
>
> Using some sort of hash of the machine ID + the proposed version would
> probably have the behaviour you want, of choosing a different x% of
> machines to be the early-adopter set for each update?

Yes, this is what I think would be the right approach.

> > This would then mean for the server that it would first serve
> > foobar_47.11_3.raw which would be version 47.11 of the OS, and 3% of
> > the hosts would update to it. And then, once you collected enough
> > feedback you'd rename the file to foobar_47.11_25.raw and 25% of the
> > hosts would switch over. Finally you'd set the value to 100 (or maybe
> > just drop it, which should be considered equivalent to 100), and then
> > all remaining hosts would update.
>
> If you're using a hash of the machine ID + the proposed version as
> your randomization, then I think you'd want to have a single image (or
> version ID, or some other unique identifier) for each proposed update, and
> separately, a metadata field that sets *x* in the instruction "if you have
> figured out that you are in the first x% of machines, upgrade". Otherwise,
> publishing foobar_47.11_3.raw followed by foobar_47.11_25.raw would be
> more likely to result in approximately (3% + 25% = 28%) of machines
> upgrading[1], because the client doesn't know that it's actually the
> same update and would "re-roll the dice" for each republished name.

My thinking was that clients would look at multiple entries which only
differ by the percentage (i.e. are identical in name and version) and
drop all of them but the one with the highest percentage, and ignore
all others.

Lennart

--
Lennart Poettering, Berlin


Re: systemd-sysupdate support for slow rollout (aka A/B testing)

2024-01-02 Thread Simon McVittie
On Tue, 02 Jan 2024 at 11:16:15 +0100, Lennart Poettering wrote:
> The idea so far was always that the server is dumb, and the client
> picks the release it wants.
> 
> I have thought about this usecase a while back, and my thinking was
> that such a staged update logic should be driven by the machine
> ID. i.e. we should teach sysupdate a simple logic that allows pattern
> matching of new versions based on some arithmetic of the machine
> ID. More specifically, include some value in the URL pattern that
> indicates the percentage of hosts that shall update to this
> release. Then, each client takes its machine ID, treats it as an
> integer and calculates modulo 100 of it or so, and then checks if the
> resulting value is below the intended percentage, and if so it
> updates, otherwise it doesn't.
> 
> (or something like that, the above is probably not ideal, since it
> would mean it's always the same hosts that try a new release first,
> and it probably should be evened out across the set of clients).

Prior art: Debian/Ubuntu apt does slow rollout for packages like
this, with simple filesystem-based http mirrors combined with "smart"
clients. It works by adding a Phased-Update-Percentage field to the
metadata of each package. The client calculates some sort of ID for itself
(I don't know precisely how), and then takes the upgrade if it finds that
its ID is in the first x% of the available range.

If I understand correctly, Ubuntu is using this mechanism in production
but Debian is not.

Using some sort of hash of the machine ID + the proposed version would
probably have the behaviour you want, of choosing a different x% of
machines to be the early-adopter set for each update?

> This would then mean for the server that it would first serve
> foobar_47.11_3.raw which would be version 47.11 of the OS, and 3% of
> the hosts would update to it. And then, once you collected enough
> feedback you'd rename the file to foobar_47.11_25.raw and 25% of the
> hosts would switch over. Finally you'd set the value to 100 (or maybe
> just drop it, which should be considered equivalent to 100), and then
> all remaining hosts would update.

If you're using a hash of the machine ID + the proposed version as
your randomization, then I think you'd want to have a single image (or
version ID, or some other unique identifier) for each proposed update, and
separately, a metadata field that sets *x* in the instruction "if you have
figured out that you are in the first x% of machines, upgrade". Otherwise,
publishing foobar_47.11_3.raw followed by foobar_47.11_25.raw would be
more likely to result in approximately (3% + 25% = 28%) of machines
upgrading[1], because the client doesn't know that it's actually the
same update and would "re-roll the dice" for each republished name.

smcv

[1] more precisely, (0.03 + 0.25 - (0.03 * 0.25)) because of how
conditional probabilities combine


Re: sysupdate: Limit update to at most one major version

2024-01-02 Thread Lennart Poettering
On Di, 02.01.24 13:49, Nils Kattenbeck (nilskem...@gmail.com) wrote:

> > I'd be fine with adding MaxVersion=. Happy to review a patch, merge
> > something like this (at least file an RFE issue)
>
> Should that be inclusive or exclusive? Naming it MaxVersion would
> imply it to be inclusive though an exclusive bound would likely be
> more useful most of the time. One could then specify MaxVersion=1.3.0
> in their 1.2.x images and once they have an upgrade path they would
> explicitly raise the max version in e.g. 1.2.15. Otherwise they would
> have to specify 1.99.99.
> In retrospect a VersionBound= property with syntax similar to
> ConditionKernelVersion= would have been better though I guess that
> ship has sailed - or is it? Is sd-sysupdate still considered
> experimental? Not sure if this warrants such a change though :shrug:

We do not allow "=" nor "<" in version strings, as per
https://uapi-group.org/specifications/specs/version_format_specification/.

Hence we could use that fact and say: "MaxVersion= <=47.11",
"MaxVersion= <47.11" could be used to make the type of version
comparison explicit. This would implement a tiny subset of the
ConditionKernelVersion= logic, and simply default to imply <= if the
comparison is not specified explicitly.

Of course, a similar logic should then be implemented for MinVersion,
i.e. >= and >

> Should we continue this discussion on the mailing list or an issue?

Issue is better.

Lennart

--
Lennart Poettering, Berlin


Re: sysupdate: Limit update to at most one major version

2024-01-02 Thread Nils Kattenbeck
> I'd be fine with adding MaxVersion=. Happy to review a patch, merge
> something like this (at least file an RFE issue)

Should that be inclusive or exclusive? Naming it MaxVersion would
imply it to be inclusive though an exclusive bound would likely be
more useful most of the time. One could then specify MaxVersion=1.3.0
in their 1.2.x images and once they have an upgrade path they would
explicitly raise the max version in e.g. 1.2.15. Otherwise they would
have to specify 1.99.99.
In retrospect a VersionBound= property with syntax similar to
ConditionKernelVersion= would have been better though I guess that
ship has sailed - or is it? Is sd-sysupdate still considered
experimental? Not sure if this warrants such a change though :shrug:

Should we continue this discussion on the mailing list or an issue?

Kind regards, Nils


Re: sysupdate: Limit update to at most one major version

2024-01-02 Thread Lennart Poettering
On So, 31.12.23 14:43, Nils Kattenbeck (nilskem...@gmail.com) wrote:

> Hello,
>
> we are currently using sd-sysupdate to roll out updates and we're wondering
> if there is any possibility to limit updates to consider at most one next
> major version. This would allow us to write the software to handle only
> data migrations from the previous major version instead of any version
> beforehand.
> The only thing I have been able to find is MinVersion= which seems to do
> exactly the opposite of what we would want to do.

I'd be fine with adding MaxVersion=. Happy to review a patch, merge
something like this (at least file an RFE issue)

Lennart

--
Lennart Poettering, Berlin


Re: systemd-sysupdate support for slow rollout (aka A/B testing)

2024-01-02 Thread Lennart Poettering
On Mi, 20.12.23 19:04, Nils Kattenbeck (nilskem...@gmail.com) wrote:

> Hey everyone,
>
> does sysupdate currently support any way to slowly roll out updates
> where the server providing the files can be in control? This would be
> used to slowly make a new version available and have it at e.g. 1%
> adoption for a day to monitor regressions before increasing the
> coverage. I was unable to find any information about it in the
> documentation.

This is currently not available, no.

The idea so far was always that the server is dumb, and the client
picks the release it wants.

I have thought about this usecase a while back, and my thinking was
that such a staged update logic should be driven by the machine
ID. i.e. we should teach sysupdate a simple logic that allows pattern
matching of new versions based on some arithmetic of the machine
ID. More specifically, include some value in the URL pattern that
indicates the percentage of hosts that shall update to this
release. Then, each client takes its machine ID, treats it as an
integer and calculates modulo 100 of it or so, and then checks if the
resulting value is below the intended percentage, and if so it
updates, otherwise it doesn't.

(or something like that, the above is probably not ideal, since it
would mean it's always the same hosts that try a new release first,
and it probably should be evened out across the set of clients).

This would then mean for the server that it would first serve
foobar_47.11_3.raw which would be version 47.11 of the OS, and 3% of
the hosts would update to it. And then, once you collected enough
feedback you'd rename the file to foobar_47.11_25.raw and 25% of the
hosts would switch over. Finally you'd set the value to 100 (or maybe
just drop it, which should be considered equivalent to 100), and then
all remaining hosts would update.

The effect of this is that client's could still explicitly upgrade if
they want, and the updates would be entirely driven by the clients,
but simply via naming the download images the server can control that
"by default" only the chosen number of clients update.

> Currently it seems like I would have to implement a different service
> which calls the sysupdate binary (or uses dbus once #28134 has landed)
> and then decides based on some other information.
>
> One idea I had would be that systemd-pull could send the machine-id
> based on which the server could then decide to provide the newer file
> (e.g. last two chars == "00" would roll it out to ~1/255). Though I am
> not sure if sd-pull is supposed to be "anonymous", i.e. do not provide
> this identifying information. Another drawback of this would be that
> stateless systems which reboot often get a new machine-id each boot,
> thus having an increased chance to get the newer version.

So this idea is not entirely different from my idea, I was just
thinking about pushing this into sysupdate rather than pull.

> Does anything like this already exist or is planned? Or should that be
> done by different applications on the client side?

I think it makes a ton of sense to add this to sysupdate. Would love
to review/merge a patch for that.

> I also remember there being a discussion about plugging in different
> sd-pull like implementations/backends[1] to support delta updates,
> other transports, or TLS client authentication. This could at least be
> adapted to support my idea to send the machine-id as an HTTP header
> (e.g. X-MACHINE-ID).

If we can avoid it, I'd always adopt a logic whether identifying info
doesn't have to be sent to the server. After all the logic should be
generic and applicable in scenarios where the client should get
anonymity as much as it wants.

The machine-id we usually consider a "half-secret", i.e. all local
programs get access to it (unless sandboxed), but they are not
supposed to be send it across the wire. If they really need to send
some identifier across the wire they should derive an app-specific ID
instead, which we make easy to acquire via
sd_id128_get_machine_app_specific().

But better than app-specific machine IDs are no machine IDs at all in
the protocol, if we can get away with it. Hence, my idea of doing the
rollout percentage logic client-side.

Lennart

--
Lennart Poettering, Berlin