Re: sysupdate: Limit update to at most one major version
Continuing in https://github.com/systemd/systemd/issues/30695 On Tue, Jan 2, 2024 at 2:06 PM Lennart Poettering wrote: > > On Di, 02.01.24 13:49, Nils Kattenbeck (nilskem...@gmail.com) wrote: > > > > I'd be fine with adding MaxVersion=. Happy to review a patch, merge > > > something like this (at least file an RFE issue) > > > > Should that be inclusive or exclusive? Naming it MaxVersion would > > imply it to be inclusive though an exclusive bound would likely be > > more useful most of the time. One could then specify MaxVersion=1.3.0 > > in their 1.2.x images and once they have an upgrade path they would > > explicitly raise the max version in e.g. 1.2.15. Otherwise they would > > have to specify 1.99.99. > > In retrospect a VersionBound= property with syntax similar to > > ConditionKernelVersion= would have been better though I guess that > > ship has sailed - or is it? Is sd-sysupdate still considered > > experimental? Not sure if this warrants such a change though :shrug: > > We do not allow "=" nor "<" in version strings, as per > https://uapi-group.org/specifications/specs/version_format_specification/. > > Hence we could use that fact and say: "MaxVersion= <=47.11", > "MaxVersion= <47.11" could be used to make the type of version > comparison explicit. This would implement a tiny subset of the > ConditionKernelVersion= logic, and simply default to imply <= if the > comparison is not specified explicitly. > > Of course, a similar logic should then be implemented for MinVersion, > i.e. >= and > > > > Should we continue this discussion on the mailing list or an issue? > > Issue is better. > > Lennart > > -- > Lennart Poettering, Berlin
Re: Masking swap.target to disable swaps (on old systemd versions)
On 02.01.2024 18:47, Max Gautier wrote: Hi, Is masking `swap.target` a reliable/supported way to disable all swaps (partition, file, whatever) ? I used that approach in Kubespray[1] (a k8s installer) while refactoring our "disable swap" steps, but it looks likes it does not work on Centos 7 [2] (more specifically, systemd 219) ; and in fact while it seemed obvious to me at the time, I didn't find it documented in systemd.special for instance. It does work on more recent version on systemd, my problem is that I don't know how recent that is. 255 is fine, 219 is not. Apparently, the .swap unit generated by systemd-fstab-generator does shows as "inactive" in `systemctl status` (see [2] last comment), but the swap **is activated** regardless. I've searched issues, PRs and commits grepping for swap.target, but I have not found anything relevant (the only thing was https://github.com/systemd/systemd/commit/90060fa6605446bef7078867423b691e4effa575, but it's way before 219...) ● dev-mapper-centos\x2dswap.swap - /dev/mapper/centos-swap Loaded: loaded (/etc/fstab; bad; vendor preset: disabled) Active: inactive (dead) since Tue 2024-01-02 02:15:31 EST; 2h 6min ago What: /dev/mapper/centos-swap The `bad` part in the Loaded line makes me think maybe something else is creating/activating that swap than systemd-fstab-generator ? From the systemctl man page this conveys that "the unit file is invalid or some other error occurred", but the generated unit looks fine to me... If anyone has had some success trying similar things on old systemd versions, please let me know if you can. [1]: https://github.com/kubernetes-sigs/kubespray [2]: https://github.com/kubernetes-sigs/kubespray/issues/10741 As clearly visible in "systemctl show" output WantedBy=dev-mapper-centos\x2dswap.device So swap is activated when systemd gets notification that device appeared. Thanks a lot for your time
Masking swap.target to disable swaps (on old systemd versions)
Hi, Is masking `swap.target` a reliable/supported way to disable all swaps (partition, file, whatever) ? I used that approach in Kubespray[1] (a k8s installer) while refactoring our "disable swap" steps, but it looks likes it does not work on Centos 7 [2] (more specifically, systemd 219) ; and in fact while it seemed obvious to me at the time, I didn't find it documented in systemd.special for instance. It does work on more recent version on systemd, my problem is that I don't know how recent that is. 255 is fine, 219 is not. Apparently, the .swap unit generated by systemd-fstab-generator does shows as "inactive" in `systemctl status` (see [2] last comment), but the swap **is activated** regardless. I've searched issues, PRs and commits grepping for swap.target, but I have not found anything relevant (the only thing was https://github.com/systemd/systemd/commit/90060fa6605446bef7078867423b691e4effa575, but it's way before 219...) ● dev-mapper-centos\x2dswap.swap - /dev/mapper/centos-swap Loaded: loaded (/etc/fstab; bad; vendor preset: disabled) Active: inactive (dead) since Tue 2024-01-02 02:15:31 EST; 2h 6min ago What: /dev/mapper/centos-swap The `bad` part in the Loaded line makes me think maybe something else is creating/activating that swap than systemd-fstab-generator ? From the systemctl man page this conveys that "the unit file is invalid or some other error occurred", but the generated unit looks fine to me... If anyone has had some success trying similar things on old systemd versions, please let me know if you can. [1]: https://github.com/kubernetes-sigs/kubespray [2]: https://github.com/kubernetes-sigs/kubespray/issues/10741 Thanks a lot for your time -- Max Gautier
Re: Can mkosi replace Kickkstart / Calamares?
On Mo, 25.12.23 02:39, Patrick Schleizer (patrick-mailingli...@whonix.org) wrote: > Hi, > > I am maintaining a systemd, Debian-based Linux distribution (Kicksecure) and > am considering moving to mkosi as the "base image creation tool". > > It seems mkosi is a fine OS image builder. With systemd-repart, you even > solved the resizing of partitions at the first boot, which is magic. > > Suppose a Linux distribution is providing an OS image that can be written to > USB. Maybe soon, even to a CD/DVD. [1] > > Suppose that OS image is supposed to be able to act as an installer, so the > user can use it to install it on an internal hard drive. > > Is something like Kickstart or Calamares still required? It seems (at least > Calamares, whose code I am reading) is kind of "yet another OS image > builder". It doesn't build an image but instead writes to a hard drive. > However, I find it problematic that a lot of code (creating partition > tables, creating file systems, making bootable) is duplicated. [2] I don't really know what Kickstart/Calamares really do. But it's certainly our intention to allow systemd-repart to operate like an installer, in the sense that you boot from a USB stick and you can use systemd-repart to copy the relevant partitions you just booted from to a target disk very efficiently, which will then be basically the same OS, just with maybe differently sized data/home partitions, new uuids, different crypto keys and such. More specifically, systemd-repart + bootctl install + systemd-firstboot is supposed to be enough to do what a classic installer disk can do on traditional OSes. Note that currently there are still some gaps, but people are workng on this in various places. > Do you have any suggestions? > > Did you envision replacing installers, or do you already have tools for > that? Well, depends on what you mean by "installers". We certainly have no interest to replace a package-based installer. But we certainly do want to provide you with basic tools which you can combine into an A/B image-based OS installer > [2] But what about installer questions, customization such as time zone, > keyboard layout? I think the crucial question for an installer is the target > drive, and that's it. Perhaps partitioning and file system choices, but that > is more for geeks. How about time zone, keyboard layout? Valid points. But I > think those would be better handled through a first-boot GUI wizard. systemd-firstboot is supposed to be just that – but it only covers the offline and console cases. It's also supposed to be useful as a blueprint to implement something similar in a graphical tool. systemd-firstboot can be used in two modes. In "offline" mode, where you call it from the cmdline and specify --root= or --image= to let it operate directly on an OS tree you mounted somewhere or on a block device/image file you have accessible. Or in "online" mode where it is run at first boot, and asks the user interactively. systemd-firstboot covers hostname, locale, keyboard, timezone, root pw currently. In systemd git main you also fine a "homectl firstboot" command which can prompt the user interactively for a user to create at boot. Regarding partitioning: my thinking was that installers would ship multiple alternative sets of repart .conf files, of which the first that can be applied is applied or of which the user can pick one explicitly, depending on the use case. The focus is clearly on automatic partitioning here though, if people want to manually and precisely set the sizes of each partition in a UI, then repart is not the tool they should use. Lennart -- Lennart Poettering, Berlin
Re: systemd-sysupdate support for slow rollout (aka A/B testing)
On Di, 02.01.24 14:40, Nils Kattenbeck (nilskem...@gmail.com) wrote: > > > does sysupdate currently support any way to slowly roll out updates > > > where the server providing the files can be in control? [...] > > > > This is currently not available, no. > > > > The idea so far was always that the server is dumb, and the client > > picks the release it wants. > > I feel like it would be more flexible to have the client mostly > handling transferring and applying the data and any additional logic > should be handled by either the server or secondary applications which > call into sysupdate (or its future dbus API). Well, our idea was really that you can use a bog standard static http server to serve this stuff and get as much of the feature set as you could possibly get. > > I have thought about this usecase a while back, and my thinking was > > that such a staged update logic should be driven by the machine > > ID. i.e. we should teach sysupdate a simple logic that allows pattern > > matching of new versions based on some arithmetic of the machine > > ID. More specifically, include some value in the URL pattern that > > indicates the percentage of hosts that shall update to this > > release. Then, each client takes its machine ID, treats it as an > > integer and calculates modulo 100 of it or so, and then checks if the > > resulting value is below the intended percentage, and if so it > > updates, otherwise it doesn't. > > > > (or something like that, the above is probably not ideal, since it > > would mean it's always the same hosts that try a new release first, > > and it probably should be evened out across the set of clients). > > Any logic based on the machine ID would also have the problem I > mentioned below that the ratios would be skewed for stateless devices > which cannot persist their machine id to disk. > > One would at least be able to override it with something persistent > like a MAC address though this could be exposed as some argument or > environment variable which a secondary application could set before > calling sysupdate. Yeah, we have something similar with the seed logic of repart already (by default repart derives partition uuids to create from the machine ID but you can also specify a seed explicitly), I see no problem with adding the same to sysupdate. That should be trivial and just adopt a scheme we already introduced at one more place. I am fully on board with that. I'd also be fine with a kernel cmdline option or so which allows fine-tuning where PID 1 takes the machine ID from if it generates a new one. Right now it takes the SMBIOS ID when running in a VM, and has a similar mechanism for containers. We could probably add something to optionally tell it to pull the ID to use from smbios/dt even on physical systems, or even from the TPM. (I am not sure how far the MAC thing would work. AFAIK on a lot of embedded systems the MAC is expected to be randomly generated by software, hence would not be useful as an identifier here. Also, it's directly publically visible, which makes it too easily guessable. And there's the raciness issue: usually device drivers for such auxiliary hardware are loaded relatively late, and we want the machine ID relatively early.) > > > I also remember there being a discussion about plugging in different > > > sd-pull like implementations/backends[1] to support delta updates, > > > other transports, or TLS client authentication. This could at least be > > > adapted to support my idea to send the machine-id as an HTTP header > > > (e.g. X-MACHINE-ID). > > > > If we can avoid it, I'd always adopt a logic whether identifying info > > doesn't have to be sent to the server. After all the logic should be > > generic and applicable in scenarios where the client should get > > anonymity as much as it wants. > > If the client automatically applies updates the server could always > deliver an image which exposes information by e.g. simply updating the > Path= to include %m somewhere in it. > Though I agree that always sending such information in headers would > not be optimal. > > I also found out that sd-import drops query parameters from the URL. I guess we could change this. if a query parameter is explicitly specified I see no reason to unconditionally drop it. > If this were not the case my use case would already be possible by > embedding the machine ID as part of the query. > This would also make it possible to opt in to sending the information. > > The problem I think is that there are two user groups of sysupdate > with different requirements. > On one hand we have end user distributions with A/B style updates > where the distribution only has limited to no interest over precise > control of updates and user devices and the users wish for anonymity. > On the other hand though are enterprises which deploy sysupdate for > (I)IoT devices. In these case devices commonly have to be registered > anyhow, and the enterprise controls how updates are
Re: systemd-sysupdate support for slow rollout (aka A/B testing)
> > does sysupdate currently support any way to slowly roll out updates > > where the server providing the files can be in control? [...] > > This is currently not available, no. > > The idea so far was always that the server is dumb, and the client > picks the release it wants. I feel like it would be more flexible to have the client mostly handling transferring and applying the data and any additional logic should be handled by either the server or secondary applications which call into sysupdate (or its future dbus API). > I have thought about this usecase a while back, and my thinking was > that such a staged update logic should be driven by the machine > ID. i.e. we should teach sysupdate a simple logic that allows pattern > matching of new versions based on some arithmetic of the machine > ID. More specifically, include some value in the URL pattern that > indicates the percentage of hosts that shall update to this > release. Then, each client takes its machine ID, treats it as an > integer and calculates modulo 100 of it or so, and then checks if the > resulting value is below the intended percentage, and if so it > updates, otherwise it doesn't. > > (or something like that, the above is probably not ideal, since it > would mean it's always the same hosts that try a new release first, > and it probably should be evened out across the set of clients). Any logic based on the machine ID would also have the problem I mentioned below that the ratios would be skewed for stateless devices which cannot persist their machine id to disk. One would at least be able to override it with something persistent like a MAC address though this could be exposed as some argument or environment variable which a secondary application could set before calling sysupdate. > This would then mean for the server that it would first serve > foobar_47.11_3.raw which would be version 47.11 of the OS, and 3% of > the hosts would update to it. And then, once you collected enough > feedback you'd rename the file to foobar_47.11_25.raw and 25% of the > hosts would switch over. Finally you'd set the value to 100 (or maybe > just drop it, which should be considered equivalent to 100), and then > all remaining hosts would update. > > The effect of this is that client's could still explicitly upgrade if > they want, and the updates would be entirely driven by the clients, > but simply via naming the download images the server can control that > "by default" only the chosen number of clients update. The explicit update by clients is definitely a nice bonus though this can also be achieved by a secondary set of definitions looking for files under s3.domain.com/rc/. > > Currently it seems like I would have to implement a different service > > which calls the sysupdate binary (or uses dbus once #28134 has landed) > > and then decides based on some other information. > > > > One idea I had would be that systemd-pull could send the machine-id > > based on which the server could then decide to provide the newer file > > (e.g. last two chars == "00" would roll it out to ~1/255). Though I am > > not sure if sd-pull is supposed to be "anonymous", i.e. do not provide > > this identifying information. Another drawback of this would be that > > stateless systems which reboot often get a new machine-id each boot, > > thus having an increased chance to get the newer version. > > So this idea is not entirely different from my idea, I was just > thinking about pushing this into sysupdate rather than pull. > > > Does anything like this already exist or is planned? Or should that be > > done by different applications on the client side? > > I think it makes a ton of sense to add this to sysupdate. Would love > to review/merge a patch for that. > > > I also remember there being a discussion about plugging in different > > sd-pull like implementations/backends[1] to support delta updates, > > other transports, or TLS client authentication. This could at least be > > adapted to support my idea to send the machine-id as an HTTP header > > (e.g. X-MACHINE-ID). > > If we can avoid it, I'd always adopt a logic whether identifying info > doesn't have to be sent to the server. After all the logic should be > generic and applicable in scenarios where the client should get > anonymity as much as it wants. If the client automatically applies updates the server could always deliver an image which exposes information by e.g. simply updating the Path= to include %m somewhere in it. Though I agree that always sending such information in headers would not be optimal. I also found out that sd-import drops query parameters from the URL. If this were not the case my use case would already be possible by embedding the machine ID as part of the query. This would also make it possible to opt in to sending the information. The problem I think is that there are two user groups of sysupdate with different requirements. On one hand we have end user distributions with A/B
Re: systemd-sysupdate support for slow rollout (aka A/B testing)
On Di, 02.01.24 13:11, Simon McVittie (s...@collabora.com) wrote: > Prior art: Debian/Ubuntu apt does slow rollout for packages like > this, with simple filesystem-based http mirrors combined with "smart" > clients. It works by adding a Phased-Update-Percentage field to the > metadata of each package. The client calculates some sort of ID for itself > (I don't know precisely how), and then takes the upgrade if it finds that > its ID is in the first x% of the available range. > > If I understand correctly, Ubuntu is using this mechanism in production > but Debian is not. > > Using some sort of hash of the machine ID + the proposed version would > probably have the behaviour you want, of choosing a different x% of > machines to be the early-adopter set for each update? Yes, this is what I think would be the right approach. > > This would then mean for the server that it would first serve > > foobar_47.11_3.raw which would be version 47.11 of the OS, and 3% of > > the hosts would update to it. And then, once you collected enough > > feedback you'd rename the file to foobar_47.11_25.raw and 25% of the > > hosts would switch over. Finally you'd set the value to 100 (or maybe > > just drop it, which should be considered equivalent to 100), and then > > all remaining hosts would update. > > If you're using a hash of the machine ID + the proposed version as > your randomization, then I think you'd want to have a single image (or > version ID, or some other unique identifier) for each proposed update, and > separately, a metadata field that sets *x* in the instruction "if you have > figured out that you are in the first x% of machines, upgrade". Otherwise, > publishing foobar_47.11_3.raw followed by foobar_47.11_25.raw would be > more likely to result in approximately (3% + 25% = 28%) of machines > upgrading[1], because the client doesn't know that it's actually the > same update and would "re-roll the dice" for each republished name. My thinking was that clients would look at multiple entries which only differ by the percentage (i.e. are identical in name and version) and drop all of them but the one with the highest percentage, and ignore all others. Lennart -- Lennart Poettering, Berlin
Re: systemd-sysupdate support for slow rollout (aka A/B testing)
On Tue, 02 Jan 2024 at 11:16:15 +0100, Lennart Poettering wrote: > The idea so far was always that the server is dumb, and the client > picks the release it wants. > > I have thought about this usecase a while back, and my thinking was > that such a staged update logic should be driven by the machine > ID. i.e. we should teach sysupdate a simple logic that allows pattern > matching of new versions based on some arithmetic of the machine > ID. More specifically, include some value in the URL pattern that > indicates the percentage of hosts that shall update to this > release. Then, each client takes its machine ID, treats it as an > integer and calculates modulo 100 of it or so, and then checks if the > resulting value is below the intended percentage, and if so it > updates, otherwise it doesn't. > > (or something like that, the above is probably not ideal, since it > would mean it's always the same hosts that try a new release first, > and it probably should be evened out across the set of clients). Prior art: Debian/Ubuntu apt does slow rollout for packages like this, with simple filesystem-based http mirrors combined with "smart" clients. It works by adding a Phased-Update-Percentage field to the metadata of each package. The client calculates some sort of ID for itself (I don't know precisely how), and then takes the upgrade if it finds that its ID is in the first x% of the available range. If I understand correctly, Ubuntu is using this mechanism in production but Debian is not. Using some sort of hash of the machine ID + the proposed version would probably have the behaviour you want, of choosing a different x% of machines to be the early-adopter set for each update? > This would then mean for the server that it would first serve > foobar_47.11_3.raw which would be version 47.11 of the OS, and 3% of > the hosts would update to it. And then, once you collected enough > feedback you'd rename the file to foobar_47.11_25.raw and 25% of the > hosts would switch over. Finally you'd set the value to 100 (or maybe > just drop it, which should be considered equivalent to 100), and then > all remaining hosts would update. If you're using a hash of the machine ID + the proposed version as your randomization, then I think you'd want to have a single image (or version ID, or some other unique identifier) for each proposed update, and separately, a metadata field that sets *x* in the instruction "if you have figured out that you are in the first x% of machines, upgrade". Otherwise, publishing foobar_47.11_3.raw followed by foobar_47.11_25.raw would be more likely to result in approximately (3% + 25% = 28%) of machines upgrading[1], because the client doesn't know that it's actually the same update and would "re-roll the dice" for each republished name. smcv [1] more precisely, (0.03 + 0.25 - (0.03 * 0.25)) because of how conditional probabilities combine
Re: sysupdate: Limit update to at most one major version
On Di, 02.01.24 13:49, Nils Kattenbeck (nilskem...@gmail.com) wrote: > > I'd be fine with adding MaxVersion=. Happy to review a patch, merge > > something like this (at least file an RFE issue) > > Should that be inclusive or exclusive? Naming it MaxVersion would > imply it to be inclusive though an exclusive bound would likely be > more useful most of the time. One could then specify MaxVersion=1.3.0 > in their 1.2.x images and once they have an upgrade path they would > explicitly raise the max version in e.g. 1.2.15. Otherwise they would > have to specify 1.99.99. > In retrospect a VersionBound= property with syntax similar to > ConditionKernelVersion= would have been better though I guess that > ship has sailed - or is it? Is sd-sysupdate still considered > experimental? Not sure if this warrants such a change though :shrug: We do not allow "=" nor "<" in version strings, as per https://uapi-group.org/specifications/specs/version_format_specification/. Hence we could use that fact and say: "MaxVersion= <=47.11", "MaxVersion= <47.11" could be used to make the type of version comparison explicit. This would implement a tiny subset of the ConditionKernelVersion= logic, and simply default to imply <= if the comparison is not specified explicitly. Of course, a similar logic should then be implemented for MinVersion, i.e. >= and > > Should we continue this discussion on the mailing list or an issue? Issue is better. Lennart -- Lennart Poettering, Berlin
Re: sysupdate: Limit update to at most one major version
> I'd be fine with adding MaxVersion=. Happy to review a patch, merge > something like this (at least file an RFE issue) Should that be inclusive or exclusive? Naming it MaxVersion would imply it to be inclusive though an exclusive bound would likely be more useful most of the time. One could then specify MaxVersion=1.3.0 in their 1.2.x images and once they have an upgrade path they would explicitly raise the max version in e.g. 1.2.15. Otherwise they would have to specify 1.99.99. In retrospect a VersionBound= property with syntax similar to ConditionKernelVersion= would have been better though I guess that ship has sailed - or is it? Is sd-sysupdate still considered experimental? Not sure if this warrants such a change though :shrug: Should we continue this discussion on the mailing list or an issue? Kind regards, Nils
Re: sysupdate: Limit update to at most one major version
On So, 31.12.23 14:43, Nils Kattenbeck (nilskem...@gmail.com) wrote: > Hello, > > we are currently using sd-sysupdate to roll out updates and we're wondering > if there is any possibility to limit updates to consider at most one next > major version. This would allow us to write the software to handle only > data migrations from the previous major version instead of any version > beforehand. > The only thing I have been able to find is MinVersion= which seems to do > exactly the opposite of what we would want to do. I'd be fine with adding MaxVersion=. Happy to review a patch, merge something like this (at least file an RFE issue) Lennart -- Lennart Poettering, Berlin
Re: systemd-sysupdate support for slow rollout (aka A/B testing)
On Mi, 20.12.23 19:04, Nils Kattenbeck (nilskem...@gmail.com) wrote: > Hey everyone, > > does sysupdate currently support any way to slowly roll out updates > where the server providing the files can be in control? This would be > used to slowly make a new version available and have it at e.g. 1% > adoption for a day to monitor regressions before increasing the > coverage. I was unable to find any information about it in the > documentation. This is currently not available, no. The idea so far was always that the server is dumb, and the client picks the release it wants. I have thought about this usecase a while back, and my thinking was that such a staged update logic should be driven by the machine ID. i.e. we should teach sysupdate a simple logic that allows pattern matching of new versions based on some arithmetic of the machine ID. More specifically, include some value in the URL pattern that indicates the percentage of hosts that shall update to this release. Then, each client takes its machine ID, treats it as an integer and calculates modulo 100 of it or so, and then checks if the resulting value is below the intended percentage, and if so it updates, otherwise it doesn't. (or something like that, the above is probably not ideal, since it would mean it's always the same hosts that try a new release first, and it probably should be evened out across the set of clients). This would then mean for the server that it would first serve foobar_47.11_3.raw which would be version 47.11 of the OS, and 3% of the hosts would update to it. And then, once you collected enough feedback you'd rename the file to foobar_47.11_25.raw and 25% of the hosts would switch over. Finally you'd set the value to 100 (or maybe just drop it, which should be considered equivalent to 100), and then all remaining hosts would update. The effect of this is that client's could still explicitly upgrade if they want, and the updates would be entirely driven by the clients, but simply via naming the download images the server can control that "by default" only the chosen number of clients update. > Currently it seems like I would have to implement a different service > which calls the sysupdate binary (or uses dbus once #28134 has landed) > and then decides based on some other information. > > One idea I had would be that systemd-pull could send the machine-id > based on which the server could then decide to provide the newer file > (e.g. last two chars == "00" would roll it out to ~1/255). Though I am > not sure if sd-pull is supposed to be "anonymous", i.e. do not provide > this identifying information. Another drawback of this would be that > stateless systems which reboot often get a new machine-id each boot, > thus having an increased chance to get the newer version. So this idea is not entirely different from my idea, I was just thinking about pushing this into sysupdate rather than pull. > Does anything like this already exist or is planned? Or should that be > done by different applications on the client side? I think it makes a ton of sense to add this to sysupdate. Would love to review/merge a patch for that. > I also remember there being a discussion about plugging in different > sd-pull like implementations/backends[1] to support delta updates, > other transports, or TLS client authentication. This could at least be > adapted to support my idea to send the machine-id as an HTTP header > (e.g. X-MACHINE-ID). If we can avoid it, I'd always adopt a logic whether identifying info doesn't have to be sent to the server. After all the logic should be generic and applicable in scenarios where the client should get anonymity as much as it wants. The machine-id we usually consider a "half-secret", i.e. all local programs get access to it (unless sandboxed), but they are not supposed to be send it across the wire. If they really need to send some identifier across the wire they should derive an app-specific ID instead, which we make easy to acquire via sd_id128_get_machine_app_specific(). But better than app-specific machine IDs are no machine IDs at all in the protocol, if we can get away with it. Hence, my idea of doing the rollout percentage logic client-side. Lennart -- Lennart Poettering, Berlin