Re: [systemd-devel] /etc/os-release but for images
On Mi, 23.03.22 17:14, Davide Bettio (davide.bet...@secomind.com) wrote: > I opened this PR: https://github.com/systemd/systemd/pull/22841 > > This doesn't enable full semver support since that would require allowing > A-Z range, but I'm not sure if it is something we really want to enable > (uppercase in semver looks quite uncommon by the way). how does semver suggest uppercase chars are handled? is "0.1.1a" newer than "0.1.1A"? > I would use the UUID as a further metadata in addition to IMAGE_VERSION, > also for the reasons I describe later here in this mail. Sounds like something you could just add as suffix to IMAGE_VERSION, no? > > > Compared to other options an UUID here would be both reliable and easy to > > > handle and generate. > > > > UUID is are effectively randomly generated. That sucks for build > > processes I am sure, simply because they hence aren't reproducible. > > > > Using a reliable digest, such as the one that can be generated with `casync > digest`, was my first option, however if you think about an update system > such such as RAUC and its bundles, you might still have the same exact > filesystem digest mapping to a number of different bundles, since they can > bring different hook scripts and whatever. > I'm also aware of scenarios where the same filesystem tree has been used to > generate different flash images in order to satisfy different flash sizes / > NAND technologies. > So from a practical point of view having an UUID, and forcing a different > one in /etc/os-release every time a filesystem image / RAUC bundle is > created allows us to have a reasonable 1:1 mapping between the update > artifact and the system image that is on the device. > Last but not least having it in /etc/os-release makes it pretty convenient > to read it (and for sure using an UUID is easier than trying to embed the > digest of the filesystem where /etc/os-release is kept ;) ) > Also there is an interesting bonus: UUID is globally unique also in > scenarios where users try to delete and recreate version tags without > incrementing the version number (or other messy scenarios). Shouldn't you use the fs header uuid? or the GPT partition or overall uuids? Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] /etc/os-release but for images
Hello, Il giorno mer 23 mar 2022 alle ore 13:56 Lennart Poettering < lenn...@poettering.net> ha scritto: > > Also sadly IMAGE_VERSION doesn't allow + which is used from semver for > > build metadata (such as 1.0.0+21AF26D3 or 1.0.0+20130313144700). > > Ah, interesting. This looks like something to fix in our syntax > descriptio though. I am pretty sure we should allow all characters > that semver requires. > > Can you file an RFE issue about this on github? Or even better, submit > a PR that adds that? > I opened this PR: https://github.com/systemd/systemd/pull/22841 This doesn't enable full semver support since that would require allowing A-Z range, but I'm not sure if it is something we really want to enable (uppercase in semver looks quite uncommon by the way). That said, I'd always include some time-based counter in automatic > builds, so that the builds can be ordered. Things like sd-boot's boot > menu entry ordering relies on that. > This is indeed a good advice. > > That's pretty useful when storing a relation between the exact update > > bundle that has been used to flash a device, and the system flashed on a > > device. It turns out to be pretty useful when collecting stats about a > > device fleet or when remote managing system versions and their updates. > > So what I would do on os-release would be storing an UUID that is > generated > > every time a system image is generated, that UUID can be collected/sent > at > > runtime from a device to a remote management service. > > Why wouldn't the IMAGE_VERSION suffice for that? Why pick a UUID where > a version already works? > I would use the UUID as a further metadata in addition to IMAGE_VERSION, also for the reasons I describe later here in this mail. > > > Compared to other options an UUID here would be both reliable and easy to > > handle and generate. > > UUID is are effectively randomly generated. That sucks for build > processes I am sure, simply because they hence aren't reproducible. > Using a reliable digest, such as the one that can be generated with `casync digest`, was my first option, however if you think about an update system such such as RAUC and its bundles, you might still have the same exact filesystem digest mapping to a number of different bundles, since they can bring different hook scripts and whatever. I'm also aware of scenarios where the same filesystem tree has been used to generate different flash images in order to satisfy different flash sizes / NAND technologies. So from a practical point of view having an UUID, and forcing a different one in /etc/os-release every time a filesystem image / RAUC bundle is created allows us to have a reasonable 1:1 mapping between the update artifact and the system image that is on the device. Last but not least having it in /etc/os-release makes it pretty convenient to read it (and for sure using an UUID is easier than trying to embed the digest of the filesystem where /etc/os-release is kept ;) ) Also there is an interesting bonus: UUID is globally unique also in scenarios where users try to delete and recreate version tags without incrementing the version number (or other messy scenarios). BTW, there's now also this: > > https://systemd.io/BUILDING_IMAGES/#image-metadata > Thanks, this feature looks pretty interesting. Regards, Davide Bettio.
Re: [systemd-devel] /etc/os-release but for images
On Mi, 23.03.22 13:38, Davide Bettio (davide.bet...@secomind.com) wrote: > > That's the idea: take the packages, build an image, and then append > > IMAGE_ID/IMAGE_VERSION to it? > > Sure, sounds pretty convenient, here my point was about blindly appending > those additional fields (trusting the distribution didn't already set > them). I don't know your distro. But I'd certainly view it as a bug if your distro fills in these two fields but doesn't actually work based on pre-built images, but is solely package-based. > > > I cook a new image, furthermore I plan to replace the whole operating > > > system image (that I keep read-only) in order to update it, so BUILD_ID > > > would change at every update (so it sounds slightly different from the > > > original described semantic). > > > > BUILD_ID is not for that. You are looking for IMAGE_VERSION. > > IMAGE_VERSION didn't look to me a good option for identifying nightly > builds, or any kind of build that wasn't cooked from a tagged image build > recipe. I think it should be fine for that. > Also sadly IMAGE_VERSION doesn't allow + which is used from semver for > build metadata (such as 1.0.0+21AF26D3 or 1.0.0+20130313144700). Ah, interesting. This looks like something to fix in our syntax descriptio though. I am pretty sure we should allow all characters that semver requires. Can you file an RFE issue about this on github? Or even better, submit a PR that adds that? That said, I'd always include some time-based counter in automatic builds, so that the builds can be ordered. Things like sd-boot's boot menu entry ordering relies on that. > That's pretty useful when storing a relation between the exact update > bundle that has been used to flash a device, and the system flashed on a > device. It turns out to be pretty useful when collecting stats about a > device fleet or when remote managing system versions and their updates. > So what I would do on os-release would be storing an UUID that is generated > every time a system image is generated, that UUID can be collected/sent at > runtime from a device to a remote management service. Why wouldn't the IMAGE_VERSION suffice for that? Why pick a UUID where a version already works? > Compared to other options an UUID here would be both reliable and easy to > handle and generate. UUID is are effectively randomly generated. That sucks for build processes I am sure, simply because they hence aren't reproducible. BTW, there's now also this: https://systemd.io/BUILDING_IMAGES/#image-metadata Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] /etc/os-release but for images
Hello, Il giorno mer 23 mar 2022 alle ore 11:32 Lennart Poettering < lenn...@poettering.net> ha scritto: > On Mi, 23.03.22 10:51, Davide Bettio (davide.bet...@secomind.com) wrote: > > Well, if the distribution people build both packages and disk images, > they can set IMAGE_ID/IMAGE_VERSION for the latter. But this should > always be part of building images, not of building packages. > > > Can I safely just append those fields at the end of > > the copy of the /etc/os-release file? > > That's the idea: take the packages, build an image, and then append > IMAGE_ID/IMAGE_VERSION to it? > Sure, sounds pretty convenient, here my point was about blindly appending those additional fields (trusting the distribution didn't already set them). > > Speaking of BUILD_ID, according to the spec sounds like a field > > reserved to > > BUILD_ID? That's a different thing... > > https://www.freedesktop.org/software/systemd/man/os-release.html#IMAGE_ID= > vs. > https://www.freedesktop.org/software/systemd/man/os-release.html#BUILD_ID= > > > distributions: "BUILD_ID may be used in distributions where the original > > installation image version is important", from my side what I need is to > > identify the git revision + build date of the recipe I'm using to cook > the > > image installed on the system, also my plan is to change that ID every > time > > I cook a new image, furthermore I plan to replace the whole operating > > system image (that I keep read-only) in order to update it, so BUILD_ID > > would change at every update (so it sounds slightly different from the > > original described semantic). > > BUILD_ID is not for that. You are looking for IMAGE_VERSION. > IMAGE_VERSION didn't look to me a good option for identifying nightly builds, or any kind of build that wasn't cooked from a tagged image build recipe. Also sadly IMAGE_VERSION doesn't allow + which is used from semver for build metadata (such as 1.0.0+21AF26D3 or 1.0.0+20130313144700). Here my aim was to provide a way to find which build recipe revision has been used or when an image has been built, that is also useful when debugging a device. > Last but not least, I was looking for a machine parsable unique id, so I > > plan to use BUILD_UUID if it is not kept reserved for other usages, that > > will be an UUID that is freshly generated every time I cook a new image. > > What's this for? > That's pretty useful when storing a relation between the exact update bundle that has been used to flash a device, and the system flashed on a device. It turns out to be pretty useful when collecting stats about a device fleet or when remote managing system versions and their updates. So what I would do on os-release would be storing an UUID that is generated every time a system image is generated, that UUID can be collected/sent at runtime from a device to a remote management service. Compared to other options an UUID here would be both reliable and easy to handle and generate. Thanks again for the kind and helpful conversation, regards, Davide Bettio.
Re: [systemd-devel] /etc/os-release but for images
On Mi, 23.03.22 10:51, Davide Bettio (davide.bet...@secomind.com) wrote: > Hello, > > First of all, thanks for your answers. > > It wasn't really clear to me that the /etc/os-release file was editable > from a 3rd party other than the distribution maintainers, so thanks for the > clarifications. Well, it's not precisely supposed to be something users or admins should edit. But image builders may. > Are the distributions required to leave IMAGE_ID and > IMAGE_VERSION empty? Well, if the distribution people build both packages and disk images, they can set IMAGE_ID/IMAGE_VERSION for the latter. But this should always be part of building images, not of building packages. > Can I safely just append those fields at the end of > the copy of the /etc/os-release file? That's the idea: take the packages, build an image, and then append IMAGE_ID/IMAGE_VERSION to it? > Speaking of BUILD_ID, according to the spec sounds like a field > reserved to BUILD_ID? That's a different thing... https://www.freedesktop.org/software/systemd/man/os-release.html#IMAGE_ID= vs. https://www.freedesktop.org/software/systemd/man/os-release.html#BUILD_ID= > distributions: "BUILD_ID may be used in distributions where the original > installation image version is important", from my side what I need is to > identify the git revision + build date of the recipe I'm using to cook the > image installed on the system, also my plan is to change that ID every time > I cook a new image, furthermore I plan to replace the whole operating > system image (that I keep read-only) in order to update it, so BUILD_ID > would change at every update (so it sounds slightly different from the > original described semantic). BUILD_ID is not for that. You are looking for IMAGE_VERSION. > Last but not least, I was looking for a machine parsable unique id, so I > plan to use BUILD_UUID if it is not kept reserved for other usages, that > will be an UUID that is freshly generated every time I cook a new image. What's this for? Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] /etc/os-release but for images
Hello, First of all, thanks for your answers. It wasn't really clear to me that the /etc/os-release file was editable from a 3rd party other than the distribution maintainers, so thanks for the clarifications. Are the distributions required to leave IMAGE_ID and IMAGE_VERSION empty? Can I safely just append those fields at the end of the copy of the /etc/os-release file? Speaking of BUILD_ID, according to the spec sounds like a field reserved to distributions: "BUILD_ID may be used in distributions where the original installation image version is important", from my side what I need is to identify the git revision + build date of the recipe I'm using to cook the image installed on the system, also my plan is to change that ID every time I cook a new image, furthermore I plan to replace the whole operating system image (that I keep read-only) in order to update it, so BUILD_ID would change at every update (so it sounds slightly different from the original described semantic). Last but not least, I was looking for a machine parsable unique id, so I plan to use BUILD_UUID if it is not kept reserved for other usages, that will be an UUID that is freshly generated every time I cook a new image. Regards, Davide Bettio. Il giorno mar 22 mar 2022 alle ore 18:09 Lennart Poettering < lenn...@poettering.net> ha scritto: > On Di, 22.03.22 17:10, Davide Bettio (davide.bet...@secomind.com) wrote: > > > Hello, > > > > I would like to figure out if anyone has proposed any kind of standard > for > > storing metadata about a system image. > > The IMAGE_ID= and IMAGE_VERSION= fields from /etc/os-release are > supposed to be used for that. > > i.e. the idea was that you can take a generic distro (fedora, debian, > …) build an image off it, and it will put its own os info in > /usr/lib/os-release, and make /etc/os-release a symlink to it. Your > image build would then replace /etc/os-release with a file, that > incorporates /usr/lib/os-release and adds in IMAGE_ID=/IMAGE_VERSION=. > > Each time you rebuild the image your image building tool would repeat > that step. i.e. it would be the image builder tool's job to extend the > generic OS data from /usr/lib/ with info about the image and place the > result in /etc/. > > Lennart > > -- > Lennart Poettering, Berlin >
Re: [systemd-devel] /etc/os-release but for images
On Di, 22.03.22 17:10, Davide Bettio (davide.bet...@secomind.com) wrote: > Hello, > > I would like to figure out if anyone has proposed any kind of standard for > storing metadata about a system image. The IMAGE_ID= and IMAGE_VERSION= fields from /etc/os-release are supposed to be used for that. i.e. the idea was that you can take a generic distro (fedora, debian, …) build an image off it, and it will put its own os info in /usr/lib/os-release, and make /etc/os-release a symlink to it. Your image build would then replace /etc/os-release with a file, that incorporates /usr/lib/os-release and adds in IMAGE_ID=/IMAGE_VERSION=. Each time you rebuild the image your image building tool would repeat that step. i.e. it would be the image builder tool's job to extend the generic OS data from /usr/lib/ with info about the image and place the result in /etc/. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] /etc/os-release but for images
Hi Davide, On 22/03/2022 16:10, Davide Bettio wrote: I would like to figure out if anyone has proposed any kind of standard for storing metadata about a system image. We need to track metadata about an image that has been flashed to a device. /etc/os-release is really good for tracking information about the operating system, but it doesn't allow us to distinguish between 2 images of the same operating system having a different set of packages and configurations. Our idea was to ship a /etc/base-image file What's wrong with using /etc/os-release as intended along with the keys in the spec e.g. IMAGE_ID and IMAGE_VERSION ? Thanks Chris
[systemd-devel] /etc/os-release but for images
Hello, I would like to figure out if anyone has proposed any kind of standard for storing metadata about a system image. We need to track metadata about an image that has been flashed to a device. /etc/os-release is really good for tracking information about the operating system, but it doesn't allow us to distinguish between 2 images of the same operating system having a different set of packages and configurations. Our idea was to ship a /etc/base-image file that contains something like: ``` NAME=my-rpi-image VERSION=1.0.9 BUILD_ID=20220425-180033-ad3498f8 FINGERPRINT=08da53cd-81b4-4d32-a72d-9aa04885e8d3 ``` We would use the fingerprint to know which specific system image has been flashed, while the other entries are valuable information for debugging, etc... Is anyone aware of anything similar to this (including WIP specs)? Is this of interest? Regards, Davide Bettio.
