Re: Changing file system for Cloud images to ext4, xfs or something custom
On Thu, Apr 11, 2024 at 5:25 AM Tadej Janež wrote: > > On Tue, 2024-04-09 at 11:04 -0400, Dusty Mabe wrote: > > > > > > On 4/9/24 10:00, Neal Gompa wrote: > > > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež wrote: > > > > > > > > Hey Fedora Cloud WG! > > > > > > > > Firstly, thanks for providing Fedora for the popular cloud > > > > providers! > > > > > > > > Since Fedora 35, the Fedora Cloud images use btrfs by default > > > > [1]. > > > > For my deployments, I would like to use ext4 or xfs, so my > > > > question > > > > are: > > > > 1) Is it possible to change the root file system at deploy time? > > > > 2) If not, how could one change the cloud images to use a > > > > different > > > > root file system? > > > > > > > > > > It is not possible to change at deployment time, you would need to > > > build your own custom images. > > > > We do have Fedora CoreOS, which may or may not fit your use case > > (i.e. > > run software in containers, automatic updates, rpm-ostree as the > > update mechanism), which can re-provision the root filesystem on > > first > > boot and does provide images for various clouds: > > > > - https://fedoraproject.org/coreos/download?stream=stable > > - > > https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfiguring_the_root_filesystem > > > > This may or may not be an option for you, but figured I'd mention. > > Thanks for mentioning this, this looks very nice! > > Does it copy the whole rootfs to RAM, erases and reformats it with a > new file system and copies the contents from RAM back to it? > This is basically what ignition does, yes. -- 真実はいつも一つ！/ Always, there's only one truth! -- ___ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Changing file system for Cloud images to ext4, xfs or something custom
On Thu, Apr 11, 2024 at 5:58 AM Tadej Janež wrote: > > > > On Wed, 2024-04-10 at 11:22 -0400, Neal Gompa wrote: > > On Wed, Apr 10, 2024 at 10:54 AM Tadej Janež wrote: > > > > > > On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote: > > > > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež > > > > wrote: > > > > > > > > > > Hey Fedora Cloud WG! > > > > > > > > > > Firstly, thanks for providing Fedora for the popular cloud > > > > > providers! > > > > > > > > > > Since Fedora 35, the Fedora Cloud images use btrfs by default > > > > > [1]. > > > > > For my deployments, I would like to use ext4 or xfs, so my > > > > > question > > > > > are: > > > > > 1) Is it possible to change the root file system at deploy > > > > > time? > > > > > 2) If not, how could one change the cloud images to use a > > > > > different > > > > > root file system? > > > > > > > > > > > > > It is not possible to change at deployment time, you would need > > > > to > > > > build your own custom images. > > > > > > > > > > Ok, thanks for clarifying! > > > > > > > Our images are now defined here: > > > > https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud.xml > > > > > > > > If you want to use something else, you'd want to have your own > > > > version > > > > of the definitions and modify that file to use the filesystem of > > > > your > > > > choice. > > > > > > I see, one would need to modify the "filesystem" key of the > > > selected > > > Cloud-Base- image. > > > > > > Are there instructions on how to build a custom cloud image? > > > > > > > The repository README includes a quickstart section for building your > > own images. > > > > > However, even if I can easily build a custom cloud image, the > > > overhead > > > of maintaining custom cloud images is very high. > > > > > > For every Fedora release and cloud provider, I would need to build > > > the > > > image and then with IaC (e.g. Terraform), handle uploading the > > > custom > > > image and using it... > > > > > > > Yeah, it is more overhead, for sure. > > > > Thinking out loud... the modification in my case would not be a custom > tailored image for my use case but more similar to how Fedora project > builds different spins if people want to use a non-GNOME desktop > environment or ALT images for people using alternative architectures. > > Is there a possibility to reuse the existing image building > infrastructure and add tasks for building images with e.g. ext4 as the > rootfs? > I think we could consider adding some overlay samples that show different ways to leverage the descriptions for building custom stuff on top of ours in the pagure.io/fedora-cloud namespace. I know David and I were already talking about doing that after we got done with the F40 cycle. > > > > > > > > Is there a particular reason you want to use ext4 or xfs for your > > > > rootfs? Typically the pattern we see is that people attach a > > > > secondary > > > > volume or use S3 and put their data on that instead of the > > > > rootfs. > > > > > > > > > > Yes, I use that pattern as well. Usually, /srv or /var would be on > > > a > > > separate block storage and formatted with the file system of > > > choice. > > > > > > The reasons why I would want to use, e.g. ext4 for the rootfs, > > > would > > > be: > > > > > > 1. Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS > > > underneath) for the last 2 decades. I know the tools and I know > > > what > > > "care" such filesystems need. > > > > > > 2. Maturity. Ext2/3/4 have been round for quite longer than btrfs > > > and > > > there are very little "unknowns" or "surprises" with it. > > > > > > 3. Simplicity. When provisioning machines with a cloud provider, I > > > actually don't need the LVM+ext4 combination because the cloud > > > provider > > > would typically handle the things LVM would handle for a non-cloud > > > machine, e.g. increasing the block storage size, snapshotting, ... > > > > > > Please, don't read this as a critique against btrfs, just me trying > > > to > > > explain why I would find it nicer to just use ext4 for the rootfs > > > as > > > well. > > > > > > > That's totally fair. I will point out that one of the big reasons we > > use Btrfs is actually to enable space efficiency (which can lead to > > cost savings across the board). The used storage is roughly 40% less > > than on ext4 due to leveraging both transparent compression and > > reflinks. > > > > The other reason is to make it easy to do replication for backups and > > other purposes. > > > > You may also find it to be beneficial for easily avoiding certain > > kinds of failure cases that can happen on ext4 (such as inode > > exhaustion with containers), since btrfs dynamically allocates inodes > > per subvolume. > > > > You can also grow (and shrink!) the volume live while the system is > > mounted and online safely. > > > > Thanks for enumerating these benefits. And I agree that these benefits > alone justify the reason for switching to
Re: Changing file system for Cloud images to ext4, xfs or something custom
On Wed, 2024-04-10 at 11:22 -0400, Neal Gompa wrote: > On Wed, Apr 10, 2024 at 10:54 AM Tadej Janež wrote: > > > > On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote: > > > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež > > > wrote: > > > > > > > > Hey Fedora Cloud WG! > > > > > > > > Firstly, thanks for providing Fedora for the popular cloud > > > > providers! > > > > > > > > Since Fedora 35, the Fedora Cloud images use btrfs by default > > > > [1]. > > > > For my deployments, I would like to use ext4 or xfs, so my > > > > question > > > > are: > > > > 1) Is it possible to change the root file system at deploy > > > > time? > > > > 2) If not, how could one change the cloud images to use a > > > > different > > > > root file system? > > > > > > > > > > It is not possible to change at deployment time, you would need > > > to > > > build your own custom images. > > > > > > > Ok, thanks for clarifying! > > > > > Our images are now defined here: > > > https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud.xml > > > > > > If you want to use something else, you'd want to have your own > > > version > > > of the definitions and modify that file to use the filesystem of > > > your > > > choice. > > > > I see, one would need to modify the "filesystem" key of the > > selected > > Cloud-Base- image. > > > > Are there instructions on how to build a custom cloud image? > > > > The repository README includes a quickstart section for building your > own images. > > > However, even if I can easily build a custom cloud image, the > > overhead > > of maintaining custom cloud images is very high. > > > > For every Fedora release and cloud provider, I would need to build > > the > > image and then with IaC (e.g. Terraform), handle uploading the > > custom > > image and using it... > > > > Yeah, it is more overhead, for sure. > Thinking out loud... the modification in my case would not be a custom tailored image for my use case but more similar to how Fedora project builds different spins if people want to use a non-GNOME desktop environment or ALT images for people using alternative architectures. Is there a possibility to reuse the existing image building infrastructure and add tasks for building images with e.g. ext4 as the rootfs? > > > > > > Is there a particular reason you want to use ext4 or xfs for your > > > rootfs? Typically the pattern we see is that people attach a > > > secondary > > > volume or use S3 and put their data on that instead of the > > > rootfs. > > > > > > > Yes, I use that pattern as well. Usually, /srv or /var would be on > > a > > separate block storage and formatted with the file system of > > choice. > > > > The reasons why I would want to use, e.g. ext4 for the rootfs, > > would > > be: > > > > 1. Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS > > underneath) for the last 2 decades. I know the tools and I know > > what > > "care" such filesystems need. > > > > 2. Maturity. Ext2/3/4 have been round for quite longer than btrfs > > and > > there are very little "unknowns" or "surprises" with it. > > > > 3. Simplicity. When provisioning machines with a cloud provider, I > > actually don't need the LVM+ext4 combination because the cloud > > provider > > would typically handle the things LVM would handle for a non-cloud > > machine, e.g. increasing the block storage size, snapshotting, ... > > > > Please, don't read this as a critique against btrfs, just me trying > > to > > explain why I would find it nicer to just use ext4 for the rootfs > > as > > well. > > > > That's totally fair. I will point out that one of the big reasons we > use Btrfs is actually to enable space efficiency (which can lead to > cost savings across the board). The used storage is roughly 40% less > than on ext4 due to leveraging both transparent compression and > reflinks. > > The other reason is to make it easy to do replication for backups and > other purposes. > > You may also find it to be beneficial for easily avoiding certain > kinds of failure cases that can happen on ext4 (such as inode > exhaustion with containers), since btrfs dynamically allocates inodes > per subvolume. > > You can also grow (and shrink!) the volume live while the system is > mounted and online safely. > Thanks for enumerating these benefits. And I agree that these benefits alone justify the reason for switching to btrfs by default. I might be tempted to check out the btrfs status page again and start playing with it. -- ___ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org Do not reply to spam, report it:
Re: Changing file system for Cloud images to ext4, xfs or something custom
On Tue, 2024-04-09 at 11:04 -0400, Dusty Mabe wrote: > > > On 4/9/24 10:00, Neal Gompa wrote: > > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež wrote: > > > > > > Hey Fedora Cloud WG! > > > > > > Firstly, thanks for providing Fedora for the popular cloud > > > providers! > > > > > > Since Fedora 35, the Fedora Cloud images use btrfs by default > > > [1]. > > > For my deployments, I would like to use ext4 or xfs, so my > > > question > > > are: > > > 1) Is it possible to change the root file system at deploy time? > > > 2) If not, how could one change the cloud images to use a > > > different > > > root file system? > > > > > > > It is not possible to change at deployment time, you would need to > > build your own custom images. > > We do have Fedora CoreOS, which may or may not fit your use case > (i.e. > run software in containers, automatic updates, rpm-ostree as the > update mechanism), which can re-provision the root filesystem on > first > boot and does provide images for various clouds: > > - https://fedoraproject.org/coreos/download?stream=stable > - > https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfiguring_the_root_filesystem > > This may or may not be an option for you, but figured I'd mention. Thanks for mentioning this, this looks very nice! Does it copy the whole rootfs to RAM, erases and reformats it with a new file system and copies the contents from RAM back to it? On the subject if this fits my use case, the answer is no. The systems don't run applications in containers. They are set up to run services via classical systemd units. -- ___ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Changing file system for Cloud images to ext4, xfs or something custom
On Wed, Apr 10, 2024 at 10:54 AM Tadej Janež wrote: > > On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote: > > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež wrote: > > > > > > Hey Fedora Cloud WG! > > > > > > Firstly, thanks for providing Fedora for the popular cloud > > > providers! > > > > > > Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. > > > For my deployments, I would like to use ext4 or xfs, so my question > > > are: > > > 1) Is it possible to change the root file system at deploy time? > > > 2) If not, how could one change the cloud images to use a different > > > root file system? > > > > > > > It is not possible to change at deployment time, you would need to > > build your own custom images. > > > > Ok, thanks for clarifying! > > > Our images are now defined here: > > https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud.xml > > > > If you want to use something else, you'd want to have your own > > version > > of the definitions and modify that file to use the filesystem of your > > choice. > > I see, one would need to modify the "filesystem" key of the selected > Cloud-Base- image. > > Are there instructions on how to build a custom cloud image? > The repository README includes a quickstart section for building your own images. > However, even if I can easily build a custom cloud image, the overhead > of maintaining custom cloud images is very high. > > For every Fedora release and cloud provider, I would need to build the > image and then with IaC (e.g. Terraform), handle uploading the custom > image and using it... > Yeah, it is more overhead, for sure. > > > > Is there a particular reason you want to use ext4 or xfs for your > > rootfs? Typically the pattern we see is that people attach a > > secondary > > volume or use S3 and put their data on that instead of the rootfs. > > > > Yes, I use that pattern as well. Usually, /srv or /var would be on a > separate block storage and formatted with the file system of choice. > > The reasons why I would want to use, e.g. ext4 for the rootfs, would > be: > > 1. Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS > underneath) for the last 2 decades. I know the tools and I know what > "care" such filesystems need. > > 2. Maturity. Ext2/3/4 have been round for quite longer than btrfs and > there are very little "unknowns" or "surprises" with it. > > 3. Simplicity. When provisioning machines with a cloud provider, I > actually don't need the LVM+ext4 combination because the cloud provider > would typically handle the things LVM would handle for a non-cloud > machine, e.g. increasing the block storage size, snapshotting, ... > > Please, don't read this as a critique against btrfs, just me trying to > explain why I would find it nicer to just use ext4 for the rootfs as > well. > That's totally fair. I will point out that one of the big reasons we use Btrfs is actually to enable space efficiency (which can lead to cost savings across the board). The used storage is roughly 40% less than on ext4 due to leveraging both transparent compression and reflinks. The other reason is to make it easy to do replication for backups and other purposes. You may also find it to be beneficial for easily avoiding certain kinds of failure cases that can happen on ext4 (such as inode exhaustion with containers), since btrfs dynamically allocates inodes per subvolume. You can also grow (and shrink!) the volume live while the system is mounted and online safely. -- 真実はいつも一つ！/ Always, there's only one truth! -- ___ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Changing file system for Cloud images to ext4, xfs or something custom
On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote: > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež wrote: > > > > Hey Fedora Cloud WG! > > > > Firstly, thanks for providing Fedora for the popular cloud > > providers! > > > > Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. > > For my deployments, I would like to use ext4 or xfs, so my question > > are: > > 1) Is it possible to change the root file system at deploy time? > > 2) If not, how could one change the cloud images to use a different > > root file system? > > > > It is not possible to change at deployment time, you would need to > build your own custom images. > Ok, thanks for clarifying! > Our images are now defined here: > https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud.xml > > If you want to use something else, you'd want to have your own > version > of the definitions and modify that file to use the filesystem of your > choice. I see, one would need to modify the "filesystem" key of the selected Cloud-Base- image. Are there instructions on how to build a custom cloud image? However, even if I can easily build a custom cloud image, the overhead of maintaining custom cloud images is very high. For every Fedora release and cloud provider, I would need to build the image and then with IaC (e.g. Terraform), handle uploading the custom image and using it... > > Is there a particular reason you want to use ext4 or xfs for your > rootfs? Typically the pattern we see is that people attach a > secondary > volume or use S3 and put their data on that instead of the rootfs. > Yes, I use that pattern as well. Usually, /srv or /var would be on a separate block storage and formatted with the file system of choice. The reasons why I would want to use, e.g. ext4 for the rootfs, would be: 1. Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS underneath) for the last 2 decades. I know the tools and I know what "care" such filesystems need. 2. Maturity. Ext2/3/4 have been round for quite longer than btrfs and there are very little "unknowns" or "surprises" with it. 3. Simplicity. When provisioning machines with a cloud provider, I actually don't need the LVM+ext4 combination because the cloud provider would typically handle the things LVM would handle for a non-cloud machine, e.g. increasing the block storage size, snapshotting, ... Please, don't read this as a critique against btrfs, just me trying to explain why I would find it nicer to just use ext4 for the rootfs as well. Regards, Tadej -- ___ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Changing file system for Cloud images to ext4, xfs or something custom
On 4/9/24 10:00, Neal Gompa wrote: > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež wrote: >> >> Hey Fedora Cloud WG! >> >> Firstly, thanks for providing Fedora for the popular cloud providers! >> >> Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. >> For my deployments, I would like to use ext4 or xfs, so my question >> are: >> 1) Is it possible to change the root file system at deploy time? >> 2) If not, how could one change the cloud images to use a different >> root file system? >> > > It is not possible to change at deployment time, you would need to > build your own custom images. We do have Fedora CoreOS, which may or may not fit your use case (i.e. run software in containers, automatic updates, rpm-ostree as the update mechanism), which can re-provision the root filesystem on first boot and does provide images for various clouds: - https://fedoraproject.org/coreos/download?stream=stable - https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfiguring_the_root_filesystem This may or may not be an option for you, but figured I'd mention. > > Our images are now defined here: > https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud.xml > > If you want to use something else, you'd want to have your own version > of the definitions and modify that file to use the filesystem of your > choice. > > Is there a particular reason you want to use ext4 or xfs for your > rootfs? Typically the pattern we see is that people attach a secondary > volume or use S3 and put their data on that instead of the rootfs. > > > -- ___ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Changing file system for Cloud images to ext4, xfs or something custom
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež wrote: > > Hey Fedora Cloud WG! > > Firstly, thanks for providing Fedora for the popular cloud providers! > > Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. > For my deployments, I would like to use ext4 or xfs, so my question > are: > 1) Is it possible to change the root file system at deploy time? > 2) If not, how could one change the cloud images to use a different > root file system? > It is not possible to change at deployment time, you would need to build your own custom images. Our images are now defined here: https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud.xml If you want to use something else, you'd want to have your own version of the definitions and modify that file to use the filesystem of your choice. Is there a particular reason you want to use ext4 or xfs for your rootfs? Typically the pattern we see is that people attach a secondary volume or use S3 and put their data on that instead of the rootfs. -- 真実はいつも一つ！/ Always, there's only one truth! -- ___ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Changing file system for Cloud images to ext4, xfs or something custom
Hey Fedora Cloud WG! Firstly, thanks for providing Fedora for the popular cloud providers! Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are: 1) Is it possible to change the root file system at deploy time? 2) If not, how could one change the cloud images to use a different root file system? Thanks and regards, Tadej Janež [1] https://fedoraproject.org/wiki/Changes/FedoraCloudBtrfsByDefault -- ___ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
