Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
Wout Mertens writes: FWIW, btrfs and ZFS also do non destructive writes, they're Copy on Write. As a bonus you get unlimited instant snapshots and lots of other wonderful things. I prefer btrfs as it has more of a desktop focus and feature set as well as being in the kernel, but ZFS is more mature. I shall probably go back to using btrfs. One nice property of NILFS is that every write results in a new checkpoint, which makes it very cheap to check whether a filesystem has been modified (other than via low-level disk-editing). Tim --- GPG public key available at: http://phasechangeit.com/~trb/gpg-key or http://subkeys.pgp.net:11371 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
On Mon, Oct 6, 2014 at 8:04 AM, Tim Barbour t...@categorical.net wrote: One nice property of NILFS is that every write results in a new checkpoint, which makes it very cheap to check whether a filesystem has been modified (other than via low-level disk-editing). On btrfs you can look at the generation: http://www.tummy.com/blogs/2010/11/01/fun-with-btrfs-what-files-have-changed/ Wout. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
Wout Mertens writes: On Mon, Oct 6, 2014 at 8:04 AM, Tim Barbour t...@categorical.net wrote: One nice property of NILFS is that every write results in a new checkpoint, which makes it very cheap to check whether a filesystem has been modified (other than via low-level disk-editing). On btrfs you can look at the generation: http://www.tummy.com/blogs/2010/11/01/fun-with-btrfs-what-files-have-changed/ Thank you! That looks like what I need. Tim --- GPG public key available at: http://phasechangeit.com/~trb/gpg-key or http://subkeys.pgp.net:11371 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
On Tue, Sep 30, 2014 at 1:08 PM, Tim Barbour t...@categorical.net wrote: NILFS2 is a log-structured filesystem which is now in the Linux kernel source tree, and supported by GRUB2. It should appeal to NixOS users because it avoids destructive update (changing a file produces a new version of the file). I have installed NixOS with all filesystems on NILFS2, but encountered some problems along the way. Looking at the details of NILFS2, I saw that it does a linear search within directories. This is far from ideal for the /nix/store as this directory is HUGE. This might have a noticeable cost at the start-up of a computer / programs. -- Nicolas Pierron http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
FWIW, btrfs and ZFS also do non destructive writes, they're Copy on Write. As a bonus you get unlimited instant snapshots and lots of other wonderful things. I prefer btrfs as it has more of a desktop focus and feature set as well as being in the kernel, but ZFS is more mature. Wout. On Oct 5, 2014 11:39 AM, Nicolas Pierron nicolas.b.pier...@gmail.com wrote: On Tue, Sep 30, 2014 at 1:08 PM, Tim Barbour t...@categorical.net wrote: NILFS2 is a log-structured filesystem which is now in the Linux kernel source tree, and supported by GRUB2. It should appeal to NixOS users because it avoids destructive update (changing a file produces a new version of the file). I have installed NixOS with all filesystems on NILFS2, but encountered some problems along the way. Looking at the details of NILFS2, I saw that it does a linear search within directories. This is far from ideal for the /nix/store as this directory is HUGE. This might have a noticeable cost at the start-up of a computer / programs. -- Nicolas Pierron http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
Nicolas Pierron writes: Looking at the details of NILFS2, I saw that it does a linear search within directories. This is far from ideal for the /nix/store as this directory is HUGE. This might have a noticeable cost at the start-up of a computer / programs. Ouch! I missed that bit. I shall probably have to go back to btrfs. Tim --- GPG public key available at: http://phasechangeit.com/~trb/gpg-key or http://subkeys.pgp.net:11371 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
aszlig writes: On Tue, Sep 30, 2014 at 09:08:41PM +1000, Tim Barbour wrote: The most serious problem is that NILFS2 needs to update /etc/mtab when mounting a filesystem, so that it can store information about the [...] I don't have any experience with NILFS2 yet, but you might want to take a look at how it is handled when using NFS in fileSystems: I think your point is that NFS also needs to keep track of some state when filesystems are mounted and unmounted. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/tasks/filesystems/nfs.nix I had a look at that file, but I could not see where it is handling the state. Nice :-) Have you considered submitting your patch for 0.41 upstream? If it doesn't get upstreamed too soon, I could include it as well while cleaning up the mess^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^Hupdating blivet/nixpart. That would be great. But as mentioned in the other mail, I'm working on nixpart 1.0 already, which should get rid of the monkey patching with 0.17. I attach a tarball with patches for Blivet (note that the 0.41 patch is untested). Yah, probably would make sense to test against 0.41, because I think we shouldn't aim for old versions, especially in terms of nixpart. Are you able to build a current Blivet to test with ? I just checked, and it looks as if Blivet is up to about 0.65 now: https://apps.fedoraproject.org/packages/python-blivet Tim --- GPG public key available at: http://phasechangeit.com/~trb/gpg-key or http://subkeys.pgp.net:11371 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
On 09/30/2014 12:08 PM, Tim Barbour wrote: NILFS2 is a log-structured filesystem which is now in the Linux kernel source tree, and supported by GRUB2. It should appeal to NixOS users because it avoids destructive update (changing a file produces a new version of the file). I have installed NixOS with all filesystems on NILFS2, but encountered some problems along the way. The most serious problem is that NILFS2 needs to update /etc/mtab when mounting a filesystem, so that it can store information about the nilfs_cleanerd process associated with the mounted filesystem. If it cannot write to /etc/mtab, it does not start nilfs_cleanerd, and thus does not do garbage collection. In NixOS, /etc/mtab is just a symlink to /proc/mounts. I have worked around this manually so far, by replacing the symlink with a copy of /proc/mounts, then remounting all the NILFS2 filesystems. I am not sure how to resolve this properly. It is possible to mount using the '-n' option, but then NILFS2 will not start nilfs_cleanerd; it can be started later, but that requires writing to /etc/mtab. Perhaps the right way would be to mount using '-n', then start nilfs_cleanerd separately, but care would be necessary to make sure that nilfs_cleanerd is started exactly once (per filesystem), and shutdown appropriately at unmount. Another problem is that nixpart does not work with NILFS2, because Blivet does not support it. I modified Blivet (0.41) to support NILFS2, but the NixOS Blivet packages use a very old version (0.17-1), so I cannot easily test my change against the latest version of Blivet. I have back-ported it to 0.17-1, and used that to setup partitions and filesystems for the above NixOS installation. I noticed one problem: it did not label any of the volumes, not even swap, even though I specified labels. I suspect this is a deficiency of Blivet 0.17-1, and imagine it is already fixed in Blivet 0.41 . I considered updating the NixOS Blivet package(s), but they do some tricky-looking processing on the source code, and I doubt that I could update that properly for the current source code. If the maintainer could update the Blivet packages, then I would test my patch against the current Blivet, and send it to the upstream maintainers. I attach a tarball with patches for Blivet (note that the 0.41 patch is untested). Tim --- GPG public key available at: http://phasechangeit.com/~trb/gpg-key or http://subkeys.pgp.net:11371 Looks to me that all the scary-looking block in the blivet package does is replace hardcoded paths with paths to the nix-store that nix provides it with, a common practice. I think if you build and try to run the newer version in chroot, it should quickly become apparent what new changes have to be made for 0.41. If someone doesn't step up then I think you might actually be the most qualified person to do the update. Notably there's no listed maintainer for the blivet package. -- Mateusz K. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
Good morning, On Tue, Sep 30, 2014 at 12:29:12PM +0100, Mateusz Kowalczyk wrote: Looks to me that all the scary-looking block in the blivet package does is replace hardcoded paths with paths to the nix-store that nix provides it with, a common practice. I think if you build and try to run the newer version in chroot, it should quickly become apparent what new changes have to be made for 0.41. Well, nixpart in the current version is more of a hack (it's an under- statement - there's heavy monkey-patching involved) rather than a sane implementation, and still relies on Kickstart syntax. However, for version 1.0 I intend to use a native Nix syntax for specifying the partition layout and use blivets device tree directly rather involving pykickstart _at_ all. i've scheduled version 1.0 of nixpart (and nixos-assimilate) for NixOS 14.10, so i'm going to pick up my (not yet pushed) branches in the next few days. The reason why blivet in nixpkgs is at 0.17 at the moment is that at least right now it's solely used by nixpart. And i've already created a nixpart0 a while ago in order to be prepared for the transition in NixOps (so that older builds of NixOps won't fail horribly with updated nixpart and/or dependencies). Notably there's no listed maintainer for the blivet package. Probably should have added myself to the list of maintainers back then. a! -- aszlig signature.asc Description: Digital signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NILFS2, NixOS, nixpart and Blivet
On Tue, Sep 30, 2014 at 09:08:41PM +1000, Tim Barbour wrote: The most serious problem is that NILFS2 needs to update /etc/mtab when mounting a filesystem, so that it can store information about the nilfs_cleanerd process associated with the mounted filesystem. If it cannot write to /etc/mtab, it does not start nilfs_cleanerd, and thus does not do garbage collection. In NixOS, /etc/mtab is just a symlink to /proc/mounts. I have worked around this manually so far, by replacing the symlink with a copy of /proc/mounts, then remounting all the NILFS2 filesystems. I am not sure how to resolve this properly. It is possible to mount using the '-n' option, but then NILFS2 will not start nilfs_cleanerd; it can be started later, but that requires writing to /etc/mtab. Perhaps the right way would be to mount using '-n', then start nilfs_cleanerd separately, but care would be necessary to make sure that nilfs_cleanerd is started exactly once (per filesystem), and shutdown appropriately at unmount. I don't have any experience with NILFS2 yet, but you might want to take a look at how it is handled when using NFS in fileSystems: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/tasks/filesystems/nfs.nix Another problem is that nixpart does not work with NILFS2, because Blivet does not support it. I modified Blivet (0.41) to support NILFS2, but the NixOS Blivet packages use a very old version (0.17-1), so I cannot easily test my change against the latest version of Blivet. I have back-ported it to 0.17-1, and used that to setup partitions and filesystems for the above NixOS installation. I noticed one problem: it did not label any of the volumes, not even swap, even though I specified labels. I suspect this is a deficiency of Blivet 0.17-1, and imagine it is already fixed in Blivet 0.41 . Nice :-) Have you considered submitting your patch for 0.41 upstream? If it doesn't get upstreamed too soon, I could include it as well while cleaning up the mess^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^Hupdating blivet/nixpart. But as mentioned in the other mail, I'm working on nixpart 1.0 already, which should get rid of the monkey patching with 0.17. I attach a tarball with patches for Blivet (note that the 0.41 patch is untested). Yah, probably would make sense to test against 0.41, because I think we shouldn't aim for old versions, especially in terms of nixpart. a! -- aszlig signature.asc Description: Digital signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
