Re: [Nix-dev] NixOS on a machine with only one user

[Matthias Beyer]

On 25-04-2015 17:00:18, Kosyrev Serge wrote:
> 
> My only pain is that the various ~/.* folders and files don't quite fit
> into this..  otherwise it'd be pure bliss.
> 

What do you mean by this?

-- 
Mit freundlichen Grüßen,
Kind regards,
Matthias Beyer

Proudly sent with mutt.
Happily signed with gnupg.


pgpzQ3dJrKjDS.pgp
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Please test Nix store auto-optimise

[Ertugrul Söylemez]

> IMO much of the time overhead is basically unavoidable, as you need to
> do lots of random probes to a huge table (order of million entries,
> e.g.  1.5M in my case). Putting a single Bloom filter in front would
> shave I/O of most *failed* probes, but otherwise we can't get much
> better, I believe (being a PhD student in data structures).

There is some overhead, but it doesn't have to be anywhere near the one
we have right now.  The filesystem stores a lot more information than
necessary.  Basically making the on-disk structure smaller and more
domain-specific makes the operation noticably faster, and I can't think
of a smaller structure to store hash-maps than patricia trees.  A simple
hash-to-block mapping is all we need.

I'm not sure whether the Bloom filter would really help.  Since it is
impossible to predict the required size you would have to use some form
of growing Bloom filter or proactively create a filter that is so large
that it might destroy its own purpose.


> Anyway, this deduplication work seems better delegated to some other
> SW/library. Surely there are some already. If we wanted to deduplicate
> *similar* (unequal) files, we would even need to get FS-dependent. In
> the typical ext4 case, we don't even have reflinks AFAIK...

That's what I'm saying.  A separate tool should handle this, and it
would depend on the FS (in the sense of being a function of the FS), but
not be bound to any particular one.  For traditional FSes like ext4 it
would do hard-linking, but perhaps in a slightly smarter way.


Greets,
Ertugrul


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Please test Nix store auto-optimise

[Vladimír Čunát]

On 04/25/2015 03:56 PM, Ertugrul Söylemez wrote:
> Tl;dr:  The current linking strategy has a noticable space and time
> overhead that is large enough to be unacceptable for me, so I turned it
> off.

I turned it off due to time overhead on slow HDD (5.4 krpm), as it was
slowing down very much when downloading fast from binary cache; most of
those files I can GC very soon anyway, as I'm often just testing some
builds. (I believe space overhead is low enough on btrfs, as I posted in
this thread.)

IMO much of the time overhead is basically unavoidable, as you need to
do lots of random probes to a huge table (order of million entries, e.g.
1.5M in my case). Putting a single Bloom filter in front would shave I/O
of most *failed* probes, but otherwise we can't get much better, I
believe (being a PhD student in data structures).

Anyway, this deduplication work seems better delegated to some other
SW/library. Surely there are some already. If we wanted to deduplicate
*similar* (unequal) files, we would even need to get FS-dependent. In
the typical ext4 case, we don't even have reflinks AFAIK...


Vladimir




smime.p7s
Description: S/MIME Cryptographic Signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] NixOS on a machine with only one user

[Ertugrul Söylemez]

> > Assuming bigger things usually gives you a better or at least more
> > scalable result, and in most cases the additional cost is very close
> > to zero. For example every single-disk system I set up uses a
> > two-disk RAID array with one missing disk from the point of view of
> > mdadm. If it does happen that I actually want to add more disks, it
> > will be a few short online commands to do it instead of a tedious
> > reinstallation of the entire system, and it costs only a few
> > megabytes. The additional disk might as well be an external
> > emergency disk that starts synchronising as soon as I plug it in.
>
> How did you manage to make mdadm do that?

It allows you to create arrays with missing disks, and you can activate
them as long as enough disks are present.  It's like creating an array
with a failed disk right from the beginning:

mdadm --create /dev/md/blah-system \
-l 1 -n 2 \
/dev/sda2 missing

Notice the `missing` argument.  It tells mdadm that the first disk is
`/dev/sda2` and the second disk is currently missing.  When you plug in
a second disk, just add it to the array.  When you plug in a third disk,
convert the array to RAID5 (can be done online in the background, see
`--grow`) and add the third disk.  And so on.


Greets,
Ertugrul


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Please test Nix store auto-optimise

[Ertugrul Söylemez]

> Once I wondered if using reflinks instead of hardlinks might be better
> from some point of view, but it probably won't be a big difference.

Tl;dr:  The current linking strategy has a noticable space and time
overhead that is large enough to be unacceptable for me, so I turned it
off.  The overhead can be removed in two ways:  Use reflinking when
available and more importantly get rid of the `.links` directory.

When the filesystem supports reflinking, it should be used.  The
advantages are:

  * From the viewpoint of the FS user this is completely transparent.

  * Most modern FSes support block-level deduplication.

  * The FS ensures consistency.  No potentially dangerous assumptions
about the atomicity of FS operations.

  * Can be done in the background even on systems without nix-daemon.

  * Can be used safely to copy files into the store when `src` points to
the local filesystem.

  * The `.links` approach would no longer work, forcing us to do
something more sensible, for example a database file.

To deduplicate new builds, simply maintain a fast database of block
hashes and compare the newly written files against it.  If a match is
found, instruct the filesystem to deduplicate and that's it.  After GC
check to see if the GCed blocks still exist, otherwise remove them from
the database.

Should it happen that the database becomes inconsistent with the store
(for example because the system crashed during deduplication), nothing
bad will happen except that a few blocks might be wasted.

It would also help developers.  For example it's not uncommon for me to
point `src` to the local filesystem.  And since I'm doing a lot with
machine learning nowadays, often enough there are some huge files in a
`data` or `examples` directory.  Each time I do nix-build those are
copied verbatim to the store, which not only takes an unnecessarily long
time, but also wastes space.  As a btrfs user I would definitely benefit
From reflinking.  It would still require reading the entire file for
hashing, but the space waste would be gone.

The requirement to have this huge .links directory would be gone, making
both the optimisation process and garbage collection faster.  Just
listing its contents takes more than a minute for me with most of the
time spent disk-thrashing.  I'm not even sure if it's really required at
all, since for book-keeping a database file would be a better option.

I'd also like to note that store optimisation is probably better handled
by a separate program.


Greets,
Ertugrul


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Vagrant stuff - images and plugin - still alive?

[Christian Theune]

Hi,

I’ve been working on getting vagrant-based setups working nicely in the last 
weeks and found

https://github.com/oxdi/vagrant-nixos 
https://github.com/oxdi/nixos 
and
https://github.com/zimbatm/nixbox 

The zimbatm has images available, I didn’t get around to using the packer 
templates, yet.

The plugin is nice but is not compatible with NixOS 14.12 in the released 
version. The code has everything fixed already and just making a release of 
that code works nicely for me. However, I’d love if the gem would be publicly 
available.

So, is anyone on this list using Vagrant with NixOS? Is the author of the 
plugin still around? Alternatives? Should I fork this and start making releases?

Christian

—
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] NixOS on a machine with only one user

[Matthias Beyer]

On 24-04-2015 20:07:49, Arseniy Seroka wrote:
> Do as you will.
> I keep my system profile as minimal as I can.

I do the exact opposite: I put almost everything into the system
packages.

Why:

I have several machines (currently only one on NixOS, actively used,
another one not used by now). I have the configuration.nix split into
several files and put everything under version control. I have several
"sets" of packages (base, gaming, development, media, etc) and I have
one main file for each host. This way I can do DRY. I put a new
package in, say, developPackages.nix and all new machines get it after
a short git pull && nixos-rebuild switch.

If someone wants to know more: http://beyermatthias.de/tags/nixos.html

-- 
Mit freundlichen Grüßen,
Kind regards,
Matthias Beyer

Proudly sent with mutt.
Happily signed with gnupg.


pgpLMVEZ3YSOJ.pgp
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Please test Nix store auto-optimise

[Wout Mertens]

I opened https://github.com/NixOS/nix/issues/377 a while ago and then I ran
some small tests and it didn't seem to matter. The links dir is of course
huge but it still performs fine on sane filesystems. IIRC, the only time
you do readdir on it is during GC to remove the ones with 1 link count.

Wout.

On Sat, Apr 25, 2015 at 3:16 AM Jascha Geerds  wrote:

> Thank you for the hint, that saved 12GB on my machine. Anyway, wouldn't
> it be better if we use an extra level of directories to reduce the size
> of /nix/store/.links ?
>
> > $ ls -l /nix/store/.links/ | wc -l
> > 374716
>
> "ls" is quite slow and this is just my desktop machine and not a busy
> server like hydra or something like that. Such a huge directory would
> kill slower file manager or other applications. I just think of
> PyCharm's "open file" dialog which sometimes tries to open my
> /nix/store... this operation freezes the application so that I have to
> kill it.
>
> For example git or git-annex introduced an extra level of directories to
> reduce the amount of files inside a single directory. Something like
> this:
>
> $ tree --charset=ascii /tmp/nix/store/.links/
> /tmp/nix/store/.links/
> |-- aab
> |   |-- aab4f31a9b3119c2f6bb3b45c27c4cb33ff45a8f6e2
> |   `-- aabd31b843d1faf29e4d3e50c135d6539ac9fe77ffd
> |-- aal
> |   |-- aal4f31a9b3119c2f6bb3b45c27c4cb33ff45a8f6e2
> |   `-- aald31b843d1faf29e4d3e50c135d6539ac9fe77ffd
> |-- ace
> |   |-- ace4f31a9b3119c2f6bb3b45c27c4cb33ff45a8f6e2
> |   `-- aced31b843d1faf29e4d3e50c135d6539ac9fe77ffd
> |-- ehw
> |   |-- ehw4f31a9b3119c2f6bb3b45c27c4cb33ff45a8f6e2
> |   `-- ehwd31b843d1faf29e4d3e50c135d6539ac9fe77ffd
> |-- eia
> |   |-- eia4f31a9b3119c2f6bb3b45c27c4cb33ff45a8f6e2
> |   `-- eiad31b843d1faf29e4d3e50c135d6539ac9fe77ffd
> |-- ial
> |   |-- ial4f31a9b3119c2f6bb3b45c27c4cb33ff45a8f6e2
> |   `-- iald31b843d1faf29e4d3e50c135d6539ac9fe77ffd
> `-- ode
> |-- ode4f31a9b3119c2f6bb3b45c27c4cb33ff45a8f6e2
> `-- oded31b843d1faf29e4d3e50c135d6539ac9fe77ffd
>
> 7 directories, 14 files
>
>
> --
>   Jascha Geerds
>   j...@ekby.de
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Possible breaking changes from staging

[Luca Bruno]

NM 1.0 and GNOME 3.16.1 have been merged into staging. Hopefully we'll get
binaries soon and merge to master. I also hope I didn't break things a lot
;)

Best regards,

On Wed, Apr 22, 2015 at 11:05 AM, Luca Bruno  wrote:

> In this period I'm working on GNOME 3.16 and NetworkManager 1.0 [1]. In
> addition to that, glib has been bumped.
> I'm currently compiling the PR for staging and then merge it if
> everything works.
>
> Below the possible breakages you must be aware of. These updates are
> necessary to forward our software, we'll fix eventual bugs with the time.
>
> * glib 2.42 -> 2.44
>
> This update is minor yet not entirely trivial. It's probable that
> software using gdbus may break because of bugs fixed in 2.44.
>
> * NM 0.9.8.10 -> 1.0
>
> It's big but shouldn't be breaking anything. In particular, you'll see a
> vboxnet0 interface in nm-applet if you enabled the virtualbox nixos
> service, I hope that will go away with further upstream updates.
>
> We haven't upgraded it until now because previously it showed tons of
> vboxnet interfaces, making the applet unusable.
>
> * GNOME 3.12 -> 3.16
>
> I did my best to keep both 3.12 and 3.16 in our repository, but I won't
> guarantee that 3.12 will keep working as well as before, I won't
> maintain it anymore.
>
> In case you switch to GNOME 3.16, running GTK 3.12 apps will be ugly
> because they broke some css stuff. Therefore I suggest you to add in
> packageOverrides a gnome3 = gnome3_16; and forget about GNOME 3.12, it
> will be dropped as soon as possible.
>
> [1] https://github.com/NixOS/nixpkgs/pull/7357
>
>


-- 
NixOS Linux 
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev