Re: [Nix-dev] gentoo like nixpkgs globala useflags in nixpkgs config - would it be worth it?
>> Issue pulseSupport -> does it make sense to have a global 'enable >> pulse for all packgages' config.pulseSupport = true flag? >> >> packages could be using: config.pulseSupport or false to disable by >> default then. > > In my opinion baking global implicit state into nixpkgs is not a good > idea. It's not *state*, is it? We're basically already doing global configuration, and I think the proposal is about actually using this to get a USE-like system. IMO it wouldn't hurt, as long as the defaults are sensible, because Hydra will most likely only build those (the number of possible configurations it could build increases exponentially with every new setting, so it couldn't possibly build all of them). In other words: please don't turn NixOS into Gentoo by forcing us to compile everything. =) signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Should we drop 9P?
> 9P is used by NixOS to share host's nix store with Qemu virtual > machines. Such technique is used in the build process, in the > test-driver, so to say in the critical places. > Recently few bugs in 9P were found (#23957 #23020 #22695) which > reveals that 9P code is not very mature and perhaps NixOS is the first > team which uses 9P heavily and relies on it in production. > > Shouldn't we replace 9P with something battle-tested like NFS or > Samba? It may also improve the performance because 9P server works in > qemu process, in user mode and there are as many servers as virtual > machines running. In terms of performance getting rid of QEMU where possible is probably the better option. Containers are fairly mature these days, and then sharing file-systems is a matter of bind-mounting. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Move NodeJS to separate overlay repository?
> There are a few package groups that I believe are not used in NixOS > core (boot, containers, ...) and are not updated as much as they could > be. Node, Haskell, others? Could the Haskell packages still be built by Hydra? I highly appreciate that, because it makes Haskell development a very smooth experience with Nix, especially on lower-end systems that I have to use from time to time. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Fwd: Why having releases if you break things in it often
>>> I dont want to be a 100% full time nixos developer / sysadmin just >>> to solve such stupid problem. Except somebody would pay me for that, >>> then I would think about it. >> >> Ah, so you won't do it for free but you expect the army of experts to >> rush for help and do it for free for you, do I read that correctly? > > "rush" the bug is open since over a month, I would not call that > "rush". That's not the point, is it? NixOS is not a commercial organisation. It's entirely driven by voluntary work. Yes, even the companies that contribute to NixOS are volunteers! In other words you can do one of four things: * Contribute patches yourself. * Kindly (!) ask for help and with a bit of luck receive voluntary (!) support from the community. Remember that nobody is obliged to help you, but if you're nice and respectful, people will usually try their best in my experience. * Use a different solution. * Oblige somebody to help you by paying them. Did you pay someone? If yes, talk to them and be as rude as you want, because you have a contract. If no, then pick one of the other three options. > If thats only a very unproffessional project, why creating a > professional looking web site? Why not? We are not obliged to *anything*, not even a terrible website. =) > So you find time for much PR, but you cant answer a bug report in a > month? Sure, why not? Who are you to tell me what I should do in my spare time? Pick one of the four options above. Greets ertes signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Arch is dropping i686. Should we?
> As far as I know, Nix has no way to gather statistics on how many people > are using different architectures. This makes questions like this a bit > awkward. > > One person on IRC said they think they may have used it once. Does > anyone on this list use NixOS on i686? anyone on this list know anyone > using NixOS on i686? Arch has a different philosophy, so it makes sense for them to drop legacy support sooner than other distributions. It's also important to consider that they actually have to manage two sets of packages, so their maintenance cost may be a lot higher, while in Nixpkgs we can get away with a single expression for both (and more) architectures most of the time. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Malicious installation methods
>> I ask you to PLEASE remove this installation method from the >> recommendations on the page because it makes it look like you don't >> care about computer secuirty one bit. > > Now, that's an interesting point. Are there many people who never > installed nix because the installer is the recommended installation > method? I have actually witnessed a few people (in person) who expressed concerns about that method. When questioned, the concerns turned out to be unreasonable, because you are going to trust code from that server to run as root blindly anyway, but this is one of those "evil pattern" triggers that make security-minded people jump. It's like seeing "strcpy" in C code: not *necessarily* bad, but triggers the warning bell. For marketing reasons it may be beneficial to attach a security note to that command, such that people understand why it's really not any less secure than other methods. Alternatively get rid of the pattern and distribute a bunch of tarballs instead. In other words: perform the installation on Hydra, tar it, then provide platform-specific tars. One interesting point is that little of the installation really requires root permissions. Users could be asked to create the `/nix` directory and optionally a bunch of build users as root and then let the script do its job without root. That way the script never needs to switch users. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Haskell NG: Local packages
Hello fellow Haskellers,
I have a bunch of local Haskell packages, mostly libraries. The package
X lives in the directory `ertes-src/X/main`. There are lots of direct
and indirect dependencies between them. My current solution is ad hoc
and rather ugly. I'm using the following hand-crafted `shell.nix` file
in every project:
{ pkgs ? import nixpkgs {} }:
let inherit (pkgs.haskellngPackages) callPackage;
extPkg = path: deps:
(callPackage (import path) deps).override (args: args // {
mkDerivation = expr:
args.mkDerivation (expr // {
src = pkgs.fetchdarcs { url = path; };
});
});
thisPkg = callPackage (import ./devel.nix) {
direct-dep1 = extPkg ertes-src/dep1/main {
indirect-dep1 = ...;
indirect-dep2 = ...;
};
direct-dep2 = extPkg ertes-src/dep2/main {};
};
in thisPkg.env
Do not pay too much attention to the `extPkg` function. It just works
around cabal2nix' lack of support for Darcs URLs.
My question is: Is there a nicer and more principled way to solve this?
Ideally I could simply add all of my local packages in some way to my
`~/.nixpkgs/config.nix`. Then they could just depend on each other
regularly and I wouldn't need any shell.nix hackery. I'm guessing that
I need to override haskellngPackages in some way or perhaps create my
own using some override.
Greets,
Ertugrul
signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Haskell NG: Local packages
I believe Peter's message 87mw5qvj9x@write-only.cryp.to could
answer my question. I will give it a try.
Hello fellow Haskellers,
I have a bunch of local Haskell packages, mostly libraries. The package
X lives in the directory `ertes-src/X/main`. There are lots of direct
and indirect dependencies between them. My current solution is ad hoc
and rather ugly. I'm using the following hand-crafted `shell.nix` file
in every project:
{ pkgs ? import nixpkgs {} }:
let inherit (pkgs.haskellngPackages) callPackage;
extPkg = path: deps:
(callPackage (import path) deps).override (args: args // {
mkDerivation = expr:
args.mkDerivation (expr // {
src = pkgs.fetchdarcs { url = path; };
});
});
thisPkg = callPackage (import ./devel.nix) {
direct-dep1 = extPkg ertes-src/dep1/main {
indirect-dep1 = ...;
indirect-dep2 = ...;
};
direct-dep2 = extPkg ertes-src/dep2/main {};
};
in thisPkg.env
Do not pay too much attention to the `extPkg` function. It just works
around cabal2nix' lack of support for Darcs URLs.
My question is: Is there a nicer and more principled way to solve this?
Ideally I could simply add all of my local packages in some way to my
`~/.nixpkgs/config.nix`. Then they could just depend on each other
regularly and I wouldn't need any shell.nix hackery. I'm guessing that
I need to override haskellngPackages in some way or perhaps create my
own using some override.
Greets,
Ertugrul
signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Haskell NG: Local packages
I believe Peter's message 87mw5qvj9x@write-only.cryp.to could
answer my question. I will give it a try.
It does and works well. Thanks anyway!
I have a bunch of local Haskell packages, mostly libraries. The package
X lives in the directory `ertes-src/X/main`. There are lots of direct
and indirect dependencies between them. My current solution is ad hoc
and rather ugly. I'm using the following hand-crafted `shell.nix` file
in every project:
{ pkgs ? import nixpkgs {} }:
let inherit (pkgs.haskellngPackages) callPackage;
extPkg = path: deps:
(callPackage (import path) deps).override (args: args // {
mkDerivation = expr:
args.mkDerivation (expr // {
src = pkgs.fetchdarcs { url = path; };
});
});
thisPkg = callPackage (import ./devel.nix) {
direct-dep1 = extPkg ertes-src/dep1/main {
indirect-dep1 = ...;
indirect-dep2 = ...;
};
direct-dep2 = extPkg ertes-src/dep2/main {};
};
in thisPkg.env
Do not pay too much attention to the `extPkg` function. It just works
around cabal2nix' lack of support for Darcs URLs.
My question is: Is there a nicer and more principled way to solve this?
Ideally I could simply add all of my local packages in some way to my
`~/.nixpkgs/config.nix`. Then they could just depend on each other
regularly and I wouldn't need any shell.nix hackery. I'm guessing that
I need to override haskellngPackages in some way or perhaps create my
own using some override.
signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Haskell NG: Local packages
Hi Jeff,
I'm using a similar approach for non-Haskell packages. However, for
Haskell packages it doesn't interact well with cabal2nix. Hence my
question. I have now solved it the following way in
`~/.nixpkgs/config.nix`:
haskellPackageOverrides = self: super:
let asDarcs = path:
(self.callPackage (import path) {}).override (args: args // {
mkDerivation = expr:
args.mkDerivation (expr // {
src = nixos.pkgs.fetchdarcs { url = path; };
});
});
in {
myxmonad = self.callPackage ertes-config/myxmonad {};
mytaffybar = self.callPackage ertes-config/mytaffybar {};
testpkg1 = asDarcs ertes-src/testpkg1/main;
testpkg2 = asDarcs ertes-src/testpkg2/main;
};
The testpkg2 executable depends on the testpkg1 library, and it works.
The expressions are generated by cabal2nix, so no manual work is
involved. So it seems to do exactly what it should.
It's not specific to Haskell (and I haven't tried haskellng at all
yet) but for personal packages I have mypkgs/pkgname/default.nix,
and then a top-level mypkgs/default.nix:
[...]
I add that to my NIX_PATH so I can `import mypkgs` anywhere to get
nixpkgs overridden with the stuff I'm working on.
signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NixOS on a machine with only one user
Assuming bigger things usually gives you a better or at least more scalable result, and in most cases the additional cost is very close to zero. For example every single-disk system I set up uses a two-disk RAID array with one missing disk from the point of view of mdadm. If it does happen that I actually want to add more disks, it will be a few short online commands to do it instead of a tedious reinstallation of the entire system, and it costs only a few megabytes. The additional disk might as well be an external emergency disk that starts synchronising as soon as I plug it in. How did you manage to make mdadm do that? It allows you to create arrays with missing disks, and you can activate them as long as enough disks are present. It's like creating an array with a failed disk right from the beginning: mdadm --create /dev/md/blah-system \ -l 1 -n 2 \ /dev/sda2 missing Notice the `missing` argument. It tells mdadm that the first disk is `/dev/sda2` and the second disk is currently missing. When you plug in a second disk, just add it to the array. When you plug in a third disk, convert the array to RAID5 (can be done online in the background, see `--grow`) and add the third disk. And so on. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Please test Nix store auto-optimise
Once I wondered if using reflinks instead of hardlinks might be better from some point of view, but it probably won't be a big difference. Tl;dr: The current linking strategy has a noticable space and time overhead that is large enough to be unacceptable for me, so I turned it off. The overhead can be removed in two ways: Use reflinking when available and more importantly get rid of the `.links` directory. When the filesystem supports reflinking, it should be used. The advantages are: * From the viewpoint of the FS user this is completely transparent. * Most modern FSes support block-level deduplication. * The FS ensures consistency. No potentially dangerous assumptions about the atomicity of FS operations. * Can be done in the background even on systems without nix-daemon. * Can be used safely to copy files into the store when `src` points to the local filesystem. * The `.links` approach would no longer work, forcing us to do something more sensible, for example a database file. To deduplicate new builds, simply maintain a fast database of block hashes and compare the newly written files against it. If a match is found, instruct the filesystem to deduplicate and that's it. After GC check to see if the GCed blocks still exist, otherwise remove them from the database. Should it happen that the database becomes inconsistent with the store (for example because the system crashed during deduplication), nothing bad will happen except that a few blocks might be wasted. It would also help developers. For example it's not uncommon for me to point `src` to the local filesystem. And since I'm doing a lot with machine learning nowadays, often enough there are some huge files in a `data` or `examples` directory. Each time I do nix-build those are copied verbatim to the store, which not only takes an unnecessarily long time, but also wastes space. As a btrfs user I would definitely benefit From reflinking. It would still require reading the entire file for hashing, but the space waste would be gone. The requirement to have this huge .links directory would be gone, making both the optimisation process and garbage collection faster. Just listing its contents takes more than a minute for me with most of the time spent disk-thrashing. I'm not even sure if it's really required at all, since for book-keeping a database file would be a better option. I'd also like to note that store optimisation is probably better handled by a separate program. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Please test Nix store auto-optimise
IMO much of the time overhead is basically unavoidable, as you need to do lots of random probes to a huge table (order of million entries, e.g. 1.5M in my case). Putting a single Bloom filter in front would shave I/O of most *failed* probes, but otherwise we can't get much better, I believe (being a PhD student in data structures). There is some overhead, but it doesn't have to be anywhere near the one we have right now. The filesystem stores a lot more information than necessary. Basically making the on-disk structure smaller and more domain-specific makes the operation noticably faster, and I can't think of a smaller structure to store hash-maps than patricia trees. A simple hash-to-block mapping is all we need. I'm not sure whether the Bloom filter would really help. Since it is impossible to predict the required size you would have to use some form of growing Bloom filter or proactively create a filter that is so large that it might destroy its own purpose. Anyway, this deduplication work seems better delegated to some other SW/library. Surely there are some already. If we wanted to deduplicate *similar* (unequal) files, we would even need to get FS-dependent. In the typical ext4 case, we don't even have reflinks AFAIK... That's what I'm saying. A separate tool should handle this, and it would depend on the FS (in the sense of being a function of the FS), but not be bound to any particular one. For traditional FSes like ext4 it would do hard-linking, but perhaps in a slightly smarter way. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NixOS on a machine with only one user
NixOS noob here. I'm using NixOS on a machine with only one user (me). What's the best strategy for installing packages? [...] Installing into a normal user profile In general I tend to just assume that my system is an actual multi-user system. I'm a normal user, so I don't make any assumptions on the way imaginary fellow users use their account. A very small number of packages will have to be used by the majority of the users on the system, for example because they are relevant to the network topology or purpose. Most of them will be installed in the form of NixOS services. All other packages are user-specific. Installing user packages into the user profile has the additional advantage that it is safe and secure to install your own packages, which you may want to do from time to time, perhaps because you may want to use Nixpkgs-master for some bleeding edge version or because you may want to use Nix for a custom package or development. Remember that Nix handles not only code. You can equally well write and deploy a bunch of static files using it. Assuming bigger things usually gives you a better or at least more scalable result, and in most cases the additional cost is very close to zero. For example every single-disk system I set up uses a two-disk RAID array with one missing disk from the point of view of mdadm. If it does happen that I actually want to add more disks, it will be a few short online commands to do it instead of a tedious reinstallation of the entire system, and it costs only a few megabytes. The additional disk might as well be an external emergency disk that starts synchronising as soon as I plug it in. Fortunately this is possible with NixOS. Many distributions, including all Anaconda-based ones (RHEL, CentOS, Fedora, etc.), do not allow you to do this. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] cabal2nix: Local Darcs URLs
Hello fellow Haskellers, how do I specify a Darcs repository on the local filesystem when invoking haskell-ng's cabal2nix? Note: I need the Darcs variant, not the local filesystem variant, so just giving a filesystem path does not work. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to develop with chicken and emacs
The way I do this is to fire up a Makefile from Emacs. The Makefile uses nix-shell to start the actual builder: nix-shell --pure --command ./Setup build This is an indirection, but it makes sure that the environment the builder sees is (fairly close to) the environment the build script would see when you use nix-build. And indeed, you can actually use nix-build as well, although you probably don't want to, because it rebuilds the whole thing all the time. That's a good tip! I will start with this. But it still wouldn't allow me to use the interactive features of emacs, if I understand correctly? It would be great to evaluate expressions inside emacs without building the whole app. Well, you can invoke nix-shell instead of the regular shell from within Emacs. Getting an inferior mode or any other tighter integration to work, if there is one, could be more difficult. One thing you can do is to make Emacs part of a project-specific development environment using myEnvFun. The wiki should help you with this. Alternatively many integration modes allow you to specify the command used for invoking the helper programs. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to develop with chicken and emacs
Hi there Eike, I found out about the egg2nix program and saw, that the 'chicken-eggs.nix' file contains many eggs and that the egg2nix program uses it. So, right now I'm starting a shell like this nix-shell -p egg2nix This drops me in a shell, where the eggs are setup via CHICKEN_REPOSITORY_EXTRA and CHICKEN_INCLUDE_PATH. That's a step forward, but it still feels strange. Maybe just because I'm not used to it… Is this a recommended way for developing? (I probably should create my own nix file to make this env I guess) My experience with Chicken is limited, but what you are describing sounds about right. It is indeed the way we develop software with the help of Nix. In Haskell land cabal2nix generates both an expression for your project and a (separate) expression that drops you into a shell with all the dependencies in scope. But the real problem is, that my emacs doesn't know about this environment. I don't know how to handle this (besides starting a new emacs from the nix-shell shell, which I don't want). The way I do this is to fire up a Makefile from Emacs. The Makefile uses nix-shell to start the actual builder: nix-shell --pure --command ./Setup build This is an indirection, but it makes sure that the environment the builder sees is (fairly close to) the environment the build script would see when you use nix-build. And indeed, you can actually use nix-build as well, although you probably don't want to, because it rebuilds the whole thing all the time. I hope this helps. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Haskell NG: Still no binaries
Hello everybody,
today I've attempted to switch to Haskell NG again. The new tree itself
seems to work for me, but I'm still not getting prebuilt binary
packages. The only binary I get is GHC, while every other Haskell
package seems to be compiled from source code.
I've tried with the following reasonably minimal setup. My
~/.nixpkgs/config.nix contains the following code:
nixos : {
packageOverrides = pkgs : rec {
hsEnv = pkgs.haskellngPackages.ghcWithPackages (hs: with hs; [
cabal-install
]);
};
}
In my initial attempt I would have liked to switch to the new GHC as
well, so I've also tried the following override:
hsEnv = pkgs.haskell-ng.packages.ghc7101.ghcWithPackages (hs: with hs; [
cabal-install
]);
I'm on the nixos-unstable channel:
# nix-channel --list
nixos https://nixos.org/channels/nixos-unstable
The commands I've tried to get binary packages were:
nix-env \
--option extra-binary-caches https://hydra.nixos.org \
--option extra-binary-caches https://hydra.cryp.to \
-iA nixos.pkgs.hsEnv
nix-env \
--option extra-binary-caches http://hydra.nixos.org \
--option extra-binary-caches http://hydra.cryp.to \
-iA nixos.pkgs.hsEnv
Neither of them gave me binaries. Am I doing something wrong there?
Background: As noted in an earlier thread my current development machine
is too weak to build everything from source. The build of the GHC
7.10-based environment above took about 30 minutes, and that doesn't
include GHC itself.
Greets,
Ertugrul
signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Haskell NG: Still no binaries
Hi Peter, nix-env \ --option extra-binary-caches https://hydra.nixos.org \ --option extra-binary-caches https://hydra.cryp.to \ -iA nixos.pkgs.hsEnv what happpens when you use that command to install the attribute nixos.pkgs.haskell-ng.ghc784.cabal-install and/or nixos.pkgs.haskell-ng.ghc7101.cabal-install? Does your machine get binaries for those attributes? Both are built from source, no binaries except GHC. You are running x86_64-linux, right? That's right. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Haskell NG: Still no binaries
That did it! Since I'm running NixOS I am indeed running nix-daemon. The following setting did the trick: nix.binaryCaches = [ https://cache.nixos.org/; https://hydra.nixos.org/; ]; IMHO, nix-env should pass those options on to the daemon, i.e. it should not be necessary to hard-code hydra.cryp.to as a global binary cache for this to work. Actually I'm not sure whether this is such a good idea. If it did, it would be a backdoor into fellow system users. An attacker could construct a Nix expression that matches exactly another system user's expression. Then the attacker builds it, but they tell Nix that they have a binary cache available for it, which delivers an infected version of the derivation. When the other system user tries to build the same expression, they find that it is already built, but it is actually the infected substitute injected by the attacker. Just out of curiosity, did you configure nix.trustedBinaryCaches = [ http://hydra.nixos.org http://hydra.cryp.to ]; in your configuration.nix? I didn't. Now that you mention it I briefly remember Nix telling me something about the untrusted binary cache. I just ignored it, assuming that Nix would go ahead and use it anyway. I will try with that setting. But yes, because of the above it's totally sensible that Nix doesn't just use any cache that you tell it to use. Thanks! Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Per-user default binary caches
Hello everybody, is there a way to set a default binary cache for my regular user without making it a systemwide default, perhaps in `~/.nixpkgs/config.nix`? Basically I'd like to get rid of the --option argument when dealing with Haskell packages. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Binary trust (was: Haskell NG: Still no binaries)
Hi Kirill, nix-env \ --option extra-binary-caches https://hydra.nixos.org \ --option extra-binary-caches https://hydra.cryp.to \ -iA nixos.pkgs.hsEnv Might it be the case that you are running nix in daemon mode and thus it ignores `binary-caches`? That did it! Since I'm running NixOS I am indeed running nix-daemon. The following setting did the trick: nix.binaryCaches = [ https://cache.nixos.org/; https://hydra.nixos.org/; ]; Thanks a lot! Unfortunately hydra.cryp.to does not seem to support TLS. That's why I left it out. But that raises an interesting question: Where do the hash values for the binary packages come from? At this point since we lack deterministic builds I would assume that they come from the same host that delivers the substitutes. A related question is: Are the hashes signed? If the hashes are not trusted, then a plain-text connection would be a huge security risk regardless of whether you trust the host. Even a malicious user or an infected machine on your local network could replace binary packages on their way and get arbitrary code onto your machine. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Let's switch the default shell in Nixpks / NixOS to csh
This makes me wonder whether maybe we should switch all shell scripting in Nixpkgs to csh? Wouldn't that solve a lot of problems? I've heard experts say that csh is generally considered superior for scripting tasks because of its more intuitive syntax. I really dislike csh's syntax, and that of tcsh, and having to learn it in order to write scripts for Nix would be undesirable to say the least. What you are saying is basically: Switching away from Bash is not the proper solution. The proper solution is not to assume a certain shell at all and let the builder itself choose it. And I agree. This would be easy to support by stdenv. Basically all we need is to make $stdenv/setup portable (i.e. parametric). Also I agree that Guix is in a number of ways superior. My only problem with Guix is Scheme, because I have some language-design issues with it. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use GHC 7.10.1 as default Haskell compiler in nixpkgs
The last time I tried to switch, Nix started to build everything from source, including GHC, even with the options described in the Haskell NG thread. what architecture do you use? Is it i686-linux by any chance? I'm asking because Hydra isn't building binaries for that platform at the moment. You should always get binaries of GHC itself, though. Those are part of the standard nixos-unstable channel. It's x86_64 using the unstable channel. Since it has been a while I will just retry with the original instructions and see what happens. I'll report my result here. Thanks! Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use GHC 7.10.1 as default Haskell compiler in nixpkgs
If you haven't migrated to Haskell NG yet ... now would be a good time to do it. (I'm looking at you, Agda.) I'm currently forced to work on a rather weak machine, so building from source is prohibitively expensive. Is there a very short summary of how to do it and still get binary packages? The last time I tried to switch, Nix started to build everything from source, including GHC, even with the options described in the Haskell NG thread. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to add an additional IP address besides the DHCP configured address?
I would like to add the IP address 172.16.48.17/28 to my enp0s3 interface but I would also like to keep the DHCP configured address. [...] Is this possible or should I use networking.localCommands? The proper way to do this is to instruct the DHCP *server* to assign the addresses. Most servers can assign IP addresses based on MAC addresses. All options below are hacks, because they circumvent the lease system of DHCP, so they are unsafe. The most proper unsafe way is to configure the DHCP *client* to assign the additional address. Unfortunately when you configure dhcpcd to assign a static address, it does not ask the server to get a lease at all. Perhaps you can figure out a way to do both. The relevant option is `networking.dhcpcd.extraConfig` and the `static` option from dhcpcd.conf(5). A less proper way is to use a hook and assign the additional address yourself. The most general interface to that seems to be `networking.dhcpcd.runHook`. It's less proper, because then the client does not manage that address. See also the section 3rdparty link management (sic) in dhcpcd(8). It seems to offer an addtional option, but I did not follow all the references. If all else fails you can disable the DHCP client *daemon* altogether and do one-shot setups instead. This is a horrible setup that cries for network problems, because now even the regular dynamic address falls outside of the lease system. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Missing documentation
What I want to say is that it might be boring to maintain such a comprehensive help system, but it’s totally worth it, and it’s wonderful that someone volounteers to do that. BTW it would be nice to have a generic framework for writing docs and then doing things like generating man-pages and context-sensitive help automatically. Maybe such a framework already exists? That's what I want to do, but I'm still not sure whether the community is behind me. I don't want to spend days to develop a nice solution just to find that nobody actually wants it. I did not count formally, but so far my proposal was met with mixed feelings. If there is a definite, stop talking, please just do it, then I will do it. My changes would include minor documentation improvements, but most notably a context-sensitive help system that is easy to use especially for Nix developers. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Missing documentation
Umm, this is kinda the point that I was making, you were the one who suggested to write that regular users should use this feature, so I explained that you can't differentiate between regular and advanced users to you need to treat all users equally and just document the feature in a clear and straightforward way. Apparently we never disagreed. We just failed to communicate our notions properly. =) One thing to note is that some functionality is genuinely unsafe, but sometimes still useful to users doing less obvious, experimental or innovative things. There is nothing wrong with adding big warning signs to the documentation in the form of: This is an unsafe operation that can violate assumptions X and Y. Make sure you understand how to apply it safely before use!. Sometimes there are use cases that the developers simply did not anticipate. This is actually quite common. For example the `--set` flag to `nix-env` should definitely be documented, because it is both safe and useful when you do more with environments than managing user-specific package sets. If you follow this line of thought to its conclusion, then the set of interfaces is really a union of safe (assumptions-preserving) and unsafe (potentially assumptions-violating) interfaces. The notion of an internal interface is informal and more often than not ill-defined. I believe that all interfaces should be fully exposed and documented, but the unsafe ones should carry a warning sign along with the rules you need to follow to preserve the assumptions, i.e. how to use unsafe interfaces safely. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Missing documentation
Software should be fully documented. If you don't know where the line between regular users and advanced users is, don't draw one in the first place. It's not so much a question of regular vs. advanced use, but whether something is a stable interface. If we document a command like nix-store --register-validity (which is mostly a hack to support the nixos-install bootstrap), we'd pretty much commit to supporting it in the future. If it's undocumented, we can change or remove it in the future. Good point. Nevertheless it's a useful feature, so perhaps it's time to consider stabilising it or providing a stable wrapper. At this point it is practically the only way to install Nix-based distributions, and you really want access to it when you're building lots of machines and containers, i.e. whenever you don't want to use any of the predefined installation routines. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Missing documentation
Regarding --register-validity, it's a bit of internal command that I'm not sure should be documented. But yeah, ‘exportReferencesGraph’ shouldn't refer to it :-) Well, it is apparently a helpful utility when you change core parts of NixOS, when you create your own Nix-based distribution or when you create virtual machine images without using NixOS tools. In fact it may be the only way to do it properly. I believe everything should be documented at least in a technical manual. But there is nothing wrong with putting a note there: Regular users should not need to use this command. The format is as follows (see decodeValidPathInfo() in store-api.cc): line containing the store path if --hash-given is used, a line containing the hash of the contents of the path (nix-store -q --path) if --hash-given is used, a line containing the size of the contents of the path (nix-store -q --size) the deriver an integer containing the number of references the references, one per line This is repeated until EOF. See pkgs/build-support/kernel/paths-from-graph.pl for an example of a script that generates registration info in this format. Thanks. That's a bit difficult for shell scripts to parse, because it's context-sensitive. Is there a context-free or regular format or a predefined parser that I can use from within a small shell script? I need to copy the full closure of a derivation in a builder. Regarding the overhaul of `--help` I will just do it and see what people think about it. Note that I removed --help on purpose because I didn't want to maintain two sets of option documentation. Invoking man is also what tools like Git do, so it's not entirely uncommon. One advantage of the way Git does it is that it's context-sensitive. It does not just fire up a general man-page and lets you search it. Nix tools have a large and growing set of options, so at least we should have that. Would you agree to reintroduce it, if it contributes to a better UX? I will think of ways to keep everything in one place. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Missing documentation
I believe everything should be documented at least in a technical manual. But there is nothing wrong with putting a note there: Regular users should not need to use this command. I wouldn't be in such a hurry to document things in this manner , because the basic assumption is if it's documented then it's supported and we welcome users to use it, and how would you differentiate between regular and advanced users anyway? Of course you should introduce the feature or functionality with the use-case that it's aimed at (i.e. Why do I need this?), but documenting something and then disclaimering it with but actually you don't really want to use this coz it's risky is usually just an invitation for mayhem. Bottom line IMHO - either you educate users with the correct information and then trust them to use the feature responsibly, or you don't expose it. That doesn't seem to be a coherent argument. Would you suggest that the `-f` option to `rm` should not be documented? Would you deny access to the manual of `dd` or `sgdisk` while being root? You can seriously break stuff with those even worse than with any Nix command. Software should be fully documented. If you don't know where the line between regular users and advanced users is, don't draw one in the first place. Chances are the line doesn't exist and you're just trying to patronise your users. I hate software that patronises me. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Missing documentation
+1 to doc fixes! :-) I'll gladly help with peer review for the content or whatever I can within the prose capacity :-) Nobody else? I would just do it, but at the very least I need an explanation of the missing parts. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Missing documentation
+1 to doc fixes! :-) I'll gladly help with peer review for the content or whatever I can within the prose capacity :-) Nobody else? I would just do it, but at the very least I need an explanation of the missing parts. The best way to find out might be to use git blame and see commit history. Okay, I will try that. Regarding the overhaul of `--help` I will just do it and see what people think about it. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] nix-store --register-validity
Hi everybody, could someone please explain the `--register-validity` flag of `nix-store` to me? I'm mostly interested in the format, because I need to use `exportReferencesGraph` in a derivation. But most likely from the search results I've got when searching for its documentation, later I will need the flag itself as well. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Missing documentation
Hi again everybody, I noticed that the documentation of various Nix tools is incomplete and often too vague, so I'd like to contribute documentation patches. From the top of my head the following two flags are currently undocumented: * `nix-store --register-validity` * `nix-env --set` The latter seems pretty straightforward from the way it is used in NixOS scripts, but the former is a bit of a mystery to me. If someone could give an explanation in simple prose, I'm happy to update the Nix documentation. Another thing I would like to do is to add an actual help system to all Nix tools. This means that `--help` no longer opens the man page, but instead displays context-sensitive help. For example nix-store -q --help would display the documentation of the query functionality of `nix-store`. I believe, this is much more useful than opening the man page. Please tell me what you think about that. Finally with some help from the community we can fix all dead references. For example when you read the documentation for `exportReferencesGraph`, it refers to the format used by `nix-store --register-validity`, and there the reference chain breaks, because the latter is not documented anywhere except perhaps in the source code. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Remember that we're talking about very different paradigms. I can totally see how you would want and love macros in CL. In CL you encode your solution in such a way that using macros is the natural thing to do, and then lexical manipulation is probably something that comes in handy. In Haskell your encoding is entirely different, so you wouldn't use macros in the first place. Instead you would use non-strict higher-order functions together with type classes. And of course Common Lisp does have higher-order functions; do type classes add any encoding power? I always thought they were needed to keep track of all that happens, and working duck typing with optional multiple dispatch allows to write the same code but with less safety. Type classes are essentially just principled overloading. They allow you to write reusable code bound to certain *features* (rather than specific types): multiCoreMapReduce :: (Monoid b) = (a - b) - [a] - b multiCoreMapReduce f = fold . parMap rseq f Whenever `b` is a monoid, the above function is a multi-core MapReduce implementation. Here `Monoid` is a class. There is a myth on the Lisp side that Common Lisp has a paradigm that means it wouldn't benefit from some more static annotation and analysis features; there is a myth in many non-Lisp places that feature X makes macros unneeded. Both are myths. Just a trivial with-fields (in its many many forms) is an example of a thing that is trivial with macros, useful enough to be built into Object Pascal and doesn't perform its primary task (reducing clutter) if implemented without macros. I couldn't find any information online about what `with-fields` is, but From its name I'm just going to assume that it brings stuff into scope semi-automatically, i.e. it does something a regular function cannot do. Does a CL programmer benefit from this? Absolutely! Would programmers of most languages benefit from this, for example Object Pascal? Likely! Would a Haskell programmer benefit from this? Well, probably not as much as you would want them to. Let's turn the table: Laziness is a very powerful feature in Haskell. Could I convince you of that? Probably not, because you're not a Haskell programmer and you wouldn't find it very useful in CL. But I may be crazy about laziness and think: How can anyone deny the usefulness of laziness?. You're suffering from the same fallacy w.r.t. macros. Let me explain. There is a *very common* myth on the CL side that macros are so powerful and so useful in CL that they must be powerful and useful *everywhere*. That may easily lead you into the trap of thinking that any language that lacks macros has a serious drawback compared to CL. But that's a myth. I'll assume that right now you're thinking: Oh, come on, macros allow you to manipulate lexical features, scope, introduce new views to data, build functions automatically, etc. How can they be useless in any language?. I could fall into the same trap: Oh, come on, laziness allows you to build virtual data structures, specify if, how and when to evaluate (e.g. in parallel), apply semantically huge/infinite transforms in O(1), use action DSLs, etc. How can it be useless in any language?. Yet it would still be pretty useless in CL. Let me go one step further: Macros would do more harm than good in Haskell, if people would actually be going to use them. In fact we do have a similarly powerful macro system called TH. We even have a system for custom syntax called QuasiQuotes, if you need it. Our TH and QQ are even type-safe! Yet most experienced Haskell programmers try to avoid both of them as much as possible, and for a good reason, which is beyond the scope of this mail. Now macros-considered-harmful probably goes even worse against your world-view than macros-considered-useless. But consider this: In a similar way laziness would not only be pretty useless in CL, it would do more harm than good, because it entails the kind of non-determinism that requires all side effects to be expressed in terms of a lazy domain-specific action language (which is our `IO`). You probably don't want that. Combined with macros it would have the potential to render CL unpredictable. Modern van Laarhoven lenses are a powerful abstraction for accessing data structures, mainly because a lens does not necessarily correspond to a certain concrete field (it can correspond to many fields or to a logical alternate view of certain data) and also because both lenses and lens operators are first class values. _Which_ lenses should be used, actually? It looks like there is a controversy about which packages are the correct ones to use and which should be considered just a proof-of-concept playground for new ideas. It doesn't actually matter. One nice thing about van Laarhoven lenses is that all the lens packages are compatible with each other. Most people use the `lens` library though, because it is very
Re: [Nix-dev] Use Haskell for Shell Scripting
I think you are thinking too big-system-design instead of quick-scripting about that. Let's say I don't draw a line between small systems and large systems when it comes to abstractions, because algebraic abstractions are cheap. IMO a minimum requirement for a language to be called functional is that functions are cheap. I mean cheap as in almost free, just like mutable variables are almost free in imperative programming. Quantification (generics in most languages) must be entirely free. It's not in C++, but in boxing languages it is. Now if quantification is free and your functions are almost free, then algebraic abstractions are also almost free. You might as well use them everywhere, even in your quick-n-dirty scripts, because they make your code shorter, more maintainable and usually more readable as well (especially the types). While you're reading this you should not think of OOP. OO abstractions are expensive in every way. They make your program slower and your source code longer. They are not suitable for quick-n-dirty scripts. I have added an external module to StumpWM that goes completely against the default logic of connecting screens, workspaces and windows. It just moves windows between the workspaces when it needs to. I still use the hard parts of a WM (X protocol handling) and just added my own implementation of the fun parts. Yeah, that sounds pretty much like what I do right now, except with a different WM at the core. I think I was not clear with my idea about Ratpoison. I meant the Unix way of doing things: there is Ratpoison that can easily receive commands to change the splits and move the windows. Your logic is in Haskell and it just sends commands to Ratpoison (which has been debugged once). The Unix philosophy can manifest in a number of ways. Usually you make programs pipe-friendly and perhaps add a configuration file. That's the traditional approach, which has one huge drawback: When communicating all structure must be flattened, because pipes understand only the stream of bytes notion. Then at the receiving end the structure must be recovered from the byte-stream. For long-running programs like a WM it also means that you need to act non-blockingly and handle errors sensibly. An interesting alternative is that you don't make your program a program in the first place. After all what is a program? It's a library together with a `main` procedure. Just leave out `main`. Now it's just a library and configuring amounts to the user writing an actual program that uses the library. That way there is no need to go through the flatten-and-recover process from above. You can stay within the language's type system. That's what most Haskell programs do, including xmonad. So I'm probably already doing what you're suggesting, except that it doesn't look like Unix, because there is no pipe involved. =) signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Data type declarations are not free in any case, I think. Compared to what? Algebraic abstractions usually compile to exactly the code you would have written if you had not used abstraction. Well, you started talking like you were considering some limitation of XMonad hard to work around. I am. I'm using a tree-shaped set of workspaces, but I need to encode this tree within the names of the workspaces, which is incredibly awkward. So I'm probably already doing what you're suggesting, except that it doesn't look like Unix, because there is no pipe involved. =) Well, with sockets it is easier to do decoupling, and cross-language development. Decoupling seems unrelated, but the cross-language point is valid of course. Also sockets can work over a network. So yeah, it's a tradeoff. For my window manager I really don't want to go through that complexity. It's written in Haskell, I'd like to configure in Haskell and I'm not planning to offload my WM state to another machine. So I'm glad I can just use xmonad as a library. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Data type declarations are not free in any case, I think. Compared to what? Algebraic abstractions usually compile to exactly the code you would have written if you had not used abstraction. Data type declarations have to be written. Store-everything-in-hash- tables is slower but quicker to write as long as I can keep the entire program structure in my head. Also I am trying to learn to write reasonable Haskell code for a small data-crunching script without just writing Fortran into Haskell; well, many of its features are minor annoyances in that mode even if they are useful for giving structure to a large program. It's seldom that you have to write your own data types, if you don't want to. Basic types, functions, products and coproducts can express anything you want that isn't a tightly packed array of machine words. But if you really want to dump everything into a table-like data structure, you can use Data.Map or Data.IntMap for everything. Together with GHC's OverloadedLists extension this should make your code just as short as the usual everything-is-a-hash-table in most scripting languages. However, you may want to write type signatures anyway. It doesn't increase your development time considerably. Well, you started talking like you were considering some limitation of XMonad hard to work around. I am. I'm using a tree-shaped set of workspaces, but I need to encode this tree within the names of the workspaces, which is incredibly awkward. Well, I think you should be able just to write alternative UI for workspace selection simply ignoring the old one, no? Unfortunately not. Everything in xmonad goes through the core data structure StackSet. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
It's seldom that you have to write your own data types, if you don't want to. Basic types, functions, products and coproducts can express anything you want that isn't a tightly packed array of machine words. But if you really want to dump everything into a table-like data structure, you can use Data.Map or Data.IntMap for everything. Together with GHC's OverloadedLists extension this should make your code just as short as the usual everything-is-a-hash-table in most scripting languages. Actually, I use a ton of tuples-as-records in my crunching code in Common Lisp or Julia. Some of the shell tricks based on expansions are portable to Lisp, not worth it in Julia and definitely too costly in Haskell (learning Template Haskell is definitely outside my plans). I don't really know TH either. Occasionally I use TH actions defined in a library (for example to derive safecopy instances or, less commonly, to auto-generate lenses). But TH somehow feels wrong and ugly. However, you may want to write type signatures anyway. It doesn't increase your development time considerably. I also need to write matching to extract data from deep structures, no? I'm not sure what you mean by that. Perhaps you can drop me an off-list mail, so we can talk about your specific application. I'm using a tree-shaped set of workspaces, but I need to encode this tree within the names of the workspaces, which is incredibly awkward. Well, I think you should be able just to write alternative UI for workspace selection simply ignoring the old one, no? Unfortunately not. Everything in xmonad goes through the core data structure StackSet. Why is it a problem? For hierarchically structured workspaces you just tell XMonad core to select the low-level workspace. The trouble is that xmonad's idea of set of workspaces is encoded as a list zipper. It does not allow me to encode the tree. It only ever holds *lists* of workspaces and does not abstract over this data structure. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] UI management (Was: Use Haskell for Shell Scripting)
Also, I sometimes tend to use closing unneeded windows as a way of keeping track of short-term TODO list. I think there doesn't have to be semantic difference between closing a window and dropping it into a virtual drawer. Your short-term to-dos would be elsewhere. Unfortunately it entails removing systemd from the top of the process tree, which is a lot of work for a single person, but I'm highly motivated. By the way, this is not my usual systemd hatred. There are technical reasons to remove it from the top. Will you share it already? I do have a bootable systemd-less system based on NixPkgs, and would probably contribute some service definitions to the thing you develop. I'm very close to a bootable system. Allow me to arrive there first, then we can compare and exchange ideas and code. I will definitely let you know, because I'm interested in your way of doing it. Publishing a working prototype works better for me. I'm not sure why. This is not about a certain UX concept, but just about a window manager being sufficiently general that I could start to experiment. Well, because what you said in that paragraph is close to what I tried to do at some time, and there are hidden costs. When they become revealed (and they are different for different people, of course), you will take the things you have by this time and change the direction of development according to new data. Oh, that might happen of course. [...] In the same way I always thought that our filesystem concept is wrong. There is no inherent reason to have files in folders. I believe that a disk should act more like a giant mutable variable. At some point I saw RelFS and tried to use it for tagging files. Then I hit its limitations and written my own QueryFS. Now I use it for many things, but not for file tagging because hierarchical structure made it simpler for me just to think in terms of hierarchy. So directory structure was a good idea for file storage, in my opinion. Sure, it is great for some things, but terrible for others. The point is that we don't get a choice. Everything is designed around the files in directories notion. That's why an sqlite database is completely opaque from the point of view of the operating system. You need special tools with a special UI to inspect databases. My vision for software in general is what I call organic software. [...] How can we write software that isn't moronware? Good design and proper responses to exceptional circumstances can make our programs faster, more responsive, more flexible and less like untrained animals. They make our programs *dependable* and in the end *trustworthy*, because we can feed tasks to our program and focus on other things or even just pick up a cup of coffee, because we *know* that it will do its best to complete as much as possible without our intervention. Well, this vision is what I consider one of the problems of modern software development. It would be yet another tangent, though; do you want to go in that direction? In short, I want (but don't yet know how to achieve in some cases) predictable transparent tools usable as IA in Engelbart speak, not DWIM helpers with wannabe-AI. I want tools that can be easily transparently wrapped to try to do as much as possible or to stop at the first problem. I find this topic very interesting. I believe it needs to be discussed, but this mailing list is probably not the right place to do it. For lack of a better place, feel free to write me off-list, if you would like to discuss this further. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Some of the shell tricks based on expansions are portable to Lisp, not worth it in Julia and definitely too costly in Haskell (learning Template Haskell is definitely outside my plans). I don't really know TH either. Occasionally I use TH actions defined in a library (for example to derive safecopy instances or, less commonly, to auto-generate lenses). But TH somehow feels wrong and ugly. In both Julia and Common Lisp I use macros for many tasks and they make life much more comfortable. Of course, Haskell type system may make it hasrder to use macros. Oh, Common Lisp (CL) macros don't correspond to TH, but rather to regular functions in Haskell. We have first class actions together with lazy evaluation. What is code is data is code in CL is actions are first class values in Haskell. You only need TH when you need to generate something that is not first class, for example class instances. However, you may want to write type signatures anyway. It doesn't increase your development time considerably. I also need to write matching to extract data from deep structures, no? I'm not sure what you mean by that. Perhaps you can drop me an off-list mail, so we can talk about your specific application. Well, it looks like field names are not scoped and if I use plain ADT I have to write pattern matching to extract data from a member of a member of a structure. You can chain both functions and lenses. Extraction via functions: (innerField . outerField) value Extraction via lenses: value ^. outerField . innerField Update via lenses: (outerField . innerField .~ 15) value You can even get completely imperative by using a state monad: outerField . innerField .= 15 outerField . otherInnerField .= blah The trouble is that xmonad's idea of set of workspaces is encoded as a list zipper. It does not allow me to encode the tree. It only ever holds *lists* of workspaces and does not abstract over this data structure. So? Keep your own structure and keep pointers to XMonad-stored entries. Yes, I do that, arguably not in the most elegant way possible, because the predefined workspace switchers and layout algorithms need to understand my concept as well. If xmonad would simply abstract over StackSet, it would be very easy and transparent to do. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] UI management (Was: Use Haskell for Shell Scripting)
Actually, my idea of the core boot sequence is contrary to your goals:
for core system I think in terms I want to easily run recovery after
a USB boot and I want to describe my system in terms of its imperative
recovery procedure started from initramfs. It is likely that my
service management will likely also be like that; but I knew you were
doing something for staring a NixPkgs-like-structured service
repository and so decided to get by with a minimum set of daemons
until I can choose to just write service definitions for your approach
or start my own set of definitions.
You don't have to wait. If your services are defined in a functional
manner, you won't have much trouble translating them to my concept.
That means that a service is entirely self-contained and receives its
full configuration from its arguments. It then returns an attribute set
that specifies at the very least the filesystem directories it needs to
access and a non-daemon program that represents the service:
{config, pkgs}: {
description = nginx web server: + f config;
init = builtins.toFile nginx-init.sh ...;
mounts = [/var/www];
}
You should assume that the init program runs as PID 1 and that it will
be killed from outside by the usual SIGTERM-wait-SIGKILL sequence.
SIGTERM is only sent to the init program. If it does not quit fast
enough, the entire process group is killed by SIGKILL. Also you should
assume that the init program can only access the Nix store and the
directories explicitly mentioned in `mounts`. It should assume that the
root directory is a read-only RAM-disk and that `/proc`, `/sys` and
`/dev` are already mounted appropriately.
Hm, want to try out QueryFS?
It looks very interesting. Currently I'd have little use for it, but if
it's available online, I will certainly try it out when a use case pops
up.
OK. Well, are you OK with me inviting some other people from the
beginning? Not sure who of them will join.
Sure, go ahead.
signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Oh, Common Lisp (CL) macros don't correspond to TH, but rather to regular functions in Haskell. We have first class actions together with lazy evaluation. What is code is data is code in CL is actions are first class values in Haskell. You only need TH when you need to generate something that is not first class, for example class instances. And many actions are first class values in CL. But there are many things that are not first class even in Haskell. Macros can manipulate lexical environments in many convenient ways, for example. Also, while you could make a function out of a complex iteration macro, it is guaranteed to be simpler to use. Remember that we're talking about very different paradigms. I can totally see how you would want and love macros in CL. In CL you encode your solution in such a way that using macros is the natural thing to do, and then lexical manipulation is probably something that comes in handy. In Haskell your encoding is entirely different, so you wouldn't use macros in the first place. Instead you would use non-strict higher-order functions together with type classes. Ah, so Haskell has no usable records without lenses? The standard record system is very limited, even though some language extensions try to improve it (without much success). However, you would most likely want to use lenses anyway, even if standard records were richer. Modern van Laarhoven lenses are a powerful abstraction for accessing data structures, mainly because a lens does not necessarily correspond to a certain concrete field (it can correspond to many fields or to a logical alternate view of certain data) and also because both lenses and lens operators are first class values. Where `temps` is a list of temperatures in degrees Celsius, the following is that list with three degrees Fahrenheit added to each item: (traverse . fahrenheit +~ 3) temps The thing in parentheses is a function that takes any traversable data structure (e.g. lists or trees), interprets its values as temperatures, applies a Fahrenheit view and then adds 3. The result is the modified data structure (i.e. it's pure). The following is the same function, except that it only adds if the original temparature is greater than 10. All lower temperatures remain untouched: traverse . filtered ( 10) . fahrenheit +~ 3 Finally as said lenses and lens operators are first class, so you can take them as arguments, return them as results and put them into data structures. Let `fields` be a list of lenses. Then the following is a list of lenses pointing deeper into the individual data structures (into the `field` subfield of whatever each lens points to): (traverse %~ (. field)) fields That's why we love lenses. They are more than just a recovery from our lack of a decent record system. And they are defined in plain Haskell, just to show you how far we can go without macros. =) signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Also the Nix model allows us to compile all our scripts easily (just apply a function), which might hold some benefit in terms of startup and switch times. There is little reason to use interpreted scripts when you have a fast compiler. So would you say that this is preferable to the shell-monad idea? Yes. Even when compiling a Haskell DSL to a shell script you need to compile the DSL itself first into a program that can output the script. If you are going to compile anyway, just compile the source language to a runnable binary in the first place. One huge benefit of that is that you can use the libraries and abstractions of the source language. As a simple example by compiling to Bash you give up the option of using Haskell's concurrency, exception handling, etc. And those two are particularly difficult to translate to Bash, or pretty much every other shell. I'd definitely say that NixOS is the most state-of-the-art GNU+Linux distribution in existence, [...] To be fair, there is also the GNU OS, which uses Guix, although the underlying ideas and build system (nix-daemon) are the same. [...] for people who rarely use computers, the potential difference is a lot lower, so UI changes should not be forced upon them.) I think it is important to define your target audience. If you are specifically targetting, as you call them, computer illiterates or people who rarely use computers, it is entirely reasonable to offer them an easy, slowly evolving UI. However, for the general public I would say that our current state of the art in UIs is so broken that the faster we move on the better for humanity. For example anyone who tells me that touch screens are intuitive does not understand the word *intuitive*. Hint: It does not mean pictures, nor does it mean pointing. The same holds for virtual UIs that we access through HIDs. For a concrete example just try to use style sheets *and* be productive at the same time in your office suite of choice (don't worry, they all suck). Try to handle hundreds of e-mails per day (*handle*, not skim!) in multiple mailboxes *and* use less than 50% of your work time for dealing with e-mail in any MUA (yes, I'm including mutt and Emacs). Now don't get me wrong. You do have a point there. If a UI was perfectly reasonable and has been replaced by something less reasonable, I can totally understand your dissatisfaction. That does happen from time to time when marketing people try to be smart. But if the sole reason that you are unhappy is because you have to learn something new, then you shouldn't suggest moving to purely functional languages for scripting. Remember that contributors are humans as well, many of them will face having to learn something new and nontrivial, and some of them will probably be just as angry as you are when your UI has been replaced without the option to go back. Again don't get me wrong. I suggest that we do it, as long as we think that we're moving along the right track. Innovation will almost by necessity cause some dissatisfaction. I'm really just saying that there is not much difference between innovation in programming languages (programmers are humans) and user experience (users are humans). For some of them innovation will be a problem in both cases. Either you choose to respect that, or you choose to move on regardless. It depends on your target audience. I'm also only interested in the pure-functional languages. I'm not interested in so-called ‘multiparadigm’ functional/imperative languages like Scheme as this does not provide the deterministic purity guarantees that pure-functional languages are based-on. I would love to use Haskell for everything, but it's simply not a realistic option. Currently GHC is practically our only compiler. It is huge, compiling takes time, etc. We can fix all of these things, but we haven't so far and GHC will not get smaller or faster any time soon. Let me clarify: GHC is not slow per se, if you consider that Haskell has an extremely high signal/noise ratio, not just in syntax, but especially in semantics. Two lines of Haskell can easily *mean a lot* (i.e. encode a program that does lots of things or very complicated things). If we measure by time spent per semantic content, then GHC is lightning-fast. However, that measure is only interesting for larger projects. For small scripts the measure time spent per line of code counts, and GHC does not perform too well there. The other static functional languages I have tried were either too noisy, too immature or did not provide a reasonable FFI. Things will improve and new options will pop up, but right now our choices are limited. True, but the idea is to minimise impure code. The documentation and tutorials of pure languages encourage the practice of factoring-out any necessary impure code and writing most code to be pure. Can the same be said about multiparadigm languages? You may need to adjust your
Re: [Nix-dev] Use Haskell for Shell Scripting
To be fair, there is also the GNU OS, which uses Guix, although the underlying ideas and build system (nix-daemon) are the same. [...] As such, it's probably best for me to not talk about GNU OS, Guix, and Guile until I get the time/willpower to explain why my viewpoint against that project is not malicious and why NixOS is – perhaps counterintuitively to some libre software supporters – more beneficial to the software freedom movement. Do yourself a favour by not getting into that kind of argument. They see benefit in their way of doing things, and I would strongly support their project. Both GNU OS and NixOS are heading into a much better direction than pretty much all other distributions in my personal view, and I would happily encourage them to carry on. At some point in time we will benefit from each other, and in fact I see a huge benefit even for NixOS in the sole existence of GNU OS, if only to have some GNU people following the functional distribution approach. If nothing else the fact that two projects happily follow the same idea may well indicate to some people that we're on the right track. We as in the GNU OS and NixOS people. In short: Better stop arguing against them, regardless of what your reasons are. The only thing you can gain by that is a week of heated flamewars with no conclusion other than that they will think that NixOS people are jerks. I'd prefer us to work together instead of against each other. Nevertheless, I still think that human interface choice is important, and XMonad is an example of the original point that I was trying to get at. It is a classic user interface, a tiling window manager, using a pure-functional language. Just because something is written using modern techniques doesn't mean to say that the user interface itself is ‘modern’. I use the same combination (xmonad + taffybar) and could complain all day long about it. Of course this is constructive criticism, and I'm an idealist to some extent, but still there is a lot of room for improvement. For example xmonad is too rigid. It has a fixed data structure that corresponds directly to what is displayed on each screen. Workspace names are strings, which is hugely inconvenient. Something like tag-based volatile workspaces are not possible, although with some really ugly (and slow) hacks you could do it. In the ideal graphical environment I wouldn't see any reason to close or move a single window -- ever. Xmonad gets closer, but is still far away. ‘multiparadigm’ is ambiguous as to what multiple paradigms that refers to, but I clarified that by saying “functional/imperative”. I did mean that Haskell supports imperative programming just as well as it does support declarative programming. Some people say that it's the best imperative language out there. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Maybe you need to script it properly? Maybe StumpWM is less rigid from the beginning, though (I use it and extended it for my needs, so I can probably answer your questions if you wonder about it; it is in Common Lisp, by default all the splits are manual, and there are many hooks to perform actiosn on events like window addition) Thanks! However, I'd better not switch to a WM that I wouldn't enjoy configuring. Also I think that I've bent xmonad pretty much to its limits and hit the abstraction wall. Nowadays my configuration is a whole cabalised package that I install via Nix. If you're interested, it's online [1]. The abstraction wall I hit is StackSet. It is highly sequential (workspaces are lists of windows, there is a mapping from screens to workspaces, etc.) and offers absolutely no choice to organise windows differently. What I mean by too rigid is having f :: Blah X - M () instead of f :: (SomeAppropriateClass f) = f X - M () thereby enforcing `Blah` instead of an `f` of my choice. [1]: http://hub.darcs.net/ertes-m/config/browse/myxmonad signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Can we please have text-only on this ML?
My issue with text-only is that I don't know how to turn off hard-wrapping in my client. By ‘hard-wrapping’ I mean: That's always something that bothered me about e-mail (among other things). There are only two formats that are widely supported, and none of them get it right. The text/plain format is too much 1980's terminal or perhaps 1900's typewriter to be useful as a standard electronic format, especially given our nowadays rich diversity in potential viewing devices. Certainly text/plain hurts on my small phone. There is an extension to turn wrapping off globally, but obviously that's just desparation rather than a solution. Also most MUAs interpret this as: Make the paragraphs of this message use the entire horizontal space. That works for phones, but typographically it's a bad idea for a MUA displayed on a large 16:9 display, unless you make the window smaller. On the other hand text/html is too much. It comes with huge implementation costs, even when you use ready-made libraries for parsing and rendering. It comes with an unnecessarily high potential for abuse. If you think about it, in e-mail it is really an extremely reduced variant of HTML to get rich text formatting and independence of the source text layout. These are features some of us (including me) would like, but without the complexity of HTML. So what we really need is something inbetween. I would request that: * it can easily be typed by hand even in very poor editors, but especially well in advanced editors like Emacs or Vim, * that it can be read without rendering, so that the barrier to writing a new MUA does not become much worse (it's horrible enough). * WYSIWYG editors for it are reasonably easy to write and would render to readable source text, * it allows us to be completely independent of source text layout except in parts where we specifically want it, * it is a standard format that all agree to interpret the same way. You may be thinking: Except for the last point that's just Markdown!. Indeed, a Markdown standard is currently in development as far as I'm informed. And I'm happy to adopt it as the standard format for email. In fact I wouldn't be surprised if some people already configured their MUAs to pass text/plain parts through Pandoc or something similar. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
In short: Better stop arguing against them, [...] I'm not; [...] Alright. =) In the ideal graphical environment I wouldn't see any reason to close or move a single window -- ever. Xmonad gets closer, but is still far away. What?!! That sounds very abstract and cool! Maybe esoteric. Can you elaborate? It's not that esoteric. Think about the average 2013 laptop or PC with plenty of RAM. When you're done with a certain task, you close its window, simply because you're used to that and perhaps because you draw a relationship to the physical world, where you prefer your desk to be clean and organised. The truth is that a virtual desktop does not really (have to) correspond to a real one. There is no technical reason to close the window other than the limits imposed by your amount of RAM and swap. In fact you shouldn't close windows, unless you're absolutely sure that you will never need them again. Most of the windows you close you just find yourself reopening again later, going through all the procedures from firing up a program to navigating to a file to scrolling to the part you want to view or edit. It's not *much* work, but it's repetetive and unnecessary. Think of the average Windows user, who is used to the fact that there is only one desktop. When they need to suspend a task to temporarily work on another one, they start to close or minimise windows, only to have to find and recall all of them later. This takes a considerable amount of time, probably wasting a good ten minutes before you can get back up to speed again. Why? Because it's the most unnatural thing you can do. When you need to interrupt your current work in the physical world, do you drop everything to the floor or stuff everything back into the drawer? Perhaps you do the first few times, but then you get a second desk or a larger one. That's probably the number one reason why Windows users generally have at least two screens and would prefer to have more. So what's the solution? Simple: Workspaces must be cheap, dynamic and extremely easy to manage. There should not be a rigid mapping between workspaces and windows. Windows should easily be able to belong to multiple workspaces. A generalised xmonad could do it, but the current one can't. I'd be happy to discuss this further, but I'm afraid we're getting way out of topic. =) signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Thanks! However, I'd better not switch to a WM that I wouldn't enjoy configuring. Also I think that I've bent xmonad pretty much to its limits and hit the abstraction wall. Nowadays my configuration is a whole cabalised package that I install via Nix. If you're interested, it's online [1]. If you hit an abstraction wall, maybe you would actually enjoy doing what you originally wanted with access to lower-level abstraction. For xmonad StackSet is the lowest level. Using something different amounts to writing a new window manager, a generalisation of xmonad. Well, StumpWM allows arbitrary splits, very useful for Lazarus, Gimp, etc. Oh, this is not about splits. Xmonad allows that, too, via layouts. This is about the core data structure holding the current window configuration (the associations between windows and workspaces, and between workspaces and screens). It is a core piece of the window manager and cannot be modified by an addon. Haskell allows to abstract over this data structure easily, i.e. quantify over what is currently StackSet. The problem is: Xmonad does not do that. There is no inherent technical difficulty in doing it, it just doesn't do it. The equivalent in C++ would be to use templates instead of concrete functions, but there is no good translation for type classes. You really want type classes for this kind of thing. Maybe you should try Ratpoison and give it commands from a Haskell decision-making piece of code? Ratpoison also has arbitrary splits, it is simpler than StumpWM and can do way less. Driving StumpWM externally shoud also be easy, although with larger overhead. Generalising the xmonad source code would probably work better for me, because it allows me to stay in Haskell land and use types the way I'm used to. Also I don't believe that a WM written in C can have what I'm looking for at all (abstracting over the core data structure). It would have to be implemented by horrible pointer magic, and I don't want to touch that, even if it were there. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
If only we had a way to produce multiple outputs from a single package... :) We have. The `derivation` function supports producing multiple outputs, and I'm sure that `mkDerivation` forwards this ability. Greets, Ertugrul signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Yes, I’m planning to work on multiple outputs for ghc soon :) Great! I'm looking forward to that. =) signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
What about other languages as Python, Perl etc.? I know it is against our purity standards, but they are a far superior to Bash scripting. Well, that's the current state of the art. They are both used in Nixpkgs and some Nix-related tools. They are an improvement over Bash scripting, but I believe we might as well go all the way and use a functional language. Also the Nix model allows us to compile all our scripts easily (just apply a function), which might hold some benefit in terms of startup and switch times. There is little reason to use interpreted scripts when you have a fast compiler. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
Curry is indeed a beautiful language, and is essentially a conservative extension of Haskell. I am surprised that more Haskell folk have not adopted it. Well, it does lack quite a few things right now, including on the language level. For example its way to deal with ad-hoc polymorphism is very ugly, because it has no type classes yet: (==) :: a - a - Bool Constraint programming can be done reasonably in Haskell, it just won't be as pretty and implicit as in Curry. So I can live with Haskell's lack of built-in constraint programming, but I can't live with Curry's lack of type classes. In other words, there is still a lot of room for improvement, and it caught my attention so much that I will definitely check from time to time for Curry news. The perfect programming language in my view would combine the best of Agda, Curry, Haskell, Idris and this crazy data flow language whose name I forgot (it has negative and reciprocal types representing lack of information). PAKCS compiles Curry to Prolog (typically SICStus), which drags in the Prolog system. To get a binary executable, a better choice would be MCC (compiles Curry to native code) or KiCS2 (compiles Curry to Haskell, which can go into ghc): PAKCS was the only implementation I could find in Nixpkgs, and it was sufficient to get a feeling for the language. signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Use Haskell for Shell Scripting
At this current point in time, GHC is packaged in a poor manner, with GHC being unbelievably huge. Dynamic linking is the answer, which isn't done by default. I have actually experimented with using Haskell (and a few other FP languages) as a substitute for shells. It is feasible if you disable dynamic linking. The non-Haskell libraries are still linked dynamically, but the reference to the GHC derivation is then gone. This brings the closure of a Haskell hello-world script from a huge 1.1 GiB down to a mere 131 MiB (on my x86_64 system), which makes it on par with shell scripts. However, static linking is probably not a good idea. The resulting scripts are on the order of megabytes and can quickly approach a few tens of them. To really fix this and make Haskell viable as a shell substitute we need to split the GHC derivation. There should be a pure library derivation and a separate compiler derivation. The former should be as small as possible. Ideally there would be one derivation per library. The other languages I have tried are Scheme (via Chicken), Curry (via PAKCS), SML (via mlton) and Idris. Before I present my results, let me clarify what I think a script is: It is a string that I can run through a simple Nix function, which gives me a derivation that contains a runnable version of that string, either binary or shebanged. This derivation pulls a reasonably sized closure along with it. I can choose to combine many such runnable scripts to a single derivation using buildEnv, which is often very useful. In other words: For the language blah there is a simple, deterministic, unconfigurable function that would have the following signature in a hypothetical typed Nix: blahScript : String - Derivation This function can be a special case of a slightly more powerful function that takes a directory and a main entry point, because if we choose to use a better language, we might as well choose to utilise its module system, if it has one, for some of our larger scripts. Now to my results: All of the above languages, except Curry, work more or less, if all you need to do is to start programs or move files around. As soon as you need to do operating-system-specific stuff (e.g. `unshare` on Linux) it gets less juicy, because unless someone has written a nice high-level library you need to touch the FFI. Chicken Scheme worked best for that, because rather than trying to model the syscall in the language, you can just dump C code into it. Not a nice and clean solution, but a working one for the many cases when you just need to -- you know -- get stuff done. Haskell works, because lots of the OS bindings can be found on Hackage, including Linux-specific libraries. But it does require a slightly more expressive 'haskellScriptWith' function. You need to be able to tell it what you depend on. SML works and produces surprisingly small executables. It loses at the library end, because there aren't many OS-specific libraries around (or I couldn't find them). Also some of the advanced FFI tooling that I'm used to from Haskell seems to be missing. Finally I would say that the syntax is too verbose for quick scripting (but that's subjective -- I have seen people use VB.NET for scripting). You might be interested why Curry didn't work. Simple: I couldn't figure out how to write a program. Actually I went through the whole tutorial, did all the exercises (they aren't really difficult to a Haskell programmer) and then skimmed through the whole PAKCS manual. I could write extremely elegant algorithmic code and was quite amazed at the beauty of this language, even compared to Haskell. But in the end I still didn't know how to turn all this beautiful Curry code into an executable file that I can run without invoking PAKCS explicitly. Something with a shebang or ideally something binary. It would probably be possible to write wrapper scripts, but let's just wait until one of the implementations becomes mature enough for systems programming. Finally there is Idris. It is a beautiful language that comes with reasonable editor integration and a lightweight syntax. It compiles to executable binary code and has a carefully designed yet useful FFI. Sounds good for scripting. On the other hand it is very young and documentation is far from mature. Not that I would mind its youth, but I do mind the barrier to entry at this point. At the very least when other authors don't understand my code, it should be reasonably obvious where to look for answers. Also the library landscape is very flat, so bootstrapping might use most of your time, if you choose to use Idris for systems-level scripting at this point. The most viable options seem to be Chicken Scheme and Haskell. Both are well documented and have a usable FFI. Chicken produces much smaller executables, and programs are very memory-efficient. By design it compiles via C; because of that instead of providing a carefully designed FFI it
Re: [Nix-dev] Announcing support for GHCJS
Hi Charles, I'm happy to announce that support for the GHCJS compiler [...] has been merged into master [...]. This introduces a new top-level package set: haskellPackages_ghcjs. I was going to try it out and found the compiler itself, but couldn't figure out how to make a derivation for a JS package. Is there any documentation on that? If not, in which files should I look? Something like ghcjsWithPackages and cabal.mkJsDerivation would be great. And now for credit where credit is due: Ryan Trinkle and John Wiegly [...], Bas van Dijk [...] and Peter Simons [...]. Thanks guys! Thanks from me as well! Greets Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Announcing support for GHCJS
Hi everybody, I was going to try it out and found the compiler itself, but couldn't figure out how to make a derivation for a JS package. Is there any documentation on that? If not, in which files should I look? Something like ghcjsWithPackages and cabal.mkJsDerivation would be great. Nevermind, I think I got it. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Multiple MySQL instances (or nginx/apache/...)
Hi Luca, Let's not break the discussion in two places. It started on the github issue and can be kept there. one advantage of breaking it up is that I can take note of it and add that I'm currently working on a prototype of my earlier algebraic services proposal, which among other things allows you to run as many MySQL and other service instances as you like. =) But don't hold your breath. It could take a week or two before I can present my first prototype. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Multiple MySQL instances (or nginx/apache/...)
Hi Marc, But don't hold your breath. It could take a week or two before I can present my first prototype. Maybe you can sketch your idea for the sake of discussion? Allow me to send you to the thread in question: http://lists.science.uu.nl/pipermail/nix-dev/2014-December/015362.html The original discussion was about replacing systemd, but you can just scroll down to my proposal. I really should have started a thread of its own for that. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] less: When assumptions ruin the world
Hi Charles, Perhaps suggesting shell aliases would be sufficient for those that may want the pager by default? Yes, that's a good solution. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] less: When assumptions ruin the world
Hi Michael, When I explicitly launch man, I am going to read the top of the man page and some place in the middle, so less is useful. Absolutely, `man` is supposed to be an interactive program, and it usually displays formatted text that wouldn't fit on one page anyway. But honestly I wouldn't even mind if `man` just dumped everything on stdout. Since we have a Unix-like system I can easily set my shell up to pipe its output through PAGER by default. When I launch some nix build-related tool I expect to monitor current progress, and read the last 50 lines if something fails (something like tail -f, but for a stream). Your terminal emulator does exactly that. People, especially text-based UX programmers, often forget that there is a terminal that displays everything, and that the terminal is a non-trivial piece of software that offers everything you need to view long output, etc. You only really need an additional program when you need to save the output to a file, but nix-build writes build logs by itself anyway. But the PAGER really matters for man, so I keep it set to 'less -R' anyway. And then a command like the following uses it as well: nix-env --list-generations I'm mostly only interested in the last few lines, so just printing everything to stdout would be the correct solution for me. The original poster probably has other motivations and workflow patterns, but that's my experience. I think I have similar workflow patterns, but my point is that Nix 1.8 makes assumptions on the workflow patterns, and that's horrible. It helps ruining the world of UX, slowly, piece by piece. Systemd makes the same assumptions. That's why invoking most systemd commands is a horrible experience. Either I write `--no-pager` with every command, or I have to live with the fact that they abuse PAGER and my output disappears when I press `q`, when all they should really do is to print to stdout. That's what stdout is for after all! Assumptions ruin the world. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] less: When assumptions ruin the world
Hi Vladimír, If you set PAGER=cat or PAGER= you get pretty much the old behavior. Also, if the output isn't a terminal, less isn't used, e.g. ``nix-env -q | cat`` or ``nix-env -q | grep foo`` works the old way for me. I'm aware of that, and it's a terrible solution. I want to have PAGER=less for other programs, I just don't want Nix to use the pager. `nix-env -q` using the pager has already been trouble for me, but now even things like `--list-generations` use it. I see no harm in having the option. It's not an option. It's builtin and enforced behaviour since Nix 1.8. And it annoys me to such an extent that I'm almost willing to fork Nix and remove this option, or to write a command-line front-end to Nix, because Nix is no longer a command line program. It has become an interactive program. Think about it: Do you want `ls` or `find` to force-use the pager? Do you want to opt out by setting PAGER to something that isn't a pager? Or by piping every invocation of them through `cat`? If you really want interactive behaviour, there should be an interactive frontend. Most importantly, people can choose their preferred way of displaying long output with one variable common to many programs, which IMO rather helps those with special needs (blind; or e.g. PAGER=tail if you always want the end, etc.). No, they can't choose. I would choose to use `less` as a pager when I need it, and otherwise I would use my terminal. I can't make that choice without corrupting the PAGER variable (which I use for other things!). Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] less: When assumptions ruin the world
Hi Paul, No, they can't choose. I would choose to use `less` as a pager when I need it, and otherwise I would use my terminal. I can't make that choice without corrupting the PAGER variable (which I use for other things!). I'm not a fan of using less for nix tools. But it seems you're inconsistent here. Which other programs use $PAGER for legitimate reasons? And what reasons are good for that? I'm asking not to insult you, but rather to make my own future decisions better. Personally I don't use PAGER much anymore, because most of my use cases have vanished by replacing applications. For example my mail client is Emacs now, so I don't need an external pager any longer. I just use Emacs itself to view, for example, text attachments. Of course new use cases could pop up any time. For example I could choose to pipe the output of `ledger reg` through PAGER by default. The reasonable use case of PAGER is the scenario where you have an interactive program that either does not have a built-in pager or gives you the choice to use an external one, similar to how the EDITOR and BROWSER variables are used. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] less: When assumptions ruin the world
Hi Michael, I know; I was answering the question of why it is a problem to choose a single PAGER value. Sorry, I'm still getting used to my new mail setup. =) nix-env --list-generations I use ls instead of this command. That's actually a good idea, thanks. Well, systemd is intentionally breaking my workflow in ways that cannot be worked around with trivial shell scripts, and I don't care what pager journalctl when it takes multiple minutes to do the job that tail -f does in tens of milliseconds. So I let's not compare oversight with malice.. Sure, systemd is broken in too many ways to enumerate. Nix has just taken a small step into the wrong direction. Nothing will explode because of that, but we should be aware of that and step back. Of course all of this is my personal opinion. If the majority of the community disagrees, then I will have to live with it. I have opened this thread because I think that many people will actually agree. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] less: When assumptions ruin the world
Hi Bjørn, Hm, I've never thought of automatic less as a misfeature before. I didn't have anything to do with its inclusion in nix, but I remember when I started using git that I was pleasantly surprised by it's use of less, which made git more fun to work with. (If I was a nix developer I might have added automatic less myself.) I guess that you're not the only one who likes a more interactive user experience, which is probably why Nix got this feature in the first place. But it assumes a workflow where interactivity is achieved through the program in question. In my case it is achieved by my shell and my terminal emulator and that fits the way I use Nix. In particular I don't want Nix to enforce the invoke-pager-then-quit pattern on me, even if I do use a pager, because that's not the way I work. Just to give you a data point: I've always been pissed off by Git's auto-invocation of the pager. But that's only one and a rather small addition to my long list of reasons why I hate Git. So what's the best way forward? Using $NIX_PAGER first, as suggested by Michael Jones earlier in this thread? That should cover your use case, right? (_And_ preserve the current behaviour for people liking it interactive? That sounds like a reasonable solution to me, as long as it's strictly opt-in. In other words, you have to set NIX_PAGER to pipe through a pager by default, otherwise it just prints to stdout. The sole fact of using a pager on the wrong level (program-level instead of shell-level) will make NIX_PAGER completely useless for me, even when I use a pager. There is good reason why I insist that piping should always be done by the shell, not by the programs you invoke. However, I really think that a better solution is to use your shell properly. There is absolutely no reason for Nix to do the piping, and there is plenty of reason not to do it. Remember that programs should only do one thing and do it well? Shells are for piping, and they do it well. Nix is for building and managing derivations and environments, and it does it well. There is a very good reason for this principle. If a program does more than what it's intended to do, then it hurts composability. That will piss people like me off who really love the Unix philosophy, because it fits well with my algebraic way of thinking and therefore with the assumptions I make about my system. Bonus: Making programs less interactive is always a win, because you can always make an interactive frontend to a noninteractive program. But you cannot make a noninteractive frontend to an interactive one. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] less: When assumptions ruin the world
Hi Domen, My 2c: if Nix offers a way to disable the pager then I'd rather see 1% of users for such needs to make the bash alias instead of the other way around. why do you insist on an interactive-by-default Nix? There should be an interactive frontend, something like an interactive nox. [...] 1% [...] Also let's not throw around unfounded numbers here. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] less: When assumptions ruin the world
Hi Eelco, There is a very good reason for this principle. If a program does more than what it's intended to do, then it hurts composability. There shouldn't be an issue with composability here, because Nix will only run the pager when stdout is a terminal. So things work fine if you pipe Nix into another command. Let me give you an example where this assumption fails: Listing the current generations from your shell profile. In fact something very similar happened to me, which motivated me to start this thread: The change broke my assumption that Nix can be used safely from a shell script. Nothing bad happened, but one day was wasted, because a script stopped for `less` without my knowledge. I really believe that projects should start as non-interactive script- and command-line-friendly programs /by default/. I'd go as far as to call this a good design principle. Frontends can always be made. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] less: When assumptions ruin the world
Hi Eelco, If it helps at all, git seems to use GIT_PAGER first and then falls back to PAGER and then to `less` for the commands that use it. Perhaps nix could respect a NIX_PAGER env var? I've implemented this now, with the same semantics as GIT_PAGER and SYSTEMD_PAGER. Thanks a lot! What about issue #2? Would you agree to switch from `man` to short context-sensitive help-messages? If yes, I can refresh my rusty C++ skills and implement it. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] less: When assumptions ruin the world
Hi everybody, since Nix 1.8 much of the output of nix commands is piped through `less`, which is certainly meant to be a convenience, but that convenience depends on a few assumptions: * the user uses a crappy terminal emulator, * the user is interested in all of the output, * the user reads using their eyes. The first assumption is wrong for me and probably for a lot of other people as well. The second assumption is wrong for me most of the time; in most cases only the last few lines are interesting. The third one is wrong for blind people. If they can't afford an external braille terminal (not those single-line pads, which are expensive enough, but a real terminal), they will rely on text-to-speech. If not all three assumptions hold, then this feature makes Nix consistently `less` convenient to use. Not only do I have to press `q` now, but `less` is an ncurses program, which means that the output disappears after quitting. This does not happen when I just use my terminal emulator to scroll, and yes, most of them if not all allow scrolling with the keyboard (commonly Shift+PageUp and Shift+PageDown). Furthermore we hold in our hands an operating system that follows the Unix philosophy, which is about composability. There is simply no need to precompose everything with `less`. As humans we are smart beings, and we have a powerful shell, so that every user can decide for themselves when to use a pager. Also remember that we may have disabled users. A blind person will dislike this new anti-feature even more than I do, because `less` is not exactly a text-to-speech-friendly program. Feature request 1: Please do your part in saving the software world and remove this anti-feature. Don't even consider making it optional. It's useless. Remove it. In the future, whenever you think that some UX-related automation will make things easier, think again. You are most likely wrong. Just look at the huge damage that Microsoft has done by assuming that users are complete idiots. Feature request 2: Also I strongly believe that `--help` should absolutely never open a man-page. When we need a manual, we will use the `man` command, but in most cases we just need a quick reminder of what that one option was called that we're failing to recall, so I would much prefer the original purpose of the `--help` option: Print a short summary to the terminal. If this appears superficial to you, please remember that many little bad things can do a lot of damage in the long run. The purpose of this post is to wake everybody up before it's too late. I hope that some people agree with me, and if yes, I will turn these into actual feature requests on Github. I'm also happy to help with implementing the latter one, because writing a good `--help` system for commands with subcommands is not necessarily easy. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Service composition
Hi Ludovic, As a first approximation there would be a Service monoid, which would combine certain typical attributes about a service, including startup scripts, required bind-mounts, required system resources and if really necessary shutdown scripts (most programs should shutdown properly when the container goes down). Service : Mon Its identity idS would be the empty service, which you can think of as an empty group of processes with no mounts, no resources, etc. The Bitlbee service would be constructed through a regular function that takes the typical bitlbee options. Nothing special here: bitlbee : BitlbeeConf - Service OK. I suppose ‘Service’ would specify required user accounts, PAM services, etc., right? Yes. It specifies everything to build a machine for the purpose of carrying out the task that the service denotes, but in such a way that two such machines can be combined into a single machine. I still need to figure out a few details though, especially since I'm relying on a lot of Linux technology here. I'd like to keep it sufficiently general that users of other kernels (most notably FreeBSD) can benefit from it as well. The nginx function is more interesting. It involves a second monoid, the monoid of web services (identity idW): WebService : Mon Then nginx is a monoid morphism, nginx : WebService - Service that is a function with additional structure, in particular: nginx x ◇ nginx y ≡ nginx (x ◇ y) nginx idW ≡ idS Very interesting. I wonder what other types of services would benefit from similar treatment. It would split the view on the configuration into two parts: services and environment. For example networking belongs to the environment, so requirements on the network as well as networking options will be part of a service. This means that creating a bridge device amounts not to adding a service, but to applying a service morphism: withBridge brCfg (br: service1 br service2 br) This reminds me that it would be great if Nix would allow some way to define custom infix operators or at least provide a few generic ones. At this point we will need to use prefix notation, which might make this a bit uglier: withBridge brCfg (br: compose (service1 br) (service2 br)) Although it's just monoid composition, so using a list argument would do the trick. Just a nice-to-have. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
Hi Alexander, We can use a monoid system to construct configurations, but the socket activation standard for example, is centered on optimizing the activation script itself. What are your thoughts on the activation script? I have no concrete thoughts at this point, just throwing around ideas. Socket activation, for those daemons that support it, should be something sufficiently transparent from the point of view of the configuration author. I can easily see that using systemd might be overkill and way too complex for a container-based system, so I think there is something to research here. The trouble with systemd for this particular purpose is that it wants everything to have an identity, so basically you need to map a compositional system to a system of global variables. In many ways you go through the same trouble when compiling a functional language to an imperative architecture. I also think upgrading services doesn't really seem to work in systemd or in the current setup. Similarly to how, when using a distributed docker setup, we have a load-balancer that atomically switches instances, we should not need to take down the old instance service before the new one is created. Rather, the upgraded service should be started in isolation (using containers), and after ensuring that it has started, is working etc, then should the switch happen using namespaces, routing entries etc. This should be the preferred way to deal with non-transactional services (i.e. non-database stuff). That's pretty much the way I want upgrading to work. The idea that the old service must be stopped before the new one is started is based on what I think is a conflation of concerns, namely treating the whole service state as global state. Yes, that's exactly what I want to get rid of. Ironically that's something systemd does very well. Systemd gets the stuff right that we don't want in the first place. =) Instead a lot of services can be treated as a sequence of isolated containers, and a small set of load-balanced, mutable service entry points. Namespace magic can make a lot of previously global state, local, such as ports, pid-files, log files etc. For these kinds of services, you don't need graceful shutdown, which is a great simplification, similarly to what docker typically provides. I feel that a great design in this area should blend well with distributing services across machines, failover, etc. which are concerns that systemd doesn't start to cover. That's a very good summary of why I want to implement my proposal. Let me see if I can get a working prototype. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd
Hi Ludovic, FWIW, Guix uses an approach along these lines: the ‘bitlbee-service’ function, for instance, returns a ‘service’ object as a monadic value (see http://www.gnu.org/software/guix/manual/guix.html#Defining-Services for details.) The way I understand the section on the store monad is that it's really a state monad with a view on the current store (they should really show the underlying types to make this easier to understand). That's not related to the way they define services, but rather just how the store itself is used. Nix has the store built into the language. Side effects happen to build store paths and, once built, they appear like regular immutable values within the language. It's convenient, but makes the assumption that build outputs depend solely on their inputs, which unfortunately, as we know, is not always true at this point. However, the way they define services seems indeed a lot better than what we currently have. In part they already have what I think we should have (functional services), but with no algebraic structure behind it other than the store monad. Unlike in NixOS, the service implementation doesn’t have access to the rest of the system configuration, which I think is a good thing, as you note. Absolutely. Everything they depend on should be an argument. What’s unsatisfying (and thus subject to change) is that ‘operating-system’ objects (which are pure declarations) end up with monadic values in their ‘services’ field. That makes it inconvenient to, say, filter items from that list, or to tweak their configuration, because one first needs to bind them. That said, it’s probably not a problem for Nix, because every Nix function is really a function in what we call the “store monad”. You could think of it that way, if you wanted. I prefer to think of what Nix does as the same magic that functional languages do with other resources like memory (alloc and GC). In your example, what would ‘bitlbee’, ‘nginx’ etc. return exactly? An attribute set describing the service? As a first approximation there would be a Service monoid, which would combine certain typical attributes about a service, including startup scripts, required bind-mounts, required system resources and if really necessary shutdown scripts (most programs should shutdown properly when the container goes down). Service : Mon Its identity idS would be the empty service, which you can think of as an empty group of processes with no mounts, no resources, etc. The Bitlbee service would be constructed through a regular function that takes the typical bitlbee options. Nothing special here: bitlbee : BitlbeeConf - Service The nginx function is more interesting. It involves a second monoid, the monoid of web services (identity idW): WebService : Mon Then nginx is a monoid morphism, nginx : WebService - Service that is a function with additional structure, in particular: nginx x ◇ nginx y ≡ nginx (x ◇ y) nginx idW ≡ idS The most important part is to get the equivalence `≡` relation right, i.e. when do we regard two services as equivalent? This requires a lot of careful thought, because services in the real world are messy and effectful. Example: 1. One nginx instance serves two websites x and y. 2. Load balancer scenario: Three nginx instances. One serves x, one serves y, the other redirects to the correct instance depending on the request. 3. Mapping identities: A hundred load-balanced nginx instances, one of them serving x, a second one serving y and all others serving nothing at all (returning 404 unconditionally). These three examples really describe the same service. One may be a more efficient implementation, but they should be equivalent with respect to `≡`, because they are indistinguishable from the outside other than in exceptional cases (case 3 is more likely to run out of memory than case 1). That enables us to use equational reasoning for correctness. It also enables us to apply automatic semantics-preserving optimisations. Example: The host may be serving two sites. At the same time a system monitoring daemon might generate health reports periodically that are also served via HTTP. Such a configuration might look like: nginxStatic site1.host /var/www/site1 nginxStatic site2.host /var/www/site2 systemHealth health.host A semantics-preserving optimistation (which is really not an optimistation, but the way web services are combined) would collapse all of that to a single instance of nginx and one entry in the crontab for the health reporter. I hope this gives a first impression of how everything would fit together. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
Hi Colin, This sounds a lot like Disnix! How would it be similar? How would it be different? Disnix and NixOps are machine-oriented. You have a network of machines with configurations. The goal is a service-oriented architecture. Every service itself is a machine, and those machines can be combined to larger machines. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
Hi Nicolas, I think as a first step to get rid of systemd and gain a much more sensible services model as well we should move away from NixOS modules for services. The module system is made in such a way that one can build on top of the work of the other without having any computer scientist knowledge. I hope you realise that my proposal makes everything *simpler* both for module/service writers and configuration authors. Services will be simpler and less tedious to write (simple functions). They will compose naturally. The algebraic part is *design* and perhaps terminology. It enables *those who want* to exploit the additional structure especially for equational reasoning and containerisation, but understanding abstract algebra is in no way a requirement for the regular NixOS user. Let me respond to your objection anyway: If we can have a configuration DSL that gives us a major benefit, including much simpler modules, provable safety, full automation of tedious tasks, etc., then I'm sorry for the I don't want to learn something new cavemen, but I'm more than happy to leave them far behind. Exploring and using new ideas require acquiring new knowledge from time to time. NixOS itself is a great example of this. One thing I'd like to see *by default* in NixOS is containerisation, even for the regular desktop user who only uses Firefox and LibreOffice, in the spirit of Qubes OS. It's technically possible, supported by Linux by and does not require setup. This is the motivating purpose for my proposal. Let me give you a very compelling example: services = withBridge bridgeConfig (br: container (nginx ... openssh ...) container (openvpn ... br ...)) openssh ... This simple configuration would define a whole network of containerised applications. This is also the reason why I am pushing to abolish functions as option definitions. Most of the time if you want to use a function, you can express it as a set of input (option declaration) which are being processed to provide a lower level of abstraction given by another module (option definitions). Perhaps NixOS with its *functional* package management and *functional* configuration language is just the wrong distribution for you. ;) /joke Jokes aside, when a function makes sense, I go for it, because functions make things simpler. Please stop pushing that way. For those who aren't programmers, functions are just syntax anyway. The module system does not prevent us from doing anything, it provides a new way of formatting which makes things declarative and extensible. I won't go as far as to argue that declarative is a meaningless term. Also I'm not sure what you mean by formatting (syntax?). The extensibility of the current module system is limited in the same way global variables limit the extensibility of a program. Everything is global, and every module can affect every other module in completely unrestricted and unpredictable ways. Any kind of data feedback requires something you would classify as computer scientist knowledge (least-defined fixed points of nonstrict recursive functions). Again, this has bitten me in the past. There is no way to reason about your configuration other than to look into the resulting derivation. This is a bad situation. The more modules and options we get, the more difficult it will get to keep everything together. If a module does not serve what you expect, you can always write another module and use it instead of the previous one. Having multiple instances of a service is an issue with the module, not a limitation in the module system. I guess we have very different philosophical views, so it is difficult to communicate the advantage of my proposal to you without actually implementing a prototype. I admit that to a non-FP programmer the concept I propose must look very similar to what we have today except that you can add the same service multiple times. The advantage is only apparent when you start ot use the functional and algebraic features. Otherwise all that changes is that the syntax becomes simpler (still a small benefit though). Being able to reason about the resulting system is very important. Have you tried nixos-option ? This question suggests that you are unfamiliar with equational reasoning. To answer your question: Yes, I'm aware of nixos-option, and it's orthogonal to the issue I'm raising. Also I should point out explicitly that I'm not proposing to replace the module system. I'm proposing not to use the module system for services. If my proposal works as intended and finds acceptance we will quickly find other things that would benefit from this, for example the network configuration. I would prefer and do propose an algebraic solution (view services as a toolbox from which you can pick the services you want and compose them together). A services configuration then might look like this:
Re: [Nix-dev] How to get rid of systemd (was: Modifying the initsystem (introducing S6 supervision suite))
Hi Michael, Well, with NixPkgs having multiple instances of a package installed requires zero support from the package expression itself. I do think that services becoming more like packages would be a good thing, but currently I have no idea how to do this except by just giving up the module system. I don't think we need to or even should give up the module system. Some settings are global (like the hostname and most boot options), and the module system makes those a lot easier to manage. This proposal is only for services and service-like entities. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
Hi there Tobias, One thing most of us seem to agree about [...] Maybe that's true (I don't see a consensus on the list, only the usual sparse FUD. I don't follow IRC) but arguments that start this way always give me the willies. I understand why, but I'm not very good at English rhetoric. Please don't let that overshadow the rest of the mail. [...] is that conceptually NixOS does not depend as much on something like systemd as most other distributions do. Something like systemd? Nix doesn't magically make *nix less *nix, for better or worse. The meat of systemd is still service management, which NixOS needs as much as any distribution. NixOS needs service *management*, but it doesn't necessarily need a service *manager*. In traditional distributions services are managed by destructive update (enable/disable). Normally we don't have that. Switching to it was a huge step back from the ideals of NixOS, because it represents all the traditional views on what a system should look like: a giant pile of interconnected mutable variables. That's what a running system is, though. Immutability *is* just for file systems. You plant a pretty tree, and hope it describes a sane system that will do the right thing most of the time. Then, something will happen that will kick all your pretty algebraic assumptions off the table very quickly. Most of your OS is in buggy C, whether you like it or not... The assumptions are not that the programs will do what I expect them to do, but rather that two instances of nginx each with one virtual host is equivalent one instance of nginx with two virtual hosts. This does not assume anything about the correctness of nginx itself, but it does help building systems, containers and networks componentwise. You seem to have some very nice ideas about describing services (or something better), if still very rough around the edges. But if you can't implement your ideas on top of systemd, so much for init-agnosticism. And probably conceptual purity, too. This may be a misunderstanding. Init-agnosticism means that the complexity of translating this concept to an actual set of services that systemd can work with is the responsibility of a single function/script/whatever. We achieve separation of concerns. Should we, at any point, decide to turn our back on systemd or even just provide the option to use another manager, then all we need to do is to reimplement this one function/script/whatever. Those who don't understand systemd are doomed to reinvent it. Poorly. :-) I do understand systemd very well and still don't like it. However, reinventing it is certainly not something I have planned for the near future. =) As a final bonus this is so difficult and ugly to solve with systemd that we would feel a much greater temptation to get rid of it. =) Because your plan is centred entirely around it not being systemd. Not at all. It would be a nice bonus, but I doubt that it's going to happen. The motivation for my proposal, as the motivation for most of the things I do right now, is easy, composable, transparent containerisation as well as a configuration DSL that is easy to reason about. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
Hi there Anderson,
I want to know if it's possible to use an alternate to Systemd as init
system on NixOS.
this has been asked many times now, and I'm in favour of switching to an
alternative. Unfortunately it's not easy, because a considerable
portion of NixOS currently depends on systemd, most notably the services
infrastructure and pretty much all container-related stuff. Note that
systemd is not just an init system.
One thing most of us seem to agree about is that conceptually NixOS does
not depend as much on something like systemd as most other distributions
do. In principle most of what systemd does could be handled by the
activation script and perhaps a custom daemon (or even just a set of
scripts) that allows us to manage the services that are part of the
current system derivation. NixOS can deal with things like cgroups or
ACPI events in a more sensible, repeatable and transparent way.
I would go as far as to say that systemd is the number one thing that
cripples us in many ways. Switching to it was a huge step back from the
ideals of NixOS, because it represents all the traditional views on what
a system should look like: a giant pile of interconnected mutable
variables. Remember that immutability is not only about the filesystem.
How to get rid of systemd
=
I think as a first step to get rid of systemd and gain a much more
sensible services model as well we should move away from NixOS modules
for services. At this point we are trapped in the traditional view that
services preexist and we get to enable/disable them. Our system is
defined by a large set of switches and global settings. This needlessly
robs us of a lot of flexibility, for example to run multiple instances
of the same daemon or to give us a choice to use our already running
preferred webserver when installing a web application as a service.
It also corrupts our ability to reason about the resulting system,
because enabling a certain service might trigger settings in some other
part of the system configuration to change. This has actually bitten me
in the past. Being able to reason about the resulting system is very
important.
I would prefer and do propose an algebraic solution (view services as a
toolbox from which you can pick the services you want and compose them
together). A services configuration then might look like this:
services =
bitlbee { port = 1; stateDir = /var/lib/bitlbee1; }
bitlbee { port = 10001; stateDir = /var/lib/bitlbee2; }
nginx { httpConfig = ...; }
postfix { postMasterAlias = blah; }
The big difference is that now services are first class values and form
a monoid under composition, which brings a number of advantages:
* It allows us to run multiple instances of a daemon with slightly
different configurations, as those are really just similar but
otherwise unrelated services.
* Services can take services as arguments, which would become the
primary way to construct a dependency graph. Think of a web
application depending on a webserver or the other way around.
* We could develop a fully automated way for services in different
containers to communicate with each other.
* Now people like me can actually use equational reasoning to
understand and make predictions about the resulting system. I can
use the fact that the `` operator is associative, that certain
services are idempotent with respect to `` and also that certain
functions are morphisms in a rich category. For example if the
nginx function is a monoid morphism, then I know that regardless of
how I compose multiple webservers, the result will be a sensible
combination of either separate daemons or multiple configurations.
This would be extremely valuable.
The last point is very important, because it basically tells that a
service is not directly related to what programs will run on the
resulting system. Even if you compose ten nginx services, there will be
only one nginx *program* with a suitable configuration file to represent
this composition. It also means that when a service itself comes with
nginx, then adding another nginx does not actually mean that two
instances will be running. If we go as far as to containerise services
by default, we can make predictions about the container's structure.
As a final bonus this is so difficult and ugly to solve with systemd
that we would feel a much greater temptation to get rid of it. =)
Let me know what you think. If this idea finds acceptance, I'm happy to
develop a formal model and a prototype.
Greets,
Ertugrul
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Proposal: Standard installation procedure
On Mon, 20 Oct 2014 09:50:15 +0200
Eelco Dolstra eelco.dols...@logicblox.com wrote:
The basic idea is this: To build a system, you use a standard
function from Nixpkgs and pass it a configuration module. The
result is a derivation that represents a complete self-contained
system.
How does that differ from how NixOS is currently implemented? You can
do:
$ nix-build 'nixpkgs/nixos' -A system
which gives you a complete system in ./result, where
e.g. ./result/init is the stage 2 init script that runs the activation
script and invokes systemd.
How would I refer to multiple systems with different configurations from
within a Nix expression? The change would allow this:
{ system1 = import nixos/build-system.nix module1;
system2 = import nixos/build-system.nix module2;
system3 = import nixos/build-system.nix module3; }
Programs like nixos-rebuild, nixos-install, nixos-container and even
NixOps are really just wrappers around this (except that they add
various levels of provisioning).
They all make the assumption that there is exactly one current system
configuration.
In other words a container-based NixOS system is really just a Nix
store, nothing more. It doesn't need to contain any additional
files or directories and building it does not require us to
understand the complexity of nixos-install, because it will all be
moved to the system initialisation phase.
I don't think nixos-install is that complex. Almost all of the
initialisation it does in the target file system is to make nix-build
work in the chroot. The NixOS initialisation is done by the activation
script.
Check out nixos-container create - it does almost no initialisation
of the container file system, since everything is done by the
activation script during the first boot of the container.
That's already most of what I need. What remains is a way to build
multiple configurations and deploy them to different systems in a
functional way, in particular without environment variables.
Greets,
Ertugrul
--
Ertugrul Söylemez ert...@gmx.de
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Proposal: Standard installation procedure
Hi there fellow Nixers,
I'd like to propose an installation standard that would make it a lot
easier and more reliable to deploy NixOS to virtual and container-based
systems, including qemu, LXC and Docker. As a nice side effect it would
greatly simplify most NixOS tools, including nixos-install and
nixos-rebuild.
First of all: Yes, I'm aware of Nixops, and I don't want to use it for
various reasons. Currently I'm using my own variant of
make-system-tarball.nix (really just to avoid the expensive and in my
case unnecessary compression step) together with a bunch of Makefiles.
It's a mess and the sooner I can dispense with it, the better.
Proposal
The basic idea is this: To build a system, you use a standard function
from Nixpkgs and pass it a configuration module. The result is a
derivation that represents a complete self-contained system. The
initialisation program ($out/init) is not systemd directly, but a script
that first checks whether the filesystem has already been initialised.
If not, it sets up all necessary directories (like /etc) and then starts
systemd as normal.
In other words a container-based NixOS system is really just a Nix
store, nothing more. It doesn't need to contain any additional files or
directories and building it does not require us to understand the
complexity of nixos-install, because it will all be moved to the system
initialisation phase. Ideally we should factor out the initialisation
into a separate script $out/pre-init that accepts a prefix argument.
Then $out/init would simply look like this:
#! /bin/sh
/nix/store/abcxyz-nixos/pre-init --prefix=/
exec /nix/store/abc123-systemd/bin/systemd
If we can also move pre-boot initialisation (like installing the boot
loader) into the system itself (say, $out/pre-boot), then we could build
entire virtual machines completely without relying on a hypervisor and
the complexity of having to communicate with the installation process.
Just set up the disk image, install the system and use the resulting
$out/pre-boot script with appropriate override arguments:
out=`nix-build my-machine.nix`
# ... copy the closure ...
$out/pre-boot --override-grub-device=disk.img
This would greatly simplify the implementation of nixos-install (just
build the system and invoke the pre-boot script without overrides), but
more importantly it would provide a very simple and standardised
installation procedure for virtual machines and containers that we can
rely on.
How to use it
=
Create a file that defines the system derivation (build-system.nix
contains a currently hypothetical function that we would implement as
part of this proposal):
# my-machine.nix
let cfg = {config, pkgs, ...}: ...;
in import nixos/build-system.nix cfg
To build the system, we simply use nix-build:
% nix-build my-machine.nix
/nix/store/abcxyz-nixos
It is now very easy to create the filesystem for the container. Just
copy the full closure of the derivation and point the container driver
to the proper initialisation script (or link /sbin/init to it).
Liberated system environments
=
Another advantage of this proposal is that we can easily put those
builds in a local user environment and copy only the closure to the
containers (I use *non-remote* containers a lot). We can even update
them off-line. That prevents a lot of duplication, because the
containers can be garbage-collected after every change and will contain
only the bare minimum required for operation. The actual history is
saved on a separate management machine and uses sharing where
appropriate.
In fact we could switch multiple VMs on multiple hosts almost fully
transactionally. Build and upload everything, then tell every machine
to switch. Finally update the environment and collect garbage on the
target machines.
Updating the local system would follow the same scheme, so even
nixos-rebuild would become very simple.
Final remarks
=
The main point of this is that it becomes really easy to reliably
implement something like a custom nixos-install, a remote nixos-rebuild,
Nixops or even make-system-tarball.nix. No deep knowledge is required
and the instructions and relevant information can be put on a fairly
simple wiki page.
What do you think? I would love to provide a prototype, but I really
need help with this one.
Greets,
Ertugrul
--
Ertugrul Söylemez ert...@gmx.de
Key-ID: F9B5212A
Fingerprint: 8D89 7AC9 21CF F868 F074 9058 30CB D56F F9B5 212A
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Package archeology in nixpkgs
On Sat, 30 Aug 2014 23:31:42 -0400 Daniel Peebles pumpkin...@gmail.com wrote: I've had a sudden urge to do some Haskell archeology and that led me to a question about how we feel philosophically about keeping abandoned projects and old versions of live projects in nixpkgs. I think it's a very good idea, but ... How do people feel about this? Is it something I should maintain independently of nixpkgs or does it belong in the main repo? ... please keep it out of the main Nixpkgs, at least for now. Reason is that currently Nix performance drops linearly with the number of packages. Until we have a caching solution of some sort, I'd prefer to keep Nixpkgs slim. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Nixos basic Qs
Ben B bburde...@gmail.com wrote: - I have xmonad as my window manager, and it defaults to slim as display manager/login gadget. My problem is that one of those two opens a terminal by default. If you type 'exit' into that terminal it kiboshes all open programs and exits the window manager. I use the ~/.xsession file. It's (usually) a shell script, and you can just execute `xmonad` from there. It will override whatever login script slim uses: #! /usr/bin/env sh exec xmonad Note that the file must be executable and XMonad must be either installed in your profile or enabled in your configuration.nix. Greets, Ertgrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] how does NixOS ensure focus?
On Mon, 28 Jul 2014 14:08:38 + hasufell hasuf...@gentoo.org wrote: Whenever you install NixOS, you're really forking it, and your installation becomes your own distribution. You can then choose to merge some or all changes in the official NixOS channel into yours (you might call this upgrading). That also works the other way around. Contributing to NixOS really means that you make a change to your own distribution and send a patch or pull-request. So unlike other distributions we do not actually have contributors or maintainers in the traditional sense. As far as I can tell, this model is unique. We have a similar concept in gentoo where you can overwrite ebuilds (package definitions) by adding 3rd party repositories called overlays. So you can in fact mix gentoo with funtoo with sabayoon with pentoo, set priorities of those overlays and even force package foo from a specific overlay. However, this usually gives a lot of complicated problems where packages rely on certain behavior or even patches of other packages. Maybe NixOS doesn't have these problems due to it's technical concept, but I find it hard to tell at this stage. This can happen in the form of state (configuration) interactions. For example it's probably a bad idea to run two versions of the same program, when they share a state directory. To prevent this some programs use/allow version-specific state directories. For example I remember seeing directories like ~/.kde3 and ~/.kde4. However versions aren't the only things that affect this. The set of available plugins may, too. What NixOS can do for you here is to ensure that there are no unintended stateless interactions. This includes libraries, programs, stuff you would normally find in /usr/share, etc. In other words, there are no package-level problems. In addition, most user overlays have very poor quality (pretty much what arch linux has with AUR) and regularly break non-trivial libraries like SDL for example. I see that as a big problem of decentralized packaging: quality assurance and consistency (consistency in a very low-level sense, as in: packages work with each other). From my experience it doesn't work to randomly review other peoples work and encourage them to more QA etc. There has to be a very strong attraction for people to have their work merged at some point. This has to be done on the community level. For example everything that eventually makes it into Nixpkgs is reviewed by the committers. I don't think we have any formal standards here, but that may well establish at some point. Of course that doesn't mean that third-party forks don't screw things up. The current NixOS state of the art is that Nixpkgs is a safe choice and for regular operation you don't need extra sources. Forks are usually used to update or add new packages, and then they are merged. What does happen primarily outside of the upstream Nixpkgs is experiments and custom site-specific customisations. This is the normal and accepted way to do it. [...] After half a century we have mostly figured out how to develop software productively, using decentralised methods, forks, branches, merges, etc. Compared to this almost all distributions still use stone-age methodologies. NixOS is a step into the right direction, not only regarding packaging, development and deployment, but also community structure. This sounds great, in a way. However, worst case scenario is that there are so many forks and people working on so different ideas that none of these can compete with up2date distributions and that getting a consistent experience involves a lot of research, manual fiddling and following unofficial tutorials and hacks. I don't want to paint things too black, I'm just saying that it really depends on details and what actually happens, probably even a culture thing. This is a theoretical possibility, but our current workflow and community mindset makes such a scenario unlikely. I'm pretty confident that it will stay this way for the foreseeable future. But I guess I'd have to get involved in order to actually know. You could install NixOS and pick a program that you would like to see in Nixpkgs. Then just add it to your local installation and send a patch/PR to Nixpkgs, just to get a feeling of how it works, both technically and socially. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] how does NixOS ensure focus?
this model successfully, although reasons for this are probably a bit more complicated. Gentoo here is almost the complete opposite with 200 core developers and it is rapidly losing focus and consistency every year. Decision making is a cumbersome process, there are too many diverging ideas, although we have a council to settle global issues (doesn't work very well). That in turn leads to less people caring about global issues and new ideas or about fixing non-trivial fundamental problems. There is also no enforced internal review workflow. People want to get things done and avoid unnecessary reviews. That is dangerous and has led to serious problems in the past. At the same time, a lot of our contribution channels have failed and are too complicated. Because the number of random contributors is decreasing, we add more core developers to the team, causing even more internal problems. I almost think that a centralized distribution model is doomed to fail. That does not only include the workflow with review platforms, DVCS tools etc., but also the political and organizational structure. I don't see much information about these points on your website (only technical stuff). What are your solutions to the mentioned issue, if any? How do you ensure focus and collaboration at the same time? ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Fw: Re: @nixpkg
[forgot reply-all] Hi there Steven, Steven Shaw ste...@steshaw.org wrote: I only knew about nixos_org. I didn't know about the tips a/c. Thing is, this is about the package system. Not the Linux distribution. Not about tips. Does that make sense? Someone already replied to one of my tweets saying that they hadn't understood that you could use Nix as a package manager separate from NixOS. Where there is 1, there are many. I rest your honour. I'm a bit torn here. One one hand I agree with you that there should be more resources for Nix itself as a development and deployment system. On the other hand I agree with Oliver that at our current community size fragmentation could become a serious problem. My two cents is that at this point it's probably better to post this information via @nixostips. However, you could use your new account for retweets from the more well known sources, whenever you find something relevant to non-NixOS users. That should make the information flow sufficiently hierarchial to avoid fragmentation or at least make it apparent to followers. In either case the name @nixpkg is very unfortunate and especially misleading to existing NixOS users (because of an actual project called Nixpkgs). A name like @nixtips would be more appropriate, should you decide to keep maintaining it. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de Key-ID: F9B5212A Fingerprint: 8D89 7AC9 21CF F868 F074 9058 30CB D56F F9B5 212A pgpGIcrt9i9qD.pgp Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Hi, I'm new!
Hi there Matthias, On Sun, 29 Jun 2014 11:01:26 +0200 Matthias Beyer m...@beyermatthias.de wrote: I don't know if this is wanted on this list (isn't there a users-list somewhere?), but I want to introduce myself before sending my bunch of questions! I'm a 22 year old CS student from south germany. I heard from nixos the first time at UnFUCK[0], a conference at my university, I organized myself. ATM I'm an Archlinux user (been for ~7 years), but within the next year I want to build a Tower-PC and I'm heavily thinking about nixos for it. This is currently our main mailing list. Sadly introducing yourself to a web community is a custom that seems to have gone into oblivion, so I'm glad that you're remining us that it still exists. Also south Germany (where I live, too) may benefit from more NixOS users. =) Is it appropriate to send my questions to this ML? I guess I will send them in seperate mails if you ACK this, as some of them may lead to longer discussuions (who knows). Go ahead. As Cillian noted there is also the #nixos channel, if you prefer a less rigid form of communication. Greets Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
On Mon, 16 Jun 2014 10:19:59 -0400 Eelco Dolstra eelco.dols...@logicblox.com wrote: Well, let's not try to decide the undecidable. Would it be so bad to have explicit dependencies? One of Nix's main selling points is that you *don't* need to specify runtime dependencies explicitly. That has worked great for the last ten years, so I'm rather hesitant to change this. Of course, you can always make runtime dependencies explicit in the way you describe, but it's important to be aware that packages that require this are broken (e.g. they won't work with Nix's --repair flag and multiple output support, because these use hash rewriting). We will probably have to find a solution sooner or later. Perhaps what we really need is the ability to give each package its own view of the Nix store. They should always find other packages where they would expect in their own virtualised environment. The Linux kernel has the necessary support to implement this securely. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
On Fri, 13 Jun 2014 18:54:53 +0200 Eelco Dolstra eelco.dols...@logicblox.com wrote: The path-rewriting proposal is a very bad idea and will cause a lot of breakage. For many/enough applications rewriting will not work at all, because they might encode paths in data structures or be using a non-UTF8 multi-byte encoding. Packages that store paths in UTF-16 don't work with Nix anyway, independent from hash rewriting, because Nix finds runtime dependencies by scanning for plain ASCII hashes. To my knowledge we've never encountered such a package in Nixpkgs. This paper has a small evaluation of hash rewriting: http://nixos.org/~eelco/pubs/secsharing-ase2005-final.pdf (section 6.1) We certainly have. As said, Haskell works that way: while hardcoded paths are commonly stored in UTF-8, serialized text is more likely in UTF-16, because that's the internal format of the Text type. Also continuous path strings are very uncommon, because we use platform-independent combinators to construct paths. Nix' dependency scanning still works most of the time, because paths to other Haskell packages are stored at least in a GHC registration file as continuous strings. It could fail for other requisites like data files from non-Haskell packages. For example the CA certificates are stored in their own derivation. If not half of the system would depend on that derivation, it would most likely be GCed away and Haskell programs would be unable to find them. Rewriting is even worse, because now you are interpreting a structurally rich file as a mere byte-string and then changing things based on vague assumptions. This is dangerous, in particular for ELF files, which are in fact Turing-complete (ELF itself, not the code it contains). Just as a theoretical point consider that the question whether your rewriting algorithm is correct is undecidable in general. More practically you would break half of Hackage. The breakage would range from being as gentle as a Haskell program failing to find certificate files to as severe as a Haskell plugin-based web application (like clckwrks) fatal-error-crashing. As a Haskell developer I must warn you about rewriting. This /will/ cause trouble at least for Haskell. Even the current dependency scanning mechanism is a ticking time bomb for Haskell users, though it did not yet explode for me. Or even better, ask the admin to use pam_namespace: http://www.linux-pam.org/Linux-PAM-html/sag-pam_namespace.html And hopefully, one day users won't need to be root to do bind mounts. That would be very useful, yeah. It's probably already possible anyway given the right capability, but I don't know. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
On Sat, 14 Jun 2014 09:36:21 -0400 Shea Levy s...@shealevy.com wrote: Please point to a specific example of a package with an unretained dependency due to UTF-16 storage I can't. I'm just pointing out a potential problem here. If you disagree, I'll be happy to construct a proof of concept. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
On Sat, 14 Jun 2014 13:43:24 -0400 Shea Levy s...@shealevy.com wrote: I can't. I'm just pointing out a potential problem here. If you disagree, I'll be happy to construct a proof of concept. Of course it could happen, but we have a huge number of packages in nixpkgs, including a significant subset of hackage, and it hasn't actually happened yet as far as anyone knows. No one claims hash scanning/hash rewriting is 100% sound. It is, however, extremely convenient and has worked remarkably well up until now. If we start finding real world use cases where it fails, then we can start thinking about alternatives, but until then why is this a big deal? It's probably not a big deal. On the other hand consider that it only happens in a rather special scenario and may go unnoticed for quite some time. Consider an application (like a game) with a bunch of separate sets of data files, each optional and referring to each other, compressed to save space. This is an actual design that I'm following in one of my projects. I'll know to make it Nix-friendly, but non-NixOS developers may fail to do so. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
On Sat, 14 Jun 2014 22:09:21 +0400
Michael Raskin 7c6f4...@mail.ru wrote:
It's probably not a big deal. On the other hand consider that it only
happens in a rather special scenario and may go unnoticed for quite
some time.
Consider an application (like a game) with a bunch of separate sets of
data files, each optional and referring to each other, compressed to
save space. This is an actual design that I'm following in one of my
projects. I'll know to make it Nix-friendly, but non-NixOS developers
may fail to do so.
I think that any proposal would be seriously considered. Any already
existing application creating such a problem would increase your chances
to make someone else propose a solution… Otherwise — I think the problem
can surface, but I have no idea ho to fix it sanely… I think many other
people think the same.
Well, let's not try to decide the undecidable. Would it be so bad to
have explicit dependencies? That seems to be the only sound solution.
A straightforward way is to add a small dotfile via the builder script
with paths to the output. The only problem at this point is that there
is pretty much no support for doing that, so it's error-prone. I could
totally see something like this happening:
stdenv.mkDerivation {
...
requisites = [cacert dataFiles1 dataFiles2];
...
}
All it does is to register the given derivations as requisites
unconditionally, perhaps simply by writing the aforementioned dotfile
at $out/.nixdeps.
Greets,
Ertugrul
--
Ertugrul Söylemez ert...@gmx.de
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
Vladimír Čunát vcu...@gmail.com wrote: For some previous discussion you can see https://github.com/NixOS/nix/issues/16 So far there's IMHO been no really good solution proposal. And probably none is currently providing binaries to use. The path-rewriting proposal is a very bad idea and will cause a lot of breakage. For many/enough applications rewriting will not work at all, because they might encode paths in data structures or be using a non-UTF8 multi-byte encoding. For example rewriting will fail for pretty much all Haskell packages, because UTF-16 is a common encoding and path handling is not stringly typed in Haskell. In many cases there is an actual data structure for paths with separated path segments in memory. Many other strongly typed languages will cause the same trouble. In other words: Do not rewrite paths! If your proposal requires violently manipulating binaries, it is most likely a bad idea. One simple and safe way to do this would involve using private mounts with chrooting: Create a private bind-mount of / somewhere, then bind-mount the Nix store at /nix/store. Run the application within a chroot in that directory. To the application the Nix store will appear to be mounted at /nix/store. The drawback of this method is that it requires the administrator to allow one SetUID executable for the setup, or perhaps a bunch of entries in the fstab. After that no further support from the administrator is required. Another way would be to hook the path handling procedures of the libc. Rewrite /nix/store to whatever is in the NIX_STORE variable. This would be a lot of hard work, but would work much more reliably than rewriting paths in binaries, and it would work without help from root. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de Key-ID: F9B5212A Fingerprint: 8D89 7AC9 21CF F868 F074 9058 30CB D56F F9B5 212A pgp7GDP_cdOm9.pgp Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Fw: Re: Linux Action Show is calling
Oops, this should go to the list. Some day I'll learn to press C-R instead of C-r. =) Thor Arne Hopland thor...@gmail.com wrote: I made this post on the LAS Reddit page because I thought they needed a better introduction. http://www.reddit.com/r/LinuxActionShow/comments/27xoqd/nixos_a_challanger_appears/ I hope I represented NixOS correctly :/ I could nitpick (some details are wrong), but your presentation is fair. In particular I liked the way you presented the Nix suite as a fully fledged (continuous) deployment solution rather than a mere package manager. I don't want Nix to stand alongside apt, pacman and Portage. =) But anwyays, they've said they've tried to contact you many times. They want one of the NixOS developers to come on the show and talk about NixOS! This is a great marketing oppertunity, you guys. Tons of people watch Linux Action Show and it might lead to us getting more pullers, committers and packagers (we need moar). Well, this is controversial. Of course I want NixOS to gain popularity, because that would get us at least new Nixpkgs developers. On the other hand the lack of large-scale success has helped us preserve and maintain the soundness of the system. As SPJ once said (about Haskell): Avoid success at all costs!. Yet I can't deny that I would like to see an LAS episode featuring NixOS. I'm not a core developer, but I contribute stuff from time to time (patches, snippets and support on Freenode), and I use both NixOS and the Nix suite to most of their potential. I can both praise and complain about them a lot. =) I'll offer my soul, if nobody else has time, on the following conditions: * +1 from Eelco, * more +1s than -1s from the Nixpkgs committers (note: 0 ≯ 0). Whoami: I'm ertes on Github and mm_freak on Freenode. Some of you may know me as @ertesx, too. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de Key-ID: F9B5212A Fingerprint: 8D89 7AC9 21CF F868 F074 9058 30CB D56F F9B5 212A -- Ertugrul Söylemez ert...@gmx.de Key-ID: F9B5212A Fingerprint: 8D89 7AC9 21CF F868 F074 9058 30CB D56F F9B5 212A pgpYNEpfJHnlG.pgp Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Extra manpage per service?
Hi Bjørn, that sounds amazing, I would love that. I wouldn't know where to start either, though. Greets, Ertugrul On Mon, 9 Jun 2014 12:02:35 +0200 Bjørn Forsman bjorn.fors...@gmail.com wrote: I've been messing around with the nixos 'graphite' service lately. I thought the module was a bit thin on the documentation side, so I started to search online. I came across this graphite puppet module[1], and what struck me is that there is no way that our current man configuration.nix can provide that kind of detail for each service, without being completely unreadable. So, how about we add a manpage for each service.name? I think it would be a good idea to still have everything in man configuration.nix, although man nixos-service-TAB-TAB would probably be just as easy to navigate. Does that sound like a good idea to you? Is it easy to do? I don't know where to begin :-) -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
On Thu, 05 Jun 2014 23:39:34 +0200 Vladimír Čunát vcu...@gmail.com wrote: Hydra has and uses priorities. Anyway, building OpenSSL itself is very quick, but rebuilding all that (transitively) depends on it is worse. And there are CVE fixes for stdenv stuff sometimes (glibc)... Yes, and the basic idea is that you could have high priority packages like OpenSSL, OpenVPN and nginx. Whenever Hydra sees a job of higher priority it starts doing it (potentially aborting whatever it is currently doing). Once all jobs of the same priority are done, it runs the tests of the same priority and updates the channel. Then it goes to the next highest priority. That way security updates won't take longer than necessary. When we use priorities generously we could avoid a lot of delay even in less critical cases. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
On Thu, 5 Jun 2014 21:01:59 +0100 Shell Turner cam.t...@gmail.com wrote: So is the argument that it should be possible to update the channel with the new package definition before the binary cache has finished building, thus letting people rebuild their systems locally if need be? That seems reasonable. I think a nice solution would be to add build priorities to Hydra. When a security update is required quickly, then update the OpenSSL expression, assign a high build priority to OpenSSL and the common server packages and let Hydra do the building. Most people will build on weaker machines, so I think that waiting for Hydra is the way to go, even when you can't watch the actual build process and thus feel that nothing is happening. All we need is to make sure that Hydra builds those quickly enough. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] New website
Eelco Dolstra eelco.dols...@logicblox.com wrote: Actually, a major goal in the redesign was to *get rid* of all the non-NixOS stuff. It's the NixOS homepage, after all, not the Nixpkgs homepage. Confronting visitors with a lot of other projects is just confusing. While non-NixOS stuff belongs into the wiki, nixpkgs itself is not quite non-NixOS. I very much like the structure of the Arch Linux homepage with its small introduction (no large fonts), news as the primary content and recent activity in the package database. On the right hand side you find links to documentation and other stuff. NixOS differs from other distributions in that it is actually a collection of projects that work together. As such a projects menu should appear somewhere on the page and you should not have to unroll it first. It should include at least (here in alphabetical order): * Hydra, * Nixops, * NixOS, * nixpkgs. I agree that helper packages like PatchELF do not belong there. It would be nice to have a Packages link in the navbar though, like on archlinux.org, linking to a easily browseable list of available packages. Also, the front page could have a blurb about the number of packages available in NixOS, which could link to Nixpkgs. This should be easy enough by using nix-env's XML output together with a suitable XSLT stylesheet. Hydra could take care of maintaining it. * The front page is serving new visitors and active users, it should pick one. I considered removing all the news from the homepage, but 1) it seems useful to show activity on the homepage, just to convey that the project is alive; 2) visitors don't have to read it, it's easy to mentally filter it, so I felt it doesn't hurt much to have it there. News are fine. They are useful both to newcomers and to active users. Videolan.org actually has a pretty similar layout to nixos.org: a description + big download button, followed by news/blogs. I would compare VideoLAN less to a distribution and more to a monolithic software package similar to Firefox or LibreOffice. Also I wouldn't like people to mistake NixOS as just another distro. It is, among other things, a development and (continuous) deployment platform. The homepage should make this clear. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de Key-ID: F9B5212A Fingerprint: 8D89 7AC9 21CF F868 F074 9058 30CB D56F F9B5 212A pgpU5_7dQ32YB.pgp Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] New website
On Sat, 31 May 2014 17:29:12 -0700 M. P. Ashton d...@imap.cc wrote: I don't mind the new website at all. I think it looks nice, and it does seem more professional than the old one. If you need to look like some Enter Prise Service (typos intended) to deserve the professional label, I'm happy to look unprofessional. [...] I'm afraid I must not have written very clearly. I certainly didn't intend to imply that you were wrong about anything. I just thought I'd tell Dr Dolstra and the list about my own experience, since it was obviously very different to yours and the other fellow's. (And was also not wrong.) I'm sorry for the confusion. As I'm sure you and the other knowledgeable participants will have no trouble working out these weighty matters amongst yourselves, and I actually don't care that much about this anyway, I'll just step aside and let you all do that. No worries, I didn't mean to be offensive. This is not about my experience, but about my opinion, and I'm just stating that we do not have to follow the lead of large websites, particularly since the goal of our website is to convey useful information. It does not provide a web service in the traditional sense. The new website does have a few new features, but most of them look like stopgaps to me, like the commit history, which isn't really useful on the home page. I would prefer the subprojects themselves to take most of the space with a very short introduction at the top. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] New website
On Fri, 30 May 2014 13:27:00 -0700 M. P. Ashton d...@imap.cc wrote: I don't mind the new website at all. I think it looks nice, and it does seem more professional than the old one. If you need to look like some Enter Prise Service (typos intended) to deserve the professional label, I'm happy to look unprofessional. It also has a more promotional bent, which may or may not fit the project's goals -- I don't know if world domination is an aim for the project right now, but the new website seems to satisfy that goal much better than the old one. I can't speak for the whole community, but I would say the project's aim is to provide a decent deployment system. A growing community is certainly helpful, yet we have managed to get this far with a comparatively small one. In either case there is no need for an Enter Prise Web Site (typos intended) to be successful. I don't care for websites sending me to external servers, mostly because the load time tends to become much longer, and occasionally the external servers don't cooperate. But for me it's not such a big deal. I'm pretty sure Google and NSA and Facebook know more about me than I do, even if I studiously use DuckDuckGo (which is probably a trigger word). As Jay explained we shouldn't actively contribute to their power. Anyway, this problem has been fixed. Someone said that they couldn't find the manual -- I found it very quickly under HELP. I usually look for Documentation first, but Help seemed a reasonable second guess. At least two of us overlooked the top bar the first time, including myself, and neither of us is a novice in web-browsing. This is very strong evidence that there must be something wrong with the design. This bar is more important than everything else on the page, so it should be very visible. In fact I'd go as far as to say that a single-line bar isn't even sufficient. For example the NixOS projects should be visible all the time. They are our bread and butter. So, thanks for working on this -- I appreciate it, FWIW. Don't get me wrong. I do appreciate the work, but the process behind it was completely wrong. The website was switched before the community even had a chance to review it. I'd ask all committers to pay more attention to what the community says before applying patches. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] New website
Hello there, I'm looking at the new website with mixed feelings. Being less static is a good idea, so I appreciate the news, blog posts and commits sections. On the other hand it's way uglier and less lucid compared to the old website. These are minor design issues that we can talk about and fix. However, one issue with the new site I would rate as critical: As a good web developer NEVER EVER download anything from external servers unless it is necessary, especially not from entities like Google, Facebook or Twitter. If at all, do it server-side. The new website unnecessarily downloads jQuery from the Google servers, not only compromising our privacy, but also every NoScript or Ghostery user will be told: This website compromises your privacy!. And for what? For a dropdown menu? Come on! You don't even need JavaScript for that. CSS alone can handle it much nicer. I have managed to keep my browser from sending my browsing habits to Google for a long time now. Indeed, I don't even use Google as a search engine (there's DuckDuckGo). And today my very Linux distribution forces me to allow access to Google servers. That's not going to happen, so currently I'm unable to navigate the website at all. This is the top issue, so as kindly as my current anger allows, I'm asking you to fix this as soon as possible. I hope I'm not the only privacy-minded NixOS user. As SPJ once said, avoid success at all costs, because this is what happens when you don't. I'm not sure the old website really needed to be replaced, but since it was, please remove the badies and bring back the goodies. Also in this case please don't tell me to send a pull request. This is web development! What would take the original developer five minutes would take me hours. By the way, the Hydra frontend has the same issue. Greets, Ertugrul -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] New website
On Fri, 30 May 2014 17:28:28 +0200 Wout Mertens wout.mert...@gmail.com wrote: Shell: why don't you make a mockup or create a pull request: https://github.com/NixOS/nixos-homepage Ertugrul: Looks like your wish was granted: https://github.com/NixOS/nixos-homepage/commit/6803987bce4ff4bca1a5482ef25bfa99f4023538 Yeah, gladly my issue was easy to resolve. Shell's point is more fundamental and I could probably help in my spare time to approach a solution. I wonder what you think of using Hakyll to generate the website instead of the current XML-based toolkit. It would allow us to use lightweight markup languages like Markdown and generate CSS from something more modular like Clay. It does require some basic Haskell knowledge though. -- Ertugrul Söylemez ert...@gmx.de ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
