On Thu, Jul 24, 2014 at 12:52 PM, Vladimír Čunát <[email protected]> wrote:
> > On 07/23/2014 11:29 AM, Mathijs Kwik wrote: >>> >> If stuff causes regressions, revert it and move it to a feature-branch >> to debug. >> > > Perhaps. I'm not very clear about that. Typically I first try to fix the > regressions within days, if possible. > > Note that this revert+move workflow tends to hit a problem with unclear > semantics of reverts+merges. The catch is that in the case of *temporarily* > undoing some changes, people typically understand revert the other way than > practically all VCS (including git), which leads to surprises. IMO it's > also connected to being state-centric instead of (arguably more natural) > change-centric approach. > > Details: > https://raw.githubusercontent.com/tonyg/revctrl.org/master/output-raw/Rollback > > So let me describe a little of my recent activities. I wanted the firefox from https://github.com/NixOS/nixpkgs/commit/872860e6de8e460fb11a79c46b8092bdcff35da8, the quassel from ~/nixpkgs, and the rest of the system from the latest hydra channel; I ended up making nixpkgs checkouts for each of those, hard-coding their paths in import statements, and manually adding each package (and some of their dependencies) to packageOverrides. But even that wasn't enough, because I also needed to override the NixOS module for quassel; unfortunately it seems the list of imports is hard-coded and there is no moduleOverride function. So, I made yet another nixpkgs checkout, at exactly the revision of the hydra channel, with my module changes on top, and set NIX_PATH to point there. Note that, throughout this process, I was *creating* and *picking*, but I never needed to revert a commit. I just checked out older versions. I did end up trying out a single rollback, when I was looking at Quassel's NixOS module; however, it was completely useless, as the problem was in systemd's unit dependencies and thus *all* of my versions were broken. NixOS is inherently a state-centric system; it's not "this change broke my system" but "version V of thing X is broken". However, to set up my system the way I wanted it (i.e., so that all my programs worked), I did need 4 different nixpkgs checkouts. And I've only been using NixOS for a few weeks; as I set up more programs and services I'm probably going to need even more than that. Looking at https://github.com/MarcWeber/nixpkgs/branches/active it seems he has almost 40 branches. His workflow is slightly different from mine because he uses TopGit instead of letting Nix do most of the work, but the branch-explosion is similar. When we start using nixpkgs-monitor, the problem is going to get even worse: a steady, near-realtime stream of patches that bump versions, URL's, and hashes, with no QA or error checking at all. We already have this, except not automated, with scripts like hackage2nix. What branches should these updates go to? AFAIK there is no good answer. They can't go to master or staging, because they might break Hydra's builds. A new topic branch per patch means we get several thousand branches, i.e. the branch explosion again. There are 126 open pull requests, and probably hundreds more sitting in offline git repositories; as the project grows this will presumably get worse. The branch approach doesn't scale; if I do a lot of version-picking, all of these nixpkgs checkouts will end up in my configuration.nix! The checkouts are a waste of disk space and contain irrelevant information. Not only that, they're fragile; if I change the checkout, then the next time I do a nixos-rebuild the changes get picked up. It's basically a hack, hard-coding all of these checkouts into Nix. The problem is that, currently, Nix expressions are used for two purposes - building packages and specifying systems. Due to integration in the nixpkgs repository, the coupling between these has grown strong. But, as I hope my example has shown, these are orthogonal and independent activities. Sometimes I want to build packages, e.g. when I want to "build the latest version of Firefox from mozilla-central". Other times I want to specify my system, e.g., "use this known-good version of Firefox". If you look closer, you can see the difference: in the first case, I am referring to a dynamic, changing process. In the second case, I just need a store path. However, currently Nix is using the extensional model which intermingles these two. For me this is most obvious in fetchurl; our current fetchurl is basically a hard-coded store path, with a special exception of "fixed-output derivations" hacked into Nix. With an intensional store model, that goes away; a fetchurl is literally "fetchurl.sh <url>" and nothing more. If the URL breaks, then the store path changes and you can just keep using the old store path. If the content moves to a different URL, you can just change the URL and the content hash is still the same, with no fixed-output hack needed. And of course these advantages apply in other cases besides network resources, e.g. system time dependency, cosmic radiation, evil hackers/crackers, etc. On the other side, the release process and picking process becomes easier. Hydra fetches all of the store paths, and it's just a matter of examining the results and picking the builds that seem to work and pass tests. A release is basically a list of known-good store paths; there should be no dependency at all on the source NixExpr's, just on the derivations. The OpenSSL problem goes away, because now patching binaries to point to a different store path is a required operation (it's used for storing derivations in the first place!) rather than a quick sed hack. There's still the question of the actual command syntax for saying "build this tarball with these dependencies" and how/where to store the path-rewriting operation, but these are basically just bikeshedding. Regardless, it should be Hydra or NixOS and their respective configurations that pick "stdenv uses gcc 4.9", not nixpkgs. We could share the configuration and call it "nixreleases" or something. The other thing this allows is using e.g. Guix's GCC, which again currently requires a bunch of hacks. After this, my vision is that the nixpkgs repository will only contain very high-level instructions, for example "to install firefox: got to mozilla ftp directory; load a folder (branch); download tarball; install to $out }". This will mean that the majority of updates (stable or unstable) will be completely automatic and invisible from a nixpkgs perspective, which means that the repository explosion will go away. We can merge in projects like hackage2nix and others since they will not be lost in the update noise. The only development that will happen in nixpkgs will be adding new download sources and fixing build scripts. One master branch should be sufficient for that, modulo forks or global refactorings. Hydra/NixOS/Nix will do everything else - picking versions, running programs, calculating/storing hashes, etc. Anyways, I filed https://github.com/NixOS/nix/issues/296 a week ago, and received no response besides Dolstra tagging it as "feature" and "Nix 2.0". So my conclusion is that nobody actually cares enough for this to be implemented now, besides possibly me (I'm already doing a GSOC; two projects at once is probably not a good idea). But I could be wrong, please convince people to work on #296 if you want to remove the hacks and get these nice features. :-) -- Mathnerd314
_______________________________________________ nix-dev mailing list [email protected] http://lists.science.uu.nl/mailman/listinfo/nix-dev
