Re: [Nix-dev] Haskell Development and Deployment Strategies (was: Stackage Support Will Be Discontinued)
This is almost the same setup we’re using in Picus Security. The difference is
that we’re building our project with master and let hydra find sweet spots to
merge. We then update our projects submodule hash to the next stable hash.
Our major problem is not sticking to a stable setup but to intergrate our
project with new and updated packages.
Oh btw, Peter, I really appreciate your efforts. You put a hard work to make
our haskell ecosystem reliable. Not only writing automation tools but also
pushing upstream projects to fix their bugs.
—
aycan
> On 10 Jun 2016, at 18:17, Bas van Dijk wrote:
>
> Just as a point of reference, in my job at LumiGuide I'm in camp 4: "I want
> total control over my entire package set."
>
> * We run a high-profile production system consisting of multiple NixOS
> machines deployed using nixops to physical machines across the Netherlands
> and to machines on Hetzner. I also consider our NixOS development
> workstations to be part of the system. The configuration of all machines is
> defined by a single repository.
>
> * In that repository we set our nixpkgs to our own fork of nixpkgs which
> basically tracks the latest release (currently
> https://github.com/NixOS/nixpkgs-channels/tree/nixos-16.03) and adds a few of
> our own commits on top. We use a variation of Rok Garbas's "Reproducable
> development environments" trick [1].
>
> * Every 3 weeks or so I run a script which:
> 1. shows me the git log between our revision of nixpkgs and the latest
> upstream
> 2. set our revision of nixpkgs to the latest upstream with our own commits
> rebased on top of it
> 3. push that to a special "staging" branch.
>
> * We run our own hydra CI server which starts building the staging branch.
> When it builds successfully I merge staging into master and update all our
> machines (including our development workstations).
>
> * We also extend and override nixpkgs using its packageOverrides
> functionality. The following is the Haskell part of our override function:
>
> super:
> let self = super.pkgs;
> haskellOverrides = import ./haskell super;
> in rec {
> haskellPackages =
> super.haskellPackages.override haskellOverrides;
> haskell = super.haskell // {
> packages = super.haskell.packages // {
> ghcjs =
> super.haskell.packages.ghcjs.override
> haskellOverrides;
> };
> };
> profiledHaskellPackages =
> super.haskellPackages.override {
> overrides = self: super: {
> mkDerivation = args:
> super.mkDerivation (args // {
> enableLibraryProfiling = true;
> });
> } // haskellOverrides.overrides self super;
> };
> ...
> }
>
> * The Haskell overrides function defined in ./haskell/default.nix adds all
> our LumiGuide specific Haskell packages and overrides some existing packages
> to newer versions:
>
> pkgs :
> {
> overrides = self : super :
> let hsLibsFromDir = dir :
> pkgs.lib.genAttrs
> (builtins.attrNames (builtins.readDir dir))
> (name : self.callPackage
> (dir + "/${name}") {});
> inhsLibsFromDir ./.
> // hsLibsFromDir ../../hs-libs
> // hsLibsFromDir ../../hs-js-libs;
> }
>
> * Note that we aren't using the lts package set. I'm not sure if we should...
>
> I just want to provide this as a datapoint of one actual use of the Haskell
> infrastructure of nixpkgs.
>
> Regards,
>
> Bas
>
> [1] https://garbas.si/2015/reproducible-development-environments.html)
>
> On 10 June 2016 at 13:23, Peter Simons wrote:
> Fellow Haskell Hackers,
>
> Nix gives you great freedom to choose how to develop and deploy your
> Haskell software. That's good, but at the same time that flexibility can
> be confusing and feel like a burden. Therefore, I'd like to shed some
> light on this issue and discuss different types of strategies and their
> respective pros and cons. Which strategy is best for you depends on what
> you would like to achieve:
>
> 1. I want the latest version of every package as quickly as possible
> because I love it when build attempts result in hundreds of
> compiler errors that I can fix.
>
> 2. I want my Haskell packages to be up-to-date, but I also want my
> builds to be reliable. I don't mind fixing the occasional build
> error, but I mean, like, *occasionally*.
>
> 3. I want my Haskell package set to be rock solid. I don't want any
> updates unless there is a major security issue or a severe bug that
> threatens the integrity of my system.
>
> 4. I want total control over my entire package set.
>
> Now, let's go through the options those types of users have.
>
> If you favor features over stability as in (1), then you should develop
> with 'cabal-install', really. Nixpkgs might be useful for installing
> your favorite development tools like GHC, c
Re: [Nix-dev] Haskell Development and Deployment Strategies (was: Stackage Support Will Be Discontinued)
Just as a point of reference, in my job at LumiGuide I'm in camp 4: "I want
total control over my entire package set."
* We run a high-profile production system consisting of multiple NixOS
machines deployed using nixops to physical machines across the Netherlands
and to machines on Hetzner. I also consider our NixOS development
workstations to be part of the system. The configuration of all machines is
defined by a single repository.
* In that repository we set our nixpkgs to our own fork of nixpkgs which
basically tracks the latest release (currently
https://github.com/NixOS/nixpkgs-channels/tree/nixos-16.03) and adds a few
of our own commits on top. We use a variation of Rok Garbas's
"Reproducable development environments" trick [1].
* Every 3 weeks or so I run a script which:
1. shows me the git log between our revision of nixpkgs and the latest
upstream
2. set our revision of nixpkgs to the latest upstream with our own
commits rebased on top of it
3. push that to a special "staging" branch.
* We run our own hydra CI server which starts building the staging branch.
When it builds successfully I merge staging into master and update all our
machines (including our development workstations).
* We also extend and override nixpkgs using its packageOverrides
functionality. The following is the Haskell part of our override function:
super:
let self = super.pkgs;
haskellOverrides = import ./haskell super;
in rec {
haskellPackages =
super.haskellPackages.override haskellOverrides;
haskell = super.haskell // {
packages = super.haskell.packages // {
ghcjs =
super.haskell.packages.ghcjs.override
haskellOverrides;
};
};
profiledHaskellPackages =
super.haskellPackages.override {
overrides = self: super: {
mkDerivation = args:
super.mkDerivation (args // {
enableLibraryProfiling = true;
});
} // haskellOverrides.overrides self super;
};
...
}
* The Haskell overrides function defined in ./haskell/default.nix adds all
our LumiGuide specific Haskell packages and overrides some existing
packages to newer versions:
pkgs :
{
overrides = self : super :
let hsLibsFromDir = dir :
pkgs.lib.genAttrs
(builtins.attrNames (builtins.readDir dir))
(name : self.callPackage
(dir + "/${name}") {});
inhsLibsFromDir ./.
// hsLibsFromDir ../../hs-libs
// hsLibsFromDir ../../hs-js-libs;
}
* Note that we aren't using the lts package set. I'm not sure if we
should...
I just want to provide this as a datapoint of one actual use of the Haskell
infrastructure of nixpkgs.
Regards,
Bas
[1] https://garbas.si/2015/reproducible-development-environments.html)
On 10 June 2016 at 13:23, Peter Simons wrote:
> Fellow Haskell Hackers,
>
> Nix gives you great freedom to choose how to develop and deploy your
> Haskell software. That's good, but at the same time that flexibility can
> be confusing and feel like a burden. Therefore, I'd like to shed some
> light on this issue and discuss different types of strategies and their
> respective pros and cons. Which strategy is best for you depends on what
> you would like to achieve:
>
> 1. I want the latest version of every package as quickly as possible
> because I love it when build attempts result in hundreds of
> compiler errors that I can fix.
>
> 2. I want my Haskell packages to be up-to-date, but I also want my
> builds to be reliable. I don't mind fixing the occasional build
> error, but I mean, like, *occasionally*.
>
> 3. I want my Haskell package set to be rock solid. I don't want any
> updates unless there is a major security issue or a severe bug that
> threatens the integrity of my system.
>
> 4. I want total control over my entire package set.
>
> Now, let's go through the options those types of users have.
>
> If you favor features over stability as in (1), then you should develop
> with 'cabal-install', really. Nixpkgs might be useful for installing
> your favorite development tools like GHC, cabal, stack, alex, happy,
> Emacs, vi, etc., but installing bleeding-edge Haskell libraries via Nix
> is not a use-case we try to fulfill. If you absolutely *want* bleeding
> edge libraries in Nix, then you'll have to define those builds yourself
> with cabal2nix. The Nixpkgs user manual [1] explains how to do this.
>
> Users who want a balance of features and stability as in (2) should base
> their efforts on haskell.packages.lts, which will become haskellPackages
> after the imminent re-organization. That package set is continuously
> tested on hydra.nixos.org (or rather: it will be soon) and therefore
> tends to compile successfully on all platforms. It receives point
> updates that fix bugs and security issues, yet package APIs don't change
> so updates are unlikely to cause trouble. On th
[Nix-dev] Haskell Development and Deployment Strategies (was: Stackage Support Will Be Discontinued)
Fellow Haskell Hackers, Nix gives you great freedom to choose how to develop and deploy your Haskell software. That's good, but at the same time that flexibility can be confusing and feel like a burden. Therefore, I'd like to shed some light on this issue and discuss different types of strategies and their respective pros and cons. Which strategy is best for you depends on what you would like to achieve: 1. I want the latest version of every package as quickly as possible because I love it when build attempts result in hundreds of compiler errors that I can fix. 2. I want my Haskell packages to be up-to-date, but I also want my builds to be reliable. I don't mind fixing the occasional build error, but I mean, like, *occasionally*. 3. I want my Haskell package set to be rock solid. I don't want any updates unless there is a major security issue or a severe bug that threatens the integrity of my system. 4. I want total control over my entire package set. Now, let's go through the options those types of users have. If you favor features over stability as in (1), then you should develop with 'cabal-install', really. Nixpkgs might be useful for installing your favorite development tools like GHC, cabal, stack, alex, happy, Emacs, vi, etc., but installing bleeding-edge Haskell libraries via Nix is not a use-case we try to fulfill. If you absolutely *want* bleeding edge libraries in Nix, then you'll have to define those builds yourself with cabal2nix. The Nixpkgs user manual [1] explains how to do this. Users who want a balance of features and stability as in (2) should base their efforts on haskell.packages.lts, which will become haskellPackages after the imminent re-organization. That package set is continuously tested on hydra.nixos.org (or rather: it will be soon) and therefore tends to compile successfully on all platforms. It receives point updates that fix bugs and security issues, yet package APIs don't change so updates are unlikely to cause trouble. On the rare occasion that updates do cause trouble, Nixpkgs users usually commit fixes for these issues rather quickly. Every now and then, a fundamental package like "aeson" or "ghc" releases an update that breaks the API but that is desirable to have for any number of reasons, and then API-breaking updates *will* appear in that package set and they will require you to deal with them in your software. That kind of thing happens once or twice per year, and it's always announced and discussed beforehand by the Stackage team. If the term "API change" just caused your blood pressure to spike because the last thing you want is to deal with any API-breaking updates, then you belong into category (3) and you should never use the 'unstable' version of Nixpkgs. Instead, work with haskell.packages.lts in the release-16.03 branch (or haskellPackages in the release-16.09 branch, once it comes out). The Haskell package sets in release branches hardly ever change, and if they do, then the changes will be (a) minor and (b) important. API-breaking changes occur only in life-and-death situations, and there will be some kind of announcement and discussion before the change hits the release branch. Last but not least, people who run high-profile production systems tend prefer option (4). Now, if you want complete control over your system, then by definition you'll have to manage that system yourself. You can pick any version of Nixpkgs you like, check out the appropriate revision from the Git repository, and configure: export NIX_PATH="nixpkgs=$HOME/src/nixpkgs" There you go: this system won't change unless you want it to. Suppose you *do* want it to change for some reason, then you can ... a) update the checkout to a different revision, b) cherry-pick commits into your copy from other branches or even other repositories, or c) configure overrides in configuration.nix or ~/.nixpkgs/config.nix that replace certain packages with those versions that you'd like to have. The Nixpkgs manual explains how to do this at [2]. Obviously, there can be variations of those development strategies, i.e. it's always possible to extend the Haskell package set in Nixpkgs with an override that adds some particular package version you'd like to use or which undoes an update Nixpkgs made but that you don't want to have. To some extend you can also influence the decisions made in Nixpkgs by editing the hackage2nix configuration file [3] or by committing an override in [4]. Furthermore, the tools that generate the Haskell code in Nixpkgs are all open-source [5] and you can use them to roll your own personal distribution that does exactly what you want. Now, no doubt there will be users who see themselves somewhere between categories (3) and (4) who'd like to have the benefits of Nix available but who don't want to bother extending or managing the package set themselves in any way. These users probably run businesses and producti
