Hope I didn't kill the thread. :-( The TL;DR of my post was that I think go2nix is great and needed for many Go projects that don't vendor go-dependencies. But, from my Go development experience, one shouldn't try to boil-the-ocean and solve all Go packaging for everything.
I've also started a PR to add vendoring to github.com/hashicorp/otto. [1] Cheers, ~ Scott [1] https://github.com/hashicorp/otto/pull/519 On Fri, Jun 17, 2016 at 10:07 AM, Scott Devoid <[email protected]> wrote: > Hi Kamil, > > First thank you for working on this! Having go2nix as a tool certainly > makes packaging Go programs much easier! I don't do a lot of Nix > packaging but I do write Go software---so maybe I have a different > perspective on this. Anyhow here are some thoughts: > > - Many Go software projects try to be careful to vendor their > dependencies. For example, the Caddy server does this and it works > very well. Nix only needs to record the top-level package's SHA1 and > use fetchFromGithub + buildGoPacakge [1]. No need to recursively > expose all dependencies. > > - An exception to this rule are projects that make use of C or C++ > linked libraries. [2] For these you would need to add explicit > packages. > > - Then I take a look at Otto's Makefile and the updatedeps target [3] > and I am shocked! This is a terrible practice that the Hashicorp folks > should fix. They do things correctly with Vault so I don't see why > they can't fix it here. In any case, here go2nix is awesome since it > can effectively discover each dependency (tooling around the `go list` > command and the contents of GOPATH) and produce Nix derivations for > each component. > > - However, putting all of these in a "flat" go-packages.nix seems > incorrect. At the very least, two Go programs may require different > versions of the same package. And, as you mention, there's the > question of how to consistently name these packages. This is even more > complicated since packages may be named one thing but reside somewhere > else. And some packages get "folded in" to the standard library with > subsequent Go releases. [4] Or terrible redirect "services" like > gopkg.in [5] > > - What's the goal here? I would expect that Nix has packages for > 'major' Go programs and tooling to make it easy to package new > programs or update releases. The tooling should be designed so that an > update to one program doesn't break or alter dependencies for other > programs. I think this implies that the tooling produces a > 'go-libs.json' per-program as opposed to a single top-level > go-packages.nix file. This might cause redundant downloads of common > dependencies but that cost seems relatively minor compared to the > organizational headache that a flat-namespace would require. > > ~ Scott > > [1] > https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/caddy/default.nix > [2] https://golang.org/cmd/cgo > [3] https://github.com/hashicorp/otto/blob/master/Makefile#L35 > [4] See golang.org/x/net/context -> net/context in go1.7 > [5] http://labix.org/gopkg.in > > On Fri, Jun 17, 2016 at 6:55 AM, Kamil Chmielewski <[email protected]> > wrote: >> Hi, >> >> few days ago I started a PR called [WIP] Rework goPackages >> https://github.com/NixOS/nixpkgs/pull/16017 which is now on master and can >> leads to confusion for everyone creating goPackages before. I was also a >> little surprised how fast it was merged but I thought it was the effect of >> how many people feel the same pain as me, trying to manage or introduce new >> goPackages so there wasn't a lot of discussion there. But now it starts in >> few places and I think that this will be better place to make decisions >> about Go packaging in Nix. >> >> I want to start with the motivation which led me to this topic. Few months >> ago I wanted to introduce Otto (https://www.ottoproject.io/) to Nix. I was >> starting with Nix back then i tried to do things as they were in >> go-packages.nix. >> So I created otto as buildFromGithub[1] and use its bin output in >> all-packages.nix[2]. Next nix-build -A otto and I got 23 dependencies >> missing[3]. So I started to add dependencies manually doing nix-build -A >> otto tens of times because each of them depends on new things that `go >> build` tells me only when I add new sources for build... [4] >> After many hours of doing nix-build and new edits in go-packages.nix I ended >> up with otto done in 400 lines of Nix [5] I don't want do maintain. Besides >> new packages I also need to change version of things that were there before, >> potentially breaking other things. I can't even image how many hours of work >> it'll cost me to update it to new version :( So there no otto in nixpkgs. >> Instead of making otto I've started go2nix[6] prototype where I want to use >> Go toolchain because it can find out all transitive dependencies in one go. >> It wasn't hard to write a prototype in Go because everthing for it is there. >> So after a minutes I could automatically generate a list of all dependencies >> that I need to build any Go binary! Next step was to generate working Nix >> expression with this list of dependencies that can be used in goPackages. >> How hard can it be? :) It turned out that it's impossible to do this reusing >> goPackages in its current state. How could I know that: >> * github.com/ugorji/go sould be ugorji.go [7] >> * gopkg.in/flosch/pongo2.v3 should be pongo2 [8] >> * google.golang.org/api should be google-api-go-client [9] >> * github.com/odeke-em/google-api-go-client shoul be >> odeke-em.google-api-go-client [10] >> * github.com/stathat/go should be stathat [11] >> ... >> >> So there were 2 ways of dealing it: >> * generate new goPackage without any imports from go-packages.nix and >> abandon it completely >> * change go-packages.nix into something that can be readable for something >> that knows how packages are named in Go land >> I thought that go-packages.nix, python-packages.nix ...... says that we want >> to reuse common dependencies somewhere but it should be possible to override >> some of them if needed. And that was the beginning of [WIP] Rework >> goPackages with go-packages.nix "converted" into libs.json that was also >> influenced by temptation to make it possible to automate future updates[12]. >> >> As a troublemaker I want to involve all interested people to find out what >> next moves we should do to make: >> * master branch at least as usable as it was before rework >> * working with Nix and Go the best experience for both Nixers and Gophers >> >> So lets start with the first one. Most comments I see is related to >> libs.json. >> Does it mean we want go-packages.nix but in form that can be generated and >> imported like libs.json or we don't want it at all moving all the >> dependencies to separate derivations? Or maybe we want go-packages.nix as it >> was before? >> >> -- >> Kamil >> >> [1] >> https://github.com/kamilchm/nixpkgs/pull/3/files#diff-8631dcd3e45d5f2ceaedeb96a23bee6fR2214 >> [2] >> https://github.com/kamilchm/nixpkgs/pull/3/files#diff-036410e9211b4336186fc613f7200b12R12219 >> [3] >> https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013#file-nix-build-a-otto >> [4] https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013 >> [5] https://github.com/kamilchm/nixpkgs/pull/3 >> [6] https://github.com/kamilchm/go2nix >> [7] >> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L683 >> [8] >> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2070 >> [9] >> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L861 >> [10] >> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L872 >> [11] >> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2700 >> [12] https://github.com/NixOS/nixpkgs/pull/13819 >> >> >> _______________________________________________ >> nix-dev mailing list >> [email protected] >> http://lists.science.uu.nl/mailman/listinfo/nix-dev >> _______________________________________________ nix-dev mailing list [email protected] http://lists.science.uu.nl/mailman/listinfo/nix-dev
