Hi, I refactored all Go applications, moving from deps.json to deps.nix and encapsulating program dependecies instead of single shared libs.json. I'm waiting for feedback in https://github.com/NixOS/nixpkgs/pull/17254 especially from Nix maintainers ;)
-- Kamil 2016-06-21 8:20 GMT+02:00 Scott Devoid <[email protected]>: > Hope I didn't kill the thread. :-( > > The TL;DR of my post was that I think go2nix is great and needed for > many Go projects that don't vendor go-dependencies. But, from my Go > development experience, one shouldn't try to boil-the-ocean and solve > all Go packaging for everything. > > I've also started a PR to add vendoring to github.com/hashicorp/otto. [1] > > Cheers, > ~ Scott > > [1] https://github.com/hashicorp/otto/pull/519 > > On Fri, Jun 17, 2016 at 10:07 AM, Scott Devoid <[email protected]> wrote: > > Hi Kamil, > > > > First thank you for working on this! Having go2nix as a tool certainly > > makes packaging Go programs much easier! I don't do a lot of Nix > > packaging but I do write Go software---so maybe I have a different > > perspective on this. Anyhow here are some thoughts: > > > > - Many Go software projects try to be careful to vendor their > > dependencies. For example, the Caddy server does this and it works > > very well. Nix only needs to record the top-level package's SHA1 and > > use fetchFromGithub + buildGoPacakge [1]. No need to recursively > > expose all dependencies. > > > > - An exception to this rule are projects that make use of C or C++ > > linked libraries. [2] For these you would need to add explicit > > packages. > > > > - Then I take a look at Otto's Makefile and the updatedeps target [3] > > and I am shocked! This is a terrible practice that the Hashicorp folks > > should fix. They do things correctly with Vault so I don't see why > > they can't fix it here. In any case, here go2nix is awesome since it > > can effectively discover each dependency (tooling around the `go list` > > command and the contents of GOPATH) and produce Nix derivations for > > each component. > > > > - However, putting all of these in a "flat" go-packages.nix seems > > incorrect. At the very least, two Go programs may require different > > versions of the same package. And, as you mention, there's the > > question of how to consistently name these packages. This is even more > > complicated since packages may be named one thing but reside somewhere > > else. And some packages get "folded in" to the standard library with > > subsequent Go releases. [4] Or terrible redirect "services" like > > gopkg.in [5] > > > > - What's the goal here? I would expect that Nix has packages for > > 'major' Go programs and tooling to make it easy to package new > > programs or update releases. The tooling should be designed so that an > > update to one program doesn't break or alter dependencies for other > > programs. I think this implies that the tooling produces a > > 'go-libs.json' per-program as opposed to a single top-level > > go-packages.nix file. This might cause redundant downloads of common > > dependencies but that cost seems relatively minor compared to the > > organizational headache that a flat-namespace would require. > > > > ~ Scott > > > > [1] > https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/caddy/default.nix > > [2] https://golang.org/cmd/cgo > > [3] https://github.com/hashicorp/otto/blob/master/Makefile#L35 > > [4] See golang.org/x/net/context -> net/context in go1.7 > > [5] http://labix.org/gopkg.in > > > > On Fri, Jun 17, 2016 at 6:55 AM, Kamil Chmielewski <[email protected]> > wrote: > >> Hi, > >> > >> few days ago I started a PR called [WIP] Rework goPackages > >> https://github.com/NixOS/nixpkgs/pull/16017 which is now on master and > can > >> leads to confusion for everyone creating goPackages before. I was also a > >> little surprised how fast it was merged but I thought it was the effect > of > >> how many people feel the same pain as me, trying to manage or introduce > new > >> goPackages so there wasn't a lot of discussion there. But now it starts > in > >> few places and I think that this will be better place to make decisions > >> about Go packaging in Nix. > >> > >> I want to start with the motivation which led me to this topic. Few > months > >> ago I wanted to introduce Otto (https://www.ottoproject.io/) to Nix. I > was > >> starting with Nix back then i tried to do things as they were in > >> go-packages.nix. > >> So I created otto as buildFromGithub[1] and use its bin output in > >> all-packages.nix[2]. Next nix-build -A otto and I got 23 dependencies > >> missing[3]. So I started to add dependencies manually doing nix-build > -A > >> otto tens of times because each of them depends on new things that `go > >> build` tells me only when I add new sources for build... [4] > >> After many hours of doing nix-build and new edits in go-packages.nix I > ended > >> up with otto done in 400 lines of Nix [5] I don't want do maintain. > Besides > >> new packages I also need to change version of things that were there > before, > >> potentially breaking other things. I can't even image how many hours of > work > >> it'll cost me to update it to new version :( So there no otto in > nixpkgs. > >> Instead of making otto I've started go2nix[6] prototype where I want to > use > >> Go toolchain because it can find out all transitive dependencies in one > go. > >> It wasn't hard to write a prototype in Go because everthing for it is > there. > >> So after a minutes I could automatically generate a list of all > dependencies > >> that I need to build any Go binary! Next step was to generate working > Nix > >> expression with this list of dependencies that can be used in > goPackages. > >> How hard can it be? :) It turned out that it's impossible to do this > reusing > >> goPackages in its current state. How could I know that: > >> * github.com/ugorji/go sould be ugorji.go [7] > >> * gopkg.in/flosch/pongo2.v3 should be pongo2 [8] > >> * google.golang.org/api should be google-api-go-client [9] > >> * github.com/odeke-em/google-api-go-client shoul be > >> odeke-em.google-api-go-client [10] > >> * github.com/stathat/go should be stathat [11] > >> ... > >> > >> So there were 2 ways of dealing it: > >> * generate new goPackage without any imports from go-packages.nix and > >> abandon it completely > >> * change go-packages.nix into something that can be readable for > something > >> that knows how packages are named in Go land > >> I thought that go-packages.nix, python-packages.nix ...... says that we > want > >> to reuse common dependencies somewhere but it should be possible to > override > >> some of them if needed. And that was the beginning of [WIP] Rework > >> goPackages with go-packages.nix "converted" into libs.json that was also > >> influenced by temptation to make it possible to automate future > updates[12]. > >> > >> As a troublemaker I want to involve all interested people to find out > what > >> next moves we should do to make: > >> * master branch at least as usable as it was before rework > >> * working with Nix and Go the best experience for both Nixers and > Gophers > >> > >> So lets start with the first one. Most comments I see is related to > >> libs.json. > >> Does it mean we want go-packages.nix but in form that can be generated > and > >> imported like libs.json or we don't want it at all moving all the > >> dependencies to separate derivations? Or maybe we want go-packages.nix > as it > >> was before? > >> > >> -- > >> Kamil > >> > >> [1] > >> > https://github.com/kamilchm/nixpkgs/pull/3/files#diff-8631dcd3e45d5f2ceaedeb96a23bee6fR2214 > >> [2] > >> > https://github.com/kamilchm/nixpkgs/pull/3/files#diff-036410e9211b4336186fc613f7200b12R12219 > >> [3] > >> > https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013#file-nix-build-a-otto > >> [4] https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013 > >> [5] https://github.com/kamilchm/nixpkgs/pull/3 > >> [6] https://github.com/kamilchm/go2nix > >> [7] > >> > https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L683 > >> [8] > >> > https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2070 > >> [9] > >> > https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L861 > >> [10] > >> > https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L872 > >> [11] > >> > https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2700 > >> [12] https://github.com/NixOS/nixpkgs/pull/13819 > >> > >> > >> _______________________________________________ > >> nix-dev mailing list > >> [email protected] > >> http://lists.science.uu.nl/mailman/listinfo/nix-dev > >> >
_______________________________________________ nix-dev mailing list [email protected] http://lists.science.uu.nl/mailman/listinfo/nix-dev
