Hello Cyril, just answering a few points quickly.
> [...] “Plugins” can wait a little. If I understand correctly, you want to focus on crowdsec the program first. In this case, starting with 'dh-make-golang make -type program' is the right way to go. The source package will be named 'crowdsec', and produce a binary package named 'crowdsec'. Later on when you want to provide a library package, you will just add a new binary package to debian/control, named 'golang-github-crowdsecurity-crowdsec-dev', then add the file `debian/golang-github-crowdsecurity-crowdsec-dev.install`, and that's pretty much done already. Many packages do that, for example nomad, consul, containerd. > Maintainer: Debian Go Packaging Team <[email protected]> > Uploaders: Cyril Brulebois <[email protected]> > [...] I also think you're right here. All the build depends, ie. packages you don't necessarily care about, and that might also be used by other programs, are better maintained within the team. So that there's no friction to work on it, and everyone from the team can fix or update them when needed. All these small packages are "common ground" more or less. As for the main package crowdsec, that you care about, I guess it also makes sense to put yourself as the maintainer, or to create a "Corwdsec Packaging Team" as you suggest. Right now I maintain the docker.io package, it's the same situation, it lives in the "Docker Packaging Team". > Does everything make sense so far? Yep. > One which seemed easy enough was the following, and the rev-deps: > - golang-github-oschwald-maxminddb-golang > - golang-github-oschwald-geoip2-golang > - syncthing So basically, when you update libraries, you should check that the rev build deps don't break. We have a tool for that, it's ratt (rebuild all the things). It automates rebuilding all the reverse build deps with sbuild, while injecting in the build environment the new package. If ratt succeeds, then you can upload. If you're still unsure, better ask on the Debian Go mailing list before uploading, it won't hurt. Some packages are sensitive, or have many many reverse build deps. Better ask before upload until you're sure :) If ratt fails (ie. some packages fail to build with the new library that you just packaged), then you should investigate why, and fix the reverse build-deps for the new package that you intend to upload. If ratt fails and you don't care, and upload your package all the same, well, nothing will stop you, but it's not nice at all. You're putting more work on other people's shoulders. And it can be a lot of work. Please don't do that :) > Speaking of rev-deps, do we have any specific tooling to inspect all the > reverse dependencies of a given package, to make sure I wouldn't miss > anything? As mentioned above, check ratt, which relies on dose-ceve to find the rev build-deps, if I'm not mistaken. Apart from that, a few comments from my experience. Sometimes, a little patching is much better than a lot of blind packaging. For example, if you can patch a feature away, and save yourself packaging and maintaining a tree of 10+ dependencies, then maybe you should do that. You can see that at work in docker-registry, where support for relic and azure was removed. Nobody complained. You should exclude the 'vendor' directory if there's one. It's done in the Files-Excluded field of debian/copyright. However, for the "big packages", it's fairly common to keep a few of the vendored libraries, either because it saves us from a complicated build failure, or because some dependencies are not packaged. You can see that at work in the debian/copyright file of "big packages", once again: nomad, consul, containerd. docker.io, ... However, 1) the Files-Excluded list is a bit of a pain to maintain, 2) You're out of luck with crowdsec, they don't have a vendor directory. Other than that, good luck with crowdsec, and welcome to the Go Team, we have many packages and need more hands! Cheers, Arnaud -- Arnaud Rebillout -- https://arnaudr.io On Wed, Dec 16, 2020 at 5:29 PM Cyril Brulebois <[email protected]> wrote: > Hi everyone, > > > Context > ------- > > Let's start with some words about CrowdSec for context. From its > website: > > CrowdSec is a free, open-source, security automation tool using > local IP behavior detection & an online community-driven IP > reputation system. > > More at: https://crowdsec.net/ > > There's also a company behind it, who's asked my help to get the package > into Debian. You'll find my proposed plan below, and I'd be happy to get > some feedback about it since Go packaging is rather new to me, and I'd > like to make sure not to add any burden on the shoulders of the Debian > Go Packaging Team. > > > New packages > ------------ > > CrowdSec would be a program-oriented package. I didn't dive too deep > into dh-make-golang, but I would expect it not to be entirely crazy to > start with a program profile, and possibly add a binary or two after a > while, if we'd like to make some “plugin mechanism” (for the lack of a > better term) available in the future. This would make it possible for > other packages to have in their build dependencies a -dev package > produced by crowdsec. I'd like to start with some kind of “minimal > viable package” anyway, since there's lot to do already, hence my > current focus on starting with the program aspect (some postinst/wizard > will require some attention as well)… “Plugins” can wait a little. > > > This package would depend on quite a number of new packages. Since those > packages aren't in Debian at the moment, and would then only be used as > Build-Depends for the src:crowdsec package, I've investigated their > initial packaging as library-only. I haven't tweaked metadata yet (as > lintian correctly spotted, looking at the quite awful descriptions), > tweaked some excludes and/or ignored test results to get everything > started, and I've reached a point where crowdsec's building process > finds all of the required dependencies, and builds fine. > > Checking the incrementally-fed reprepro repository where I've stashed my > results, the involved packages would be: > - golang-github-alecaivazis-survey 2.2.4-1 > - golang-github-antonmedv-expr 1.8.9-1 > - golang-github-appleboy-gin-jwt 2.6.4-1 > - golang-github-appleboy-gofight 2.1.2-1 > - golang-github-denisbrodbeck-machineid 1.0.1-1 > - golang-github-enescakir-emoji 1.0.0-1 > - golang-github-facebook-ent 0.5.1-1 > - golang-github-go-co-op-gocron 0.3.3-1 > - golang-github-goombaio-namegenerator 0.0.2-1 > - golang-github-go-openapi-inflect 0.19.0-1 > - golang-github-hinshun-vt10x 0.0~git20180809.d55458d-1 > - golang-github-jamiealquiza-tachymeter 2.0.0-1 > - golang-github-logrusorgru-grokky 0.0~git20180829.47edf01-1 > - golang-github-mohae-deepcopy 0.0~git20170929.c48cc78-1 > - golang-github-netflix-go-expect 0.0~git20201125.85d881c-1 > - golang-github-nxadm-tail 1.4.5-1 > - golang-github-prometheus-prom2json 1.3.0-2 > > (I haven't double-checked whether some ITPs are around for any of those > yet.) > > If the “let's start with library-only” aspect looks good to you, I was > expecting to have all those packages have the recommended (dixit > <https://go-team.pages.debian.net/packaging.html>): > > Maintainer: Debian Go Packaging Team <[email protected]> > Uploaders: Cyril Brulebois <[email protected]> > > while the crowdsec package itself would be using something like: > > Maintainer: Cyril Brulebois <[email protected]> > Uploaders: Debian Go Packaging Team <[email protected]> > > or: > > Maintainer: CrowdSec Packaging Team <[email protected]> > Uploaders: Debian Go Packaging Team <[email protected]>, > Cyril Brulebois <[email protected]> > > (Maybe some upstream developers might want to update the package on > their own after a while, but some reviewing/mentoring would be required > anyway.) > > The idea would be that help from the team would be welcome here, even if > it'd be nicer to have me and/or upstream in the loop depending on the > scope of proposed changes. > > I would be on the lookout for everything related to the packages we're > introducing, through my subscription to this very list. > > Does everything make sense so far? > > > Updated packages > ---------------- > > I'll have some specific questions (that I'll ask in separate threads) > for some aspects, like a possible package/gopath misdetection, how to > deal with vendoring properly, how to deal with an obsolete v8 vs. a v10, > etc. > > The second biggest topic for me besides new packages would be about two > sets of packages that are a bit dated in Debian. Upstream would prefer > if we could just update the relevant packages, instead of having to > “downgrade” their code, or disable features. > > One which seemed easy enough was the following, and the rev-deps: > - golang-github-oschwald-maxminddb-golang > - golang-github-oschwald-geoip2-golang > - syncthing > > Fetching “blindly” the latest versions for the first two makes crowdsec > happy, and syncthing (with its apparently extensive test suite) still > builds fine. > > I've added Alexandre Viau in copy to see if proceeding with those > upgrades would seem appropriate, or whether I should envision a > different plan. > > > The second stack seems way more packed, it's about the > golang-github-gin-gonic-gin package, and a quick debtree -R shows quite > a number of reverse dependencies. I haven't investigated rebuilding them > and checking how it goes yet, I have only checked that updating this > package (and introducing in turn new dependencies/packages) makes it > possible to build crowdsec. I've added Shengjing Zhu in copy to get some > input, would updating this set of packages be welcome? > > Here's the list of extra packages I needed to get the new upstream to > build: > - golang-github-gin-gonic-gin 1.6.3-1 > - golang-github-go-playground-assert 2.0.1-1 > - golang-github-go-playground-locales 0.13.0-1 > - golang-github-go-playground-universal-translator 0.17.0-1 > - golang-github-go-playground-validator 10.4.1-1 > - golang-github-leodido-go-urn 1.2.0-1 > > Again, I didn't investigate the rev-deps (yet!). > > > Speaking of rev-deps, do we have any specific tooling to inspect all the > reverse dependencies of a given package, to make sure I wouldn't miss > anything? > > > I'm happily taking any questions/comments/recommendations, thanks for > reading thus far! > > > Cheers, > Cyril. > -- > Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/ >
