Bug#971515: kubernetes: excessive vendoring (private libraries)
On Friday, 23 October 2020 4:20:37 AM AEDT Shengjing Zhu wrote: > However kubernetes is also a library, and the maintainer doesn't provide > it. It becomes headache for other maintainers. Yes we have to vendor "k8s.io/" all the time in multiple packages but I doubt it would be of help if Kubernetes provide a "-dev" library because it is notoriously unstable with frequent breaking changes. Ideally that's how it should be done but, pragmatically speaking, Kubernetes can never migrate to "testing" hence everything that depends on it will be unsuitable for release. > The way the kubernetes maintainer chooses, is making others difficult. > Every package that uses kubernetes as a library, has to keep a local copy > of the source. The partial solution might be to introduce another source package only with portion of Kubernetes that provides a shared library but that will require proper dependency management which is being discussed here. > But I hope Janos can cooperate with the Go packaging team to make Debian, > especially the Go packages in Debian more maintainable. Yes indeed. We should not need to argue that cooperation is important. Cooperation should always be a default mode of operation. -- Kind regards, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- Hay smells different to lovers and horses. -- Stanisław Jerzy Lec signature.asc Description: This is a digitally signed message part.
Bug#971515: kubernetes: excessive vendoring (private libraries)
On Tue, Oct 20, 2020 at 12:16:03PM -0700, Sean Whitton wrote: > > - are people trying to do cross-archive work going to find the packaging > of kubernetes getting in their way? (e.g. other Go team members > trying to update things, improve our binNMU techniques and machinery, > etc.) > The kubernetes maintainer is not in the Go packaging team, so doesn't follow the team practice. But as kubernetes vendor everything, it doesn't need to work with all other Go packages. However kubernetes is also a library, and the maintainer doesn't provide it. It becomes headache for other maintainers. We (the Go packaging team) need to vendor the kubernetes in our packages, for example: + https://sources.debian.org/src/golang-github-openshift-api/4.0+git20190508.81d064c-4/vendor/k8s.io/ + https://sources.debian.org/src/gitlab-ci-multi-runner/13.3.1+dfsg-1/vendor/k8s.io/ + https://sources.debian.org/src/flannel/0.9.1~ds1-1/vendor/k8s.io/ + https://sources.debian.org/src/consul/1.7.4+dfsg1-1/vendor/k8s.io/ And many, they can be found with: https://codesearch.debian.net/search?q=%22k8s.io%2F+filetype%3Ago=1 It has been asked at #970331, without response from maintainer. If kubernetes package want to make other package which Build-Depends on it to build successfully, it needs to decouple the dependencies from vendor dir. When building other packages, the Go toolchain can't use the vendor in kubernetes, since the Go toolchain only supports the top level vendor dir. The way the kubernetes maintainer chooses, is making others difficult. Every package that uses kubernetes as a library, has to keep a local copy of the source. It's not Janos fault, as he doesn't make things worse. Kubernetes package is in a bad state for a long time. But I hope Janos can cooperate with the Go packaging team to make Debian, especially the Go packages in Debian more maintainable. -- Shengjing Zhu signature.asc Description: PGP signature
Bug#971515: kubernetes: excessive vendoring (private libraries)
On Thursday, 22 October 2020 2:16:16 AM AEDT Sean Whitton wrote: > I think that we can all agree with everything you've written about the > reasons why packaging components separately is better. Thank you. > The problem is > that in this case the choice seems to be between not having recent > Kubernetes in Debian at all, or giving up on some of those advantages. Ultimately it is about maintainers comfort then. Because there is a compromise to un-vendor most libraries (one by one) and keep some/few strategically vendored, when a system library can not be used (determined case by case). Examples of this approach are numerous: syncthing, consul, nomad, vault, docker.io, runc, gitlab-runner, libpod, buildah, singularity-container and _most_ Golang packages. Kubernetes had no maintainer - that was the problem. Having maintainer who does not care for shared libraries because he does not want to be bothered is a different problem. -- Kind regards, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- In times of universal deceit, telling the truth becomes a revolucionary act. -- George Orwell signature.asc Description: This is a digitally signed message part.
Bug#971515: kubernetes: excessive vendoring (private libraries)
On Wednesday, 21 October 2020 8:56:53 PM AEDT Felix Lechner wrote: > How is the second one inferior, please? I think it includes a crucial > missing feature (co-installable versions). To reduce maintainers burden, you want maintainers to look after not one but multiple versions of libraries. This is a contradiction (oxymoron). How many releases (current + obsolete ones) one can meaningfully support? -- Best wishes, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- The cure for the evils of democracy is more democracy. -- H. L. Mencken signature.asc Description: This is a digitally signed message part.
Bug#971515: kubernetes: excessive vendoring (private libraries)
Hello Dmitry, On Wed 21 Oct 2020 at 11:21am +11, Dmitry Smirnov wrote: > On Wednesday, 21 October 2020 6:16:03 AM AEDT Sean Whitton wrote: >> I think that my message [1] is what makes you think that the package >> would not have got through NEW? > > It was not your message but my own experience with introducing of 100+ > packages through NEW, especially those ones with large burden of vendored > libraries, including Kubernetes. The main hassle usually is to convince FTP- > masters when some vendoring is _necessary_ (case by case) and the usual > request is to package all vendored libraries separately. With rare exceptions > some (few) vendored libraries are allowed like when a library is a fork, > customised for the particular project and therefore not re-usable by other > software. Another example is when vendored library is an obsolete software > phasing out in future releases. Few micro-libraries might be tolerated when > vendored, especially when they are not widely used. Also vendoring might be > acceptable when software components with mutual/circular dependencies are > shipped in one or several name spaces - in other words when a software code > base is not from one name space but from several. None of those cases applies > to Kubernetes. > > A specific example (libpod/podman) is mentioned in > > https://lists.debian.org/debian-devel/2020/03/msg00441.html > > "Podman was rejected due to "many embedded packages in vendor/" with only > 6 or 7 private libs versus 120 libraries removed in favour of packaged > ones." Fair enough, but again, this is about NEW as a review by experienced packagers rather than NEW as a blocker for inclusion. > If your concern is about security support then IMHO Kubernetes can not be > meaningfully supported from security prospective, with or without vendored > dependencies. Fair enough. In the -devel thread the security team indicated that they thought they could do security support in the case where there is vendoring. It would be good to get more input from them in this bug. > If Debian only cared about maintainers' convenience or reducing maintainers > efforts then Debian would not be what it is now. We favour technical elegance > often in expense of maintainers' comfort. I don't think maintainers' comfort is part of Janos' motivation -- it's the issue of not having recent Kubernetes in Debian at all. >> Are there issues the TC should think about which do not fall under this >> way of looking at things? I.e., weighing the impact on people other >> than Janos who want to work on the package, vs. the impact on people who >> want recent kubernetes to be part in the archive at all? > > Is Debian ecosystem of packaged reusable libraries worth caring about? > > If so then why grant exception to one particular package? We have several (or > more) sophisticated Golang packages using hundreds of packaged libraries. > > In the early days of packaging Kubernetes we did not even have most > components packaged and I've been spending most effort on packaging, > introducing and stabilising dependency libraries. > These days the major work has already been done and the argument for > monolithic vendoring is much weaker. > > [...] I think that we can all agree with everything you've written about the reasons why packaging components separately is better. The problem is that in this case the choice seems to be between not having recent Kubernetes in Debian at all, or giving up on some of those advantages. -- Sean Whitton
Re: Bug#971515: kubernetes: excessive vendoring (private libraries)
Hello, On Tue 20 Oct 2020 at 06:52pm -07, Felix Lechner wrote: > I think our response to the vendoring explosion is at odds with the > trends in many languages. > > It's time to retool. At the two ends of the solution spectrum, I see > > 1. Fully vendored source packages; or > 2. A packaging system that allows different vendor versions to co-exist. > > Either one allows dependent sources to consume whichever versions they > require, but in my view solution (2) is otherwise superior---provided > that the packaging process is automated. (A language's build system > also has to distinguish the installed versions.) For each language so > affected, could we make (2) our goal, and allow (1) until then? This is a very general (but of course interesting) topic. Could I ask that it be kept out of this TC bug, please? We have to figure out what to do about this package given our present tooling. -- Sean Whitton
Re: Bug#971515: kubernetes: excessive vendoring (private libraries)
Hi Dmitry, On Wed, Oct 21, 2020 at 12:50 AM Dmitry Smirnov wrote: > > Yes, at first there will > be a significant effort but then it will become easier. I know you put a lot of effort in (as did Michael Stapelberg, with whom I spent some time before he left), but I don't think our current approach makes anything easy. It is also why the world is moving in another direction. > You are advocating for disruptive > changes therefore your proposed theoretical solutions have to be available as > a proof of concept for review. Did you catch the part about different versions being co-installable? It would be similar to the freedom we grant to major numbers of shared object libraries. My point is theoretical only because we do not presently have it. > Personally I'm not satisfied with either of those inferior proposals. How is the second one inferior, please? I think it includes a crucial missing feature (co-installable versions). > Look how many packages we already have: > > > https://qa.debian.org/developer.php?login=pkg-go-maintainers%40lists.alioth.debian.org+team%2Bpkg-go%40tracker.debian.org It's an impressive list; thank you for forwarding it. I am proud to be on the Golang team. :) > In the meantime you could follow the established practice that is > demonstrated to be working on several packaged heavy Golang applications. Not sure about heavy Golang applications, but our established practice does work well for the relatively lightweight 'gocryptfs', which I maintain. That source moves very fast. I often have problems finding the proper go-fuse or xattr prerequisite required for a new version. Sometimes they are incompatible with the needs of other packages in the archive. As a side note, several "library" packages that gocryptfs relies on should really be marked "Architecture: any" to exercise their test suites properly. It is another example of the impedance mismatch in our current approach. We are confusing sources and libraries, and our method of shoehorning one into the other ought to be rethought. > Besides un-vendoring libraries can prevent some CVE issues as well. Packages could declare vendored sources (or Lintian could scan for them) for an effective match with their security status. > If we tolerate full vendoring now, because "there is no better way" yet, then > there will be no better way for sure. Please do not despair. I offered full vendoring only in the interest of compromise, which I believe is a worthwhile goal just like the consensus we are working on. As a project, we are looking for a better endpoint together. Kind regards Felix Lechner
Bug#971515: kubernetes: excessive vendoring (private libraries)
Hi Felix, On Wednesday, 21 October 2020 12:52:40 PM AEDT Felix Lechner wrote: > > We favour technical elegance often in expense of maintainers' comfort. > > Is our approach really either one of those? I think our response to > the vendoring explosion is at odds with the trends in many languages. IMHO we are managing quite admirably. Basically, to me it looks like you don't want to maintain Kubernetes the way we maintain heavy Golang packages. You would have to learn to un-vendor many libraries. Yes, at first there will be a significant effort but then it will become easier. "Too many vendored libraries to use packaged libs" is a poor excuse. We have been dealing with "explosion" for years already. Tools like "dh-make- golang" are helpful to generate initial packaging for new Golang libraries in a semi-automatic manner. FTP-masters are usually quite effective with processing of NEW packages. Look how many packages we already have: https://qa.debian.org/developer.php?login=pkg-go-maintainers%40lists.alioth.debian.org+team%2Bpkg-go%40tracker.debian.org > It's time to retool. At the two ends of the solution spectrum, I see > > 1. Fully vendored source packages; or > 2. A packaging system that allows different vendor versions to > co-exist. Personally I'm not satisfied with either of those inferior proposals. Besides un-vendoring libraries can prevent some CVE issues as well. > Either one allows dependent sources to consume whichever versions they > require, but in my view solution (2) is otherwise superior---provided > that the packaging process is automated. (A language's build system > also has to distinguish the installed versions.) For each language so > affected, could we make (2) our goal, and allow (1) until then? IMHO tools have to come first (if ever). You are advocating for disruptive changes therefore your proposed theoretical solutions have to be available as a proof of concept for review. In the meantime you could follow the established practice that is demonstrated to be working on several packaged heavy Golang applications. If we tolerate full vendoring now, because "there is no better way" yet, then there will be no better way for sure. For now using packaged system libraries whenever possible is the best way. -- Kind regards, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- Those who disdain wealth as a worthy goal for an individual or a society seem not to realize that wealth is the only thing that can prevent poverty. -- Thomas Sowell signature.asc Description: This is a digitally signed message part.
Re: Bug#971515: kubernetes: excessive vendoring (private libraries)
Hi Dmitry, On Tue, Oct 20, 2020 at 5:24 PM Dmitry Smirnov wrote: > > Let's not attempt to fabricate perception of consensus please. Consensus is a worthy goal and perhaps even possible, per below. > We favour technical elegance > often in expense of maintainers' comfort. Is our approach really either one of those? I think our response to the vendoring explosion is at odds with the trends in many languages. It's time to retool. At the two ends of the solution spectrum, I see 1. Fully vendored source packages; or 2. A packaging system that allows different vendor versions to co-exist. Either one allows dependent sources to consume whichever versions they require, but in my view solution (2) is otherwise superior---provided that the packaging process is automated. (A language's build system also has to distinguish the installed versions.) For each language so affected, could we make (2) our goal, and allow (1) until then? Kind regards Felix Lechner
Bug#971515: kubernetes: excessive vendoring (private libraries)
On Wednesday, 21 October 2020 6:16:03 AM AEDT Sean Whitton wrote: > I think that my message [1] is what makes you think that the package > would not have got through NEW? It was not your message but my own experience with introducing of 100+ packages through NEW, especially those ones with large burden of vendored libraries, including Kubernetes. The main hassle usually is to convince FTP- masters when some vendoring is _necessary_ (case by case) and the usual request is to package all vendored libraries separately. With rare exceptions some (few) vendored libraries are allowed like when a library is a fork, customised for the particular project and therefore not re-usable by other software. Another example is when vendored library is an obsolete software phasing out in future releases. Few micro-libraries might be tolerated when vendored, especially when they are not widely used. Also vendoring might be acceptable when software components with mutual/circular dependencies are shipped in one or several name spaces - in other words when a software code base is not from one name space but from several. None of those cases applies to Kubernetes. A specific example (libpod/podman) is mentioned in https://lists.debian.org/debian-devel/2020/03/msg00441.html "Podman was rejected due to "many embedded packages in vendor/" with only 6 or 7 private libs versus 120 libraries removed in favour of packaged ones." > There are a few issues tangled together here. IMHO it is really one issue of how we maintain Debian packages. If you want to distinguish few issues then they are all closely related. > NEW is mainly about license and DFSG compliance, and secondarily about > the idea that we want to avoid accepting packages where doing so would > make Debian worse, even if it would also make Debian better along other > dimensions. As a simple example, we try to avoid accepting a package > that is already packaged under a slightly different name, because in > most cases it is worse for both users and contributors to have the same > thing in the archive twice (not talking about vendoring here). It is also about preserving integrity of Debian identity. We try to prevent monolithic bundles like Kubernetes in favour of maintaining ecosystem of re- usable libraries, packaged individually. > In this case, the reason I wrote in [1] that I would probably have > rejected the package, had I come across it in NEW, is that it seemed to > me that having this package in Debian would make the archive less > maintainable by contributors other than Janos who might need to work > with the package. (After the discussion on -devel, I'm no longer so > sure about that opinion of mine.) https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_party_code If your concern is about security support then IMHO Kubernetes can not be meaningfully supported from security prospective, with or without vendored dependencies. Also I must point out that Kubernetes upstream have the worst management of vendored libraries that I have ever seen. Examples include vendoring multiple versions of the same library, etc. A particular case when upstream failed to update a problematic vendored library for years(!) practically destroys faith in upstream care for good hygiene of vendored dependencies: https://github.com/kubernetes/kubernetes/issues/27543 Note the expressed _resistance_ to upgrading a vendored library... With the above example, how can anyone have confidence in upstream security patching? After all Kubernetes have more vendored code than its own. > It's not correct to say, however, that the package "is in violation of > ftpmaster's policy for inclusion of new packages". That could only be > true is if the package met one of the "serious violations" listed in the > REJECT-FAQ, which is basically DFSG and licensing issues, and a few > obvious clangers. Instead, what we have is a situation in which there > is reason to be worried about the way the package is put together, but > the opinion of one FTP team member at one particular point in time > carries about the same weight as the opinion of any experienced > packager. There is an established practice, a tradition if you wish, that is followed all the time even if not explicitly described in REJECT-FAQ. Debian clearly tries to be modular whenever possible. > In other words, I suggest that we ignore the NEW issue entirely, and > just consider whether the way the package is currently put together > imposes an unreasonable burden on Debian contributors other than Janos > who want to work on the package, or users who want to patch it, etc. > The sorts of questions we should try to answer: > > - does the vendoring make Debian security support harder (discussion on > -devel suggests it makes it easier) > > - everyone seems to think the level of vendoring is at best a necessary > evil; Let's not attempt to fabricate perception of
Bug#971515: kubernetes: excessive vendoring (private libraries)
Hello Dmitry, others, On Thu 01 Oct 2020 at 11:15am +10, Dmitry Smirnov wrote: > I seek your judgement regarding excessive, unnecessary and unjustifiable > vendoring of private libraries in Kubernetes package: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970717 > > Some relevant argumentation can be found in > > https://lists.debian.org/debian-devel/2020/03/msg00359.html > https://lists.debian.org/debian-devel/2020/03/msg00400.html > https://lists.debian.org/debian-devel/2020/03/msg00441.html > > In short, myself and Golang packaging team spent years on stabilising > Golang ecosystem of packaged libraries for re-use by Golang software. > > To the best of my knowledge, all packaged Golang software, regardless of its > sophistication, use some packages libraries. > Except Kubernetes, that disconnected itself from cooperation by not using any > packaged libraries, instead exclusively using only private libraries in > numbers greater than any Debian package ever used. > > As a former Kubernetes maintainer and a developer who originally introduced > Kubernetes to Debian, I know that it could be maintained with only few (or > several) private libraries at most. > > Current state of Kubernetes package is in violation of ftp-master's policy > for inclusion of new packages to Debian. I think that my message [1] is what makes you think that the package would not have got through NEW? There are a few issues tangled together here. NEW is mainly about license and DFSG compliance, and secondarily about the idea that we want to avoid accepting packages where doing so would make Debian worse, even if it would also make Debian better along other dimensions. As a simple example, we try to avoid accepting a package that is already packaged under a slightly different name, because in most cases it is worse for both users and contributors to have the same thing in the archive twice (not talking about vendoring here). In this case, the reason I wrote in [1] that I would probably have rejected the package, had I come across it in NEW, is that it seemed to me that having this package in Debian would make the archive less maintainable by contributors other than Janos who might need to work with the package. (After the discussion on -devel, I'm no longer so sure about that opinion of mine.) It's not correct to say, however, that the package "is in violation of ftpmaster's policy for inclusion of new packages". That could only be true is if the package met one of the "serious violations" listed in the REJECT-FAQ, which is basically DFSG and licensing issues, and a few obvious clangers. Instead, what we have is a situation in which there is reason to be worried about the way the package is put together, but the opinion of one FTP team member at one particular point in time carries about the same weight as the opinion of any experienced packager. In other words, I suggest that we ignore the NEW issue entirely, and just consider whether the way the package is currently put together imposes an unreasonable burden on Debian contributors other than Janos who want to work on the package, or users who want to patch it, etc. The sorts of questions we should try to answer: - does the vendoring make Debian security support harder (discussion on -devel suggests it makes it easier) - everyone seems to think the level of vendoring is at best a necessary evil; if someone wants to try to reduce the level of vendoring (as Dmitry did when he was maintainer), is the current way the package is built going to make it harder for people to work on making that sort of contribution? - are people trying to do cross-archive work going to find the packaging of kubernetes getting in their way? (e.g. other Go team members trying to update things, improve our binNMU techniques and machinery, etc.) ... and this is to be weighed against the negative impact of having kubernetes in Debian lag so seriously behind upstream such that almost no-one would want to bother installing it. Are there issues the TC should think about which do not fall under this way of looking at things? I.e., weighing the impact on people other than Janos who want to work on the package, vs. the impact on people who want recent kubernetes to be part in the archive at all? If I'm right about what the question for the TC is, I hope that Janos and Dmitry can both help us discuss it in a way which sets aside the heat which characterised the -devel thread. It is completely understandable to (a) feel very frustrated at Debian not including recent versions of a useful piece of free software; and also (b) feel very frustrated when someone chooses to accept a less-than-ideal approach as necessary when one has put a lot of time into trying to find workable alternatives. The thing is, both (a) and (b) are motivated by the same basic desire to make Debian better and more useful, so perhaps we can focus on that point of commonality. [1]
Bug#971515: kubernetes: excessive vendoring (private libraries)
Package: tech-ctte Severity: normal X-Debbugs-CC: o...@debian.org Control: block 970717 by -1 Dear Technical Committee, Apologies that we were not able to resolve the problem without your help. I seek your judgement regarding excessive, unnecessary and unjustifiable vendoring of private libraries in Kubernetes package: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970717 Some relevant argumentation can be found in https://lists.debian.org/debian-devel/2020/03/msg00359.html https://lists.debian.org/debian-devel/2020/03/msg00400.html https://lists.debian.org/debian-devel/2020/03/msg00441.html In short, myself and Golang packaging team spent years on stabilising Golang ecosystem of packaged libraries for re-use by Golang software. To the best of my knowledge, all packaged Golang software, regardless of its sophistication, use some packages libraries. Except Kubernetes, that disconnected itself from cooperation by not using any packaged libraries, instead exclusively using only private libraries in numbers greater than any Debian package ever used. As a former Kubernetes maintainer and a developer who originally introduced Kubernetes to Debian, I know that it could be maintained with only few (or several) private libraries at most. Current state of Kubernetes package is in violation of ftp-master's policy for inclusion of new packages to Debian. Thank you. This matter is not urgent. -- Regards, Dmitry Smirnov --- We do our friends no favors by pretending not to notice flaws in their work, especially when those who are not their friends are bound to notice these same flaws. -- Sam Harris signature.asc Description: This is a digitally signed message part.
