Re: Bug#971515: kubernetes: excessive vendoring (private libraries)
Hi Dmitry, On Tue, Oct 20, 2020 at 5:24 PM Dmitry Smirnov wrote: > > Let's not attempt to fabricate perception of consensus please. Consensus is a worthy goal and perhaps even possible, per below. > We favour technical elegance > often in expense of maintainers' comfort. Is our approach really either one of those? I think our response to the vendoring explosion is at odds with the trends in many languages. It's time to retool. At the two ends of the solution spectrum, I see 1. Fully vendored source packages; or 2. A packaging system that allows different vendor versions to co-exist. Either one allows dependent sources to consume whichever versions they require, but in my view solution (2) is otherwise superior---provided that the packaging process is automated. (A language's build system also has to distinguish the installed versions.) For each language so affected, could we make (2) our goal, and allow (1) until then? Kind regards Felix Lechner
Bug#971515: kubernetes: excessive vendoring (private libraries)
On Wednesday, 21 October 2020 6:16:03 AM AEDT Sean Whitton wrote: > I think that my message [1] is what makes you think that the package > would not have got through NEW? It was not your message but my own experience with introducing of 100+ packages through NEW, especially those ones with large burden of vendored libraries, including Kubernetes. The main hassle usually is to convince FTP- masters when some vendoring is _necessary_ (case by case) and the usual request is to package all vendored libraries separately. With rare exceptions some (few) vendored libraries are allowed like when a library is a fork, customised for the particular project and therefore not re-usable by other software. Another example is when vendored library is an obsolete software phasing out in future releases. Few micro-libraries might be tolerated when vendored, especially when they are not widely used. Also vendoring might be acceptable when software components with mutual/circular dependencies are shipped in one or several name spaces - in other words when a software code base is not from one name space but from several. None of those cases applies to Kubernetes. A specific example (libpod/podman) is mentioned in https://lists.debian.org/debian-devel/2020/03/msg00441.html "Podman was rejected due to "many embedded packages in vendor/" with only 6 or 7 private libs versus 120 libraries removed in favour of packaged ones." > There are a few issues tangled together here. IMHO it is really one issue of how we maintain Debian packages. If you want to distinguish few issues then they are all closely related. > NEW is mainly about license and DFSG compliance, and secondarily about > the idea that we want to avoid accepting packages where doing so would > make Debian worse, even if it would also make Debian better along other > dimensions. As a simple example, we try to avoid accepting a package > that is already packaged under a slightly different name, because in > most cases it is worse for both users and contributors to have the same > thing in the archive twice (not talking about vendoring here). It is also about preserving integrity of Debian identity. We try to prevent monolithic bundles like Kubernetes in favour of maintaining ecosystem of re- usable libraries, packaged individually. > In this case, the reason I wrote in [1] that I would probably have > rejected the package, had I come across it in NEW, is that it seemed to > me that having this package in Debian would make the archive less > maintainable by contributors other than Janos who might need to work > with the package. (After the discussion on -devel, I'm no longer so > sure about that opinion of mine.) https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_party_code If your concern is about security support then IMHO Kubernetes can not be meaningfully supported from security prospective, with or without vendored dependencies. Also I must point out that Kubernetes upstream have the worst management of vendored libraries that I have ever seen. Examples include vendoring multiple versions of the same library, etc. A particular case when upstream failed to update a problematic vendored library for years(!) practically destroys faith in upstream care for good hygiene of vendored dependencies: https://github.com/kubernetes/kubernetes/issues/27543 Note the expressed _resistance_ to upgrading a vendored library... With the above example, how can anyone have confidence in upstream security patching? After all Kubernetes have more vendored code than its own. > It's not correct to say, however, that the package "is in violation of > ftpmaster's policy for inclusion of new packages". That could only be > true is if the package met one of the "serious violations" listed in the > REJECT-FAQ, which is basically DFSG and licensing issues, and a few > obvious clangers. Instead, what we have is a situation in which there > is reason to be worried about the way the package is put together, but > the opinion of one FTP team member at one particular point in time > carries about the same weight as the opinion of any experienced > packager. There is an established practice, a tradition if you wish, that is followed all the time even if not explicitly described in REJECT-FAQ. Debian clearly tries to be modular whenever possible. > In other words, I suggest that we ignore the NEW issue entirely, and > just consider whether the way the package is currently put together > imposes an unreasonable burden on Debian contributors other than Janos > who want to work on the package, or users who want to patch it, etc. > The sorts of questions we should try to answer: > > - does the vendoring make Debian security support harder (discussion on > -devel suggests it makes it easier) > > - everyone seems to think the level of vendoring is at best a necessary > evil; Let's not attempt to fabricate perception of
Bug#971515: kubernetes: excessive vendoring (private libraries)
Hello Dmitry, others, On Thu 01 Oct 2020 at 11:15am +10, Dmitry Smirnov wrote: > I seek your judgement regarding excessive, unnecessary and unjustifiable > vendoring of private libraries in Kubernetes package: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970717 > > Some relevant argumentation can be found in > > https://lists.debian.org/debian-devel/2020/03/msg00359.html > https://lists.debian.org/debian-devel/2020/03/msg00400.html > https://lists.debian.org/debian-devel/2020/03/msg00441.html > > In short, myself and Golang packaging team spent years on stabilising > Golang ecosystem of packaged libraries for re-use by Golang software. > > To the best of my knowledge, all packaged Golang software, regardless of its > sophistication, use some packages libraries. > Except Kubernetes, that disconnected itself from cooperation by not using any > packaged libraries, instead exclusively using only private libraries in > numbers greater than any Debian package ever used. > > As a former Kubernetes maintainer and a developer who originally introduced > Kubernetes to Debian, I know that it could be maintained with only few (or > several) private libraries at most. > > Current state of Kubernetes package is in violation of ftp-master's policy > for inclusion of new packages to Debian. I think that my message [1] is what makes you think that the package would not have got through NEW? There are a few issues tangled together here. NEW is mainly about license and DFSG compliance, and secondarily about the idea that we want to avoid accepting packages where doing so would make Debian worse, even if it would also make Debian better along other dimensions. As a simple example, we try to avoid accepting a package that is already packaged under a slightly different name, because in most cases it is worse for both users and contributors to have the same thing in the archive twice (not talking about vendoring here). In this case, the reason I wrote in [1] that I would probably have rejected the package, had I come across it in NEW, is that it seemed to me that having this package in Debian would make the archive less maintainable by contributors other than Janos who might need to work with the package. (After the discussion on -devel, I'm no longer so sure about that opinion of mine.) It's not correct to say, however, that the package "is in violation of ftpmaster's policy for inclusion of new packages". That could only be true is if the package met one of the "serious violations" listed in the REJECT-FAQ, which is basically DFSG and licensing issues, and a few obvious clangers. Instead, what we have is a situation in which there is reason to be worried about the way the package is put together, but the opinion of one FTP team member at one particular point in time carries about the same weight as the opinion of any experienced packager. In other words, I suggest that we ignore the NEW issue entirely, and just consider whether the way the package is currently put together imposes an unreasonable burden on Debian contributors other than Janos who want to work on the package, or users who want to patch it, etc. The sorts of questions we should try to answer: - does the vendoring make Debian security support harder (discussion on -devel suggests it makes it easier) - everyone seems to think the level of vendoring is at best a necessary evil; if someone wants to try to reduce the level of vendoring (as Dmitry did when he was maintainer), is the current way the package is built going to make it harder for people to work on making that sort of contribution? - are people trying to do cross-archive work going to find the packaging of kubernetes getting in their way? (e.g. other Go team members trying to update things, improve our binNMU techniques and machinery, etc.) ... and this is to be weighed against the negative impact of having kubernetes in Debian lag so seriously behind upstream such that almost no-one would want to bother installing it. Are there issues the TC should think about which do not fall under this way of looking at things? I.e., weighing the impact on people other than Janos who want to work on the package, vs. the impact on people who want recent kubernetes to be part in the archive at all? If I'm right about what the question for the TC is, I hope that Janos and Dmitry can both help us discuss it in a way which sets aside the heat which characterised the -devel thread. It is completely understandable to (a) feel very frustrated at Debian not including recent versions of a useful piece of free software; and also (b) feel very frustrated when someone chooses to accept a less-than-ideal approach as necessary when one has put a lot of time into trying to find workable alternatives. The thing is, both (a) and (b) are motivated by the same basic desire to make Debian better and more useful, so perhaps we can focus on that point of commonality. [1]
Next Meeting - October 21st at 18:00 UTC (tomorrow)
Dear Technical Committee members, Our monthly meeting is scheduled to take place tomorrow, October 21st at 18:00 UTC. We currently have one ongoing bug, that hasn't seen much traffic on our side (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=971515) Topics for our agenda are: * #971515 - kubernetes: excessive vendoring (private libraries) * Follow up from the DebConf20 discussion (we should decide what we want to do next) * Recruitment efforts Notes from previous meeting here: http://meetbot.debian.net/debian-ctte/2020/debian-ctte.2020-09-16-17.58.html Notes from the DebConf discussion here: https://salsa.debian.org/debian/tech-ctte/-/blob/master/talks/202008-pad.html I'm not sure if I'll be able to join the meeting. If I'm not there, please carry on without me. -- Regards, Marga
