Bug#727708: init system decision-making concerns
Dear all, I'm sincerely grateful to technical committee members for their dedication and relentless effort to thoroughly research and understand the issue in order to make the best decision possible. Although most arguments for and against various init systems were already presented I think I still have something to add. I apologise in advance to some who might consider my feedback to be obvious or redundant. This is the first time ever I'm sharing my concerns regarding init system for Debian. I think well-balanced decision on this subject would benefit from not being too technical. For instance due to controversial contributor's agreement Upstart is pretty much defunct project. Many contributors prefer to spend their time on something else rather than Upstart. If adopted Upstart will likely turn into a big liability for Debian. The very survival of Upstart may depend on whether we going to be involved or not. Canonical/Ubuntu would be very happy to use Debian resources for Upstart as if they succeed in selling Upstart to Debian they would be able to offload (i.e. outsource) a significant chunk of effort that they have to dedicate to Upstart development and maintenance otherwise. It is quite possible that Ubuntu might reduce their involvement to Upstart (and allow Debian to deal with problems) while they are likely to spend more of their resources formerly allocated to Upstart to contribute to other areas of added value. (IMHO the only major Ubuntu sell point is a concept of added value on top of Debian.) In my opinion Canonical/Ubuntu will benefit the most from Upstart adoption in Debian. Considering the possibility that in the future Ubuntu might abandon Upstart, Debian may end up with unwanted/obsolete init system. Since Upstart future is uncertain I fear that we might waste a lot of precious resources for Upstart and/or potentially became de-facto upstream for Upstart. IMHO from this prospective Upstart shall not be considered as alternative init system at all. Indeed I'm concerned about conflict of interests from DDs affiliated with Canonical and Ubuntu. When they advocate for Upstart I doubt they have Debian's best interests in mind. There is a danger for Debian to be overrun by outsiders or to fall under their influence even if some of them are working on both sides. Besides we can learn from OpenSUSE where Upstart was replaced with Systemd. Even without much investigation it should be fairly clear that there are good reasons not to use Upstart and to prefer something else. As for Systemd I do not fear its adoption. On the bright side it would be nice to reduce our differences with other distros in that area. Systemd may open some exciting opportunities to cooperate and join the efforts with other influential distros. Our users may benefit from feature rich init system and its adoption might make it easier for new users to switch to Debian. It doesn't look like Systemd survival will be influenced much from Debian involvement so from non-technical prospective Systemd is better for us due to strong upstream and wide(r) adoption. Of course there are concerns regarding integration between Systemd and GNOME but that's a different issue and perhaps not a major one as long as we use GNOME as default desktop environment. Besides GNOME already became notorious for being intrusive (e.g. it depends on pulseaudio etc.). Also I'd like to notice that shopping for most feature-rich init system might be not our goal after all. OpenRC may be the safest choice that might satisfy majority of developers as it appears to have the least number of objections. I have impression that OpenRC have far less passionate opponents than Systemd. Finally I'm sure everybody is already getting exhausted by long debates about this topic. At this point it might be tempting to approach on decision, any decision, to put this to end. This is a way to make mistakes of judgement. Unless there is a rush we all need to slow down and perhaps even take a break for several weeks to clear our heads and make a balanced, well thought decision. Taking break may be beneficial for the quality of decision making. -- Cheers, Dmitry Smirnov GPG key : 4096R/53968D1B --- Odious ideas are not entitled to hide from criticism behind the human shield of their believers' feelings. -- Richard Stallman signature.asc Description: This is a digitally signed message part.
Bug#970717: kubernetes: abuse of "embedded code copies" | hundreds of vendored libraries
Source: kubernetes Version: 1.17.4-1 Severity: serious X-Debbugs-CC: debian-ctte@lists.debian.org As discussed in debian-devel [1], Kubernetes package abuses Debian practices and Golang team policies by needlessly vendoring hundreds(!) of libraries, most of which are available in Debian. For a complex package like Kubernetes, _some_ strategic vendoring would be understandable for practical reasons. But not everything. Maintainer circumvented packaging practices and introduced re-packaged Kubernetes in a state that would have never been accepted by ftp-masters. Please consider removing redundant libraries from "vendor". In the current state, the package is unsuitable for "stable". [1]: https://lists.debian.org/debian-devel/2020/03/msg00359.html See also: https://wiki.debian.org/EmbeddedCopies -- Best wishes, Dmitry Smirnov --- We often refuse to accept an idea merely because the way in which it has been expressed is unsympathetic to us. -- Friedrich Nietzsche signature.asc Description: This is a digitally signed message part.
Bug#971515: kubernetes: excessive vendoring (private libraries)
Package: tech-ctte Severity: normal X-Debbugs-CC: o...@debian.org Control: block 970717 by -1 Dear Technical Committee, Apologies that we were not able to resolve the problem without your help. I seek your judgement regarding excessive, unnecessary and unjustifiable vendoring of private libraries in Kubernetes package: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970717 Some relevant argumentation can be found in https://lists.debian.org/debian-devel/2020/03/msg00359.html https://lists.debian.org/debian-devel/2020/03/msg00400.html https://lists.debian.org/debian-devel/2020/03/msg00441.html In short, myself and Golang packaging team spent years on stabilising Golang ecosystem of packaged libraries for re-use by Golang software. To the best of my knowledge, all packaged Golang software, regardless of its sophistication, use some packages libraries. Except Kubernetes, that disconnected itself from cooperation by not using any packaged libraries, instead exclusively using only private libraries in numbers greater than any Debian package ever used. As a former Kubernetes maintainer and a developer who originally introduced Kubernetes to Debian, I know that it could be maintained with only few (or several) private libraries at most. Current state of Kubernetes package is in violation of ftp-master's policy for inclusion of new packages to Debian. Thank you. This matter is not urgent. -- Regards, Dmitry Smirnov --- We do our friends no favors by pretending not to notice flaws in their work, especially when those who are not their friends are bound to notice these same flaws. -- Sam Harris signature.asc Description: This is a digitally signed message part.
Bug#971515: kubernetes: excessive vendoring (private libraries)
On Thursday, 22 October 2020 2:16:16 AM AEDT Sean Whitton wrote: > I think that we can all agree with everything you've written about the > reasons why packaging components separately is better. Thank you. > The problem is > that in this case the choice seems to be between not having recent > Kubernetes in Debian at all, or giving up on some of those advantages. Ultimately it is about maintainers comfort then. Because there is a compromise to un-vendor most libraries (one by one) and keep some/few strategically vendored, when a system library can not be used (determined case by case). Examples of this approach are numerous: syncthing, consul, nomad, vault, docker.io, runc, gitlab-runner, libpod, buildah, singularity-container and _most_ Golang packages. Kubernetes had no maintainer - that was the problem. Having maintainer who does not care for shared libraries because he does not want to be bothered is a different problem. -- Kind regards, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- In times of universal deceit, telling the truth becomes a revolucionary act. -- George Orwell signature.asc Description: This is a digitally signed message part.
Bug#971515: kubernetes: excessive vendoring (private libraries)
On Wednesday, 21 October 2020 8:56:53 PM AEDT Felix Lechner wrote: > How is the second one inferior, please? I think it includes a crucial > missing feature (co-installable versions). To reduce maintainers burden, you want maintainers to look after not one but multiple versions of libraries. This is a contradiction (oxymoron). How many releases (current + obsolete ones) one can meaningfully support? -- Best wishes, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- The cure for the evils of democracy is more democracy. -- H. L. Mencken signature.asc Description: This is a digitally signed message part.
Bug#971515: Request for security team input on kubernetes TC bug
On Wednesday, 28 October 2020 6:13:41 AM AEDT Moritz Mühlenhoff wrote: > The bigger issue here (independent of the whole vendoring aspect) is > how kubernetes can be supported in a stable release to begin with. > This was raised by Shengjing Zhu in #959685 before. If Kubernetes can be supported then such support will be done by upstream, but with extraordinary amount of dependencies (and upstream reluctance to manage them), I have very low confidence and low expectations for quality of such support. The problem primarily is that Kubernetes vendors hundreds of dependencies representing a large support surface. Effectively it is "#include world" (or vendor world) situation. And when it comes to problems in 3rd party vendored libraries, it iw worth remembering that Kubernetes don't own them. > This leaves Debian with two options: > * Keep it out of a stable release and accept that it's good enough > if people just install whatever deb they currently find in testing/sid > (works out well enough for most given that blob nature of Go!) IMHO this is the most reasonable option and perhaps the only viable one. > * Follow a scheme similar to Firefox ESR where in case of a security > the update either happens to the latest minor release of > the current branch or if that has stopped, happens to the next > major release. I think Kubernetes have many more vendored 3rd party libraries than Firefox. IMHO we can not expect the same level of confidence for Kubernetes... -- Best wishes, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- You have to start with the truth. The truth is the only way that we can get anywhere. Because any decision-making that is based upon lies or ignorance can't lead to a good conclusion. -- Julian Assange, 2010 signature.asc Description: This is a digitally signed message part.
Bug#971515: kubernetes: excessive vendoring (private libraries)
t easier) > > - everyone seems to think the level of vendoring is at best a necessary > evil; Let's not attempt to fabricate perception of consensus please. "Everyone" is more like "everyone who noticed the discussion and cared to express an opinion" at best, and even those were not all in agreement... > if someone wants to try to reduce the level of vendoring (as > Dmitry did when he was maintainer), is the current way the package is > built going to make it harder for people to work on making that sort > of contribution? If Debian only cared about maintainers' convenience or reducing maintainers efforts then Debian would not be what it is now. We favour technical elegance often in expense of maintainers' comfort. > Are there issues the TC should think about which do not fall under this > way of looking at things? I.e., weighing the impact on people other > than Janos who want to work on the package, vs. the impact on people who > want recent kubernetes to be part in the archive at all? Is Debian ecosystem of packaged reusable libraries worth caring about? If so then why grant exception to one particular package? We have several (or more) sophisticated Golang packages using hundreds of packaged libraries. In the early days of packaging Kubernetes we did not even have most components packaged and I've been spending most effort on packaging, introducing and stabilising dependency libraries. These days the major work has already been done and the argument for monolithic vendoring is much weaker. > If I'm right about what the question for the TC is, I hope that Janos > and Dmitry can both help us discuss it in a way which sets aside the > heat which characterised the -devel thread. It is completely > understandable to (a) feel very frustrated at Debian not including > recent versions of a useful piece of free software; and also (b) feel > very frustrated when someone chooses to accept a less-than-ideal > approach as necessary when one has put a lot of time into trying to find > workable alternatives. There is a compromise and a delicate balance between those concerns. Sometimes it may be necessary to vendor few libraries strategically. But most packaged libraries can (and should) be reused. It requires some effort, knowledge and experience. Packaging software "the Debian way" may be harder but it have benefits precisely because we do things differently. There is little value to build Kubernetes exactly the same way how upstream builds it because they already provide binary releases. Kubernetes maintainer wants to disengage from the process and do a quick but sloppy work that does not involve cooperation and mutual maintenance of individually packaged dependencies. Debian way of packaging Golang software have distinct advantages over upstream bundle. In Debian, maintainers own dependency tree, helping each other to maintain ecosystem of reusable libraries. Golang software benefits from compartmentalisation because vendored libraries are not tested on build. That is, Kubernetes is build from large number of untested vendored libraries. Individually packaged re-usable libraries are usually running their test suites on different architectures, exposing bugs with greater visibility. Another argument for using packages libraries is distro-consistency of used components. Case of Kubernetes over-vendoring is obvious. Making exception for Kubernetes threatens to open pandora's box and destroy the very identity of Debian. -- All the best, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- Within any important issue, there are always aspects no one wishes to discuss. -- George Orwell signature.asc Description: This is a digitally signed message part.
Bug#971515: kubernetes: excessive vendoring (private libraries)
Hi Felix, On Wednesday, 21 October 2020 12:52:40 PM AEDT Felix Lechner wrote: > > We favour technical elegance often in expense of maintainers' comfort. > > Is our approach really either one of those? I think our response to > the vendoring explosion is at odds with the trends in many languages. IMHO we are managing quite admirably. Basically, to me it looks like you don't want to maintain Kubernetes the way we maintain heavy Golang packages. You would have to learn to un-vendor many libraries. Yes, at first there will be a significant effort but then it will become easier. "Too many vendored libraries to use packaged libs" is a poor excuse. We have been dealing with "explosion" for years already. Tools like "dh-make- golang" are helpful to generate initial packaging for new Golang libraries in a semi-automatic manner. FTP-masters are usually quite effective with processing of NEW packages. Look how many packages we already have: https://qa.debian.org/developer.php?login=pkg-go-maintainers%40lists.alioth.debian.org+team%2Bpkg-go%40tracker.debian.org > It's time to retool. At the two ends of the solution spectrum, I see > > 1. Fully vendored source packages; or > 2. A packaging system that allows different vendor versions to > co-exist. Personally I'm not satisfied with either of those inferior proposals. Besides un-vendoring libraries can prevent some CVE issues as well. > Either one allows dependent sources to consume whichever versions they > require, but in my view solution (2) is otherwise superior---provided > that the packaging process is automated. (A language's build system > also has to distinguish the installed versions.) For each language so > affected, could we make (2) our goal, and allow (1) until then? IMHO tools have to come first (if ever). You are advocating for disruptive changes therefore your proposed theoretical solutions have to be available as a proof of concept for review. In the meantime you could follow the established practice that is demonstrated to be working on several packaged heavy Golang applications. If we tolerate full vendoring now, because "there is no better way" yet, then there will be no better way for sure. For now using packaged system libraries whenever possible is the best way. -- Kind regards, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- Those who disdain wealth as a worthy goal for an individual or a society seem not to realize that wealth is the only thing that can prevent poverty. -- Thomas Sowell signature.asc Description: This is a digitally signed message part.
Bug#971515: kubernetes: excessive vendoring (private libraries)
On Friday, 23 October 2020 4:20:37 AM AEDT Shengjing Zhu wrote: > However kubernetes is also a library, and the maintainer doesn't provide > it. It becomes headache for other maintainers. Yes we have to vendor "k8s.io/" all the time in multiple packages but I doubt it would be of help if Kubernetes provide a "-dev" library because it is notoriously unstable with frequent breaking changes. Ideally that's how it should be done but, pragmatically speaking, Kubernetes can never migrate to "testing" hence everything that depends on it will be unsuitable for release. > The way the kubernetes maintainer chooses, is making others difficult. > Every package that uses kubernetes as a library, has to keep a local copy > of the source. The partial solution might be to introduce another source package only with portion of Kubernetes that provides a shared library but that will require proper dependency management which is being discussed here. > But I hope Janos can cooperate with the Go packaging team to make Debian, > especially the Go packages in Debian more maintainable. Yes indeed. We should not need to argue that cooperation is important. Cooperation should always be a default mode of operation. -- Kind regards, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- Hay smells different to lovers and horses. -- Stanisław Jerzy Lec signature.asc Description: This is a digitally signed message part.
Bug#971515: Request for security team input on kubernetes TC bug
On Thursday, 22 October 2020 2:22:11 AM AEDT Sean Whitton wrote: > The TC are being asked about src:kubernetes, and it would be good to > hear from you about whether and how security support is a relevant > consideration in determining whether the level of vendoring in that > package is acceptable. Could you take a look at #971515 and perhaps > give us some input, please? I'm not a member of security team but I want to share my prospective on this. To us what matters is what we can do about vulnerability in situation where the problem has been identified and fixed upstream, in the particular library. When a library is packaged separately, this is a matter of patching/updating a particular library then re-building depending packages. With vendored libraries, we have no control as patching arbitrary versions of all instances of vendored library is a very bad option that is practically unsustainable. Even tracking of vendored libraries is difficult. So in case of vendored libraries we can rely on "Oracle model" when upstream releases an update of a software where vendored library is patched so we ship the whole bundle. This is the worst option because we have no control and because we have to rely on faith in good upstream maintenance. But here lies the problem. Kubernetes have bad dependency hygiene and poor abolity to control their enormous vendor mess (burden). With hundreds of vendored libraries they have large surface for problems and limited ability to track and respond. Just like we know from experience, an attempt to update a vendored library may lead to FTBFS and that's why upstream is reluctant to make changes to vendored libraries, unless when they absolutely have to. Kubernetes can not effectively address issues in "vendor/" space. So my conclusion about security support of upstream Kubernetes is that it is can not be done effectively in either case. There are greater chances for meaningful security maintenance when most dependency libraries are un- vendored. With most libraries vendored, there is less security maintenance burden to us, but the outcome is no better. -- All the best, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- No snowflake in an avalanche ever feels responsible. -- Stanisław Jerzy Lec signature.asc Description: This is a digitally signed message part.
Bug#971515: CTTE decision on "kubernetes: excessive vendoring (private libraries)"
On Saturday, 23 January 2021 3:18:52 AM AEDT Shengjing Zhu wrote: > The real complex things are, dealing license and copyright and *NEW* queue. > If this TC decision is that we just trust what upstream say, then why not > just unvendor them. Then many pieces of libraries can be reused by others. Even without packaging new libraries, we could do so much better un-vendoring some of already packaged ones, even if maintainer focuses only on those that are _easy_ to un-vendor. Vendoring _everything_ is crazy, unnecessary, irrational... -- Cheers, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- People are rarely grateful for a demonstration of their credulity. -- Carl Sagan signature.asc Description: This is a digitally signed message part.
Bug#971515: Status as of last tech-ctte meeting
On Friday, 20 November 2020 4:35:23 PM AEDT Elana Hashman wrote: > Kubernetes is a very large and active project. It has about 600 > members,[0] 1000 voters,[1] 100 committers (which I define as members of > the milestone-maintainers team),[2] and over 50,000 contributors.[3] The > project has its own security[4] and release teams,[5] and the release > team includes a software licensing team. > [...] > As such, the scale of Kubernetes is similar to that of Debian itself. I was too slow to reply to this email but I'll leave some comments for the record. Comparison of Kubernetes' team size to Debian is misleading and the comparison is not in favour of Kubernetes. Debian is compartmentalised into relatively small (mostly) well coordinated teams. Kubernetes - the monolithic project - suffers from well known problem with coordinating large teams. Frederick Brooks described the problem well in his "The Mythical Man-Month" book. Regarding number of contributors, what is strength for Debian is a weakness for Kubernetes. Kubernetes bug tracker shows the scale of the problem and I don't have impression that the team is big enough to handle bug reports effectively or have them under control. It was my observation that vendoring problems in Kubernetes are worse than everything I've seen in other projects. Is that despite the size of the team or because of it? The particular example that I did not mention enough [1] (and it is not the only one) show how little that remarkably large team cares about dependency hygiene: a trivial library update with a patch was not done in a few years time and all that was produced by the large team are excuses for not doing the update, only to finally perform it in the end as advised. Who can have confidence in the Kubernetes dependency management after that? [1]: https://github.com/kubernetes/kubernetes/issues/27543 Of course the problem have little to do with Kubernetes. The "use world" situation with over-dependency on large number of 3rd party libraries is a challenging one especially with historically poor Golang approach to dependency management a.k.a. "vendoring". I'm just saying that Kubernetes is not all that special in regards to vendoring because of the team size. > Kubernetes as an upstream project is much better resourced than the > single downstream maintainer in Debian. Nobody argues with that. But it is only "single maintainer" with monolitic packaging - a case that I'm arguing against. Golang team is strong and its capacity is impressive. Packages that use dependency libraries are always team-maintained. It is the same situation as with Kubernetes upstream where much of the 3rd party code is not maintained by Kubernetes team. > The resourcing and scale of the Kubernetes project gives us better > assurances that upstream has done due diligence for dependency > management. IMHO upstream demonstrated bad and terrible attitude with dependency management. But apparently nobody here considered that argument... :( -- Kind regards, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- We often refuse to accept an idea merely because the way in which it has been expressed is unsympathetic to us. -- Friedrich Nietzsche --- COVID-19: The Trouble With PCR Tests https://swprs.org/the-trouble-with-pcr-tests/ signature.asc Description: This is a digitally signed message part.
Bug#971515: marked as done (kubernetes: excessive vendoring (private libraries))
I'm disappointed with TC's decision. IMHO it is lacking depth of judgement. The original problem I reported is absolutism - "vendor everything". And TC apparently considered only that without trying to find the balance. What are the reasons for vendoring stable well maintained library like Logrus? None. That system library can by used easily with hardly any burden to maintenance. Same can be said probably about ~100 other libraries. We could do so much better with recommending at least to un-vendor trivial cases of 3rd party code... > The Committee declines to overrule the maintainer and accepts the level > of vendoring used in the 'kubernetes' source package. We also decline > to intervene in bugs requesting that vendored components in the > 'kubernetes' source package be installed in binary packages in such a > way that other packages can make use of them. > Our consensus is that Kubernetes ought to be considered special in the > same way that Firefox is considered special -- we treat the package > differently from most other source packages because (i) it is very large > and complex, and (ii) upstream has significantly more resources to keep > all those moving parts up-to-date than Debian does. That is based on misleading Elana's comparison that I've answered (regretfully too late) in the other email... > Our most basic reason for this point of view is that given how much > fewer resources we have than upstream Kubernetes, it is not feasible for > Kubernetes to make it into a stable release of Debian unless we take an > approach like this one. IMHO Kubernetes is not suitable for "stable" either way. > The same goes for the possibility of providing > security support. And given that a strategy for security support is > available, we do not see any reason why having the Kubernetes bundle in > a stable release alongside other copies of its vendored dependencies is > worse than not having Kubernetes in a stable release at all. It is not suppose to be "all or nothing" approach. We are talking about 3rd party code, not maintained by Kubernetes. At least some of it is perfectly feasible to take from packaged libraries instead of messy upstream bundle. > It should be noted that we think the greater resources of upstream Let's not praise greater resources unnecessarily. Upstream don't care about good versioning practices and even automatically closes bugs for inactivity without addressing them (e.g. https://github.com/kubernetes/ kubernetes/issues/17077). > is > relevant not only to keeping on top of patches and security fixes, but > also to license compliance. It is our belief that there is no reason at > the present time to be concerned that non-DFSG material would find its > way into the package, because Kubernetes upstream care about ensuring > that all vendored dependencies are free software, and they have the > resources to check. Since when we are delegating DFSG compliance to upstream based on faith?? FYI in case of "cadvisor" - a required Kubernetes component, DFSG compliance was difficult to achieve due to pre-minified source-less 3rd party javascript files... It was a hell of an effort to get Kubernetes through NEW process. Apparently I should not have bothered with all that because upstream is so "good" (it wasn't). Did you ask ftp-masters opinion/position on that?? Are they OK with volume of needless garbage in vendored bundle? Would they allow that before TC's conclusion? > This could change in the future and it may not be > true for other upstreams. I've spent about a year of effort for introducing Kubernetes to Debian - yes, I had that much faith in Kubernetes but ultimately the effort lead to disappointment in the technology and loss of confidence in upstream's ability to maintain the project. But in the course of the effort many new libraries were introduced as packages, the Debian Golang ecosystem was stabilised and we've gained a valuable experience with maintaining a complex Golang software. The hard work was accomplished and it was only the matter or maintaining Kubernetes properly... The TC's decision tells me that the effort was largely wasted because it was "too hard" to do things this way... Naturally the sloppy way is so much easier... :( -- Best wishes, Dmitry Smirnov GPG key : 4096R/52B6BBD953968D1B --- There are occasions when it pays better to fight and be beaten than not to fight at all. -- George Orwell --- "Increased Risk of Noninfluenza Respiratory Virus Infections Associated With Receipt of Inactivated Influenza Vaccine". -- https://pubmed.ncbi.nlm.nih.gov/22423139/ signature.asc Description: This is a digitally signed message part.
