Bug#971515: Status as of last tech-ctte meeting

2021-01-21 Thread Dmitry Smirnov
On Friday, 20 November 2020 4:35:23 PM AEDT Elana Hashman wrote:
> Kubernetes is a very large and active project. It has about 600
> members,[0] 1000 voters,[1] 100 committers (which I define as members of
> the milestone-maintainers team),[2] and over 50,000 contributors.[3] The
> project has its own security[4] and release teams,[5] and the release
> team includes a software licensing team.
> [...]
> As such, the scale of Kubernetes is similar to that of Debian itself.

I was too slow to reply to this email but I'll leave some comments for the
record.

Comparison of Kubernetes' team size to Debian is misleading and the
comparison is not in favour of Kubernetes.

Debian is compartmentalised into relatively small (mostly) well coordinated
teams. Kubernetes - the monolithic project - suffers from well known
problem with coordinating large teams. Frederick Brooks described the
problem well in his "The Mythical Man-Month" book. 
Regarding number of contributors, what is strength for Debian
is a weakness for Kubernetes.

Kubernetes bug tracker shows the scale of the problem and I don't have
impression that the team is big enough to handle bug reports effectively or
have them under control.

It was my observation that vendoring problems in Kubernetes are worse than
everything I've seen in other projects.
Is that despite the size of the team or because of it?

The particular example that I did not mention enough [1] (and it is not the 
only one) show how little that remarkably large team cares about dependency
hygiene: a trivial library update with a patch was not done in a few years
time and all that was produced by the large team are excuses for not doing
the update, only to finally perform it in the end as advised.
Who can have confidence in the Kubernetes dependency management after that?

[1]: https://github.com/kubernetes/kubernetes/issues/27543

Of course the problem have little to do with Kubernetes. The "use world"
situation with over-dependency on large number of 3rd party libraries is a
challenging one especially with historically poor Golang approach to
dependency management a.k.a. "vendoring". I'm just saying that Kubernetes
is not all that special in regards to vendoring because of the team size.



> Kubernetes as an upstream project is much better resourced than the
> single downstream maintainer in Debian.

Nobody argues with that. But it is only "single maintainer" with monolitic
packaging - a case that I'm arguing against.
Golang team is strong and its capacity is impressive.
Packages that use dependency libraries are always team-maintained.

It is the same situation as with Kubernetes upstream where
much of the 3rd party code is not maintained by Kubernetes team.


> The resourcing and scale of the Kubernetes project gives us better
> assurances that upstream has done due diligence for dependency
> management.

IMHO upstream demonstrated bad and terrible attitude with dependency
management. But apparently nobody here considered that argument... :(

-- 
Kind regards,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

We often refuse to accept an idea merely because the way in which it has
been expressed is unsympathetic to us.
-- Friedrich Nietzsche

---

COVID-19: The Trouble With PCR Tests
https://swprs.org/the-trouble-with-pcr-tests/


signature.asc
Description: This is a digitally signed message part.


Bug#971515: Status as of last tech-ctte meeting

2020-11-22 Thread Tollef Fog Heen
]] Philip Hands 

> Tollef Fog Heen  writes:
> 
> > ]] Shengjing Zhu 
> >
> >> Firefox is special, since for Debian desktop users, they need a browser. Is
> >> kubernetes same here?
> >
> > FWIW, the lack of Kubernetes or a similar orchestration platform (mesos,
> > nomad, docker swarm) in stable has been keeping back development of the
>   ^^
> 
> Do you really mean stable here?  I had gained the impression that there
> was no prospect of Kubernetes getting into stable, regardless of the
> details of how it ends up being packaged.  Have I misunderstood?

I do mean stable.  I was making a narrow comment about how special or
not Kubernetes is, I don't know if the goal is for it to reach stable or
not.

(My comment above was also slightly imprecise, I've since learned that
Docker Swarm is in Debian, in the docker.io package, thanks to Shengjing
Zhu who pointed it out in private email.)

-- 
Tollef Fog Heen



Bug#971515: Status as of last tech-ctte meeting

2020-11-21 Thread Philip Hands
Tollef Fog Heen  writes:

> ]] Shengjing Zhu 
>
>> Firefox is special, since for Debian desktop users, they need a browser. Is
>> kubernetes same here?
>
> FWIW, the lack of Kubernetes or a similar orchestration platform (mesos,
> nomad, docker swarm) in stable has been keeping back development of the
  ^^

Do you really mean stable here?  I had gained the impression that there
was no prospect of Kubernetes getting into stable, regardless of the
details of how it ends up being packaged.  Have I misunderstood?

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature


Bug#971515: Status as of last tech-ctte meeting

2020-11-21 Thread Tollef Fog Heen
]] Shengjing Zhu 

> Firefox is special, since for Debian desktop users, they need a browser. Is
> kubernetes same here?

FWIW, the lack of Kubernetes or a similar orchestration platform (mesos,
nomad, docker swarm) in stable has been keeping back development of the
next generation way of handling Debian services, since that will need a
reasonable container orchestation platform to build on.  The lack of a
platform is not the only reason for the delay, but it surely hasn't
helped either.

-- 
Tollef Fog Heen, speaking for himself, but as a DSA member



Bug#971515: Status as of last tech-ctte meeting

2020-11-20 Thread Jonathan Carter
Hi Josh

On 2020/11/20 01:30, Josh Triplett wrote:
> In particular, with my upstream Rust/Cargo hats on, I would love to work
> with you and others on questions of what software packaging could look
> like, and how to maintain the quality and curation *and* package
> availability of Debian in collaboration with other ecosystems of package
> and dependency management.
> 
> Other potentially interesting questions: what are the assumptions that
> go into our current tradeoffs about shared libraries vs static
> libraries, and are those still the correct tradeoffs in all cases?

There is already a lot of text to read on this bug report, please try to
refrain from adding more off-topic content to it.

thanks,

-Jonathan



Bug#971515: Status as of last tech-ctte meeting

2020-11-19 Thread Josh Triplett
On Wed, Nov 18, 2020 at 10:26:21AM -0800, Russ Allbery wrote:
> There are a lot of fascinating edge cases and precedents and philosophical
> questions about the function of a distribution here, and I think they
> rightfully attract a lot of energy, but I'm worried this is at the cost of
> losing sight of the core functionality provided by Debian.  Because that
> functionality is currently working, the people who are benefiting from it
> don't know to weigh in.
[...]
> As a Debian developer, I appreciate the arguments about vendoring and the
> desire to maintain Go libraries and the pride that we take in our work of
> really understanding software and packaging it properly.
> 
> However, as a Debian *user* of the kubernetes-client, I have to say that I
> do not care in the slightest.  The older, unvendored kubernetes-client
> package worked great.  The new, heavily vendored kubernetes-client package
> works great.  Both do exactly what I want, and I don't care at all, as a
> user, which is in Debian.  But if the package were removed from Debian, I
> would be really unhappy.  And if Helm and Argo CD were packaged for
> Debian, I would be much happier, so if unvendoring them is the obstacle,
> as a user I'm opposed to unvendoring!
> 
> I want to push back a bit against the feeling that some things are
> ill-suited for how Debian has historically packaged software and therefore
> maybe Debian isn't the place for them, and we should instead ask people to
> manage them outside of Debian (but somehow make this easy).
> 
> First, while I appreciate and cheer Phil and Sean's optimism, there have
> been a lot of plans in Debian historically to make something that isn't a
> package easy to build and install, and they have basically never worked.
> The way Debian makes something easy to build and install is by making it a
> package.  That's what our entire project is designed around, and I'm
> dubious that we're going to be able to reproduce those properties with
> something that isn't a package.
> 
> Second, the point of Debian at the end of the day is that I can install it
> on a computer and use it to get things done.  The details we're discussing
> here are important and work towards making Debian maintainable and
> sustainable and to ensure that a quality bar has been met in terms of
> security and legality and licensing, but I think it's important that they
> are also means to an end and the end is not security and licensing per se.
> We're not making a random collection of relatively secure software; we're
> giving people programs that they can run while keeping them secure.  We're
> not just a classification system for what software is free; we're giving
> people software they can use while ensuring that all of it is free.  I
> think it's worth occasionally stepping back and dwelling on that thought
> for a moment and making sure we're picking the right strategy for meeting
> our quality bar that allows us to still achieve the mission.
> 
> This is particularly true for something like vendoring or the avoidance of
> vendoring, which is not a core mission.  It's not in the social contract,
> and it's not a DFSG principle.  It's a hard-won and very valuable piece of
> experience with how best to sustainably make a distribution... but one
> that was hard-won in the era of C programs and shared libraries and that
> has generalized admirably to Perl and Python.  It may generalize to other
> languages and other mechanisms for distributing software.  It may not!  If
> it doesn't, that's significant because it's such a deeply-engrained part
> of our culture, but it's *not* a breach of our fundamental project goals.
> We can consider new approaches here without becoming untethered, and
> indeed may be able to achieve our primary goal *better* by expanding the
> scope of software that we can include in the distribution and that can
> therefore just work.
> 
> I think there's a bit of a crisis of confidence in Debian because of how
> much larger the free software ecosystem is now than it was when the
> project started, how far away from doing everything through a distribution
> a lot of developers have moved, and how many resources are flooding into
> other areas that we have difficulty keeping pace with.  One of the natural
> reactions to that crisis of confidence is to pull back from these new and
> difficult and unfamiliar areas and decrease scope to the things we know
> we're really good at: packaging primarily C, C++, Perl, and Python code.
> And that is a valid strategy; it's okay to just keep being very good at
> something one is already very good at.  But I think in the long term that
> means Debian becomes something much different than it historically has
> been.  It means Debian would become a base on which other people would
> build things, rather than a comprehensive collection of tools that covers
> all the incidental things you need from a computer, and where people need
> only reach outside of Debian for 

Bug#971515: Status as of last tech-ctte meeting

2020-11-19 Thread Shengjing Zhu
On Wed, Nov 18, 2020 at 10:26:21AM -0800, Russ Allbery wrote:
> 
> I maintain a bunch of Kubernetes clusters as part of my job.  Those
> clusters are run by other people (cloud providers, data centers, etc.).  I
> need clients to talk to those clusters.  When I first started working with
> Kubernetes, I realized I needed a client, worried about how much of a pain
> that would be, did an apt-cache search for kubernetes, found
> kubernetes-client, breathed a sigh of relief, and ran apt install
> kubernetes-client.  I have subsequently not given Kubernetes clients a
> second thought.
> 

In priciple, the kubectl compatibility matrix says it only supports one minor
version (older or newer). When you working with many clusters, the cluster
version may not be compatible with the kubectl version in Debian stable.

PS, in practice, the basic function of kubectl doesn't change much, and new
kubectl with old cluster seems to work.

As a user of upstream kubectl package user, I'm quite happy with it. I can
always install the latest, or any old version. They keep all old versions in
their apt repo. And with the nature of static link, their repo works well
other parts of the system, regardless I'm running Debian oldstable, stable,
or unstable.

IMO, we should admit that current Debian way doesn't fit well for such
softwares.


signature.asc
Description: PGP signature


Bug#971515: Status as of last tech-ctte meeting

2020-11-19 Thread Ian Jackson
Russ Allbery writes ("Bug#971515: Status as of last tech-ctte meeting"):
> [much stuff]

Oh, wow, Russ.  Thank you very much.  What an excellent piece of
writing.  I agree entirely with all of it.

Ian.

(And I speak as someone who thinks that this newfangled "vendor
everything" and "just download that shit from the internet" approach
is hideously bad engineering practice.)

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.



Bug#971515: Status as of last tech-ctte meeting

2020-11-18 Thread Russ Allbery
This is an excellent discussion and a good summary.  Thank you, Gunnar!

Gunnar Wolf  writes:

> There is a course of action that's unlikely to leave everybody happy,
> but is worth considering: Phil said:

> Phil>
> that makes it seem like what we actually need is a decent way of
> letting our users install the upstream binaries, rather than
> trying to force those packages into Debian as a special case

> Sean agreed with him; probably something as K8S due to its nature is
> not suited for Debian inclusion (but we'd still have to ensure it's
> easy to build and install, not from packages). But the quesion is
> where to draw the line:

I want to explicitly state my concern with this approach, speaking as one
of a community that I think may have been a bit underrepresented in this
discussion to date: a user of the kubernetes-client package in Debian.

There are a lot of fascinating edge cases and precedents and philosophical
questions about the function of a distribution here, and I think they
rightfully attract a lot of energy, but I'm worried this is at the cost of
losing sight of the core functionality provided by Debian.  Because that
functionality is currently working, the people who are benefiting from it
don't know to weigh in.

I maintain a bunch of Kubernetes clusters as part of my job.  Those
clusters are run by other people (cloud providers, data centers, etc.).  I
need clients to talk to those clusters.  When I first started working with
Kubernetes, I realized I needed a client, worried about how much of a pain
that would be, did an apt-cache search for kubernetes, found
kubernetes-client, breathed a sigh of relief, and ran apt install
kubernetes-client.  I have subsequently not given Kubernetes clients a
second thought.

Not having to think about things like that is exactly the point of Debian.
It's possibly the highest standard of success for a distribution.

By comparison, I also use Helm and Argo CD as part of my job.  Helm has a
snap package that's kind of maintained and sort of works but kept being
irritating plus I had to remember to upgrade a different thing.  Argo CD
is available only as a binary download from its release page and therefore
I never remember to upgrade it and run into weird problems and then have
to go download new versions.  The experience is annoying and irritating
and reminds me of when my primary working system was running Solaris.

As a Debian developer, I appreciate the arguments about vendoring and the
desire to maintain Go libraries and the pride that we take in our work of
really understanding software and packaging it properly.

However, as a Debian *user* of the kubernetes-client, I have to say that I
do not care in the slightest.  The older, unvendored kubernetes-client
package worked great.  The new, heavily vendored kubernetes-client package
works great.  Both do exactly what I want, and I don't care at all, as a
user, which is in Debian.  But if the package were removed from Debian, I
would be really unhappy.  And if Helm and Argo CD were packaged for
Debian, I would be much happier, so if unvendoring them is the obstacle,
as a user I'm opposed to unvendoring!

I want to push back a bit against the feeling that some things are
ill-suited for how Debian has historically packaged software and therefore
maybe Debian isn't the place for them, and we should instead ask people to
manage them outside of Debian (but somehow make this easy).

First, while I appreciate and cheer Phil and Sean's optimism, there have
been a lot of plans in Debian historically to make something that isn't a
package easy to build and install, and they have basically never worked.
The way Debian makes something easy to build and install is by making it a
package.  That's what our entire project is designed around, and I'm
dubious that we're going to be able to reproduce those properties with
something that isn't a package.

Second, the point of Debian at the end of the day is that I can install it
on a computer and use it to get things done.  The details we're discussing
here are important and work towards making Debian maintainable and
sustainable and to ensure that a quality bar has been met in terms of
security and legality and licensing, but I think it's important that they
are also means to an end and the end is not security and licensing per se.
We're not making a random collection of relatively secure software; we're
giving people programs that they can run while keeping them secure.  We're
not just a classification system for what software is free; we're giving
people software they can use while ensuring that all of it is free.  I
think it's worth occasionally stepping back and dwelling on that thought
for a moment and making sure we're picking the right strategy for meeting
our quality bar that allows us to still achieve the mission.

This is particularly true for something like vendoring or the avoidance of
vendoring, which is not a core mission.  It's not in 

Bug#971515: Status as of last tech-ctte meeting

2020-11-18 Thread Sean Whitton
Hello all,

On Wed 18 Nov 2020 at 11:36AM -06, Gunnar Wolf wrote:

> So... It's not like we reached a conclusion, but I do feel the meeting
> was interesting and the discussion very much worthy. Yes, this will
> surely end up in reinforcing the notion that the Technical Committee
> is a slow and bureaucratic beast :-) But... well, I'll stop
> apologizing. Only some minutes to go before the meeting, and I want to
> give the rest of the Committee the opportunity to check if I didn't
> misrepresent them or skip too much of their opinion.

Thank you for this summary.

At today's meeting one point which we thought was missing from this
summary was that no-one on the committee has any appetite for overruling
the package maintainer, so it is very unlikely that will be the outcome
of this bug.

-- 
Sean Whitton


signature.asc
Description: PGP signature


Bug#971515: Status as of last tech-ctte meeting

2020-11-18 Thread Gunnar Wolf
Hello world,

When we had our last tech-ctte meeting, 2020.10.21¹, I volunteered to
write up a summary of our positions regarding this bug. Then... Well,
life happened, and I have not had the time to sit down and write until
today -- A couple of hours before our next meeting. Several other
discussions have happened as well, most notably, the article by
Jonathan Corbet on this issue in LWN².

¹ http://meetbot.debian.net/debian-ctte/2020/debian-ctte.2020-10-21-17.58.html
  
http://meetbot.debian.net/debian-ctte/2020/debian-ctte.2020-10-21-17.58.log.html
² https://lwn.net/Articles/835599/

# Vendoring: Impedance mismatch with our long-established
# practices/traditions
##

I believe the issue of vendoring to be central to the discussion of
what Debian is, and what its role should be. We are very lucky to have
proponents of both sides of this issue in the Committee, and I'll try
to keep my ideas as centered as possible (of course, disclaimer: I do
hold a position, I really do not like vendoring; we have the luck of
having Elana in the ctte, as she is a developer of the affected
package, and Sean, who is a Debian Policy editor).

Elana started off with a very important and true point:

Elana>
I think this bug was sort of inevitable. Debian policy cruft is
clashing with how people actually build and distribute software
these days. we have an appeal to override a maintainer on the
basis of "policy" and the "Debian way being superior" without any
real technical examination of the merits of "the Debian way"

We are quite far from 1996, yes, and many languages have for a long
time delivered their own packaging systems, "freeing" the users from
the need of installing a myriad of little packages, and "freeing" the
distribution caregivers (and I don't only mean the developers, but
also the ftp-masters) and infrastructure from carrying this myriad of
little packages.

Elana and Sean seem to share the thought that each language ecosystem
could work with somewhat different rules:

Sean>
It might be reasonable to vendor like mad for something written in
Go but not for something written in Haskell, say

Elana>
Debian policy is specifically built around the distribution and
execution of dynamically linked C/C++ applications and
libraries. distros in general were. but modern software above that
base does not necessarily operate under the same assumptions and
it is silly to apply policy that was designed for applications
that are dynamically linked against programs that are statically
linked and are impossible to dynamically link

Simon mentioned that "our identity should be about shipping
high-quality packages, whatever that means", and mentioning that "our
packaging policy is designed for medium-sized packages", refereed back
to the discussions had long time ago over tiny Javascript snippets
packaged as packages, back in the starting days of the nodejs growth.

# DFSG-freeness checking
##

Sean and I shared the concern that excessive vendoring (say, having
tens to hundreds of libraries shipped as part of a single package)
place a very heavy burden on our ftp-masters when checking
DFSG-freeness; coupling this with the rapid change rate that
"new-wave" software development usually has, if adding / dropping
dependencies from already packaged software is usual practice,
DFSG-checks would not just have to be made in NEW, but as a continuous
process. Far from sustainable, and impossible to do by our ftp-master
team practices.

# Security issues
###

The issue of security support was also mentioned: If many packages
start shipping tens or hundreds of vendored libraries, how can we
ensure security support? This has long been a pride point for Debian
and, in general, for the Linux distributions model. We package
independent libraries, dynamically link them, and security supports
"just happens" for the users. Now, what happens in languages as Go or
Rust, where linking is done statically? They would have to be at least
recompiled when any of their constitutive libraries is updated. But
what if the libraries are vendored-in? Security issues are prone to be
hidden from our sight.

I'm pasting here a bit of the discussion that happened later during
the meeting: having this discussion K8S-specific, Elana mentioned that
"that is a big part of the tension. sometimes, you have an upstream
where the authors are less resourced and responsive than the
downstream. this case is almost certainly the opposite".

At this point, we found some friction as to _what_ we are discussing
on: Is this bug specifically on Kubernetes, which should be taken as a
special case? Or would we try to rule as the Technical Committee on
vendoring in general in the Debian ecosystem? Elana repeated her
assurance that the Kubernetes team is thorough in their
freeness-checking and security practices; I insisted on "not
discussing about K8S, but about a bundling practice to which K8S