Re: Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-20 Thread Felix Lechner
Hi Dmitry,

On Tue, Oct 20, 2020 at 5:24 PM Dmitry Smirnov  wrote:
>
> Let's not attempt to fabricate perception of consensus please.

Consensus is a worthy goal and perhaps even possible, per below.

> We favour technical elegance
> often in expense of maintainers' comfort.

Is our approach really either one of those? I think our response to
the vendoring explosion is at odds with the trends in many languages.

It's time to retool. At the two ends of the solution spectrum, I see

1. Fully vendored source packages; or
2. A packaging system that allows different vendor versions to co-exist.

Either one allows dependent sources to consume whichever versions they
require, but in my view solution (2) is otherwise superior---provided
that the packaging process is automated. (A language's build system
also has to distinguish the installed versions.) For each language so
affected, could we make (2) our goal, and allow (1) until then?

Kind regards
Felix Lechner



Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-20 Thread Dmitry Smirnov
On Wednesday, 21 October 2020 6:16:03 AM AEDT Sean Whitton wrote:
> I think that my message [1] is what makes you think that the package
> would not have got through NEW?

It was not your message but my own experience with introducing of 100+ 
packages through NEW, especially those ones with large burden of vendored 
libraries, including Kubernetes. The main hassle usually is to convince FTP-
masters when some vendoring is _necessary_ (case by case) and the usual 
request is to package all vendored libraries separately. With rare exceptions 
some (few) vendored libraries are allowed like when a library is a fork, 
customised for the particular project and therefore not re-usable by other 
software. Another example is when vendored library is an obsolete software 
phasing out in future releases. Few micro-libraries might be tolerated when 
vendored, especially when they are not widely used. Also vendoring might be 
acceptable when software components with mutual/circular dependencies are 
shipped in one or several name spaces - in other words when a software code 
base is not from one name space but from several. None of those cases applies 
to Kubernetes.

A specific example (libpod/podman) is mentioned in 

  https://lists.debian.org/debian-devel/2020/03/msg00441.html

"Podman was rejected due to "many embedded packages in vendor/" with only
 6 or 7 private libs versus 120 libraries removed in favour of packaged
 ones."


> There are a few issues tangled together here.

IMHO it is really one issue of how we maintain Debian packages. If you want 
to distinguish few issues then they are all closely related.


> NEW is mainly about license and DFSG compliance, and secondarily about
> the idea that we want to avoid accepting packages where doing so would
> make Debian worse, even if it would also make Debian better along other
> dimensions.  As a simple example, we try to avoid accepting a package
> that is already packaged under a slightly different name, because in
> most cases it is worse for both users and contributors to have the same
> thing in the archive twice (not talking about vendoring here).

It is also about preserving integrity of Debian identity. We try to prevent 
monolithic bundles like Kubernetes in favour of maintaining ecosystem of re-
usable libraries, packaged individually.


> In this case, the reason I wrote in [1] that I would probably have
> rejected the package, had I come across it in NEW, is that it seemed to
> me that having this package in Debian would make the archive less
> maintainable by contributors other than Janos who might need to work
> with the package.  (After the discussion on -devel, I'm no longer so
> sure about that opinion of mine.)

  https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_party_code

If your concern is about security support then IMHO Kubernetes can not be 
meaningfully supported from security prospective, with or without vendored 
dependencies.

Also I must point out that Kubernetes upstream have the worst management of 
vendored libraries that I have ever seen. Examples include vendoring multiple 
versions of the same library, etc.
A particular case when upstream failed to update a problematic vendored 
library for years(!) practically destroys faith in upstream care for good 
hygiene of vendored dependencies:

  https://github.com/kubernetes/kubernetes/issues/27543

Note the expressed _resistance_ to upgrading a vendored library...

With the above example, how can anyone have confidence in upstream security 
patching? After all Kubernetes have more vendored code than its own.



> It's not correct to say, however, that the package "is in violation of
> ftpmaster's policy for inclusion of new packages".  That could only be
> true is if the package met one of the "serious violations" listed in the
> REJECT-FAQ, which is basically DFSG and licensing issues, and a few
> obvious clangers.  Instead, what we have is a situation in which there
> is reason to be worried about the way the package is put together, but
> the opinion of one FTP team member at one particular point in time
> carries about the same weight as the opinion of any experienced
> packager.

There is an established practice, a tradition if you wish, that is followed 
all the time even if not explicitly described in REJECT-FAQ. Debian clearly
tries to be modular whenever possible.


> In other words, I suggest that we ignore the NEW issue entirely, and
> just consider whether the way the package is currently put together
> imposes an unreasonable burden on Debian contributors other than Janos
> who want to work on the package, or users who want to patch it, etc.
> The sorts of questions we should try to answer:
> 
> - does the vendoring make Debian security support harder (discussion on
>   -devel suggests it makes it easier)
> 
> - everyone seems to think the level of vendoring is at best a necessary
>   evil;

Let's not attempt to fabricate perception of 

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-20 Thread Sean Whitton
Hello Dmitry, others,

On Thu 01 Oct 2020 at 11:15am +10, Dmitry Smirnov wrote:

> I seek your judgement regarding excessive, unnecessary and unjustifiable
> vendoring of private libraries in Kubernetes package:
>
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970717
>
> Some relevant argumentation can be found in
>
>   https://lists.debian.org/debian-devel/2020/03/msg00359.html
>   https://lists.debian.org/debian-devel/2020/03/msg00400.html
>   https://lists.debian.org/debian-devel/2020/03/msg00441.html
>
> In short, myself and Golang packaging team spent years on stabilising
> Golang ecosystem of packaged libraries for re-use by Golang software.
>
> To the best of my knowledge, all packaged Golang software, regardless of its 
> sophistication, use some packages libraries.
> Except Kubernetes, that disconnected itself from cooperation by not using any 
> packaged libraries, instead exclusively using only private libraries in 
> numbers greater than any Debian package ever used.
>
> As a former Kubernetes maintainer and a developer who originally introduced
> Kubernetes to Debian, I know that it could be maintained with only few (or 
> several) private libraries at most.
>
> Current state of Kubernetes package is in violation of ftp-master's policy 
> for inclusion of new packages to Debian.

I think that my message [1] is what makes you think that the package
would not have got through NEW?  There are a few issues tangled together
here.

NEW is mainly about license and DFSG compliance, and secondarily about
the idea that we want to avoid accepting packages where doing so would
make Debian worse, even if it would also make Debian better along other
dimensions.  As a simple example, we try to avoid accepting a package
that is already packaged under a slightly different name, because in
most cases it is worse for both users and contributors to have the same
thing in the archive twice (not talking about vendoring here).

In this case, the reason I wrote in [1] that I would probably have
rejected the package, had I come across it in NEW, is that it seemed to
me that having this package in Debian would make the archive less
maintainable by contributors other than Janos who might need to work
with the package.  (After the discussion on -devel, I'm no longer so
sure about that opinion of mine.)

It's not correct to say, however, that the package "is in violation of
ftpmaster's policy for inclusion of new packages".  That could only be
true is if the package met one of the "serious violations" listed in the
REJECT-FAQ, which is basically DFSG and licensing issues, and a few
obvious clangers.  Instead, what we have is a situation in which there
is reason to be worried about the way the package is put together, but
the opinion of one FTP team member at one particular point in time
carries about the same weight as the opinion of any experienced
packager.

In other words, I suggest that we ignore the NEW issue entirely, and
just consider whether the way the package is currently put together
imposes an unreasonable burden on Debian contributors other than Janos
who want to work on the package, or users who want to patch it, etc.
The sorts of questions we should try to answer:

- does the vendoring make Debian security support harder (discussion on
  -devel suggests it makes it easier)

- everyone seems to think the level of vendoring is at best a necessary
  evil; if someone wants to try to reduce the level of vendoring (as
  Dmitry did when he was maintainer), is the current way the package is
  built going to make it harder for people to work on making that sort
  of contribution?

- are people trying to do cross-archive work going to find the packaging
  of kubernetes getting in their way?  (e.g. other Go team members
  trying to update things, improve our binNMU techniques and machinery,
  etc.)

... and this is to be weighed against the negative impact of having
kubernetes in Debian lag so seriously behind upstream such that almost
no-one would want to bother installing it.

Are there issues the TC should think about which do not fall under this
way of looking at things?  I.e., weighing the impact on people other
than Janos who want to work on the package, vs. the impact on people who
want recent kubernetes to be part in the archive at all?

If I'm right about what the question for the TC is, I hope that Janos
and Dmitry can both help us discuss it in a way which sets aside the
heat which characterised the -devel thread.  It is completely
understandable to (a) feel very frustrated at Debian not including
recent versions of a useful piece of free software; and also (b) feel
very frustrated when someone chooses to accept a less-than-ideal
approach as necessary when one has put a lot of time into trying to find
workable alternatives.

The thing is, both (a) and (b) are motivated by the same basic desire to
make Debian better and more useful, so perhaps we can focus on that
point of commonality.

[1]  

Next Meeting - October 21st at 18:00 UTC (tomorrow)

2020-10-20 Thread Margarita Manterola

Dear Technical Committee members,

Our monthly meeting is scheduled to take place tomorrow, October 21st at 
18:00 UTC.


We currently have one ongoing bug, that hasn't seen much traffic on our 
side (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=971515)


Topics for our agenda are:

* #971515 - kubernetes: excessive vendoring (private libraries)
* Follow up from the DebConf20 discussion (we should decide what we want 
to do next)

* Recruitment efforts

Notes from previous meeting here: 
http://meetbot.debian.net/debian-ctte/2020/debian-ctte.2020-09-16-17.58.html


Notes from the DebConf discussion here:
https://salsa.debian.org/debian/tech-ctte/-/blob/master/talks/202008-pad.html

I'm not sure if I'll be able to join the meeting. If I'm not there, 
please carry on without me.


--
Regards,
Marga