Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Dmitry Smirnov
On Thursday, 22 October 2020 2:16:16 AM AEDT Sean Whitton wrote:
> I think that we can all agree with everything you've written about the
> reasons why packaging components separately is better. 

Thank you.


> The problem is
> that in this case the choice seems to be between not having recent
> Kubernetes in Debian at all, or giving up on some of those advantages.

Ultimately it is about maintainers comfort then. Because there is a 
compromise to un-vendor most libraries (one by one) and keep some/few 
strategically vendored, when a system library can not be used (determined 
case by case).

Examples of this approach are numerous: syncthing, consul, nomad, vault, 
docker.io, runc, gitlab-runner, libpod, buildah, singularity-container
and _most_ Golang packages.

Kubernetes had no maintainer - that was the problem. Having maintainer who
does not care for shared libraries because he does not want to be bothered
is a different problem.

-- 
Kind regards,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

In times of universal deceit, telling the truth becomes a revolucionary
act.
-- George Orwell


signature.asc
Description: This is a digitally signed message part.


Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Dmitry Smirnov
On Wednesday, 21 October 2020 8:56:53 PM AEDT Felix Lechner wrote:
> How is the second one inferior, please? I think it includes a crucial
> missing feature (co-installable versions).

To reduce maintainers burden, you want maintainers to look after not one but 
multiple versions of libraries. This is a contradiction (oxymoron).
How many releases (current + obsolete ones) one can meaningfully support?

-- 
Best wishes,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

The cure for the evils of democracy is more democracy.
-- H. L. Mencken


signature.asc
Description: This is a digitally signed message part.


Bug#971515: Request for security team input on kubernetes TC bug

2020-10-21 Thread Sean Whitton
Hello security team,

The TC are being asked about src:kubernetes, and it would be good to
hear from you about whether and how security support is a relevant
consideration in determining whether the level of vendoring in that
package is acceptable.  Could you take a look at #971515 and perhaps
give us some input, please?

Thanks.

-- 
Sean Whitton



Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Sean Whitton
Hello Dmitry,

On Wed 21 Oct 2020 at 11:21am +11, Dmitry Smirnov wrote:

> On Wednesday, 21 October 2020 6:16:03 AM AEDT Sean Whitton wrote:
>> I think that my message [1] is what makes you think that the package
>> would not have got through NEW?
>
> It was not your message but my own experience with introducing of 100+ 
> packages through NEW, especially those ones with large burden of vendored 
> libraries, including Kubernetes. The main hassle usually is to convince FTP-
> masters when some vendoring is _necessary_ (case by case) and the usual 
> request is to package all vendored libraries separately. With rare exceptions 
> some (few) vendored libraries are allowed like when a library is a fork, 
> customised for the particular project and therefore not re-usable by other 
> software. Another example is when vendored library is an obsolete software 
> phasing out in future releases. Few micro-libraries might be tolerated when 
> vendored, especially when they are not widely used. Also vendoring might be 
> acceptable when software components with mutual/circular dependencies are 
> shipped in one or several name spaces - in other words when a software code 
> base is not from one name space but from several. None of those cases applies 
> to Kubernetes.
>
> A specific example (libpod/podman) is mentioned in 
>
>   https://lists.debian.org/debian-devel/2020/03/msg00441.html
>
> "Podman was rejected due to "many embedded packages in vendor/" with only
>  6 or 7 private libs versus 120 libraries removed in favour of packaged
>  ones."

Fair enough, but again, this is about NEW as a review by experienced
packagers rather than NEW as a blocker for inclusion.

> If your concern is about security support then IMHO Kubernetes can not be 
> meaningfully supported from security prospective, with or without vendored 
> dependencies.

Fair enough.  In the -devel thread the security team indicated that they
thought they could do security support in the case where there is
vendoring.  It would be good to get more input from them in this bug.

> If Debian only cared about maintainers' convenience or reducing maintainers 
> efforts then Debian would not be what it is now. We favour technical elegance 
> often in expense of maintainers' comfort.

I don't think maintainers' comfort is part of Janos' motivation -- it's
the issue of not having recent Kubernetes in Debian at all.

>> Are there issues the TC should think about which do not fall under this
>> way of looking at things?  I.e., weighing the impact on people other
>> than Janos who want to work on the package, vs. the impact on people who
>> want recent kubernetes to be part in the archive at all?
>
> Is Debian ecosystem of packaged reusable libraries worth caring about?
>
> If so then why grant exception to one particular package? We have several (or 
> more) sophisticated Golang packages using hundreds of packaged libraries.
>
> In the early days of packaging Kubernetes we did not even have most 
> components packaged and I've been spending most effort on packaging, 
> introducing and stabilising dependency libraries.
> These days the major work has already been done and the argument for 
> monolithic vendoring is much weaker.
>
> [...]

I think that we can all agree with everything you've written about the
reasons why packaging components separately is better.  The problem is
that in this case the choice seems to be between not having recent
Kubernetes in Debian at all, or giving up on some of those advantages.

-- 
Sean Whitton



Re: Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Sean Whitton
Hello,

On Tue 20 Oct 2020 at 06:52pm -07, Felix Lechner wrote:

>  I think our response to the vendoring explosion is at odds with the
> trends in many languages.
>
> It's time to retool. At the two ends of the solution spectrum, I see
>
> 1. Fully vendored source packages; or
> 2. A packaging system that allows different vendor versions to co-exist.
>
> Either one allows dependent sources to consume whichever versions they
> require, but in my view solution (2) is otherwise superior---provided
> that the packaging process is automated. (A language's build system
> also has to distinguish the installed versions.) For each language so
> affected, could we make (2) our goal, and allow (1) until then?

This is a very general (but of course interesting) topic.  Could I ask
that it be kept out of this TC bug, please?  We have to figure out what
to do about this package given our present tooling.

-- 
Sean Whitton



Re: Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Felix Lechner
Hi Dmitry,

On Wed, Oct 21, 2020 at 12:50 AM Dmitry Smirnov  wrote:
>
> Yes, at first there will
> be a significant effort but then it will become easier.

I know you put a lot of effort in (as did Michael Stapelberg, with
whom I spent some time before he left), but I don't think our current
approach makes anything easy. It is also why the world is moving in
another direction.

> You are advocating for disruptive
> changes therefore your proposed theoretical solutions have to be available as
> a proof of concept for review.

Did you catch the part about different versions being co-installable?
It would be similar to the freedom we grant to major numbers of shared
object libraries. My point is theoretical only because we do not
presently have it.

> Personally I'm not satisfied with either of those inferior proposals.

How is the second one inferior, please? I think it includes a crucial
missing feature (co-installable versions).

> Look how many packages we already have:
>
>   
> https://qa.debian.org/developer.php?login=pkg-go-maintainers%40lists.alioth.debian.org+team%2Bpkg-go%40tracker.debian.org

It's an impressive list; thank you for forwarding it. I am proud to be
on the Golang team. :)

> In the meantime you could follow the established practice that is
> demonstrated to be working on several packaged heavy Golang applications.

Not sure about heavy Golang applications, but our established practice
does work well for the relatively lightweight 'gocryptfs', which I
maintain. That source moves very fast. I often have problems finding
the proper go-fuse or xattr prerequisite required for a new version.
Sometimes they are incompatible with the needs of other packages in
the archive.

As a side note, several "library" packages that gocryptfs relies on
should really be marked "Architecture: any" to exercise their test
suites properly. It is another example of the impedance mismatch in
our current approach. We are confusing sources and libraries, and our
method of shoehorning one into the other ought to be rethought.

> Besides un-vendoring libraries can prevent some CVE issues as well.

Packages could declare vendored sources (or Lintian could scan for
them) for an effective match with their security status.

> If we tolerate full vendoring now, because "there is no better way" yet, then
> there will be no better way for sure.

Please do not despair. I offered full vendoring only in the interest
of compromise, which I believe is a worthwhile goal just like the
consensus we are working on. As a project, we are looking for a better
endpoint together.

Kind regards
Felix Lechner



Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Dmitry Smirnov
Hi Felix,

On Wednesday, 21 October 2020 12:52:40 PM AEDT Felix Lechner wrote:
> > We favour technical elegance often in expense of maintainers' comfort.
> 
> Is our approach really either one of those? I think our response to
> the vendoring explosion is at odds with the trends in many languages.

IMHO we are managing quite admirably. Basically, to me it looks like you 
don't want to maintain Kubernetes the way we maintain heavy Golang packages.
You would have to learn to un-vendor many libraries. Yes, at first there will 
be a significant effort but then it will become easier. "Too many vendored 
libraries to use packaged libs" is a poor excuse.

We have been dealing with "explosion" for years already. Tools like "dh-make-
golang" are helpful to generate initial packaging for new Golang libraries in 
a semi-automatic manner. FTP-masters are usually quite effective with 
processing of NEW packages. Look how many packages we already have:

  
https://qa.debian.org/developer.php?login=pkg-go-maintainers%40lists.alioth.debian.org+team%2Bpkg-go%40tracker.debian.org


> It's time to retool. At the two ends of the solution spectrum, I see
> 
> 1. Fully vendored source packages; or
> 2. A packaging system that allows different vendor versions to
> co-exist.

Personally I'm not satisfied with either of those inferior proposals.

Besides un-vendoring libraries can prevent some CVE issues as well.


> Either one allows dependent sources to consume whichever versions they
> require, but in my view solution (2) is otherwise superior---provided
> that the packaging process is automated. (A language's build system
> also has to distinguish the installed versions.) For each language so
> affected, could we make (2) our goal, and allow (1) until then?

IMHO tools have to come first (if ever). You are advocating for disruptive 
changes therefore your proposed theoretical solutions have to be available as 
a proof of concept for review.

In the meantime you could follow the established practice that is 
demonstrated to be working on several packaged heavy Golang applications.

If we tolerate full vendoring now, because "there is no better way" yet, then 
there will be no better way for sure. For now using packaged system libraries 
whenever possible is the best way.

-- 
Kind regards,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

Those who disdain wealth as a worthy goal for an individual or a society
seem not to realize that wealth is the only thing that can prevent poverty.
-- Thomas Sowell


signature.asc
Description: This is a digitally signed message part.