Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Saturday, 11 April 2020 4:45:08 AM AEST Peter Silva wrote:
> Debian policy is there to shape packages so they fit into stable. If we
> "know" that a package will never make it into stable, is Policy relevant?

Of course policy is relevant. Debian is not only "stable" but also "testing", 
"unstable", "backports" and hopefully "fasttrack" for those packages that are 
not in "stable" but (at least could be) _for_ "stable".


> Why have packagers do work that will be forever useless?

Nonsense. "Forever useless" is a useless exaggeration.

Even when package itself is not targeting "stable" (yet), it is still useful 
for ecosystem of depending libraries, if packaging is done properly.

Responsible maintainer is motivated to look after dependencies. The very 
distribution benefits from more maintainers collectively caring after common 
dependency libraries, build tools, supporting tools and procedures, etc.

-- 
Regards,
 Dmitry Smirnov.

---

No person, no idea, and no religion deserves to be illegal to insult,
not even the Church of Emacs.
-- Richard Stallman


signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Thomas Goirand]

On 4/10/20 9:50 AM, Marc Haber wrote:
> A package without upstream support is bound to be a zombie.

There's many examples of this in Debian. It all depends, and you cannot
make a generality like this.

On 4/10/20 9:43 AM, Marc Haber wrote:
> On Thu, 9 Apr 2020 08:47:16 +0200, Thomas Goirand 
> wrote:
>> On 4/8/20 10:44 PM, Peter Silva wrote:
>>> doesn´t this whole discussion just mean that k8 should just not be in
>>> Debian?
>>
>> IMO no. It means that if we have enough packaging resources (which
>> probably we don't, I can't tell),
> 
> We don't. That's the problem: We might have the personpower to
> _PACKAGE_ a monster like openstack, docker or k2s, but we definetely
> dont have the resources to actually SUPPORT the software for the
> entire stable-oldstable etc lifecycle.

Thanks, but I believe I'm doing well regarding security fixes of
OpenStack in Debian. Sure, I have given-up on old-stable, but stable has
been fine for at least the last 2 or 3 releases. I'm even trying to push
for more stable updates, but it isn't easy to get them accepted by the
stable release team. If you want to help there, please review the
changes and convince the SRT we need them...

> We NEED upstream support for
> that, and if we even don't have that right after our release (when the
> software is already half a year old), then we are not doing ourselves
> and our users a favor by providing packages at all.

I believe you're wrong here, sorry. Let me attempt to explain from my
experience with maintaining OpenStack in Debian over the last 9 years.

Packaging is actually the *much* bigger part, compared to security
fixes. The first early years of OpenStack were terrible, because on each
release, I had 20 to 30 new Python dependencies to package. But after
that initial period, it all settled, as the upstream designed was more
or less done. It eventually became smoother. I don't see why it wouldn't
be the same pattern with Kubernetes.

Maintaining a big amount of language libraries isn't *that* difficult
and time demanding. It's a bit time demanding at first, to prepare the
initial packages, then it eventually gets better once you've done the
bulk of the work.

If you have an optimized workflow, it is possible to upload dozens of
time a day (yes, *multiple* dozens of packages) to get everything
updated. So, that's some kind of commitment. I'm not sure if Janos is
ready for it, or even paid for doing it (which IMO is necessary in this
case), and has enough time, but you probably should at least give him a
chance and let him try, or simply ask him directly.

Today, with a few more dependencies getting in, the OpenStack team just
reached the milestone of 500 packages. It used to scare me. It doesn't
anymore. Especially since I've met other people in Debian maintaining so
many packages, like for example Andreas Tile, or when I heard about what
Jeremy Bicha is doing with Gnome. I'm also impressed with what Mike
Grabriel has been doing too!

On the opposite side, most security fixes need a tiny small fix of a few
lines. Most of the time, easy to backport.

There's indeed these rare cases where the security fix needs a more deep
change in the code, and in such case, we may need the help from
upstream. But we can't tell in advance if this is going to be needed
during the life of Debian stable. The only thing we can do is survey how
many big security issue we've seen over let's say the last year, and how
many of them needed a big patch to fix it.

I don't know Kubernetes enough to tell, and even less its security
record, so this really is an open question!

>> As a user, I'd prefer Kubernetes to be in Stable if possible. I'd be one
>> of these users who don't care about the latest shiny feature, and prefer
>> something stable, supported for YEARS to come, not just 3 months.
> 
> That a big package in Debian stable that has been abandoned by
> upstream months or even years ago is actually supported is fairy tale.

It depends on many factors, like the security record which I just talked
about above. In many cases, I was able to backport security fixes for
versions of OpenStack that were EOLed a long time ago, because security
fixes were 2 liners. And OpenStack got a WAY better than it used to be
(understand: one CVE each 3 to 6 months instead of each 3 weeks, literally).

> You're lying to yourself if you believe that you'd get full security
> support for a two years old kubernetes release in Debian stable.

Is Kubernetes *that* bad regarding security?!? Gosh, it's a call for
never using it then, whatever the method of deployment!

> The utmost you'd get would be an occasional security fix where an
> upstream fix would get _flagged_ as security relevant _AND_
> coincidentially backportable to our version.

Don't you also think that, over time, Kubernetes will mature, and its
upstream will eventually learn from their mistakes of never willing to
help downstream distros? How will they learn it if we don't show them
the way we're doing 

Re: What to do when DD considers policy to be optional? [kubernetes]

[Peter Silva]

On Fri, Apr 10, 2020 at 1:12 PM Russ Allbery  wrote:

> Dmitry Smirnov  writes:
>
> > Let's remember that Kubernetes was never in "stable" to begin with.
>
> > This is not to say that it couldn't be useful in "testing", "unstable"
> > or even "experimental". Many packages that may be considered not
> > suitable for "production" are nevertheless useful.
>
> Speaking as someone who previously supported having Kubernetes in Debian,
> I was using it directly from unstable.  For my purposes, it was fine that
> it was never part of a stable release.
>
> To be clear, I was mostly happy to have the clients.  The control plane
> and pod software for my purposes is provided by a platform provider, so is
> outside of my scope.
>
> --
> Russ Allbery (r...@debian.org)  
>
>
Debian policy is there to shape packages so they fit into stable. If we
"know" that a package will never make it into stable, is Policy relevant?
Why have packagers do work that will be forever useless? Getting that sort
of stuff out of unstable and testing, when we know it will never go
further, is also a net benefit reducing the noise and load for the devs.  A
release blocker on one of those packages, isn't.  Forcing people to get
stuff from unstable when they want to be running stable (for everything but
X) isn't great either.

A "package" that doesn't follow Debian Policy should be walled off and
obvious somehow. It's maintenance will be different than most of Debian.
Since Debian Policy is sort of the heart of Debian... it basically means
that such packages can be built *for* Debian, but can never be *part of*
Debian...  Having more software available *for Debian* is still a good
thing, even if it isn't part of it.  Expanding the user base by enabling
projects with ... ahem... different maintenance cultures... is still a win.
Supporting stuff to be built *for* debian allows much more newer, faster
paced, stuff to be made available for the operating system, without
jeopardizing the quality of stable.

fwiw, this is one of the things that Launchpad.net provides for Ubuntu,
that doesn't seem to be around for Debian.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Russ Allbery]

Dmitry Smirnov  writes:

> Let's remember that Kubernetes was never in "stable" to begin with.

> This is not to say that it couldn't be useful in "testing", "unstable"
> or even "experimental". Many packages that may be considered not
> suitable for "production" are nevertheless useful.

Speaking as someone who previously supported having Kubernetes in Debian,
I was using it directly from unstable.  For my purposes, it was fine that
it was never part of a stable release.

To be clear, I was mostly happy to have the clients.  The control plane
and pod software for my purposes is provided by a platform provider, so is
outside of my scope.

-- 
Russ Allbery (r...@debian.org)  

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Friday, 10 April 2020 5:43:57 PM AEST Marc Haber wrote:
> That a big package in Debian stable that has been abandoned by
> upstream months or even years ago is actually supported is fairy tale.

Let's remember that Kubernetes was never in "stable" to begin with.

This is not to say that it couldn't be useful in "testing", "unstable" or 
even "experimental". Many packages that may be considered not suitable for 
"production" are nevertheless useful.

What I've learned from my "Kubernetes affair" is that upstream is not capable 
of support (due to project's "attention deficit disorder") as they've 
seemingly lost control over issue management and they barely control enormous 
dependency tree where I've seen numerous abominations like sub-vendored 
multiple different versions of the same library...

IMHO Kubernetes demonstrated that problems in 3rd party dependency libraries 
can not be easily fixed upstream. In that regards Debian have greater 
potential due to flexibility of dependencies.

What I think is missing in this conversation is understanding that packaging 
Kubernetes (and other sophisticated software) is much about maintaining 
_ecosystem of libraries_ rather than building monolitic, unsupportable by 
design binary.

I think there is no question whether Golang ecosystem is useful in Debian or 
not. Therefore packages that maintained properly, with respect to ecosystem 
of their dependencies, are being useful, directly or indirectly.

Not every package deserves to be in "stable" and not every package outside of 
"stable" is worthless.

-- 
All the best,
 Dmitry Smirnov.

---

In theory, there is no difference between theory and practice. But, in
practice, there is.
-- Jan L. A. van de Snepscheut



signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Thu, 9 Apr 2020 09:36:22 +0200, Thomas Goirand 
wrote:
>On 4/8/20 10:58 PM, Michael Stone wrote:
>> Or, if they're not clear about what
>> adopting this tool means, I agree that isn't in their interest for them
>> to see that there's a debian package and be fooled into thinking it
>> doesn't require this deployment model just because there's a zombie
>> package in debian stable which will be essentially unsupported
>
>If the package becomes a "zombie package" as you describe, then you're
>right. Hopefully, that's not the plan!

A package without upstream support is bound to be a zombie. We don't
have the resources to provide support on the same level like an actice
upstream community does.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Wed, 8 Apr 2020 16:58:54 -0400, Michael Stone 
wrote:
>On Wed, Apr 08, 2020 at 10:36:17PM +0200, Thomas Goirand wrote:
>>I don't agree with this *at all*. It is not in the interest of our users
>>to be forced to update the software they use for their infrastructure
>>every few months.
>
>Isn't that the user's decision, when they decided to adopt a tool that 
>requires this deployment model? Or, if they're not clear about what 
>adopting this tool means, I agree that isn't in their interest for them 
>to see that there's a debian package and be fooled into thinking it 
>doesn't require this deployment model just because there's a zombie 
>package in debian stable which will be essentially unsupported and will 
>completely cut them off from the tool's community. Putting it into 
>debian provides zero benefit, and they could get the same "stability" 
>guarantee by just keeping a copy of a two year old third party .deb and 
>never updating it.

What Michael says.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Thu, 9 Apr 2020 09:21:35 +0200, Tomas Pospisek 
wrote:
>On 09.04.20 08:47, Thomas Goirand wrote:
>
>> As a user, I'd prefer Kubernets to be in Stable if possible. I'd be one
>> of these users who don't care about the latest shiny feature, and prefer
>> something stable, supported for YEARS to come, not just 3 months.
>
>To give a datapoint:
>
>Kubernetes as a Service offerings on the market are often already one,
>two or three major releases behind. Which IMHO serves as a useful
>datapoint of the "needs of the users"...

I do have serious doubts that the providers of such services are able
to have the security vulnerabilities in their outdated software fixed.
And if so, since they have their services in a controlled environment
and call roll fixes that would be unsuitable for a Debian package. And
propble working with that software professionally might probably not
need the level of upstream support that Joe B. Debianuser needs.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Thu, 9 Apr 2020 08:47:16 +0200, Thomas Goirand 
wrote:
>On 4/8/20 10:44 PM, Peter Silva wrote:
>> doesn´t this whole discussion just mean that k8 should just not be in
>> Debian?
>
>IMO no. It means that if we have enough packaging resources (which
>probably we don't, I can't tell),

We don't. That's the problem: We might have the personpower to
_PACKAGE_ a monster like openstack, docker or k2s, but we definetely
dont have the resources to actually SUPPORT the software for the
entire stable-oldstable etc lifecycle. We NEED upstream support for
that, and if we even don't have that right after our release (when the
software is already half a year old), then we are not doing ourselves
and our users a favor by providing packages at all.

>As a user, I'd prefer Kubernets to be in Stable if possible. I'd be one
>of these users who don't care about the latest shiny feature, and prefer
>something stable, supported for YEARS to come, not just 3 months.

That a big package in Debian stable that has been abandoned by
upstream months or even years ago is actually supported is fairy tale.
You're lying to yourself if you believe that you'd get full security
support for a two years old kubernetes release in Debian stable.

The utmost you'd get would be an occasional security fix where an
upstream fix would get _flagged_ as security relevant _AND_
coincidentially backportable to our version.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Jeremy Stanley]

On 2020-04-09 09:42:28 +0200 (+0200), Thomas Goirand wrote:
[...]
> I agree with all what you wrote above. However, there's still no
> LTS release in OpenStack, unfortunately. I can support Debian
> stable, though I have (understandably) given up on oldstable, yet
> even on Debian LTS.

Yep, at the moment it's running around the 3-year mark where the
community ceases to be able to maintain extensive integration
testing any longer (the Ocata release from early 2017 is at the
point where it's falling apart now CI-wise). This fluctuates a bit
though, for example I anticipate an interest in expiring releases
prior to the current one a bit faster because it will mean not
having to continue supporting Python 2.7 within the test
infrastructure for as long.

> Also, it used not to be the way you described. The OpenStack
> community has evolved from operators like Rackspace proudly
> advertising they deploy from master, to something more reasonable.
> It is currently widely accepted that running older releases of
> OpenStack is actually OK, and that upgrades aren't easy.
[...]

I don't think that goal has gone away entirely. Some public
providers are basically deploying from master or at least upgrading
to the released versions by/on release day (I'm a happy customer of
one of them and they even use Debian, though I think they're
deploying from source with Ansible so not using the OpenStack
packages you're maintaining). On the other hand, upgrades have
become far easier for users/operators than they used to be, with
most of the deployment automation projects now performing
system-wide upgrade tests on every proposed change, stable branch
policies limiting what sorts of configuration transitions are
allowable between releases, and so on.

I get that the software is still changing far faster than is
comfortable for LTS GNU/Linux distribution release cadences, and
more recent improvements like support for "fast-forward" upgrades to
get between noncontiguous major releases are only a half measure. I
still hold out hope that as the software matures (it's only 10 years
old now, which is relatively young compared to other ecosystems its
size), things will continue to stabilize and become more viable in
traditional distro packaging realms. The work you're doing to keep
up with it in Debian is massive, and I can't thank you enough for
that. Please do know that it's greatly appreciated!

The Kubernetes community is roughly 5 years behind OpenStack, so
they're just now bumping up against the challenges the OpenStack
community ran into circa 2015, and much like many of OpenStack's
struggles stem from challenges with Python software management,
Kubernetes is running into different problems with their choice of
programming language. Python was 15-16 years old when OpenStack
started, while Golang was only 5-6 years old when Kubernetes began.
In some ways this was a benefit to the Kubernetes community as it
allowed their work to help shape the language (not to say that
OpenStack hasn't had a significant impact on Python as well), but
with that also comes the joys of trying to maintain a very large
project in a rapidly-evolving language.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Thomas Goirand]

On 4/8/20 11:25 PM, Jeremy Stanley wrote:
> On 2020-04-08 22:36:17 +0200 (+0200), Thomas Goirand wrote:
> [...]
>> Also, the docker world is not the only one to be this way. It used to be
>> like this in OpenStack too. In the OpenStack world, they haven't changed
>> the way they release (ie: every 6 months), but the user survey has shown
>> that almost every user is lagging 4 or 5 versions behind, because
>> upgrading the infrastructure is both difficult and time consuming. Over
>> time, they became very helpful for back-porting fixes to EOL versions too.
>>
>> The main issue is that upstream wants to be able to do fast development,
>> and focus on the development rather than on their users. Taking care of
>> a long term release is time consuming. Taking care of multiple old
>> release is very annoying (backporting fixes may not be always obvious).
>>
>> So yeah, probably upstream will reply with "Debian is stupid". Let them
>> say it if they want to: that doesn't make them right. It only shows they
>> are completely ignorant of what their users want, and the need of
>> downstream distributions.
>>
>> The more there's going to be users going at them asking them about a 2
>> year old release, the more they will realize that Debian isn't stupid,
>> and that this is the way the final users want to consume their work. So
>> it's good for us, and beneficial to the project. It's doing our users a
>> favor, and it doesn't hurt us.
> [...]
> 
> To be fair, the OpenStack community as a whole never suggested it
> was stupid for distros to carry older versions of the software (many
> folks there have experience packaging for and in LTS GNU/Linux
> distributions too, so understand the pain). What was said is that
> it's hard to find enough people with sufficient time to maintain
> years-old versions of the software, thus the community looks to the
> distributions' package maintainers to help shoulder some of that
> burden, coordinate with each other on backporting fixes they need to
> the versions they're carrying, and so on. What has typically been
> laughed off as impractical is suggestions like not developing as
> much software so quickly.
> 
> The OpenStack community also doesn't tell users who have questions
> about 2+ year old versions of the software to go away until they
> upgrade to the bleeding edge. However, it's only natural that if
> users ask questions about old tech, they may find that the number of
> folks still around who remember how it worked (beyond what's already
> in the older versions of the software's documentation) will be
> vanishingly small.

Jeremy,

I agree with all what you wrote above. However, there's still no LTS
release in OpenStack, unfortunately. I can support Debian stable, though
I have (understandably) given up on oldstable, yet even on Debian LTS.

Also, it used not to be the way you described. The OpenStack community
has evolved from operators like Rackspace proudly advertising they
deploy from master, to something more reasonable. It is currently widely
accepted that running older releases of OpenStack is actually OK, and
that upgrades aren't easy.

I by the way completely understand that the need for more people
maintaining older releases of OpenStack (and the CI that goes with it)
has never been fixed, and probably never will, unless a big corp like
Red Hat puts serious resources on it, which would be the only serious
way to maintain an OpenStack LTS.

I don't know the Kubernets community, but probably they are facing the
same difficulties (ie: the lack of resources to maintain older versions).

Cheers,

Thomas Goirand (zigo)

Re: What to do when DD considers policy to be optional? [kubernetes]

[Thomas Goirand]

On 4/8/20 10:58 PM, Michael Stone wrote:
> On Wed, Apr 08, 2020 at 10:36:17PM +0200, Thomas Goirand wrote:
>> I don't agree with this *at all*. It is not in the interest of our users
>> to be forced to update the software they use for their infrastructure
>> every few months.
> 
> Isn't that the user's decision, when they decided to adopt a tool that
> requires this deployment model?

As a distribution, it's our role to give the choice to use the software,
without the painful points of the release management choices from upstream.

> Or, if they're not clear about what
> adopting this tool means, I agree that isn't in their interest for them
> to see that there's a debian package and be fooled into thinking it
> doesn't require this deployment model just because there's a zombie
> package in debian stable which will be essentially unsupported

If the package becomes a "zombie package" as you describe, then you're
right. Hopefully, that's not the plan!

> Putting it into
> debian provides zero benefit, and they could get the same "stability"
> guarantee by just keeping a copy of a two year old third party .deb and
> never updating it.

Though hopefully, the Debian package will be updated and fixed for any
bug found in Stable, plus security updates.

In the OpenStack world, we still find bugs after a year of using a one
year old release in production, which I insist fixing (though dealing
with the Stable release updates in Debian isn't easy, for reasons
outside of the scope of this thread). I expect the same things in the
Kubernets world too (both software stack are comparable in many ways).

Cheers,

Thomas Goirand (zigo)

Re: What to do when DD considers policy to be optional? [kubernetes]

[Tomas Pospisek]

On 09.04.20 08:47, Thomas Goirand wrote:

> As a user, I'd prefer Kubernets to be in Stable if possible. I'd be one
> of these users who don't care about the latest shiny feature, and prefer
> something stable, supported for YEARS to come, not just 3 months.

To give a datapoint:

Kubernetes as a Service offerings on the market are often already one,
two or three major releases behind. Which IMHO serves as a useful
datapoint of the "needs of the users"...

Once you have a cluster running, there is very little incentive to take
it down (which is the exact contrary of why you'd run a cluster in the
first place) and upgrade it... so one can expect Kubernetes **in
particular** to remain on the same major version once it's running and
in production.

Note that Kubernetes had major breakages between major releases
(deprecated old APIs were removed), which means that upgrading your
cluster has high potential to break your deployment. One more reason to
stay put with whatever release you're running on.

I'm saying all this not as an opinion for or against anything, but just
to present datapoints and arguments.
*t

Re: What to do when DD considers policy to be optional? [kubernetes]

[Thomas Goirand]

On 4/8/20 10:44 PM, Peter Silva wrote:
> doesn´t this whole discussion just mean that k8 should just not be in
> Debian?

IMO no. It means that if we have enough packaging resources (which
probably we don't, I can't tell), then we may just as well ignore an
upstream who's basically saying we should never package what they do, or
package the way they want, rather than the we we should.

> It should be a third party package, perhaps with a third party repo, and
> just not be in Debian at all.
> If any means of packaging for a Debian release results in a package that
> is essentially unsupported by upstream... what is the value of including
> it?  For stuff that moves too quickly... perhaps a
> different repo like *forever-sid.d.o* could be set up... and have it
> built against releases, so people
> have current software for Debian... without it being part of Debian.
> 
> That repo would have different rules for it... that loosens things up
> for this kind of hairball package that isn´t stable enough to benefit
> from Debian stability.

As a user, I'd prefer Kubernets to be in Stable if possible. I'd be one
of these users who don't care about the latest shiny feature, and prefer
something stable, supported for YEARS to come, not just 3 months.

Cheers,

Thomas Goirand (zigo)

Re: What to do when DD considers policy to be optional? [kubernetes]

[Andrei POPESCU]

On Mi, 08 apr 20, 16:44:22, Peter Silva wrote:
> doesn´t this whole discussion just mean that k8 should just not be in
> Debian?
> 
> It should be a third party package, perhaps with a third party repo, and
> just not be in Debian at all.
> If any means of packaging for a Debian release results in a package that is
> essentially unsupported by upstream... what is the value of including it?
> For stuff that moves too quickly... perhaps a
> different repo like *forever-sid.d.o* could be set up... and have it built
> against releases, so people
> have current software for Debian... without it being part of Debian.
> 
> That repo would have different rules for it... that loosens things up for
> this kind of hairball package that isn´t stable enough to benefit from
> Debian stability.

You mean something like http://fasttrack.debian.net ?

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Arnaud Rebillout]

On 3/25/20 11:45 PM, Marc Haber wrote:

On Tue, 24 Mar 2020 20:37:16 +, Jeremy Stanley 
wrote:

If this represents the actual state of building Kubernetes, it's
unclear to me why Debian would package it at all. I don't see the
value to users in consuming Kubernetes from a Debian package if the
result is compromising on Debian's vision and values so that they
can get the exact same thing they'd have if they just used the
Kubernetes community's recommended tooling to install it instead.
I'm all for using the best tool for the job, and while I've been a
die-hard Debian user for more than two decades I also don't install
every last bit of software from its packages. Some software
ecosystems have chosen to focus on tools and workflows which are
incompatible with Debian's, but that doesn't mean that either one is
inherently bad nor that they need to be integrated at all costs.

Software packages like kubernetes, docker, and many of the other "hip
tools of the day" are moving way too fast for our release scheme.
Additionally, many communities and developers stop caring for their
software once it has cleared the door and laugh upon people who are
using anything but their latest release.

I think we're not doing the world a favor packaging those software and
releasing it with our stable release [...]



Docker release cycle is not that fast anymore. The current 19.03 branch 
has been maintained by upstream for 10 months now. Not that bad. I think 
it makes it a valuable package for Debian stable.


https://www.docker.com/blog/extending-support-cycle-docker-community-edition/

https://docs.docker.com/engine/release-notes/#version-1903

Re: What to do when DD considers policy to be optional? [kubernetes]

[Jeremy Stanley]

On 2020-04-08 22:36:17 +0200 (+0200), Thomas Goirand wrote:
[...]
> Also, the docker world is not the only one to be this way. It used to be
> like this in OpenStack too. In the OpenStack world, they haven't changed
> the way they release (ie: every 6 months), but the user survey has shown
> that almost every user is lagging 4 or 5 versions behind, because
> upgrading the infrastructure is both difficult and time consuming. Over
> time, they became very helpful for back-porting fixes to EOL versions too.
> 
> The main issue is that upstream wants to be able to do fast development,
> and focus on the development rather than on their users. Taking care of
> a long term release is time consuming. Taking care of multiple old
> release is very annoying (backporting fixes may not be always obvious).
> 
> So yeah, probably upstream will reply with "Debian is stupid". Let them
> say it if they want to: that doesn't make them right. It only shows they
> are completely ignorant of what their users want, and the need of
> downstream distributions.
> 
> The more there's going to be users going at them asking them about a 2
> year old release, the more they will realize that Debian isn't stupid,
> and that this is the way the final users want to consume their work. So
> it's good for us, and beneficial to the project. It's doing our users a
> favor, and it doesn't hurt us.
[...]

To be fair, the OpenStack community as a whole never suggested it
was stupid for distros to carry older versions of the software (many
folks there have experience packaging for and in LTS GNU/Linux
distributions too, so understand the pain). What was said is that
it's hard to find enough people with sufficient time to maintain
years-old versions of the software, thus the community looks to the
distributions' package maintainers to help shoulder some of that
burden, coordinate with each other on backporting fixes they need to
the versions they're carrying, and so on. What has typically been
laughed off as impractical is suggestions like not developing as
much software so quickly.

The OpenStack community also doesn't tell users who have questions
about 2+ year old versions of the software to go away until they
upgrade to the bleeding edge. However, it's only natural that if
users ask questions about old tech, they may find that the number of
folks still around who remember how it worked (beyond what's already
in the older versions of the software's documentation) will be
vanishingly small.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Michael Stone]

On Wed, Apr 08, 2020 at 10:36:17PM +0200, Thomas Goirand wrote:

I don't agree with this *at all*. It is not in the interest of our users
to be forced to update the software they use for their infrastructure
every few months.


Isn't that the user's decision, when they decided to adopt a tool that 
requires this deployment model? Or, if they're not clear about what 
adopting this tool means, I agree that isn't in their interest for them 
to see that there's a debian package and be fooled into thinking it 
doesn't require this deployment model just because there's a zombie 
package in debian stable which will be essentially unsupported and will 
completely cut them off from the tool's community. Putting it into 
debian provides zero benefit, and they could get the same "stability" 
guarantee by just keeping a copy of a two year old third party .deb and 
never updating it.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Peter Silva]

doesn´t this whole discussion just mean that k8 should just not be in
Debian?

It should be a third party package, perhaps with a third party repo, and
just not be in Debian at all.
If any means of packaging for a Debian release results in a package that is
essentially unsupported by upstream... what is the value of including it?
For stuff that moves too quickly... perhaps a
different repo like *forever-sid.d.o* could be set up... and have it built
against releases, so people
have current software for Debian... without it being part of Debian.

That repo would have different rules for it... that loosens things up for
this kind of hairball package that isn´t stable enough to benefit from
Debian stability.




On Wed, Apr 8, 2020 at 4:36 PM Thomas Goirand  wrote:

> On 4/8/20 6:14 PM, Marc Haber wrote:
> > On Sun, 5 Apr 2020 23:16:51 +0100, Wookey  wrote:
> >> On 2020-04-05 21:15 +0200, Marc Haber wrote:
> >>> having an obsolete version of the software distributed
> >>> with/through Debian is (rightfully) seen a liabilty by some upstreams,
> >>> not as an asset.
> >>
> >> I think a more interesting/important question is whether users like
> >> it, rather than whether upstreams like it.
> >
> > I think it is also important to have our users be taken seriously by
> > upstreams. There is software that doesn't move as fast any more. Using
> > a two years old version of those is often fine.
> >
> > Kubernetes, docker etc, however, are fast moving targets. Nobody in
> > the uptream community is willing to even consider answering a question
> > about a version that is two years old. The dialog will inevitably be
> > "well, first you update to our latest version and verify whether your
> > question still applies, then come back with your question" "but I am
> > using the version in Debian stable!" "well, Debian is stupid! Use
> > ".
> >
> > This is not doing our users a favor. And it hurts the Project.
>
> I don't agree with this *at all*. It is not in the interest of our users
> to be forced to update the software they use for their infrastructure
> every few months. They don't want that. If upstream think that this is
> what users want, well upstream is wrong then. And the stability of
> Debian (understand: not a moving target, rather than bug free) is one of
> our very good point.
>
> Also, the docker world is not the only one to be this way. It used to be
> like this in OpenStack too. In the OpenStack world, they haven't changed
> the way they release (ie: every 6 months), but the user survey has shown
> that almost every user is lagging 4 or 5 versions behind, because
> upgrading the infrastructure is both difficult and time consuming. Over
> time, they became very helpful for back-porting fixes to EOL versions too.
>
> The main issue is that upstream wants to be able to do fast development,
> and focus on the development rather than on their users. Taking care of
> a long term release is time consuming. Taking care of multiple old
> release is very annoying (backporting fixes may not be always obvious).
>
> So yeah, probably upstream will reply with "Debian is stupid". Let them
> say it if they want to: that doesn't make them right. It only shows they
> are completely ignorant of what their users want, and the need of
> downstream distributions.
>
> The more there's going to be users going at them asking them about a 2
> year old release, the more they will realize that Debian isn't stupid,
> and that this is the way the final users want to consume their work. So
> it's good for us, and beneficial to the project. It's doing our users a
> favor, and it doesn't hurt us.
>
> >> Quite a lot of users just want to use stuff and so long as it works
> >> for their purposes they really don't know or care if it is 2 years
> >> old.
> >
> > And if it doesnt they want to be able to google for their issues or
> > ask the upstream community. You cannot ask the kubernetes community a
> > question about a kubernetes version that is two years old.
>
> Of course you can! If they choose to not answer, or say bad things about
> Debian, that doesn't make them smarter and us stupid. It just shows how
> careless upstream is.
>
> >> I quite often find myself in this situation. Quite a lot of
> >> software is not something you want to care about - you just want to
> >> use it.
> >
> > Quite a lot, yes. But there is software that doesn't work that way.
>
> Could you please explain why any software would be different? It's just
> upstream that has a super ego and think they are different. Nothing
> more, nothing less.
>
> >> So long as most users will find it works for them, then I think it's
> >> still really useful for us to package stuff,
> >
> > That is not going to happen with kubernetes, docker, node.js et al at
> > this current time.
>
> Like many other software stack, hopefully they will learn by this
> mistake and the release management will change.
>
> Cheers,
>
> Thomas Goirand (zigo)
>
>

Re: What to do when DD considers policy to be optional? [kubernetes]

[Thomas Goirand]

On 4/8/20 6:14 PM, Marc Haber wrote:
> On Sun, 5 Apr 2020 23:16:51 +0100, Wookey  wrote:
>> On 2020-04-05 21:15 +0200, Marc Haber wrote:
>>> having an obsolete version of the software distributed
>>> with/through Debian is (rightfully) seen a liabilty by some upstreams,
>>> not as an asset.
>>
>> I think a more interesting/important question is whether users like
>> it, rather than whether upstreams like it.
> 
> I think it is also important to have our users be taken seriously by
> upstreams. There is software that doesn't move as fast any more. Using
> a two years old version of those is often fine.
> 
> Kubernetes, docker etc, however, are fast moving targets. Nobody in
> the uptream community is willing to even consider answering a question
> about a version that is two years old. The dialog will inevitably be
> "well, first you update to our latest version and verify whether your
> question still applies, then come back with your question" "but I am
> using the version in Debian stable!" "well, Debian is stupid! Use
> ".
> 
> This is not doing our users a favor. And it hurts the Project.

I don't agree with this *at all*. It is not in the interest of our users
to be forced to update the software they use for their infrastructure
every few months. They don't want that. If upstream think that this is
what users want, well upstream is wrong then. And the stability of
Debian (understand: not a moving target, rather than bug free) is one of
our very good point.

Also, the docker world is not the only one to be this way. It used to be
like this in OpenStack too. In the OpenStack world, they haven't changed
the way they release (ie: every 6 months), but the user survey has shown
that almost every user is lagging 4 or 5 versions behind, because
upgrading the infrastructure is both difficult and time consuming. Over
time, they became very helpful for back-porting fixes to EOL versions too.

The main issue is that upstream wants to be able to do fast development,
and focus on the development rather than on their users. Taking care of
a long term release is time consuming. Taking care of multiple old
release is very annoying (backporting fixes may not be always obvious).

So yeah, probably upstream will reply with "Debian is stupid". Let them
say it if they want to: that doesn't make them right. It only shows they
are completely ignorant of what their users want, and the need of
downstream distributions.

The more there's going to be users going at them asking them about a 2
year old release, the more they will realize that Debian isn't stupid,
and that this is the way the final users want to consume their work. So
it's good for us, and beneficial to the project. It's doing our users a
favor, and it doesn't hurt us.

>> Quite a lot of users just want to use stuff and so long as it works
>> for their purposes they really don't know or care if it is 2 years
>> old.
> 
> And if it doesnt they want to be able to google for their issues or
> ask the upstream community. You cannot ask the kubernetes community a
> question about a kubernetes version that is two years old.

Of course you can! If they choose to not answer, or say bad things about
Debian, that doesn't make them smarter and us stupid. It just shows how
careless upstream is.

>> I quite often find myself in this situation. Quite a lot of
>> software is not something you want to care about - you just want to
>> use it.
> 
> Quite a lot, yes. But there is software that doesn't work that way.

Could you please explain why any software would be different? It's just
upstream that has a super ego and think they are different. Nothing
more, nothing less.

>> So long as most users will find it works for them, then I think it's
>> still really useful for us to package stuff,
> 
> That is not going to happen with kubernetes, docker, node.js et al at
> this current time.

Like many other software stack, hopefully they will learn by this
mistake and the release management will change.

Cheers,

Thomas Goirand (zigo)

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Sun, 5 Apr 2020 23:16:51 +0100, Wookey  wrote:
>On 2020-04-05 21:15 +0200, Marc Haber wrote:
>> having an obsolete version of the software distributed
>> with/through Debian is (rightfully) seen a liabilty by some upstreams,
>> not as an asset.
>
>I think a more interesting/important question is whether users like
>it, rather than whether upstreams like it.

I think it is also important to have our users be taken seriously by
upstreams. There is software that doesn't move as fast any more. Using
a two years old version of those is often fine.

Kubernetes, docker etc, however, are fast moving targets. Nobody in
the uptream community is willing to even consider answering a question
about a version that is two years old. The dialog will inevitably be
"well, first you update to our latest version and verify whether your
question still applies, then come back with your question" "but I am
using the version in Debian stable!" "well, Debian is stupid! Use
".

This is not doing our users a favor. And it hurts the Project.

>Quite a lot of users just want to use stuff and so long as it works
>for their purposes they really don't know or care if it is 2 years
>old.

And if it doesnt they want to be able to google for their issues or
ask the upstream community. You cannot ask the kubernetes community a
question about a kubernetes version that is two years old.

>I quite often find myself in this situation. Quite a lot of
>software is not something you want to care about - you just want to
>use it.

Quite a lot, yes. But there is software that doesn't work that way.

>Now of course things get sticky when the old version _doesn't_ work
>for the user's purpose (which happens sometimes) and they to go to bug
>upstream about the issue (rather than bugging debian). 

Yes.

>So long as most users will find it works for them, then I think it's
>still really useful for us to package stuff,

That is not going to happen with kubernetes, docker, node.js et al at
this current time.


>because it's good for our
>users, whatever upstreams would prefer. What proprtion of users are
>likely to find a 2-yr old version satisfactory is a judgement for
>packagers (who will be left with the resulting bugs).

Agreed.

>In cases like this its also good to document that bugs should go to
>debian, not upstream, although of course not many people read docs so
>that won't work as well as one might like. 

They don't. I remember when exim was still popular and people kept
coming to the upstream mailing list with their questions about
Debian's split config, not having read our docs. I stil cringe at the
thought.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Wookey]

On 2020-04-05 21:15 +0200, Marc Haber wrote:
> having an obsolete version of the software distributed
> with/through Debian is (rightfully) seen a liabilty by some upstreams,
> not as an asset.

I think a more interesting/important question is whether users like
it, rather than whether upstreams like it.

Quite a lot of users just want to use stuff and so long as it works
for their purposes they really don't know or care if it is 2 years
old. I quite often find myself in this situation. Quite a lot of
software is not something you want to care about - you just want to
use it.

Now of course things get sticky when the old version _doesn't_ work
for the user's purpose (which happens sometimes) and they to go to bug
upstream about the issue (rather than bugging debian). 

So long as most users will find it works for them, then I think it's
still really useful for us to package stuff, because it's good for our
users, whatever upstreams would prefer. What proprtion of users are
likely to find a 2-yr old version satisfactory is a judgement for
packagers (who will be left with the resulting bugs).

In cases like this its also good to document that bugs should go to
debian, not upstream, although of course not many people read docs so
that won't work as well as one might like. 

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Sat, 28 Mar 2020 12:41:46 +1100, Dmitry Smirnov
 wrote:
>On Friday, 27 March 2020 7:08:35 PM AEDT Marc Haber wrote:
>> Many upstreams deliver .deb packages of their current releases in
>> noticeably high quality from a Debian point of view. One should look
>> at them before putting them in production.
>
>I have a very different observations regarding upstream packaging for Debian.
>Most vendor packages are of notoriously bad quality, disconnected from Debian 
>practices and bug reports, not benefiting from QA and continuous testing, 
>heavily compromising on policy compliance, not following library transitions, 
>etc., etc.

You're right, that makes them unsuitable for a Debian release.

And yes, I have also seen "upstream" packages that were obviousy
aliened rpm packages with zero adaption to the .deb world, including
vendor instructions to dpkg --install them with --force-everything,
very obviously breaking our dependency mechanisms.

And I have also seen "upstream" packages that went ahead in their
postinst to replace /lib//libc.so.6 with their own, older
version.

But I have also seen the Ubuquiti .debs, the Upstream docker .debs and
the Upstream icinga .debs, which find a rather enticing middle way
between being of high quality _and_ offering current Software for
Debian stable. I do not have big doubts that the Upstream k8s packages
will also fall in this category (not having looked at them).

>The very existence of vendor packaging often indicates unwillingness to do 
>the job properly, up to our standards, and maybe even unwillingness to 
>cooperate with Debian to achieve proper integration. Precisely because it is 
>easier to do a quick sloppy work that won't be accepted without much 
>improvements.

Why would someone jump through our burning loops if the package is not
going to be in stable anyway, or horribly outdated when it has finally
reached stable. There are projects that don't want to support their
software in a version that has been released two years ago. That's a
valid approach, and one that I can fully understand from a software
project's point of view.

>>From vendor prospective having their own packages asserts exclusive control - 
>a something that some appreciate more than proper integration with Debian or 
>benefits of distributing software with/through Debian.

You can offer proper (or just "good enough") integration into Debian
yourself, and having an obsolete version of the software distributed
with/through Debian is (rightfully) seen a liabilty by some upstreams,
not as an asset.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Thomas Goirand]

On 3/28/20 2:41 AM, Dmitry Smirnov wrote:
> On Friday, 27 March 2020 7:08:35 PM AEDT Marc Haber wrote:
>> Many upstreams deliver .deb packages of their current releases in
>> noticeably high quality from a Debian point of view. One should look
>> at them before putting them in production.
> 
> I have a very different observations regarding upstream packaging for Debian.
> Most vendor packages are of notoriously bad quality, disconnected from Debian 
> practices and bug reports, not benefiting from QA and continuous testing, 
> heavily compromising on policy compliance, not following library transitions, 
> etc., etc.
> 
> The very existence of vendor packaging often indicates unwillingness to do 
> the job properly, up to our standards, and maybe even unwillingness to 
> cooperate with Debian to achieve proper integration. Precisely because it is 
> easier to do a quick sloppy work that won't be accepted without much 
> improvements.

+1 to all you wrote, Dmitry.

I have very rarely found good upstream packages. Even worth: very few
upstream provide the source package (and when they do, it's extremely
hard to find).

Cheers,

Thomas Goirand (zigo)

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Friday, 27 March 2020 7:08:35 PM AEDT Marc Haber wrote:
> Many upstreams deliver .deb packages of their current releases in
> noticeably high quality from a Debian point of view. One should look
> at them before putting them in production.

I have a very different observations regarding upstream packaging for Debian.
Most vendor packages are of notoriously bad quality, disconnected from Debian 
practices and bug reports, not benefiting from QA and continuous testing, 
heavily compromising on policy compliance, not following library transitions, 
etc., etc.

The very existence of vendor packaging often indicates unwillingness to do 
the job properly, up to our standards, and maybe even unwillingness to 
cooperate with Debian to achieve proper integration. Precisely because it is 
easier to do a quick sloppy work that won't be accepted without much 
improvements.

>From vendor prospective having their own packages asserts exclusive control - 
a something that some appreciate more than proper integration with Debian or 
benefits of distributing software with/through Debian.

-- 
Regards,
 Dmitry Smirnov.

---

You have enemies? Good. That means you've stood up for something,
sometime in your life.
-- Victor Hugo, "Villemain", 1845
   (often misattributed to Winston Churchill)


signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Thomas Goirand]

On 3/26/20 11:52 PM, Dmitry Smirnov wrote:
>> Applying the Debian "library" policy on go code is imho impossible -
>> there are no sonames, often no proper releases. The way how Debian
>> packages source code does not fit for go.
> 
> The way how Golang community handled dependencies was insane for years, only 
> being somewhat stabilized recently. There are no reasons to think that Goland 
> is too special when it comes to life cycle and versioning of libraries.
> More and more upstreams do the right thing and Debian approach mostly works  
> when applied properly.

I have the feeling that we get the same thing regularly, with new
language coming in. And after a few years, everything settles all by
itself slowly... :)

Thanks for all your work in Debian Dmitry!

Cheers,

Thomas Goirand (zigo)

Re: What to do when DD considers policy to be optional? [kubernetes]

[Christian Kastner]

On 27.03.20 01:57, Paul Wise wrote:
> On Thu, Mar 26, 2020 at 8:30 PM Christian Kastner wrote:
> 
>> [Well, technically, you could use your own lawyer to perform the due
>> diligence and have them submit any necessary changes to the BTS, but I
>> think it's safe to assume that that is a theoretical example.]
> 
> The OSI started ClearlyDefined, which aims to do just that (and more)
> for both Debian and other communities, using automated tools
> (scanCode) and human curation of the results.
> 
> https://clearlydefined.io/?type=debsrc=releaseDate=true
> https://clearlydefined.io/about
> https://docs.clearlydefined.io/contributing-data
> https://wiki.debian.org/CopyrightReviewTools
> https://github.com/nexB/scancode-toolkit/
> https://fosdem.org/2020/schedule/event/rust_license_clearlydefined/

Interesting. While I personally believe this would be insufficient as
due diligence [*], it would definitely be a great assistance in carrying
out thereof, and help bring "surprises" to light (eg a BSD-licensed
software containing some GPL code).

One frequent reason why proprietary software wins over FOSS in corporate
environments is because the vendor is willing to indemnify the client.
So by buying $FOO, you can spare yourself all this legal headache, the
problem is simply shifted to the vendor. That's a really hard offer to
beat, business-wise.

[*] To use extreme examples, in SCO v. IBM and Google v. Oracle cases,
they went file-by file. In the first trial to the latter case, a 9-line
function was found to be infringing. (This was later reversed, but
that's still bad enough).

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Thu, 26 Mar 2020 20:22:18 +1100, Dmitry Smirnov
 wrote:
>On Thursday, 26 March 2020 7:27:33 PM AEDT Marc Haber wrote:
>> I still wouldn't dare pulling testing or unstable packages to
>> production systems. It's like a worst-of-all-worlds approach.
>
>Not at all. Packaging is an extra layer of safety. Upstream release may be 
>outright broken, unusable or buggy with critical regressions. Presumably 
>maintainer at least tested the packages maybe even by using.

Many upstreams deliver .deb packages of their current releases in
noticeably high quality from a Debian point of view. One should look
at them before putting them in production.

With all due respect, I'd prefer not getting laughed at by the
upstream community over taking advantage of the Debian package
maintainer's work and experience for many programs if they're
sufficiently fast-paced.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Paul Wise]

On Thu, Mar 26, 2020 at 8:30 PM Christian Kastner wrote:

> [Well, technically, you could use your own lawyer to perform the due
> diligence and have them submit any necessary changes to the BTS, but I
> think it's safe to assume that that is a theoretical example.]

The OSI started ClearlyDefined, which aims to do just that (and more)
for both Debian and other communities, using automated tools
(scanCode) and human curation of the results.

https://clearlydefined.io/?type=debsrc=releaseDate=true
https://clearlydefined.io/about
https://docs.clearlydefined.io/contributing-data
https://wiki.debian.org/CopyrightReviewTools
https://github.com/nexB/scancode-toolkit/
https://fosdem.org/2020/schedule/event/rust_license_clearlydefined/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Friday, 27 March 2020 5:42:47 AM AEDT Bernd Zeimetz wrote:
> B) there are reasons why people recommend not to use the packaged
> versions of docker.io. No opinion on the others, never touched them.

The are some valid reasons, for example version in "stable" is too old and 
the package (and more importantly, its dependency tree) was stabilized only 
recently. We did a big job fixing past design failures and mistakes by re-
factoring Docker packaging significantly. Now the package is in much better 
shape than it was.

I doubt that your "life outside Debian" argument is valid when we are 
discussing Debian packaging. The approach to build Golang applications 
against packaged reusable libraries is valid.


> Applying the Debian "library" policy on go code is imho impossible -
> there are no sonames, often no proper releases. The way how Debian
> packages source code does not fit for go.

The way how Golang community handled dependencies was insane for years, only 
being somewhat stabilized recently. There are no reasons to think that Goland 
is too special when it comes to life cycle and versioning of libraries.
More and more upstreams do the right thing and Debian approach mostly works  
when applied properly.

-- 
Best wishes,
 Dmitry Smirnov.

---

The further a society drifts from the truth, the more it will hate those
that speak it.
-- George Orwell


signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Johannes Schauer]

Quoting Andrej Shadura (2020-03-26 22:24:41)
> On Thu, 26 Mar 2020 at 21:01, Russ Allbery  wrote:
> > > An example: commercial users. They need to know *exactly* what they
> > > are running and under which licenses. They often want to be holier not
> > > only than the Pope, but holier than the whole population of Poland,
> > > Italy and Spanish-speaking countries altogether (I hope I don’t offend
> > > anyone with this comparison, it’s meant as a joke).
> > Could you provide some more details about this?  Statements from those
> > companies about what they care about exactly, or open source policies that
> > you can point at?  I ask because this is contrary to my own personal
> > experience where commercial users care about the top-line license
> > (including not wanting to use licenses that we consider free) but do not
> > care about the work that Debian does beyond that and routinely use software
> > based on the declared upstream license on GitHub without giving it a second
> > though.  However, my personal experience is limited, and I'd be happy to be
> > educated!
> Car industry. They prefer to have nothing to do with GPL-3 and related
> licenses. They also want to know for sure when there’s something with
> undeclared or unknown license or something completely non-free that flew
> under the radar. As it is now, they cannot rely on debian/copyright files
> because often they’re out of date, sometimes up to ten years old. For
> Apertis, we had to build our own machinery based on scan-copyright and, in
> future, on Fossology, to attempt to mitigate that to some degree.

University. We maintain a GPL software of which we are also selling proprietary
licenses to companies. For this purpose our software must not link against any
GPL code. Recently we found by accident that a single file from an otherwise
BSD licensed 3rd party library we use was licensed under GPL. It was only a few
days before we had to hand our software to our customer so it would've spared
us a lot of headaches if that 3rd party library had a well documented
d/copyright file where we could've easily spotted that one GPL file.

signature.asc
Description: signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Andrej Shadura]

Hi,

On Thu, 26 Mar 2020 at 21:01, Russ Allbery  wrote:
> > An example: commercial users. They need to know *exactly* what they
> > are running and under which licenses. They often want to be holier not
> > only than the Pope, but holier than the whole population of Poland,
> > Italy and Spanish-speaking countries altogether (I hope I don’t offend
> > anyone with this comparison, it’s meant as a joke).

> Could you provide some more details about this?  Statements from those
> companies about what they care about exactly, or open source policies that
> you can point at?  I ask because this is contrary to my own personal
> experience where commercial users care about the top-line license
> (including not wanting to use licenses that we consider free) but do not
> care about the work that Debian does beyond that and routinely use
> software based on the declared upstream license on GitHub without giving
> it a second though.  However, my personal experience is limited, and I'd
> be happy to be educated!

Car industry. They prefer to have nothing to do with GPL-3 and related
licenses. They also want to know for sure when there’s something with
undeclared or unknown license or something completely non-free that
flew under the radar. As it is now, they cannot rely on
debian/copyright files because often they’re out of date, sometimes up
to ten years old. For Apertis, we had to build our own machinery based
on scan-copyright and, in future, on Fossology, to attempt to mitigate
that to some degree.

-- 
Cheers,
  Andrej

Re: What to do when DD considers policy to be optional? [kubernetes]

[Johannes Schauer]

Hi,

Quoting Russ Allbery (2020-03-25 03:25:49)
> Michael Lustfield  writes:
> > One last thing to consider... NEW reviews are already an intense process.
> > If this package hit NEW /and/ we allowed vendored libs, you could safely
> > expect me to never complete that particular review. I doubt I'm the only
> > one; that's essentially ~200 package reviews wrapped into 1.
> I'll repeat a point that I made earlier but put a bit of a sharper point on
> it: We should thoughtfully question whether the current approach to license
> review that we as a project ask ftpmasters to do is a correct investment of
> project resources.
> 
> The current approach to NEW license review is not a law of nature.  It's a
> choice.  Other choices are possible, such as trusting license declarations
> by upstream and then handling upstream mistakes that are later discovered
> as RC bugs.  The project is much better at sharing the load of handling RC
> bugs than it is at sharing the load of NEW license reviews.

the sub-thread as started by this mail only explores the option of reducing how
thoroughly we have to document copyright of the source code we put into Debian.
I wanted to point out that the original problem that started this was the
burden this status-quo puts on the FTP masters and how long packages remain in
NEW due to the required review effort.

So while we are contemplating our choices I wanted to point out that reducing
the quality of d/copyright is not the only possible option to tackle these
issues. The following ideas are not new but I wanted to just list them because
this thread seems to suggest that reducing our d/copyright thoroughness was the
only option we have.

For example it is also our choice that only FTP master are allowed to do the
copyright review and that the uploaded source must not leave the machine it was
uploaded to (even though it probably also lives on salsa at the same time). The
review process could be spread across more people or it could be open to the
public or to DD eyes only.  Yes, this introduces risks but taking this risk is
again a choice that we can make. It could be such that it needs X DDs to agree
and then a package clears the NEW queue or automated scripts could do a
plausibility check upfront. Especially teams that already know how the typical
layout of their domain best could work together to allow quick introduction of
new packages without reducing their quality.

What I'm saying is, that "reduce how good our d/copyright files have to be" is
not the only way to reduce burden on FTP master or to reduce the amount of time
packages spend in NEW. I'm also all in for reducing both but I would dislike it
very much if the quality of d/changelog would suffer as a result.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Christian Kastner]

On 26.03.20 19:57, Andrej Shadura wrote:
> An example: commercial users. They need to know *exactly* what they
> are running and under which licenses.

The only way to know that is by performing your own due diligence.

> They are often bound by regulations with heavy fines for violating
> them, and not only fines, but a threat of your product being banned,
> and that often means they want only specific licenses in their
> products.

There is no way whatsoever that a regulator, or a court, will accept
checking a single third-party file (debian/copyright), created by an
unpaid volunteer one has never even met, much less have a business
relation with, as proper due diligence in a copyright infringement case.

[Well, technically, you could use your own lawyer to perform the due
diligence and have them submit any necessary changes to the BTS, but I
think it's safe to assume that that is a theoretical example.]

Don't get me wrong, debian/copyright is certainly useful in general, but
the only value in a legal conflict I can see is for Debian, namely to
demonstrate our own compliance when distributing binaries (see Scott's
and Sean's replies).

Re: What to do when DD considers policy to be optional? [kubernetes]

[Russ Allbery]

Andrej Shadura  writes:

> An example: commercial users. They need to know *exactly* what they
> are running and under which licenses. They often want to be holier not
> only than the Pope, but holier than the whole population of Poland,
> Italy and Spanish-speaking countries altogether (I hope I don’t offend
> anyone with this comparison, it’s meant as a joke).

Could you provide some more details about this?  Statements from those
companies about what they care about exactly, or open source policies that
you can point at?  I ask because this is contrary to my own personal
experience where commercial users care about the top-line license
(including not wanting to use licenses that we consider free) but do not
care about the work that Debian does beyond that and routinely use
software based on the declared upstream license on GitHub without giving
it a second though.  However, my personal experience is limited, and I'd
be happy to be educated!

-- 
Russ Allbery (r...@debian.org)  

Re: What to do when DD considers policy to be optional? [kubernetes]

[Scott Kitterman]

On Thursday, March 26, 2020 3:27:18 PM EDT Kyle Edwards wrote:
> On Thu, 2020-03-26 at 19:57 +0100, Andrej Shadura wrote:
> > An example: commercial users. They need to know *exactly* what they
> > are running and under which licenses. They often want to be holier
> > not
> > only than the Pope, but holier than the whole population of Poland,
> > Italy and Spanish-speaking countries altogether (I hope I don’t
> > offend
> > anyone with this comparison, it’s meant as a joke). They are often
> > bound by regulations with heavy fines for violating them, and not
> > only
> > fines, but a threat of your product being banned, and that often
> > means
> > they want only specific licenses in their products.
> > 
> > And then there are Debian derivatives that cater to such commercial
> > users. And guess what? The users tell the makers of the derivatives
> > debian/copyright data that comes from Debian is not sufficiently
> > strict or precise, and for that they need to set up their own
> > processes to double-check and review the Debian copyright data.
> 
> Is it Debian's responsibility to cater to commercial users though?
> Debian is not receiving commercial funding, or at least not directly.
> It seems to me like the responsibility for this work falls on the
> commercial users, not Debian.
> 
> Kyle

Debian claims its users are its priority.  Not a specific sub-set of them.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Kyle Edwards]

On Thu, 2020-03-26 at 19:57 +0100, Andrej Shadura wrote:
> An example: commercial users. They need to know *exactly* what they
> are running and under which licenses. They often want to be holier
> not
> only than the Pope, but holier than the whole population of Poland,
> Italy and Spanish-speaking countries altogether (I hope I don’t
> offend
> anyone with this comparison, it’s meant as a joke). They are often
> bound by regulations with heavy fines for violating them, and not
> only
> fines, but a threat of your product being banned, and that often
> means
> they want only specific licenses in their products.
> 
> And then there are Debian derivatives that cater to such commercial
> users. And guess what? The users tell the makers of the derivatives
> debian/copyright data that comes from Debian is not sufficiently
> strict or precise, and for that they need to set up their own
> processes to double-check and review the Debian copyright data.

Is it Debian's responsibility to cater to commercial users though?
Debian is not receiving commercial funding, or at least not directly.
It seems to me like the responsibility for this work falls on the
commercial users, not Debian.

Kyle

Re: What to do when DD considers policy to be optional? [kubernetes]

[Andrej Shadura]

Hi,

On Thu, 26 Mar 2020 at 12:40, Christian Kastner  wrote:
> There's this expression in German for when one takes a policy too far:
> "Don't try to be holier than the Pope".
>
> But that's how maintaining debian/copyright has come to feel to me. We
> still apply a level of detail that seems out of place in today's world.
> It's such an odd position to be in to sit between upstream and
> contributor and discuss with them the legalities of one contribution
> you've hunted down, when it is evident that it was just "a
> contribution", and neither could understand why I have to make such a
> big deal out of it. [3]
>
> So, I ask: what for? Policy for policy's sake?

I beg to disagree. We do this for our users. They, believe it or not,
often need it.

An example: commercial users. They need to know *exactly* what they
are running and under which licenses. They often want to be holier not
only than the Pope, but holier than the whole population of Poland,
Italy and Spanish-speaking countries altogether (I hope I don’t offend
anyone with this comparison, it’s meant as a joke). They are often
bound by regulations with heavy fines for violating them, and not only
fines, but a threat of your product being banned, and that often means
they want only specific licenses in their products.

And then there are Debian derivatives that cater to such commercial
users. And guess what? The users tell the makers of the derivatives
debian/copyright data that comes from Debian is not sufficiently
strict or precise, and for that they need to set up their own
processes to double-check and review the Debian copyright data.

So if anything, we need to pay *more* attention to copyrights, not less.

-- 
Cheers,
  Andrej

Re: What to do when DD considers policy to be optional? [kubernetes]

[Bernd Zeimetz]

On 3/25/20 11:30 PM, Dmitry Smirnov wrote:
> Given that logic even re-compiling using different compiler would not be 
> trustworthy. And indeed some people make exactly that argument -- "use our 
> tested binary" as one can't be sure if re-compiling introduces any bugs.

Indeed I'd expect it that you'd at least compile it using the same
golang version.

> That questions the very usability of source code releases, whether you 
> understand it or not.
> 
> With this and your next arguments you are questioning the very usability of 
> packaging and I might agree that Kubernetes may not be worth packaged, 
> especially if we can't do it properly.


k8s is a way too fast moving target to be able to package it in a sane
way. It will be versions behind already when we release Debian.


>> What you suggest is a nice idea, but hard for go sources and impossible
>> for packages like k8s.
> 
> I don't know how to respond nicely when someone who did not maintain a 
> complex Golang application tells me that the way myself and others maintain 
> packages like docker,io, consul, nomad, vault, syncthing is "impossible"...

Not sure how I should respond nicely, but

A) you have no clue what I do (there is life outside of Debian, you know)

B) there are reasons why people recommend not to use the packaged
versions of docker.io. No opinion on the others, never touched them.



Applying the Debian "library" policy on go code is imho impossible -
there are no sonames, often no proper releases. The way how Debian
packages source code does not fit for go.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: What to do when DD considers policy to be optional? [kubernetes]

[Simon Richter]

Hi,

On 25.03.20 23:39, Dmitry Smirnov wrote:

>> Software packages like kubernetes, docker, and many of the other "hip
>> tools of the day" are moving way too fast for our release scheme.

> It is worth remembering that Debian is not only a stable release.
> Statically built Golang apps are easy to install from testing/unstable into 
> stable and packaging still provides certain benefits.

That, and in theory we also have a "volatile" archive for things that
move too fast.

> Stable Debian release protects users from sudden unexpected and disruptive 
> changes but some prefer to run their systems from "testing".

I'd expect people running container hosts to prefer stable, but if
upstream doesn't provide a "long term support" version (or anything that
actually has a version number instead of a commit ID) and at the same
time the software is too complex to branch off and maintain ourselves,
then our hands are kind of tied.

FWIW, I used to maintain software like this, and filed a "this software
is unstable and should not be part of a stable release" RC bug against
it, that's a completely valid state for a package to be in. For anything
but container host software, the existence of containers even makes this
somewhat manageable.

We have an awful lot of packages with version numbers of the forms
"0+git...", "0.0.2019..." or just a single large number, because
upstream does not believe in traditional releases, but the processes
Debian was built around expect software to be released and supported for
a while and expect maintainers to be able to forward bug reports
upstream and get a better reply than "tell your users to upgrade".

Getting Firefox and Thunderbird as ESR versions into Debian was a
massive amount of pain, but in the end it seems to have paid off,
because we managed to convince upstream that there is demand for stable
releases from users who do not want to roll out updates constantly and
deal with the resulting stability issues.

As a distribution, it is our job to communicate users' expectations to
upstream, not just to facilitate an unidirectional flow of packages.
Many upstreams already distribute .deb files from a robust mirror
infrastructure, our value proposition should be a bit stronger than "you
don't need to configure an additional source".

   Simon



signature.asc
Description: OpenPGP digital signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Scott Kitterman]

On Thursday, March 26, 2020 7:40:42 AM EDT Christian Kastner wrote:
> On 25.03.20 15:50, Scott Kitterman wrote:
> > The FTP Team review of debian/copyright is about DFSG and upstream license
> > compliance.  Most licenses require things like copyright statement
> > preservation in binary distribution and debian/copyright is how we do
> > that.  Occasionally, in the process of evaluating if Debian would be in
> > compliance with the upstream license and if the license meets our DFSG
> > requirements, we detect larger issues.
> > 
> > When I'm reviewing something for New I take upstream's copyright/license
> > statements at face value unless there's something too obvious to be
> > ignored.  I certainly don't go hunting for reasons not to like upstream
> > licensing statements, but sometimes they slap you in the face.
> > 
> > By far the most common case for rejections is incomplete debian/copyright
> > and that's unrelated to trusting upstream.  Personally, I don't think the
> > problem is that the bar is too high.  Almost all debian/copyright issue I
> > find could have been located before upload if the maintainer had simply
> > grep -ir copyright * | less and checked the results before uploading.
> It's just that this last step frequently turns out to be a very draining
> rabbit hole. [1]
> 
> Frustratingly, the smaller the contribution, the deeper the hole. Larger
> contributions tend to have explicit copyright information, so it's
> easier to determine and document. Smaller ones sometimes just document
> only the author. [2]
> 
> I find the analysis of this to be totally draining work. And while I
> could see a point to this in the past, I no longer do.
> 
> FOSS began as a movement where legal and political aspects were on the
> front line, and distributions like Debian were the primary channel to
> access FOSS, so it's easy to understand why debian/copyright was
> initially given so much importance.
> 
> Today, (IMO most of) "FOSS" is people creating, sharing and contributing
> stuff on GitHub and similar platforms. GitHub et al. have made hosting
> and contributing for/to projects so trivial that people /just do it/. It
> comes naturally. The legal and political aspects are no longer the front
> line. In many cases, they even have become overhead.
> 
> There's this expression in German for when one takes a policy too far:
> "Don't try to be holier than the Pope".
> 
> But that's how maintaining debian/copyright has come to feel to me. We
> still apply a level of detail that seems out of place in today's world.
> It's such an odd position to be in to sit between upstream and
> contributor and discuss with them the legalities of one contribution
> you've hunted down, when it is evident that it was just "a
> contribution", and neither could understand why I have to make such a
> big deal out of it. [3]
> 
> So, I ask: what for? Policy for policy's sake?
> 
> Why not just begin with a simple debian/copyright that documents, in
> very short form, what upstream documents in LICENSE and COPYING, and
> only refer to AUTHORS, etc., and only begin to look at further details
> once somebody actually /complains/ about them.
> 
> In any case, let's keep in mind that we are just one of the
> distributors. Now that it has become so easy to file complaints with
> upstream (again, thanks to GitHub et al.), I'm all but certain that any
> legal issues with the original source would first be filed upstream,
> rather than for all downstreams individually.
> 
> I think we investing far too much time in a problem that no longer
> exists, and probably wasn't even that big of a problem in the past
> (unless anyone can provide an example where a detailed debian/copyright
> has ever paid off the effort it cost).
> 
> Christian
> 
> 
> [1] It only starts with 'grep -ir copyright *'. In many jurisdictions,
> copyright applies regardless of where it is expressly claimed or not, so
> 'grep -ir author *' is probably just as important as a signal, and
> people tend to add that a lot.
> 
> [2] Because why would anyone (who is not Oracle) actually care about a
> few dozen lines of code.
> 
> [3] Except, of course, where contributions are legally formalized, for
> example with a CLA.
> 
> PS: This is my personal view of the the matter as I originally
> experienced it when becoming a maintainer, and over time when fixing RC
> bugs related to policy, resp. observing other bugs. It's quite possible
> that it is my personal view that is completely off, and that the
> ftp-master view isn't that strict, at all.

I think you assume we're looking for more than we are.  We aren't asking 
anyone to research and document undocumented but technically legally 
assertable copyright claims.  From an FTP perspective we're after license 
compliance.  

If debian/copyright includes all the copyright notices that upstream does (or 
an equivalent), then that's all that's needed (there are exceptions, I have 
reviewed packages where upstream literally 

Re: What to do when DD considers policy to be optional? [kubernetes]

[Christian Kastner]

On 25.03.20 15:50, Scott Kitterman wrote:
> The FTP Team review of debian/copyright is about DFSG and upstream license 
> compliance.  Most licenses require things like copyright statement 
> preservation in binary distribution and debian/copyright is how we do that.  
> Occasionally, in the process of evaluating if Debian would be in compliance 
> with the upstream license and if the license meets our DFSG requirements, we 
> detect larger issues.  
> 
> When I'm reviewing something for New I take upstream's copyright/license 
> statements at face value unless there's something too obvious to be ignored.  
> I certainly don't go hunting for reasons not to like upstream licensing 
> statements, but sometimes they slap you in the face.
> 
> By far the most common case for rejections is incomplete debian/copyright and 
> that's unrelated to trusting upstream.  Personally, I don't think the problem 
> is that the bar is too high.  Almost all debian/copyright issue I find could 
> have been located before upload if the maintainer had simply grep -ir 
> copyright * | less and checked the results before uploading.

It's just that this last step frequently turns out to be a very draining
rabbit hole. [1]

Frustratingly, the smaller the contribution, the deeper the hole. Larger
contributions tend to have explicit copyright information, so it's
easier to determine and document. Smaller ones sometimes just document
only the author. [2]

I find the analysis of this to be totally draining work. And while I
could see a point to this in the past, I no longer do.

FOSS began as a movement where legal and political aspects were on the
front line, and distributions like Debian were the primary channel to
access FOSS, so it's easy to understand why debian/copyright was
initially given so much importance.

Today, (IMO most of) "FOSS" is people creating, sharing and contributing
stuff on GitHub and similar platforms. GitHub et al. have made hosting
and contributing for/to projects so trivial that people /just do it/. It
comes naturally. The legal and political aspects are no longer the front
line. In many cases, they even have become overhead.

There's this expression in German for when one takes a policy too far:
"Don't try to be holier than the Pope".

But that's how maintaining debian/copyright has come to feel to me. We
still apply a level of detail that seems out of place in today's world.
It's such an odd position to be in to sit between upstream and
contributor and discuss with them the legalities of one contribution
you've hunted down, when it is evident that it was just "a
contribution", and neither could understand why I have to make such a
big deal out of it. [3]

So, I ask: what for? Policy for policy's sake?

Why not just begin with a simple debian/copyright that documents, in
very short form, what upstream documents in LICENSE and COPYING, and
only refer to AUTHORS, etc., and only begin to look at further details
once somebody actually /complains/ about them.

In any case, let's keep in mind that we are just one of the
distributors. Now that it has become so easy to file complaints with
upstream (again, thanks to GitHub et al.), I'm all but certain that any
legal issues with the original source would first be filed upstream,
rather than for all downstreams individually.

I think we investing far too much time in a problem that no longer
exists, and probably wasn't even that big of a problem in the past
(unless anyone can provide an example where a detailed debian/copyright
has ever paid off the effort it cost).

Christian


[1] It only starts with 'grep -ir copyright *'. In many jurisdictions,
copyright applies regardless of where it is expressly claimed or not, so
'grep -ir author *' is probably just as important as a signal, and
people tend to add that a lot.

[2] Because why would anyone (who is not Oracle) actually care about a
few dozen lines of code.

[3] Except, of course, where contributions are legally formalized, for
example with a CLA.

PS: This is my personal view of the the matter as I originally
experienced it when becoming a maintainer, and over time when fixing RC
bugs related to policy, resp. observing other bugs. It's quite possible
that it is my personal view that is completely off, and that the
ftp-master view isn't that strict, at all.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Thursday, 26 March 2020 7:27:33 PM AEDT Marc Haber wrote:
> I still wouldn't dare pulling testing or unstable packages to
> production systems. It's like a worst-of-all-worlds approach.

Not at all. Packaging is an extra layer of safety. Upstream release may be 
outright broken, unusable or buggy with critical regressions. Presumably 
maintainer at least tested the packages maybe even by using.

Therefore I consider packages to be a better alternative than direct use of 
upstream software, even when you need to use a package from "testing".

Also all "stable" packages were once in "testing" and one can never be sure 
if any package was tested enough before it become "stable".

Careful use of canary pre-production systems where you roll updates first, 
before deploying to production, allows to mitigate risks of regressions due 
to software upgrades.

-- 
All the best,
 Dmitry Smirnov.

---

Platitude: an idea (a) that is admitted to be true by everyone, and (b)
that is not true.
-- H. L. Mencken


signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Thu, 26 Mar 2020 09:39:56 +1100, Dmitry Smirnov
 wrote:
>On Thursday, 26 March 2020 3:45:21 AM AEDT Marc Haber wrote:
>> Software packages like kubernetes, docker, and many of the other "hip
>> tools of the day" are moving way too fast for our release scheme.
>
>It is worth remembering that Debian is not only a stable release.
>Statically built Golang apps are easy to install from testing/unstable into 
>stable and packaging still provides certain benefits.

I still wouldn't dare pulling testing or unstable packages to
production systems. It's like a worst-of-all-worlds approach.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Thursday, 26 March 2020 3:45:21 AM AEDT Marc Haber wrote:
> Software packages like kubernetes, docker, and many of the other "hip
> tools of the day" are moving way too fast for our release scheme.

It is worth remembering that Debian is not only a stable release.
Statically built Golang apps are easy to install from testing/unstable into 
stable and packaging still provides certain benefits.

Stable Debian release protects users from sudden unexpected and disruptive 
changes but some prefer to run their systems from "testing".

With rapidly changing upstream projects users can't have peace anywhere 
neither in Debian nor with upstream.

Also most complex Golang apps usually don't even get into "stable" due to 
massive removals from "testing" on bugs in any of their dependency libraries 
(direct or indirect). Stabilising the whole Golang ecosystem so few heavy 
packages could migrate is a long effort but we are getting there...

-- 
All the best,
 Dmitry Smirnov.

---

Few people are capable of expressing with equanimity opinions which
differ from the prejudices of their social environment. Most people are
even incapable of forming such opinion.
-- Albert Einstein, from "Aphorisms for Leo Baeck;
   Opinions of Albert Einstein"


signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Thursday, 26 March 2020 7:38:12 AM AEDT Bernd Zeimetz wrote:
> ... you would know that it is only tested with the
> exact versions of the libraries as it was released with.
> 
> Choosing different versions is prone to introduce subtle bugs and should
> never be done and accepted if that k8s version is supposed to be ready
> for production usage.

Given that logic even re-compiling using different compiler would not be 
trustworthy. And indeed some people make exactly that argument -- "use our 
tested binary" as one can't be sure if re-compiling introduces any bugs.

That questions the very usability of source code releases, whether you 
understand it or not.

With this and your next arguments you are questioning the very usability of 
packaging and I might agree that Kubernetes may not be worth packaged, 
especially if we can't do it properly.


> What you suggest is a nice idea, but hard for go sources and impossible
> for packages like k8s.

I don't know how to respond nicely when someone who did not maintain a 
complex Golang application tells me that the way myself and others maintain 
packages like docker,io, consul, nomad, vault, syncthing is "impossible"...


> The DD you called inexperienced has done everything right.

If you are not with me on technical reasoning, at least you have to recognise 
that he circumvented FTP-masters policy: I'm having a hard time getting a 
package with several vendored libraries accepted through NEW process (e.g. 
Podman was rejected due to "many embedded packages in vendor/" with only 6 or 
7 private libs versus 120 libraries removed in favour of packaged ones).

Thinking that compliance with NEW acceptance practices and standards is only 
a one time thing for a first upload (when subsequently you can re-introduce 
hundreds of private libraries at your discretion) is disrespectful to ftp-
masters team and to established practices.

We have to conclude that either we accept the terms for future uploads or not 
package at all.

-- 
Cheers,
 Dmitry Smirnov.

---

We have now sunk to a depth at which restatement of the obvious is the
first duty of intelligent men.
-- George Orwell


signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Moritz Mühlenhoff]

Russ Allbery  schrieb:
> Michael Lustfield  writes:
>
>> One last thing to consider... NEW reviews are already an intense
>> process. If this package hit NEW /and/ we allowed vendored libs, you
>> could safely expect me to never complete that particular review. I doubt
>> I'm the only one; that's essentially ~200 package reviews wrapped into
>> 1.
>
> I'll repeat a point that I made earlier but put a bit of a sharper point
> on it: We should thoughtfully question whether the current approach to
> license review that we as a project ask ftpmasters to do is a correct
> investment of project resources.

Full ack!

> We do not *have* to do a detailed file-by-file review of the correctness
> of upstream's license metadata when packaging.  This is a choice.  By
> choosing to do this, we absolutely catch bugs... just like we would catch
> bugs if we did a detailed file-by-file review of any other property of
> upstream code.

Or even replace it with automated license detection to spot such bugs (as
provided by tools like Fossology), which could even be an ongoing thing
for every upload instead of "once for the initial upload" and "randomly 
when new new binary  packages" appear. Plus everyone keen on reviewing
copyright files is always able to report bugs in the BTS.

Cheers,
Moritz

Re: What to do when DD considers policy to be optional? [kubernetes]

[Bernd Zeimetz]

Hi,

> As a person who originally introduced Kubernetes to Debian I can say that it 
> is indeed a lot of work to maintain this package and to reuse packaged 
> libraries. But I've demonstrated that it is possible at least to some extent.

if you've maintained k8s you would know that it is only tested with the
exact versions of the libraries as it was released with.

Choosing different versions is prone to introduce subtle bugs and should
never be done and accepted if that k8s version is supposed to be ready
for production usage.

What you suggest is a nice idea, but hard for go sources and impossible
for packages like k8s.

The DD you called inexperienced has done everything right.


Bernd

-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: What to do when DD considers policy to be optional? [kubernetes]

[Vincent Bernat]

 ❦ 25 mars 2020 15:57 +01, Jonas Smedegaard:

>> rpm packages record the package license information in a one-line License:
>> field.
>
> Is your point that 9 lines can be reduced to one, or that 100 lines can 
> be reduced to one?
>
> It is legal in Debian to write debian/copyright files looking like
> this:

Redhat also drops the license files in /usr/share/licenses/packagename
without much structure.
-- 
Don't over-comment.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Tue, 24 Mar 2020 20:37:16 +, Jeremy Stanley 
wrote:
>If this represents the actual state of building Kubernetes, it's
>unclear to me why Debian would package it at all. I don't see the
>value to users in consuming Kubernetes from a Debian package if the
>result is compromising on Debian's vision and values so that they
>can get the exact same thing they'd have if they just used the
>Kubernetes community's recommended tooling to install it instead.
>I'm all for using the best tool for the job, and while I've been a
>die-hard Debian user for more than two decades I also don't install
>every last bit of software from its packages. Some software
>ecosystems have chosen to focus on tools and workflows which are
>incompatible with Debian's, but that doesn't mean that either one is
>inherently bad nor that they need to be integrated at all costs.

Software packages like kubernetes, docker, and many of the other "hip
tools of the day" are moving way too fast for our release scheme.
Additionally, many communities and developers stop caring for their
software once it has cleared the door and laugh upon people who are
using anything but their latest release.

I think we're not doing the world a favor packaging those software and
releasing it with our stable release, making our users appear on
community support channels with a version that is many months old and
will only trigger a "first, upgrade to our current version, check
whether your issue still exists and then come back" answer from the
upstream community. In the long run, people who don't know how our
releases work will make their judgement about Debian being "always out
of date", worsening our reputation.

We do provide a rock solid base system where people can install their
software, in the case of those fast moving things in the way upstream
wants them to. This causes a much better result for all those
involved.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Pierre-Elliott Bécue]

Le mardi 24 mars 2020 à 19:08:23+, Janos LENART a écrit :
> [snip]
> I do think there is a good case for Kubernetes to be an exception from 4.13 
> for
> now, just like other Go packages effectively are.

Using other examples to say "people do bas stuff elsewhere so I'm
entitled to do the same" is not a good practice.

And, really, comparing uploads with 4, 6, 10 bundles libraries (bad)
with an upload of a package with TWO HUNDRED libraries??? You shouldn't
have done that hoping that it'd slip through.

> It is a massively popular project topped only by the Linux kernel. We
> cannot afford not to have up to date versions in Debian, or have forks
> that no one can use.

Yes we can afford not to have an up to date version in Debian. We could
also afford not having Kubernetes into Debian.

And the fact that the project is popular is not a good reason either.

I'm sorry, but the way you do things and try to justify yourself
afterwards is not really something I find normal for a DD who commited
themselves to the standard the project defined as its own.

-- 
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528  F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Florian Weimer]

* Andrey Rahmatullin:

> Or you can look at the Redhat approach as a minimal working one.
> You know it can be done much easier and still work: in Redhat.

I think you are referring to a Fedora process, not a Red Hat process.
The Red Hat process does not seem much simpler than what ftpmaster are
doing, as far as I can tell.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Christian Kastner]

On 25.03.20 15:14, Tomas Pospisek wrote:
> I can be sure that if stuff lands in Debian then I won't get screwed
> by weird, dirty, missleading, underhanded licensing rules, which
> seems to be the standard outside the Open Source world and even on
> its fringes.
The only thing you can be sure about is that the declared license is up
to Debian standards.

No such thing can be said about most of the code itself, unless you have
an actual audit trail documenting its origin.

To use an extreme example: me grabbing code from $EMPLOYER and slapping
the GPL onto it, doesn't make it GPL.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Andrey Rahmatullin]

On Wed, Mar 25, 2020 at 03:57:29PM +0100, Jonas Smedegaard wrote:
> > > >> I'd contest this. Whenever Open Source standards come up in a
> > > >> discussion, Debian is always the gold reference. You know it can be 
> > > >> done
> > > >> right and it is: in Debian.
> > > > Or you can look at the Redhat approach as a minimal working one.
> > > > You know it can be done much easier and still work: in Redhat.
> > > 
> > > (in case it hasn't already been discussed in this thread, but don't
> > > bother rehashing...): What are they doing differently?
> > rpm packages record the package license information in a one-line License:
> > field.
> 
> Is your point that 9 lines can be reduced to one, or that 100 lines can 
> be reduced to one?
I'll repeat my point:

You know it can be done much easier and still work: in Redhat.

> If you are talking about omitting some licensing, then I fail to 
> recognize how that can be a "gold standard" which I believe is what you 
> claimed above.
I'm just saying that there is an approach of providing a gold standard and
there are other approaches.
The project decides what to provide.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Jonas Smedegaard]

Quoting Andrey Rahmatullin (2020-03-25 15:46:10)
> On Wed, Mar 25, 2020 at 03:32:20PM +0100, Tomas Pospisek wrote:
> > On 25.03.20 15:19, Andrey Rahmatullin wrote:
> > > On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
> > >> On 25.03.20 14:43, Christian Kastner wrote:
> > >>
> > >>> This is not to say that licensing is an unimportant issue -- it clearly
> > >>> is. But our analyze-and-document down-to-the-file approach is on the
> > >>> other extreme end of the spectrum, and it causes lots of tiresome work
> > >>> that nobody apart from us seems to care about.
> > >>
> > >> I'd contest this. Whenever Open Source standards come up in a
> > >> discussion, Debian is always the gold reference. You know it can be done
> > >> right and it is: in Debian.
> > > Or you can look at the Redhat approach as a minimal working one.
> > > You know it can be done much easier and still work: in Redhat.
> > 
> > (in case it hasn't already been discussed in this thread, but don't
> > bother rehashing...): What are they doing differently?
> rpm packages record the package license information in a one-line License:
> field.

Is your point that 9 lines can be reduced to one, or that 100 lines can 
be reduced to one?

It is legal in Debian to write debian/copyright files looking like this:

 debian/copyright ==
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: Foo
Upstream-Contact: https://github.com/foo/foo/issues
Source: https://github.com/foo/foo

Files: *
Copyright: Foo Bar
License: WTFPL
 Do what the fuck
 debian/copyright ==

Obviously that is legal only when it actually covers the full licensing 
situation for that source code!

If you are talking about omitting some licensing, then I fail to 
recognize how that can be a "gold standard" which I believe is what you 
claimed above.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Scott Kitterman]

On March 25, 2020 1:43:26 PM UTC, Christian Kastner  wrote:
>Russ,
>
>On 25.03.20 03:25, Russ Allbery wrote:
>> I'll repeat a point that I made earlier but put a bit of a sharper
>point
>> on it: We should thoughtfully question whether the current approach
>to
>> license review that we as a project ask ftpmasters to do is a correct
>> investment of project resources.
>> 
>> The current approach to NEW license review is not a law of nature. 
>It's a
>> choice.  Other choices are possible, such as trusting license
>declarations
>> by upstream and then handling upstream mistakes that are later
>discovered
>> as RC bugs.  The project is much better at sharing the load of
>handling RC
>> bugs than it is at sharing the load of NEW license reviews.
>> 
>> Most of the ecosystems that make extensive use of vendored packages
>also
>> make extensive use of license metadata.  Sometimes that license
>metadata
>> is wrong, and we will have to have a process for dealing with those
>> errors.  But the purpose of Debian is not to be a license checking
>service
>> for the entire free software world.  It's to build a distribution
>adhering
>> to a set of principles but with the understanding that we will
>sometimes
>> make errors and fix them later as bugs, not be perfect and error-free
>at
>> any of our principles, including our licensing principles.  For
>everything
>> other than licenses, we accept the risk that we will incorporate
>upstream
>> errors in our distribution until someone points them out via a bug
>report.
>> 
>> We do not *have* to do a detailed file-by-file review of the
>correctness
>> of upstream's license metadata when packaging.  This is a choice.  By
>> choosing to do this, we absolutely catch bugs... just like we would
>catch
>> bugs if we did a detailed file-by-file review of any other property
>of
>> upstream code.  For many other types of bugs (including ones that
>arguably
>> have a more severe impact on our users than licensing bugs, such as
>> security bugs), we often make trade-off decisions in favor of
>accepting a
>> risk of upstream mistakes and having to fix them later.
>
>if I could +1000 just one post, ever, it's this one.
>
>The vast majority of people developing Open Source Software nowadays
>just publish stuff on GitHub and friends. They fork code, they merge
>other people's contributions, etc.
>
>And at no point can I see those people engage in even a fraction of the
>bureaucracy that we follow when sharing, forking, publishing, and
>contributing code.
>
>This is not to say that licensing is an unimportant issue -- it clearly
>is. But our analyze-and-document down-to-the-file approach is on the
>other extreme end of the spectrum, and it causes lots of tiresome work
>that nobody apart from us seems to care about.
>
>> Other choices are possible, such as trusting license declarations
>> by upstream and then handling upstream mistakes that are later
>discovered
>> as RC bugs.
>
>This would be fantastic. Just fantastic.

This is basically what we do now, so don't get too excited.

The FTP Team review of debian/copyright is about DFSG and upstream license 
compliance.  Most licenses require things like copyright statement preservation 
in binary distribution and debian/copyright is how we do that.  Occasionally, 
in the process of evaluating if Debian would be in compliance with the upstream 
license and if the license meets our DFSG requirements, we detect larger 
issues.  

When I'm reviewing something for New I take upstream's copyright/license 
statements at face value unless there's something too obvious to be ignored.  I 
certainly don't go hunting for reasons not to like upstream licensing 
statements, but sometimes they slap you in the face.

By far the most common case for rejections is incomplete debian/copyright and 
that's unrelated to trusting upstream.  Personally, I don't think the problem 
is that the bar is too high.  Almost all debian/copyright issue I find could 
have been located before upload if the maintainer had simply grep -ir copyright 
* | less and checked the results before uploading.

Scott K
P.S. Speaking only for myself, no hats

Re: What to do when DD considers policy to be optional? [kubernetes]

[Adam Borowski]

On Wed, Mar 25, 2020 at 03:32:20PM +0100, Tomas Pospisek wrote:
> On 25.03.20 15:19, Andrey Rahmatullin wrote:
> > On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
> >> On 25.03.20 14:43, Christian Kastner wrote:
> >>
> >>> This is not to say that licensing is an unimportant issue -- it clearly
> >>> is. But our analyze-and-document down-to-the-file approach is on the
> >>> other extreme end of the spectrum, and it causes lots of tiresome work
> >>> that nobody apart from us seems to care about.
> >>
> >> I'd contest this. Whenever Open Source standards come up in a
> >> discussion, Debian is always the gold reference. You know it can be done
> >> right and it is: in Debian.
> > Or you can look at the Redhat approach as a minimal working one.
> > You know it can be done much easier and still work: in Redhat.
> 
> (in case it hasn't already been discussed in this thread, but don't
> bother rehashing...): What are they doing differently?

Bad: much poorer _technical_ standards.
Good: single-line description of copyright.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
⢿⡄⠘⠷⠚⠋⠀   --  on #linux-sunxi
⠈⠳⣄

Re: What to do when DD considers policy to be optional? [kubernetes]

[Andrey Rahmatullin]

On Wed, Mar 25, 2020 at 03:32:20PM +0100, Tomas Pospisek wrote:
> On 25.03.20 15:19, Andrey Rahmatullin wrote:
> > On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
> >> On 25.03.20 14:43, Christian Kastner wrote:
> >>
> >>> This is not to say that licensing is an unimportant issue -- it clearly
> >>> is. But our analyze-and-document down-to-the-file approach is on the
> >>> other extreme end of the spectrum, and it causes lots of tiresome work
> >>> that nobody apart from us seems to care about.
> >>
> >> I'd contest this. Whenever Open Source standards come up in a
> >> discussion, Debian is always the gold reference. You know it can be done
> >> right and it is: in Debian.
> > Or you can look at the Redhat approach as a minimal working one.
> > You know it can be done much easier and still work: in Redhat.
> 
> (in case it hasn't already been discussed in this thread, but don't
> bother rehashing...): What are they doing differently?
rpm packages record the package license information in a one-line License:
field.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Tomas Pospisek]

On 25.03.20 15:19, Andrey Rahmatullin wrote:
> On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
>> On 25.03.20 14:43, Christian Kastner wrote:
>>
>>> This is not to say that licensing is an unimportant issue -- it clearly
>>> is. But our analyze-and-document down-to-the-file approach is on the
>>> other extreme end of the spectrum, and it causes lots of tiresome work
>>> that nobody apart from us seems to care about.
>>
>> I'd contest this. Whenever Open Source standards come up in a
>> discussion, Debian is always the gold reference. You know it can be done
>> right and it is: in Debian.
> Or you can look at the Redhat approach as a minimal working one.
> You know it can be done much easier and still work: in Redhat.

(in case it hasn't already been discussed in this thread, but don't
bother rehashing...): What are they doing differently?

Re: What to do when DD considers policy to be optional? [kubernetes]

[Andrey Rahmatullin]

On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
> On 25.03.20 14:43, Christian Kastner wrote:
> 
> > This is not to say that licensing is an unimportant issue -- it clearly
> > is. But our analyze-and-document down-to-the-file approach is on the
> > other extreme end of the spectrum, and it causes lots of tiresome work
> > that nobody apart from us seems to care about.
> 
> I'd contest this. Whenever Open Source standards come up in a
> discussion, Debian is always the gold reference. You know it can be done
> right and it is: in Debian.
Or you can look at the Redhat approach as a minimal working one.
You know it can be done much easier and still work: in Redhat.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Tomas Pospisek]

On 25.03.20 14:43, Christian Kastner wrote:

> This is not to say that licensing is an unimportant issue -- it clearly
> is. But our analyze-and-document down-to-the-file approach is on the
> other extreme end of the spectrum, and it causes lots of tiresome work
> that nobody apart from us seems to care about.

I'd contest this. Whenever Open Source standards come up in a
discussion, Debian is always the gold reference. You know it can be done
right and it is: in Debian.

Having the possibility to look up to that standard and being able to
compare to it is a value and valuable in it self.

That said, Debian doesn't have to hold up that standard if it doesn't
deem it useful of course. Me personally I deem it very valuable. I can
be sure that if stuff lands in Debian then I won't get screwed by weird,
dirty, missleading, underhanded licensing rules, which seems to be the
standard outside the Open Source world and even on its fringes.
*t

Re: What to do when DD considers policy to be optional? [kubernetes]

[Christian Kastner]

Russ,

On 25.03.20 03:25, Russ Allbery wrote:
> I'll repeat a point that I made earlier but put a bit of a sharper point
> on it: We should thoughtfully question whether the current approach to
> license review that we as a project ask ftpmasters to do is a correct
> investment of project resources.
> 
> The current approach to NEW license review is not a law of nature.  It's a
> choice.  Other choices are possible, such as trusting license declarations
> by upstream and then handling upstream mistakes that are later discovered
> as RC bugs.  The project is much better at sharing the load of handling RC
> bugs than it is at sharing the load of NEW license reviews.
> 
> Most of the ecosystems that make extensive use of vendored packages also
> make extensive use of license metadata.  Sometimes that license metadata
> is wrong, and we will have to have a process for dealing with those
> errors.  But the purpose of Debian is not to be a license checking service
> for the entire free software world.  It's to build a distribution adhering
> to a set of principles but with the understanding that we will sometimes
> make errors and fix them later as bugs, not be perfect and error-free at
> any of our principles, including our licensing principles.  For everything
> other than licenses, we accept the risk that we will incorporate upstream
> errors in our distribution until someone points them out via a bug report.
> 
> We do not *have* to do a detailed file-by-file review of the correctness
> of upstream's license metadata when packaging.  This is a choice.  By
> choosing to do this, we absolutely catch bugs... just like we would catch
> bugs if we did a detailed file-by-file review of any other property of
> upstream code.  For many other types of bugs (including ones that arguably
> have a more severe impact on our users than licensing bugs, such as
> security bugs), we often make trade-off decisions in favor of accepting a
> risk of upstream mistakes and having to fix them later.

if I could +1000 just one post, ever, it's this one.

The vast majority of people developing Open Source Software nowadays
just publish stuff on GitHub and friends. They fork code, they merge
other people's contributions, etc.

And at no point can I see those people engage in even a fraction of the
bureaucracy that we follow when sharing, forking, publishing, and
contributing code.

This is not to say that licensing is an unimportant issue -- it clearly
is. But our analyze-and-document down-to-the-file approach is on the
other extreme end of the spectrum, and it causes lots of tiresome work
that nobody apart from us seems to care about.

> Other choices are possible, such as trusting license declarations
> by upstream and then handling upstream mistakes that are later discovered
> as RC bugs.

This would be fantastic. Just fantastic.

we have choice, but no change (Re: What to do when DD considers policy to be optional? [kubernetes])

[Holger Levsen]

On Tue, Mar 24, 2020 at 07:25:49PM -0700, Russ Allbery wrote:
> I'll repeat a point that I made earlier but put a bit of a sharper point
> on it: We should thoughtfully question whether the current approach to
> license review that we as a project ask ftpmasters to do is a correct
> investment of project resources.
> 
> The current approach to NEW license review is not a law of nature.  It's a
> choice.  Other choices are possible
[...]
> The project is much better at sharing the load of handling RC
> bugs than it is at sharing the load of NEW license reviews.
[...] 

ohh, yes!

So what's the way to change the project's choice on this?


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Wednesday, 25 March 2020 2:34:03 PM AEDT Michael Lustfield wrote:
> With regard to the kubernetes package, I don't see anything to indicate it
> was abandoned.

Sorry if I did not make it clear: the package was orphaned as per #886739.
The takeover was only technological. I don't dispute the ownership of the 
package.


> One person is
> interested in maintaining the package per our standards, and another is
> interested in getting it updated.

It is all about balance... I think we can have both: a package updated with 
respect to our standards or at least with best effort to respect standards. 

I'm not interested in maintaining Kubernetes any more. I simply can't afford 
to spend more time on it after investing so much already. I'm somewhat 
disappointed in upstream and their handling of the project. I'm not even 
using Kubernetes and my concerns are about packaging practices and 
methodologies.

-- 
All the best,
 Dmitry Smirnov.

---

Doublethink means the power of holding two contradictory beliefs in one's
mind simultaneously, and accepting both of them.
-- George Orwell, 1984.


signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Michael Lustfield]

Ehm... perhaps we should practice some de-escalation techniques, please. :/

On Wed, 25 Mar 2020 13:55:50 +1100
Dmitry Smirnov  wrote:
> On Wednesday, 25 March 2020 6:08:23 AM AEDT Janos LENART wrote:
> > Debian Policy, paragraph 4.13 states:  
> 
> There are several problems with how you did it too. You did not use anyone's 
> advise, ignored Salsa repository, threw away _everything_ and made no effort 
> to understand how and why things were implemented, let alone appreciated 
> prior work or tried to improve it. What you did is technological hijack of 
> the package, a gross violation of practices.
> 
> Imagine I'll upload a package to NEW, get in reviewed and accepted for what 
> it is then re-upload as something entirely different bundled with 500+ 
> dependencies that were not reviewed then claim that policy allows it?

I missed this earlier...

With regard to the kubernetes package, I don't see anything to indicate it was
abandoned. I'll also assume that the current maintainer (Dmitry) was not
contacted by Janos. Considering how quickly this started after the upload, this
seems like a safe bet.

Janos- If that's correct, then this would indeed be considered a "hostile
takeover" of the package. It would be nice to assume that this was simple
oversight of forgotten communication.

No matter the actual facts- please remember how important communication is to
our community.

So... with Dmitry still active in Debian and still holding interest in the
package, the path forward seems obvious, doesn't it? One person is interested
in maintaining the package per our standards, and another is interested in
getting it updated.

Regarding vendor/ libraries... [Was: Re: What to do when DD considers policy to be optional? [kubernetes]]

[Michael Lustfield]

On Tue, 24 Mar 2020 19:25:49 -0700
Russ Allbery  wrote:

> Michael Lustfield  writes:
> 
> > One last thing to consider... NEW reviews are already an intense
> > process. If this package hit NEW /and/ we allowed vendored libs, you
> > could safely expect me to never complete that particular review. I doubt
> > I'm the only one; that's essentially ~200 package reviews wrapped into
> > 1.  
> 
> I'll repeat a point that I made earlier but put a bit of a sharper point
> on it: We should thoughtfully question whether the current approach to
> license review that we as a project ask ftpmasters to do is a correct
> investment of project resources.
> 
> The current approach to NEW license review is not a law of nature.  It's a
> choice.  Other choices are possible, such as trusting license declarations
> by upstream and then handling upstream mistakes that are later discovered
> as RC bugs.  The project is much better at sharing the load of handling RC
> bugs than it is at sharing the load of NEW license reviews.
> 
> Most of the ecosystems that make extensive use of vendored packages also
> make extensive use of license metadata.  Sometimes that license metadata
> is wrong, and we will have to have a process for dealing with those
> errors.  But the purpose of Debian is not to be a license checking service
> for the entire free software world.  It's to build a distribution adhering
> to a set of principles but with the understanding that we will sometimes
> make errors and fix them later as bugs, not be perfect and error-free at
> any of our principles, including our licensing principles.  For everything
> other than licenses, we accept the risk that we will incorporate upstream
> errors in our distribution until someone points them out via a bug report.
> 
> We do not *have* to do a detailed file-by-file review of the correctness
> of upstream's license metadata when packaging.  This is a choice.  By
> choosing to do this, we absolutely catch bugs... just like we would catch
> bugs if we did a detailed file-by-file review of any other property of
> upstream code.  For many other types of bugs (including ones that arguably
> have a more severe impact on our users than licensing bugs, such as
> security bugs), we often make trade-off decisions in favor of accepting a
> risk of upstream mistakes and having to fix them later.

Scott K. reminded me that the policy in question is a "should" and not a
"must." This is a pretty important distinction that I forgot about. In
practice, it's useful for when a single file is taken from a large project,
sometimes including weird build tools.

Assuming d/copyright is actually correct (it's not..), then it's not a policy
violation. However, for all of the reasons mentioned, and especially because
most of that work was already complete, I have to agree with the term sloppy.

We obviously need to continue the vendor/ discussion, but I think switching
threads would be a good idea.

-- 
Michael Lustfield

Re: What to do when DD considers policy to be optional? [kubernetes]

[Wookey]

On 2020-03-24 19:08 +, Janos LENART wrote:
> Hi Dimitry, FTP masters and others,
> 4. TESTING. The Kubernetes releases are meticulously tested, with far greater
> technical resources that Debian can collectively muster.

On how many architectures? Debian's support of multiple architectures
often differentiates it from upstream, which sometimes knows little or
nothing of !x86. I don't know if kubernetes is doing a good job of
testing on other arches, but it's important to consider those when
thinking about the tradeoffs here.

I do know this is important software on arm64 too, and agree with Russ
that it's nice to have things packaged even where they are largely
what you would get from upstream, just because one only has to learn
one package manager, and can assume that policy has been followed at
least in terms of layout.

I am not at all happy with the tendency of some developer
groups/projects to bundle like crazy, but it's also true that there
are constraints on the degree to which we can and should (try to) fix
that for them. You make some good points in defense of your approach,
and thank you for responding cogently at length, but I'm not (yet)
convinced by the argument that we shouldn't try at all in this case.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Wednesday, 25 March 2020 6:08:23 AM AEDT Janos LENART wrote:
> I know Dimitry was fighting an uphill battle with kubernetes between 2016
> and 2018 and he experienced first hand the problems posed by vendored code.

No. This is a incorrect. Largest chunk of work that I did on Kubernetes was 
in 2015..2016 when I was packaging and introducing many new libraries and 
stabilizing their relationships. A lot of this work has been done with 
various upstreams and a lot of it was in Kubernetes' dependency packages such 
as docker.io. Licensing/copyright documentation and addressing numerous DFSG 
problems took a lot of time and effort to meet requirements for new packages. 
This effort took more than a year or work. Between 2016...2018 I barely 
touched Kubernetes as it was already orphaned.

Challenge is not with "vendored code" as such, or at least not the way you 
think.


> We see more and more software making excessive use of vendored code. Pretty
> much everything that is written in Go. Some of these are crucially
> important, like Docker or Kubernetes. So I understand the concern everyone
> has about how this fits with the Debian Policy.

No you don't understand.

Vendored code is nuisance but not a problem if/when you can throw it away in 
favor of packaged, tested, reusable libraries.

You've made no effort to reuse any existing packaged libraries. Most of them 
could be used without much effort. And there are many advantages of doing so 
beyond obvious benefits for security and de-duplication.

There is integrity of build to care about, and automated QA/CI checks for 
libraries to appreciate. Most of the libraries are tested on more 
architectures that upstreams ever test. We know when there is a problem and 
can do something about it. Concept of good quality reusable components is too 
powerful to throw it away just because you think it is inconvenient.

Vendored Golang libraries run no tests on build. IMHO Kubernetes bundle (with 
vendor directory) is unmaintainable and you can find how much upstream 
struggles with it to extent when they would not upgrade a component to fix a 
problem due to fears that everything might fall apart.

Using properly packages libraries is an advantage to maintainer and a benefit 
to a whole ecosystem.


> Debian Policy, paragraph 4.13 states:

Before we discuss the policy let me focus on your attitude first.

Kubernetes -- one of the most sophisticated packages -- was you _second_ ever 
upload to Debian. You are woefully inexperienced maintainer, knowing little 
if anything about team work, packaging practices, their meanings and 
implications. IMHO your interpretation of policy is mostly irrelevant as you 
are merely trying to use the policy to justify what you did.

There are several problems with how you did it too. You did not use anyone's 
advise, ignored Salsa repository, threw away _everything_ and made no effort 
to understand how and why things were implemented, let alone appreciated 
prior work or tried to improve it. What you did is technological hijack of 
the package, a gross violation of practices.

Imagine I'll upload a package to NEW, get in reviewed and accepted for what 
it is then re-upload as something entirely different bundled with 500+ 
dependencies that were not reviewed then claim that policy allows it?


> =
> I think this is the part that has the most bearing on the vendored code
> problem, especially the footnote. I agree with this principle. But we
> should apply it to the state of affairs in 2020, and to this specific
> situation.

Nonsense. In 2020 using packaged libraries is much easier than before.
You simply have no excuse not to do so. Again, you are not understanding what 
is the problem.


> Keeping all that in mind, here are the reasons why I think it is acceptable
> for now to package Kubernetes with the vendored code, and even the best
> solution that is available currently:

I keep telling you that it is not a best solution but the sloppiest and the 
most inferior one.


> 1. OTHER EXAMPLES.
> [...]
> - docker.io (58, including some that are vendored more than once within the
> same source package, but not including the fact that docker.io itself is
> made up of 7 tarballs)

Docker.io is a special, exceptionally difficult case which should be an 
example to you that even with Docker it is possible to leverage packaged 
libraries. Docker upstream is one of the worst with their abuse of versioning 
practices. On top of that Docker code base is shipped in several name spaces 
that make it impossible to package some components separately due to mutual 
circular dependencies. In Docker we use strategically bundled components only 
when necessary.


> - kubernetes (20 for the previous version, 200 now)
> - prometheus (4)
> - golang (4)
> None of these were REJECTed, and please don't sabotage these packages now

Perhaps Nomad or recently accepted Vault would serve as a better example. 
Vault is one of packages with 

Re: What to do when DD considers policy to be optional? [kubernetes]

[Russ Allbery]

Michael Lustfield  writes:

> One last thing to consider... NEW reviews are already an intense
> process. If this package hit NEW /and/ we allowed vendored libs, you
> could safely expect me to never complete that particular review. I doubt
> I'm the only one; that's essentially ~200 package reviews wrapped into
> 1.

I'll repeat a point that I made earlier but put a bit of a sharper point
on it: We should thoughtfully question whether the current approach to
license review that we as a project ask ftpmasters to do is a correct
investment of project resources.

The current approach to NEW license review is not a law of nature.  It's a
choice.  Other choices are possible, such as trusting license declarations
by upstream and then handling upstream mistakes that are later discovered
as RC bugs.  The project is much better at sharing the load of handling RC
bugs than it is at sharing the load of NEW license reviews.

Most of the ecosystems that make extensive use of vendored packages also
make extensive use of license metadata.  Sometimes that license metadata
is wrong, and we will have to have a process for dealing with those
errors.  But the purpose of Debian is not to be a license checking service
for the entire free software world.  It's to build a distribution adhering
to a set of principles but with the understanding that we will sometimes
make errors and fix them later as bugs, not be perfect and error-free at
any of our principles, including our licensing principles.  For everything
other than licenses, we accept the risk that we will incorporate upstream
errors in our distribution until someone points them out via a bug report.

We do not *have* to do a detailed file-by-file review of the correctness
of upstream's license metadata when packaging.  This is a choice.  By
choosing to do this, we absolutely catch bugs... just like we would catch
bugs if we did a detailed file-by-file review of any other property of
upstream code.  For many other types of bugs (including ones that arguably
have a more severe impact on our users than licensing bugs, such as
security bugs), we often make trade-off decisions in favor of accepting a
risk of upstream mistakes and having to fix them later.

-- 
Russ Allbery (r...@debian.org)  

Re: What to do when DD considers policy to be optional? [kubernetes]

[Michael Lustfield]

On Wed, 25 Mar 2020 02:07:13 +0100
Marco d'Itri  wrote:
> On Mar 24, Russ Allbery  wrote:
> [...]
> The main reason for mostly forbidding vendored libraries has been that 
> the security team rightly argues that in the event of a security issue 
> it would be too much work to 1) hunt each package using a vendored 
> library and 2) patch and rebuild all of them.
> This does not really matter for Go and Rust software because 1) the list 
> of (vendored) dependencies can be extracted automatically at build time 
> and 2) all this software would have to be rebuilt anyway since these 
> languages do not support or do not use dynamic linking.
> 
> Also, shared libraries save memory when multiple programs using them are 
> run concurrently, but nowadays this kind of saving is rarely meaningful.

My point earlier was that it's not just a security problem. We have established
that Debian does, and will continue to, care about fulfilling license
obligations, including distributing license and copyright information with the
package.

Bluntly put, I have yet to see a project that doesn't treat 'vendor/' as a black
box of collected libraries. These directories are rarely reviewed and it is
trivial (default) to wind up including extra libs between builds. Even if it's
possible to restrict that list, I doubt it would be done.

I was (pleasantly) surprised to see d/copyright updated in this package.
However, this is not maintainable-
   https://sources.debian.org/src/kubernetes/1.17.4-1/debian/copyright/


> Because of these reasons maybe we should consider supporting vendored 
> libraries in some cases.

My opinion is still a very hard "No."

This sounds (to me) rather akin to- "since app devs tend to be lazy, should we
be the same?"


One last thing to consider... NEW reviews are already an intense process. If
this package hit NEW /and/ we allowed vendored libs, you could safely expect me
to never complete that particular review. I doubt I'm the only one; that's
essentially ~200 package reviews wrapped into 1.

-- 
Michael Lustfield

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marco d'Itri]

On Mar 24, Russ Allbery  wrote:

> (The Rust team is trying the package everything approach with some success
> but is uncovering other limitations in our processes and tools.)  But
"Some" success indeed. My personal experience with trying to package 
routinator has been awful, and there is still no actualy package in the 
archive after many months because it depends on a version of a library 
which is different from the version that we have in the archive, and 
there is nothing wrong with this in the Rust world.

The main reason for mostly forbidding vendored libraries has been that 
the security team rightly argues that in the event of a security issue 
it would be too much work to 1) hunt each package using a vendored 
library and 2) patch and rebuild all of them.
This does not really matter for Go and Rust software because 1) the list 
of (vendored) dependencies can be extracted automatically at build time 
and 2) all this software would have to be rebuilt anyway since these 
languages do not support or do not use dynamic linking.

Also, shared libraries save memory when multiple programs using them are 
run concurrently, but nowadays this kind of saving is rarely meaningful.

Because of these reasons maybe we should consider supporting vendored 
libraries in some cases.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Russ Allbery]

Jeremy Stanley  writes:

> If this represents the actual state of building Kubernetes, it's
> unclear to me why Debian would package it at all. I don't see the
> value to users in consuming Kubernetes from a Debian package if the
> result is compromising on Debian's vision and values so that they
> can get the exact same thing they'd have if they just used the
> Kubernetes community's recommended tooling to install it instead.

I have been very grateful over the past few months to have Kubernetes
available in Debian (and have been quite annoyed at the irritating things
I have to do get and update Helm, which at least has a snap, and Argo CD,
which doesn't have anything useful).

I have no opinion about the best solution to the huge vendoring problem.
(The Rust team is trying the package everything approach with some success
but is uncovering other limitations in our processes and tools.)  But
having these tools in Debian is hugely valuable for Debian users who need
them, which is sort of the point of Debian at the end of the day.

> I'm all for using the best tool for the job, and while I've been a
> die-hard Debian user for more than two decades I also don't install
> every last bit of software from its packages.

I do when I can because I otherwise have to remember to wire together N
different update mechanisms, which is remarkably not-fun and takes time
away from the actual work I'm trying to do.

-- 
Russ Allbery (r...@debian.org)  

Re: What to do when DD considers policy to be optional? [kubernetes]

[Jeremy Stanley]

On 2020-03-24 19:08:23 + (+), Janos LENART wrote:
> I know Dimitry was fighting an uphill battle with kubernetes
> between 2016 and 2018 and he experienced first hand the problems
> posed by vendored code.
> 
> We see more and more software making excessive use of vendored
> code. Pretty much everything that is written in Go. Some of these
> are crucially important, like Docker or Kubernetes. So I
> understand the concern everyone has about how this fits with the
> Debian Policy.
[snip rationale for packaging Kubernetes while giving up on policy]

If this represents the actual state of building Kubernetes, it's
unclear to me why Debian would package it at all. I don't see the
value to users in consuming Kubernetes from a Debian package if the
result is compromising on Debian's vision and values so that they
can get the exact same thing they'd have if they just used the
Kubernetes community's recommended tooling to install it instead.
I'm all for using the best tool for the job, and while I've been a
die-hard Debian user for more than two decades I also don't install
every last bit of software from its packages. Some software
ecosystems have chosen to focus on tools and workflows which are
incompatible with Debian's, but that doesn't mean that either one is
inherently bad nor that they need to be integrated at all costs.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Janos LENART]

Hi Dimitry, FTP masters and others,

I know Dimitry was fighting an uphill battle with kubernetes between 2016
and 2018 and he experienced first hand the problems posed by vendored code.

We see more and more software making excessive use of vendored code. Pretty
much everything that is written in Go. Some of these are crucially
important, like Docker or Kubernetes. So I understand the concern everyone
has about how this fits with the Debian Policy.

Debian Policy, paragraph 4.13 states:
(for your convenience I include it below :) )
https://www.debian.org/doc/debian-policy/ch-source.html#convenience-copies-of-code

=
4.13 Convenience copies of code

Some software packages include in their distribution convenience copies of
code from other software packages, generally so that users compiling from
source don’t have to download multiple packages. Debian packages should not
make use of these convenience copies unless the included package is
explicitly intended to be used in this way. [17] If the included code is
already in the Debian archive in the form of a library, the Debian
packaging should ensure that binary packages reference the libraries
already in Debian and the convenience copy is not used. If the included
code is not already in Debian, it should be packaged separately as a
prerequisite if possible. [18]

[18] Having multiple copies of the same code in Debian is inefficient,
often creates either static linking or shared library conflicts, and, most
importantly, increases the difficulty of handling security vulnerabilities
in the duplicated code.
=

I think this is the part that has the most bearing on the vendored code
problem, especially the footnote. I agree with this principle. But we
should apply it to the state of affairs in 2020, and to this specific
situation.

Keeping all that in mind, here are the reasons why I think it is acceptable
for now to package Kubernetes with the vendored code, and even the best
solution that is available currently:

1. OTHER EXAMPLES. If we take this paragraph completely literally and to
the extreme then other packages are also in violation of it. True, the
current packaging of kubernetes does this to a greater extent than its
predecessor for example, but perhaps this shows that this section was
always open for interpretation. Examples of some prominent packages in
Debian that bundle and use the vendored code (in parentheses is the number
of go packages bundled, estimate):
- docker.io (58, including some that are vendored more than once within the
same source package, but not including the fact that docker.io itself is
made up of 7 tarballs)
- kubernetes (20 for the previous version, 200 now)
- prometheus (4)
- golang (4)
None of these were REJECTed, and please don't sabotage these packages now
:-D The idea was only to show that, at least for now, vendoring is a fact
in Debian. There is an effort to improve the situation but in the meantime
we just go on. Not great, not terrible..

2. MAINTAINABILITY. Having every single vendored repo available as a
separate package in Debian is not feasible. It is true that some of them
are already packaged. But the expectation that all of them are (with the
exact version that is needed for Kubernetes), is not going to happen. Also,
the golang-* packages have a number of different maintainers. Hundreds of
such packages would be required to build Kubernetes. So one can be rest
assured that every future release in Debian will be blocked on waiting for
dozens of these packages to be updated. Dimitry and a few others worked
hard on trying to pull this off but even they could not do it. Since 2016 a
total of 3 Kubernetes releases made it into Debian/unstable, but there have
been 17 major and countless minor upstream releases of Kubernetes.
Thousands of issues were fixed upstream, including serious security flaws,
these never made it into Debian. Exactly because the packaging was too
difficult to maintain. So, how maintainable was that solution then, despite
the huge amount of effort put in? In my opinion this shows that the
reasoning on maintainability in DP does not apply here.

3. NO FORKS. Debian developers hacking Kubernetes source code, so it
compiles with a lucky enough version of a dependency that made it into
Debian, makes the Debian version of Kubernetes different from the standard
one that everyone expects. This is totally unwelcome by almost every user.
No sane cluster admin would dare to use this "fork", ever. There were some
attempts to get the Kubernetes contributors to update dependencies to a
specific version: https://github.com/kubernetes/kubernetes/issues/27543 .
Reading the whole thread helps to put some perspective on this. The
Kubernetes contributors were actually quite helpful throughout but they
have made it clear that they will not update dependencies for update's
sake. Maybe with some projects Debian would have the upper hand, but not
with Kubernetes.

4. TESTING. The Kubernetes 

Re: What to do when DD considers policy to be optional? [kubernetes]

[Florian Weimer]

* Paul Wise:

> On Tue, Mar 24, 2020 at 6:17 AM Vincent Bernat wrote:
>
>> Kubernetes is already using Go modules. They happen to have decided to
>> keep shipping a `vendor/` directory but this is not uncommon. It is
>> often considered as a protection against disappearing modules. So, there
>> is nothing to be done upstream. And BTW, there are currently 616
>> dependencies, pinned to a specific version.
>
> I wonder if the existence of Software Heritage could convince them
> disappearing modules aren't a problem, or if another service is
> needed.

The required versions of the modules could disappear from Debian, though.
Services like Software Heritage will not help with that.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Sean Whitton]

Hello,

On Tue 24 Mar 2020 at 05:37AM -05, Michael Lustfield wrote:

> The 'go vendor' approach is especially bad within the Debian context because 
> it
> will download any/all modules that are referenced. In some cases, 'go get 
> [..]'
> can go from downloading a single repository to downloading 200+ because one 
> (1)
> extra dependency was added for one (1) extra feature that almost nobody will
> ever use.
>
> It's nearly guaranteed that at least a large handful of those will have no
> license at all and at least one is going to have large embedded non-dfsg 
> blobs.
>
> Or, to summarize my rant...
>
> These lazy young whipper snappers don't know what good source looks like!
>
> .. back in my day, we coded on paper, had real bugs, and that's just the way 
> we
> liked it.

If there are multiple DFSG issues then I would say that this is more
than a rant.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Vincent Bernat]

 ❦ 24 mars 2020 14:18 +01, Julien Puydt:

>> There are other reasons, notably that you speed up builds by having
>> all the source code ready.
>
> Sorry, I don't know much about how go works, but : can't the developer
> just have the deps ready -- and just not commit them to the repo and
> not ship them?

These projects heavily rely on automated builds. Depending on platform,
it may or may not be easy to have shared cache for these builds. If they
have, you have to debug them as well. From their point of view, it's
simpler to deliver the artifacts with the rest of the source code.
Fast, simple and more reliable.

Moreover, the vendor directory is absolutely not a problem for Debian
(except for licensing where you would have to do a repack for stuff not
distributable). Debian just has to ignore the vendor directory and have
all the dependencies ready. The tooling around Go in Debian already
handles that. The problem is the number of dependencies.

> From what I have read in this thread, go developers seem to be about as
> sloppy as javascript ones : put junk together and throw it to the
> world...

That's not comparable. Go is not using micro-dependencies. However, they
don't have optional dependencies, so anything cloud-based has a lot of
dependencies to manage.
-- 
Make it clear before you make it faster.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Julien Puydt]

Le mardi 24 mars 2020 à 14:03 +0100, Vincent Bernat a écrit :
>  
> There are other reasons, notably that you speed up builds by having
> all
> the source code ready.

Sorry, I don't know much about how go works, but : can't the developer
just have the deps ready -- and just not commit them to the repo and
not ship them?

>From what I have read in this thread, go developers seem to be about as
sloppy as javascript ones : put junk together and throw it to the
world...

J.Puydt

Re: What to do when DD considers policy to be optional? [kubernetes]

[Vincent Bernat]

 ❦ 24 mars 2020 05:37 -05, Michael Lustfield:

>> > Kubernetes is already using Go modules. They happen to have decided to
>> > keep shipping a `vendor/` directory but this is not uncommon. It is
>> > often considered as a protection against disappearing modules. So, there
>> > is nothing to be done upstream. And BTW, there are currently 616
>> > dependencies, pinned to a specific version.  
>> 
>> I wonder if the existence of Software Heritage could convince them
>> disappearing modules aren't a problem, or if another service is
>> needed.
>
> I think this is a symptom of the tools being used. Using 'go vendor' is a
> documented step in nearly all golang-based "release tutorials." Most never 
> even
> get as far as considering that maybe their source should have a version,
> because the toolset mentality is "download latest at build time."

With Go modules, that's not true anymore. It will use the minimal
version satisfying the minimal versions specified in go.mod.
-- 
Follow each decision as closely as possible with its associated action.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Vincent Bernat]

 ❦ 24 mars 2020 10:14 +00, Paul Wise:

>> Kubernetes is already using Go modules. They happen to have decided to
>> keep shipping a `vendor/` directory but this is not uncommon. It is
>> often considered as a protection against disappearing modules. So, there
>> is nothing to be done upstream. And BTW, there are currently 616
>> dependencies, pinned to a specific version.
>
> I wonder if the existence of Software Heritage could convince them
> disappearing modules aren't a problem, or if another service is
> needed.

There are other reasons, notably that you speed up builds by having all
the source code ready. If the vendor/ directory wasn't there, the
presence of `go.sum` would ensure you get something reproducible by
downloading all the deps, but I think it implies a full checkout of all
deps, which can be lengthy. There is also a caching mechanism (local and
remote).
-- 
Make sure every module hides something.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Michael Lustfield]

On Tue, 24 Mar 2020 10:14:08 +
Paul Wise  wrote:

> On Tue, Mar 24, 2020 at 6:17 AM Vincent Bernat wrote:
> 
> > Kubernetes is already using Go modules. They happen to have decided to
> > keep shipping a `vendor/` directory but this is not uncommon. It is
> > often considered as a protection against disappearing modules. So, there
> > is nothing to be done upstream. And BTW, there are currently 616
> > dependencies, pinned to a specific version.  
> 
> I wonder if the existence of Software Heritage could convince them
> disappearing modules aren't a problem, or if another service is
> needed.

I think this is a symptom of the tools being used. Using 'go vendor' is a
documented step in nearly all golang-based "release tutorials." Most never even
get as far as considering that maybe their source should have a version,
because the toolset mentality is "download latest at build time."

The 'go vendor' approach is especially bad within the Debian context because it
will download any/all modules that are referenced. In some cases, 'go get [..]'
can go from downloading a single repository to downloading 200+ because one (1)
extra dependency was added for one (1) extra feature that almost nobody will
ever use.

It's nearly guaranteed that at least a large handful of those will have no
license at all and at least one is going to have large embedded non-dfsg blobs.

Or, to summarize my rant...

These lazy young whipper snappers don't know what good source looks like!

.. back in my day, we coded on paper, had real bugs, and that's just the way we
liked it.

-- 
Michael Lustfield

Re: What to do when DD considers policy to be optional? [kubernetes]

[Michael Lustfield]

On Mon, 23 Mar 2020 18:47:18 -0700
Sean Whitton  wrote:

> Hello Dmitry, Janos, others,
> 
> On Mon 23 Mar 2020 at 05:32PM +11, Dmitry Smirnov wrote:
> 
> > What would be best to do in such situation?  
>
> [...]
> 
> I think that I would start by filing an RC bug.

+1

If you run into issues, then you'll want to contact the ftp-masters team.

It would be helpful if the bug mentioned the two solutions I'm aware of:
- Revert the offending changes
- Migrate from main to non-free

The latter would be much easier, but would destroy all the work you put in. :(

> > [...]
> > As a person who originally introduced Kubernetes to Debian I can say that it
> > is indeed a lot of work to maintain this package and to reuse packaged
> > libraries. But I've demonstrated that it is possible at least to some 
> > extent.

As a person who temporarily introduced gitea into Debian, I fully understand.
Unfortunately, I've found that such problems are often a result of poorly
written code where the approach tends to be, "I don't know how to do this
thing, so I'll copy a module that does this and 200 other things just as
poorly." 

The lesson I learned was-
Just because something /can/ be packaged, does not mean it /should/ be packaged.
(pabs warned me about this hundreds of hours prior to me giving up)

> > I don't recall a situation when policing of how a package is maintained 
> > would
> > be necessary long after package is accepted...

It's rare, but it happens. My most recent experience was with bitlbee.
Unfortunately, our current auto-reject system is quite limited. Catching things
like this automatically is (currently) not possible and Debian relies of folks
like you. (btw- thanks for this report)

> > What do we do to ensure quality work in situation of technological hijack
> > when maintainer is unwilling to follow the practice or comply with policy?
> >
> > This is not a technical disagreement so I think that involving technical
> > committee may not be the right way to handle the problem... Or is it?  

TC is not needed. This is a clear policy violation that the new maintainer
appears to have known about, even before the upload.

It concerns me that they thought this package warranted an exception...
-- 
Michael Lustfield

Re: What to do when DD considers policy to be optional? [kubernetes]

[Paul Wise]

On Tue, Mar 24, 2020 at 6:17 AM Vincent Bernat wrote:

> Kubernetes is already using Go modules. They happen to have decided to
> keep shipping a `vendor/` directory but this is not uncommon. It is
> often considered as a protection against disappearing modules. So, there
> is nothing to be done upstream. And BTW, there are currently 616
> dependencies, pinned to a specific version.

I wonder if the existence of Software Heritage could convince them
disappearing modules aren't a problem, or if another service is
needed.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What to do when DD considers policy to be optional? [kubernetes]

[Holger Levsen]

On Mon, Mar 23, 2020 at 05:32:54PM +1100, Dmitry Smirnov wrote:
> Something interesting just happened. An inexperienced DD adopted a very 
> complicated package (kubernetes) and uploaded it with changes that would have 
> never been accepted by ftp-masters.

file bugs then. maybe that new maintainer will be very happy about them being
pointed out?!

> This is not a technical disagreement so I think that involving technical 
> committee may not be the right way to handle the problem... Or is it?

involving the TC without disagreements documented in the BTS seems premature.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

"... the premise [is] that privacy is about hiding a wrong. It's not.
 Privacy is an inherent human right, and a requirement for maintaining
 the human condition with dignity and respect." (Bruce Schneier)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Vincent Bernat]

 ❦ 24 mars 2020 03:11 +00, Paul Wise:

>> Specifically, as README.Debian states, the vendor/ subdirectory of the
>> source package contains more than two hundred Go libraries.
>
> There are a *lot* of embedded code/data copies in Debian already.
> While it would be nice to remove them, sometimes it isn't possible.
> Often the copies are forked, or upstream refuses to remove them,
> sometimes even though they forgot why they were added in the first
> place. In addition the developer culture in various communities
> encourages embedded copies. I think the best action we can do is send
> patches to upstream projects to switch from vendoring to using the
> native dependency system of the package manager for the related
> language community. ISTR reading that Go has one of those now.

Kubernetes is already using Go modules. They happen to have decided to
keep shipping a `vendor/` directory but this is not uncommon. It is
often considered as a protection against disappearing modules. So, there
is nothing to be done upstream. And BTW, there are currently 616
dependencies, pinned to a specific version.
-- 
Don't just echo the code with comments - make every comment count.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Paul Wise]

On Tue, Mar 24, 2020 at 1:47 AM Sean Whitton wrote:

> Specifically, as README.Debian states, the vendor/ subdirectory of the
> source package contains more than two hundred Go libraries.

There are a *lot* of embedded code/data copies in Debian already.
While it would be nice to remove them, sometimes it isn't possible.
Often the copies are forked, or upstream refuses to remove them,
sometimes even though they forgot why they were added in the first
place. In addition the developer culture in various communities
encourages embedded copies. I think the best action we can do is send
patches to upstream projects to switch from vendoring to using the
native dependency system of the package manager for the related
language community. ISTR reading that Go has one of those now. Where
language communities don't have a native package manager, we need to
invent one for them. Then we can use things like dh-make-perl to
package the dependencies for Debian. I have no data but I think this
approach is more likely to have success than ranting about embedded
copies, tempting though that is. Apart from trying to discourage their
use, unfortunately embedded copies are here and they will never go
away and we need to accept that fact and to deal with the
consequences; for example to ensure that all copies get fixed for
security issues, try to get them updated upstream after important
bug/performance fixes and so on.

https://wiki.debian.org/EmbeddedCodeCopies
https://salsa.debian.org/security-tracker-team/security-tracker/raw/master/data/embedded-code-copies
https://wiki.debian.org/AutomaticPackagingTools

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What to do when DD considers policy to be optional? [kubernetes]

[Sean Whitton]

Hello Dmitry, Janos, others,

On Mon 23 Mar 2020 at 05:32PM +11, Dmitry Smirnov wrote:

> Something interesting just happened. An inexperienced DD adopted a very
> complicated package (kubernetes) and uploaded it with changes that would have
> never been accepted by ftp-masters.

Specifically, as README.Debian states, the vendor/ subdirectory of the
source package contains more than two hundred Go libraries.

I can't speak for the whole FTP Team here, but if this had ended up in
NEW and I had been the FTP Team member to review it, I would indeed have
rejected it, on the grounds that accepting the upload renders Debian
less maintaintable in various ways.

> What would be best to do in such situation?

I think that I would start by filing an RC bug.

> The problem is not with DD's qualification (although this certainly plays a
> role) but with intentional non-compliance with policies and packaging
> practices.
>
> As a person who originally introduced Kubernetes to Debian I can say that it
> is indeed a lot of work to maintain this package and to reuse packaged
> libraries. But I've demonstrated that it is possible at least to some extent.
>
> New maintainer of kubernetes do not even make an attempt to build it properly
> and blatantly states the following in README which demonstrates profound lack
> of understanding of problems that were impairing maintenance of the package:
>
>"I kindly ask purist aspirations that effectively halted Kubernetes'
> release and updates in Debian for YEARS to be kept at bay."

The new maintainer presumably thinks that Policy should have an
exception for this sort of case -- let's assume good faith.

> I don't consider myself to be a purist. I have pragmatic approach to
> packaging but I feel concerned when policies and packaging practices are
> circumvented.
>
> I don't recall a situation when policing of how a package is maintained would
> be necessary long after package is accepted...
> What do we do to ensure quality work in situation of technological hijack
> when maintainer is unwilling to follow the practice or comply with policy?
>
> This is not a technical disagreement so I think that involving technical
> committee may not be the right way to handle the problem... Or is it?

I think it counts as a technical disagreement but surely there is room
for discussion in a bug report before involving the T.C.

-- 
Sean Whitton


signature.asc
Description: PGP signature

What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

Something interesting just happened. An inexperienced DD adopted a very 
complicated package (kubernetes) and uploaded it with changes that would have 
never been accepted by ftp-masters.

What would be best to do in such situation?

The problem is not with DD's qualification (although this certainly plays a 
role) but with intentional non-compliance with policies and packaging 
practices.

As a person who originally introduced Kubernetes to Debian I can say that it 
is indeed a lot of work to maintain this package and to reuse packaged 
libraries. But I've demonstrated that it is possible at least to some extent.

New maintainer of kubernetes do not even make an attempt to build it properly 
and blatantly states the following in README which demonstrates profound lack 
of understanding of problems that were impairing maintenance of the package:

   "I kindly ask purist aspirations that effectively halted Kubernetes'
release and updates in Debian for YEARS to be kept at bay."

I don't consider myself to be a purist. I have pragmatic approach to 
packaging but I feel concerned when policies and packaging practices are 
circumvented.

I don't recall a situation when policing of how a package is maintained would 
be necessary long after package is accepted...
What do we do to ensure quality work in situation of technological hijack 
when maintainer is unwilling to follow the practice or comply with policy?

This is not a technical disagreement so I think that involving technical 
committee may not be the right way to handle the problem... Or is it?

-- 
Cheers,
 Dmitry Smirnov.

---

We do our friends no favors by pretending not to notice flaws in their
work, especially when those who are not their friends are bound to notice
these same flaws.
-- Sam Harris


signature.asc
Description: This is a digitally signed message part.