Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Fri, Mar 09, 2018 at 02:07:19AM +0100, gregor herrmann wrote:
> On Thu, 08 Mar 2018 23:03:17 +0200, Adrian Bunk wrote:
> 
> > The first question should always be if/how we can provide something that 
> > is better than what is already available elsewhere.
> 
> An answer to that question might often be: "because it integrates
> into a Debian system". -- This is also an answer to the proposal
> raised here some days ago about not being shy to promote third-party
> repositories; no objections from me in general, just the remark that
> most third-party .debs I've seen are of abysmal quality.

I wasn't thinking of third-party .debs or anything else that
is Debian specific.

npm or flatpak would be examples that I had in mind.

> So even if
> we can't provide a true real-debian all-DFSG-free
> non-embedded-code-copies etc. package in main, I think offering a
> .deb (or .vdeb or flatpak or whatever)

"Debian specific flatpak" would be wrong, a flatpak app
is supposed to work equally well on all distributions.

> in the spirit of contrib/non-free
> aka DFSG#5 would be a valuable service to users, as we can offer
> packagees that Just Work™ (as in: don't install files into absurd
> locations, don't require crazy manual update steps with scripts
> placed SomeWhere etc.)

"install files into absurd locations" doesn't even matter for solutions 
like flatpat that containerize an app separated from the system and 
other apps.

Regarding "as we can offer packagees that Just Work™", I do not 
necessarily see us providing better packages than upstream here.

> > The worst case would be if we have to tell more frequently to users
> > "Please don't use the packages in our stable release." because they
> > are worse than alternatives.
> 
> We might need an archive area which is independent of our release
> suites. I guess there's lots of software out there which doesn't
> align with our release cycle, fixes bugs only in "latest release"
> etc. -- but still works from anything between oldstable and
> unstable. While this is all a bit disgusting from our traditional
> Debian point of view, users would still be better of with installing
> a package from, say, "alien", that works on a Debian system
> over having to fight with creative upstream packaging attempts or
> manual installations

Why would a Debian package from alien be better than "npm install"?

Why would a Debian package from alien be better than a flatpak
from alien?

And it is hard to see the point why alien should provide a Debian
packages when providing a flatpak would make the software built once
work equally well on all releases of all distributions, also avoiding 
all the pesky differences like different versions of the same library
having slightly different behaviour.

And for our users, a Debian package where the postinst from alien 
runs as root and the proprietary software from alien does not run 
containerized is not a good option.

> Cheers,
> gregor

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Holger Levsen]

On Fri, Mar 09, 2018 at 02:07:19AM +0100, gregor herrmann wrote:
> We might need an archive area which is independent of our release
> suites. 

.oO( PPAs )


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[gregor herrmann]

On Thu, 08 Mar 2018 23:03:17 +0200, Adrian Bunk wrote:

> The first question should always be if/how we can provide something that 
> is better than what is already available elsewhere.

An answer to that question might often be: "because it integrates
into a Debian system". -- This is also an answer to the proposal
raised here some days ago about not being shy to promote third-party
repositories; no objections from me in general, just the remark that
most third-party .debs I've seen are of abysmal quality. So even if
we can't provide a true real-debian all-DFSG-free
non-embedded-code-copies etc. package in main, I think offering a
.deb (or .vdeb or flatpak or whatever) in the spirit of contrib/non-free
aka DFSG#5 would be a valuable service to users, as we can offer
packagees that Just Work™ (as in: don't install files into absurd
locations, don't require crazy manual update steps with scripts
placed SomeWhere etc.)
 
> The worst case would be if we have to tell more frequently to users
> "Please don't use the packages in our stable release." because they
> are worse than alternatives.

We might need an archive area which is independent of our release
suites. I guess there's lots of software out there which doesn't
align with our release cycle, fixes bugs only in "latest release"
etc. -- but still works from anything between oldstable and
unstable. While this is all a bit disgusting from our traditional
Debian point of view, users would still be better of with installing
a package from, say, "alien", that works on a Debian system
over having to fight with creative upstream packaging attempts or
manual installations


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: The Doors: Break On Through


signature.asc
Description: Digital Signature

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Tue, Feb 27, 2018 at 02:13:41PM +0100, Didier 'OdyX' Raboud wrote:
>...
> In other words, vendorization is the tool that allows developers to get rid 
> of 
> distribution constraints and get on with their development through installing 
> the dependencies from their ecosystem as they see fit (non-root), in the 
> (eventually precise) version they need. But using these "upper-layer" 
> management tools (pip, npm, bower, you-name-it), one doesn't get the 
> constraints from the distribution, but one doesn't get the benefits from the 
> distribution either. And Debian (amongst others) has value to offer to these 
> layers too: DFSG-freeness, traceability, reproducibility, a common package 
> format and a set of tools, etc.

You omitted "security support", which is essential for any real-world usage.

> It would be really sad if Debian was incrementally reduced to only the 
> "boring" lower layers,
>...

#MDGA (Make Debian Great Again)

More seriously, our priority should be to help users getting working
and security-supported systems.

Distribution-agnostic formats like flatpak might become a good option
when people always want/need the latest upstream version.

An ancient version with known vulnerabilities and no security support
in stable is clearly worse than whatever the management tools of the
upstream ecosystem would install.

The first question should always be if/how we can provide something that 
is better than what is already available elsewhere.

The worst case would be if we have to tell more frequently to users
"Please don't use the packages in our stable release." because they
are worse than alternatives.

> Cheers,
>   OdyX
>...

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Tue, Feb 27, 2018 at 02:14:02PM +, Simon McVittie wrote:
>...
> Also, the security team specifically don't provide security
> support for libv8, which apparently extends to node-* packages like
> , so it's
> hard to see how tolerating embedded code copies of nodejs modules in
> particular would make their security support situation a whole lot worse:
> it's already the case that the upstream and downstream maintainers of
> these modules (or the applications that bundle them, or both) provide
> the only security maintenance they'll get. In practice, this isn't as
> awful as it first appears, because nodejs modules are often very small,
> so an individual nodejs module is relatively unlikely to contain security
> vulnerabilities even if its defect density is high, simply because there
> isn't very much code to be vulnerable.
>...

https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#libv8
"Unfortunately, this means that libv8-3.14, nodejs, and the associated 
 node-* package ecosystem should not currently be used with untrusted 
 content, such as unsanitized data from the Internet."

IMHO any package in Debian stable that uses a node* package on untrusted 
content should get an RC bug and a CVE - it is clearly documented that 
this should not be done.

> Regards,
> smcv
>...

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Sean Whitton]

Hello,

On Sun, Mar 04 2018, Didier 'OdyX' Raboud wrote:

> Le mardi, 27 février 2018, 14.13:41 h CET Didier 'OdyX' Raboud a
>écrit :
>> tl;dr: a new package format is needed, with a new non-suite-specific
>> repository is needed to bring the Debian added-value to these
>> ecosystems.
>
> FTR, my current line of thought and understanding is that guix or nix
> do pretty much what is needed here, and it would be silly to start
> something from scratch.
>
> I plan to spend some brain time playing with guix and report back; I'm
> aware of:
>
> * https://bugs.debian.org/850644 for guix
> * https://bugs.debian.org/877019 for nix
> * Diane Trout's efforts 2 years ago on
> https://github.com/detrout/debian-guix

AIUI the guix repositories care more about including only free software
than the nix repositories do.  That might be a reason for us to prefer
guix.

Thank you for investigating these options!

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Didier 'OdyX' Raboud]

Le mardi, 27 février 2018, 14.13:41 h CET Didier 'OdyX' Raboud a écrit :
> tl;dr: a new package format is needed, with a new non-suite-specific
> repository is needed to bring the Debian added-value to these ecosystems.

FTR, my current line of thought and understanding is that guix or nix do 
pretty much what is needed here, and it would be silly to start something from 
scratch.

I plan to spend some brain time playing with guix and report back; I'm aware 
of:

* https://bugs.debian.org/850644 for guix 
* https://bugs.debian.org/877019 for nix
* Diane Trout's efforts 2 years ago on https://github.com/detrout/debian-guix

Cheers,
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: What can Debian do to provide complex applications to its users?

[Paul Wise]

On Thu, 2018-03-01 at 09:53 +0100, Didier 'OdyX' Raboud wrote:

> Maybe what we need is a packaged nix and a standardized nix repository.

There is an ITP and RFS:

https://bugs.debian.org/877019
https://bugs.debian.org/877331

I assume Nix/NixOS have a standard repo already.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: What can Debian do to provide complex applications to its users?

[Sean Whitton]

Hello,

On Thu, Mar 01 2018, Didier 'OdyX' Raboud wrote:

> In pretty much the same vein as dh-virtualenv, a possibility would be
> to do install-time build, through triggers for example.

Just to note that the emacsen-common infrastructure does this too, as
another place to look for a working example.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Ian Jackson]

Holger Levsen writes ("Re: What can Debian do to provide complex applications 
to its users?"):
> On Thu, Mar 01, 2018 at 10:26:27AM +0100, Didier 'OdyX' Raboud wrote:
> > Good point: not all versions are desirable; "majors" can be installed in 
> > parallel, "minors" are updates to the formers.
> 
> I dont get this, your minor difference may make a major difference to
> me. So if you/we were allowing several versions to be instaled in
> parallel, why restrict this again?

The upstream dependencies tend to specify particular "versions".

If one wants to modify something, it is obviously undesirable to have
to modify all the callers right up the chain with different version
numbers.

So we need to be able to install actually-different packages as the
same "version"-as-far-as-upstream-tools-are-concerned, and have them
*replace* rather than appear alongside the unmodified one.

But if *upstream* give us a new package it should usually be
coinstallable with other packages, because that coinstallability is
the Debian equivalent of the version-locking that upstreams support.

In .deb, the Package: field identifies the unit of replacement
vs. coinstall: different Package's are coinstallable, whereas two
.debs with the same Package are
switchable-between-without-disturbing-other-things (more or less).

The right answer is, IMO, to put the
version-as-far-as-upstream-is-concerned into the .deb package name.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: What can Debian do to provide complex applications to its users?

[Ian Jackson]

Didier 'OdyX' Raboud writes ("Re: What can Debian do to provide complex 
applications to its users?"):
> Le mercredi, 28 février 2018, 06.06:54 h CET Sean Whitton a écrit :
> > No, but we might often have reason to maintain a small delta.  We patch
> > upstream source for all sorts of reasons; it is hard to believe it
> > wouldn't come up.
> 
> It would not be reasonable to ship binary artifacts out of unpatcheable 
> source, so I read Ian's point as getting rid of quilt in favour of git 
> commits.

In particular, getting rid of the idea of maintaining a clean patch
queue separated into individual commits.  Instead, use "git merge".
If one wants to examine the diff between the vdeb source and upstream,
one runs
  git diff upstream -- ':!debian'

(I have some tools near completion that would make it easier to
maintain a separated-into-individual-changes patch queue entirely in
git, but I think your project will have small deltas and want to go
for simplicity, at least at first.  You want to optimise for the case
of a small, nor absent, delta.)

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: What can Debian do to provide complex applications to its users?

[Ian Jackson]

Didier 'OdyX' Raboud writes ("Re: What can Debian do to provide complex 
applications to its users?"):
> Le mardi, 27 février 2018, 14.48:48 h CET Ian Jackson a écrit :
> > Instead, establish a formal convention about embedding the (stable
> > part of) the version number in the package name.  This is important
> > so that it is possible to replace a specific "version" of one of these
> > packages with a modified package which is the "same version".
> 
> Good point: not all versions are desirable; "majors" can be installed in 
> parallel, "minors" are updates to the formers.
> 
> But, assuming a new binary format, it feels really weak to abuse of the 
> package name to embed a version.  (The filename is a different question). 
> What 
> about (in control.tar's control):
> 
> Package: simplejson
> Version-Unique: 3.13
> Version: 3.13.2-1

Almost everything else in your message I agree with, or at least,
think is arguable, or a matter of detail.  This, and your next
suggestion, however, I think would be serious mistakes.

I think for the success of your enterprise you need to make at least
your .vdebs processable by the existing tools.  That means not
changing the binary format in an incompatible way.  Otherwise you are
addding "change every tool in the world which tries to process a .deb"
to your todo list.

As for the details:

The "abuse" of which you speak is already done routinely in Debian and
works very well, so there is not even a need to fear the unknown
problems it might bring: we know what problems it brings - very few.

> > These can be verified either the archive server as part of package
> > acceptance, or by apt as an annotation in the sources.list.
> 
> Again, that's the main advantage I see from a new .*deb format: if these 
> restrictions are part of the format, they can be checked and enforced at all 
> steps in the process: dpkg-buildpackage would error out if there's a 
> postinst, 
> lintian would add an error, the server would block it and dpkg would not 
> unpack the postinst, just because 'debian-binary' is 2.0-vdeb.

As for checking: the right way to think about this is as a security
and quality issue.  From a security point of view, the configuration
needs to be in the apt sources.list, because that is where the
authorisation happens on an individual system.  You will at least
initially be using a less-firmly-protected signing key for your new
archive, so you need to protect, as far as you can, users from
compromise of that key.  From a quality point of view a check in your
archive software is the right place.

The properties which are important for security are: a whitelist of
dpkg .deb features, to exclude maintainer scripts, triggers,
conffiles, etc.  A restriction on package names.  And a restriction on
paths.

In terms of code changes that means, I think, a new apt option to
enable the new behaviour, and probably a new dpkg option to implement
the checks.  If you get resistance from dpkg upstream, you can invent
a new checking tool and call that from apt at an appropriate point.


As for source package format:

> > OTOH we do not want to abandon the Debian source package format
> > completely because we have lots and lots of tools which understand it
> > well.
> 
> Although I share the sentiment, I see value in finding a suitable model for 
> the problem at hand, rather than massage our existing tools to fix the 
> problem, "just because" they are our existing tools.  I think we agree on 
> that.

If you use the existing source metdata format then existing Debian
users can use their existing Debian tools.  You may find that those
tools are too awkward of course - dgit-user(7) doesn't make for
particularly pretty reading.  More on this below.

> > Building should be done with "git clone" followed by
> > "dpkg-buildpackage -b" (or some suitable wrapper).
> 
> That's pretty much what I had in mind, yes.  I'm not even sure there's much 
> need for a complete traditional debian/ directory: iff the .vdeb ecosystem 
> does much less than normal .debs, we could aim for a single declarative 
> .vspec 
> (yes, I know what you're thinking) file for instance.  Given we're tackling 
> wide consistent (hmm) ecosystems, a set of fine tools and very minimal 
> declarative packaging per-item could do it.

You have a choice to make about how to deliver the ecosystem-specific
build knowledge.

ISTM that your basic choices are:

 1. Source in your repo is identical to upstream source with no
Debian-specific information.  Building is done by a new builder
tool which autodetects the source ecosystem, and runs appropriate
build rules (eg, by invoking dh).

 2. Source in your repo is *not* identical to upstream source.  It
   

Re: What can Debian do to provide complex applications to its users?

[Holger Levsen]

On Thu, Mar 01, 2018 at 10:26:27AM +0100, Didier 'OdyX' Raboud wrote:
> Good point: not all versions are desirable; "majors" can be installed in 
> parallel, "minors" are updates to the formers.

I dont get this, your minor difference may make a major difference to
me. So if you/we were allowing several versions to be instaled in
parallel, why restrict this again?


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Didier 'OdyX' Raboud]

Le mardi, 27 février 2018, 15.14:02 h CET Simon McVittie a écrit :
> Here is a different straw man, which I think might be similarly effective
> and a lot less work:
> 
> On Tue, 27 Feb 2018 at 14:13:41 +0100, Didier 'OdyX' Raboud wrote:
> > As Debian, we
> > are insisting that our releases ideally only contain a single version of a
> > software, that we insist is made available at system-level.
> 
> ...
> 
> > In other words, vendorization is the tool that allows developers to get
> > rid of distribution constraints and get on with their development through
> > installing the dependencies from their ecosystem as they see fit
> > (non-root), in the (eventually precise) version they need.
> 
> I can't help wondering whether vendorizing dependencies (embedded code
> copies) would be a better route for ecosystems like npm that we might
> categorise as "aggressively modular" - package the app, but not the
> dependencies, and treat the vendorized/embedded dependencies as part
> of the app. Not embedding or duplicating libraries is already an ideal
> that we have to compromise on for pragmatic reasons - browsers use
> system libraries in testing/unstable but gradually make increasing use
> of embedded code copies in stable[1], drifting further away from the
> "no embedded code copies" ideal as time goes on and the latest browser
> version becomes increasingly unbuildable against the increasingly old
> libraries in stable. It's also how many Rust dependencies are already
> managed, as far as I'm aware.

Sure.  That would just be a different use of these .vdebs where these would be 
embedded/repacked in the end application packages rather than installed 
globally and used from their installation paths.

The added value that Debian can provide are source traceability, and trust 
mechanisms for package updates (although my proposal would most certainly 
weaken that) The issue I have with npm/pypi/you-name-it ecosystems is that I 
have to rely on _binary_ distribution made by non-Debian actors (with their 
own standards), to get dependencies for my applications. (I'm not saying their 
standards are necessarily bad; just that they often are not Debian's.)

So I was talking about Debian providing a new source-to-binary_artifact 
pipeline, which I think is an area where there's something to be done, by 
Debian.  Your proposal about embedding vendorized dependencies within 
applications (through embedded code copies or through Flatpak) is good, but I 
think the two proposals are orthogonal.

The advantage of having Debian-provided version-specific dependencies is the 
centralization of the DFSG-freeness checking, of the copyright assignment, 
etc. (and of the security patching, but that's harder).

I envision a hypothetical situation where instead of having embedded code 
copies of, say, jquery in various software, we had a centralized jquery source 
repository to which any arbitrary version (even patched) could be requested 
from.  Need jQuery Core 2.2.3 ? We have it for your package, so please either:
* depend on jquery_core_2.2.3.vdeb and add a symlink to
  /opt/vdebs/jquery-core/2.2.3;
* build-depend on jquery-core_2.2.3.vdeb and add
  Built-Using: jquery-core_2.2.3

Need jQuery Core 3.3.1 with your specific patches? Add your patches to a 
branch, and get your jquery-core_3.3.1+g1231234.vdeb.

That'd be a neat way to entirely eliminate code copies, and win metadata 
(which is currently fuzzy, at best) about these.

> I'm just not sure that taking the rules we follow for C/C++ shared
> libraries and applying them to every other language's ecosystem is either
> feasible or desirable - nodejs libraries are not the same as C libraries,
> and their tradeoffs are not the same tradeoffs.

Yes.

signature.asc
Description: This is a digitally signed message part.

Re: What can Debian do to provide complex applications to its users?

[Didier 'OdyX' Raboud]

Le mercredi, 28 février 2018, 06.06:54 h CET Sean Whitton a écrit :
> > Furthermore, abandon the patch queue approach to Debian package
> > management.  We will not be able to maintain a big delta to any of
> > these packages anyway.
> 
> No, but we might often have reason to maintain a small delta.  We patch
> upstream source for all sorts of reasons; it is hard to believe it
> wouldn't come up.

It would not be reasonable to ship binary artifacts out of unpatcheable 
source, so I read Ian's point as getting rid of quilt in favour of git 
commits.

signature.asc
Description: This is a digitally signed message part.

Re: What can Debian do to provide complex applications to its users?

[Didier 'OdyX' Raboud]

Le mardi, 27 février 2018, 14.48:48 h CET Ian Jackson a écrit :
> I have some specific comments:
> > Imagine
> > * a new .vdeb format variant that:
> > ** enables for multiple versions to be installed in parallel, where files
> >are unpacked in a version-specific paths
> 
> Instead, establish a formal convention about embedding the (stable
> part of) the version number in the package name.  This is important
> so that it is possible to replace a specific "version" of one of these
> packages with a modified package which is the "same version".

Good point: not all versions are desirable; "majors" can be installed in 
parallel, "minors" are updates to the formers.

But, assuming a new binary format, it feels really weak to abuse of the 
package name to embed a version.  (The filename is a different question). What 
about (in control.tar's control):

Package: simplejson
Version-Unique: 3.13
Version: 3.13.2-1

> > ** forbids any kind of maintainer scripts
> > ** is not bound to a specific suite
> > ** is restricted to be arch:all (~ shipping interpreter scripts)
> 
> These can be verified either the archive server as part of package
> acceptance, or by apt as an annotation in the sources.list.

Again, that's the main advantage I see from a new .*deb format: if these 
restrictions are part of the format, they can be checked and enforced at all 
steps in the process: dpkg-buildpackage would error out if there's a postinst, 
lintian would add an error, the server would block it and dpkg would not 
unpack the postinst, just because 'debian-binary' is 2.0-vdeb.

> > ** (ideally) can be user-installed
> 
> This one would require some thought.  Do the target ecosystems support
> transplantation of installed directory trees, without modification ?
> 
> Perhaps the experience of other projects doing "prefixed" uses of dpkg
> might be relevant.  Eg
>   https://wiki.termux.com/wiki/FAQ#Is_Termux_a_complete_Linux_Environment.3F
> (But the prefix there is baked into the .deb.)
> 
> How is this going to work for build-dependencies ?  If one says
> "build-depends: npm-vdeb-foo", will the build system know to look in
> $HOME as well as /usr ?

user-installation embeds two different problems: prefixed unpack and non-root 
unpack.  Given this is something orthogonal to the vdebs proposal (+ a hard 
problem), it's probably easier (for a start) to just leave that requirement 
off.

> > * a repository of these .vdeb
> > ** whose DFSG-freeness is checked
> > ** which version set is known, and tracked
> > ** that can provide new versions "on-demand"
> 
> It is important to appreciate what we are *discarding* as too hard.

Yes.

> > How does that sound?
> 
> Your proposal depends on continuous and almost-automatic
> incorporation of upstream code.  Our current source package workflows
> are not suited to this.

Yes.  With Debian approval / whitelisting at various points in the lifetime of 
a "package: initial submission (as our NEW) and ideally at later points / 
versions.

> OTOH we do not want to abandon the Debian source package format
> completely because we have lots and lots of tools which understand it
> well.

Although I share the sentiment, I see value in finding a suitable model for 
the problem at hand, rather than massage our existing tools to fix the 
problem, "just because" they are our existing tools.  I think we agree on 
that.

> I would like to suggest a radical approach to the source code
> management for your system: abandon source *packages* in favour of git
> trees.  Furthermore, abandon the patch queue approach to Debian
> package management.  We will not be able to maintain a big delta to
> any of these packages anyway.
> 
> So instead, we should have a Debian branch of each upstream git tree,
> which we should "git merge".  We will have to have a tool, for each
> ecosystem, that converts the metadata provided in the upstream package
> into the Debian format in debian/.  The "update to new upstream"
> process would be:
>git fetch upstream
>git merge upstream/master
>npm-to-debian-autoconvert-update
>git commit -s -a -m npm-to-debian-autoconvert-update debian
> 
> Building should be done with "git clone" followed by
> "dpkg-buildpackage -b" (or some suitable wrapper).

That's pretty much what I had in mind, yes.  I'm not even sure there's much 
need for a complete traditional debian/ directory: iff the .vdeb ecosystem 
does much less than normal .debs, we could aim for a single declarative .vspec 
(yes, I know what you're thinking) file for instance.  Given we're tackling 
wide consistent (hmm) ecosystems, a set of fine tools and very minimal 
declarative packaging per-item could do it.

Thanks for your inputs; I should probably find some time to put a refined 
version of these thoughts in a wiki page somewhere.

Cheers,
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: What can Debian do to provide complex applications to its users?

[Didier 'OdyX' Raboud]

Le jeudi, 1 mars 2018, 06.29:37 h CET Paul Wise a écrit :
> On Tue, Feb 27, 2018 at 9:13 PM, Didier 'OdyX' Raboud wrote:
> > Now, as a strawman proposition, here's what I fiddled with in my mind for
> > some days now:
> This reminds me a bit of Nix or Gentoo Prefix.

Yes, only for the upper layers' only, on top of your good-old Debian, and "by 
Debian"™.

Maybe what we need is a packaged nix and a standardized nix repository.

> > ** is restricted to be arch:all (~ shipping interpreter scripts)
> 
> Hmm, so this isn't going to cover all npm/pypi packages.

It's not a roadmap, just a braindump :-)  The packages that need compilation 
could still be provided as classical .debs for example.

signature.asc
Description: This is a digitally signed message part.

Re: What can Debian do to provide complex applications to its users?

[Didier 'OdyX' Raboud]

Le mercredi, 28 février 2018, 06.04:27 h CET Sean Whitton a écrit :
> On Tue, Feb 27 2018, Didier 'OdyX' Raboud wrote:
> > ** is restricted to be arch:all (~ shipping interpreter scripts)
> 
> There are compiled binary ecosystems that would benefit from your
> proposal, such as Haskell, so could you say more about why you want this
> restriction?

Mostly out of wanting to (eventually) start small.  It just reduces the 
problem surface.  Arch:all packages can virtually be built anywhere, on any 
architecture, so vdebs wouldn't (really) need a buildd network; or at least 
not a multiple-architectures' buildd network.

At first glance, it would also seem to vastly facilitate reproducibility, and 
reduces potential for executables' injection.

In pretty much the same vein as dh-virtualenv, a possibility would be to do 
install-time build, through triggers for example.


signature.asc
Description: This is a digitally signed message part.

Re: What can Debian do to provide complex applications to its users?

[Didier 'OdyX' Raboud]

Le mercredi, 28 février 2018, 04.57:50 h CET Russell Stuart a écrit :
> On Tue, 2018-02-27 at 14:13 +0100, Didier 'OdyX' Raboud wrote:
> > > - we could ship those applications not as .deb but as container
> > >
> > >   and let them have their own lifecycle
> > 
> > tl;dr: a new package format is needed, with a new non-suite-specific 
> > repository is needed to bring the Debian added-value to these
> > ecosystems.
> 
> To me 'container' is the right solution.

For me, this is orthogonal.  How your binaries or artifacts are orchestrated 
and isolated from eachothers doesn't have much to do with where they come from 
and how they're built.

I'm not looking forward to having to maintain hierarchies of containers in 
which software is pulled from various different sources which don't have a 
shared agreement on what constitutes Free Software, sane security guidelines, 
etc.  Ewww, wait…

The hard problem is how to keep fostering and providing the four software 
freedoms, not (only) how to ship software.

> (a grandiose and far fetched proposal)

I don't disagree that smart containerization would make a better Debian.  But 
not if we loose track of what binary gets installed from which source.  
Reproducibility and traceability are really important, and we should not throw 
them away for sake of more convenient software deployment.

Cheers,
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: What can Debian do to provide complex applications to its users?

[Paul Wise]

On Tue, Feb 27, 2018 at 9:13 PM, Didier 'OdyX' Raboud wrote:

> Now, as a strawman proposition, here's what I fiddled with in my mind for some
> days now:

This reminds me a bit of Nix or Gentoo Prefix.

> ** is restricted to be arch:all (~ shipping interpreter scripts)

Hmm, so this isn't going to cover all npm/pypi packages.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What can Debian do to provide complex applications to its users?

[Ian Jackson]

Simon McVittie writes ("Re: What can Debian do to provide complex applications 
to its users?"):
> Here is a different straw man, which I think might be similarly effective
> and a lot less work:

FTR, even though I am trying to participate constructively and help
refine Didier's proposal, I think your proposal is a good one too.

I have a reservation which is not really related to proper security
support (which the user simply can't have in Red Queen's Race
ecosystems, whether they install things from the upstream or via some
new thing of ours).

Your proposal would involve some quite large source packages which
would, in some cases, substantially overlap with each other.  Also
these packages would have to be updated much more frequently than
normal.  Often, automatically.  Debian's release cycle is a poor fit.

I think our normal review processes, freeze policies, and so on, would
struggle.  It might even be that our mirror network would struggle.

Didier's proposal differs from yours not only in whether sources and
binaries are agglomerated, but also (implicitly) in whether we try to
do it via our existing development, review and distribution channels.

Ian.

Re: What can Debian do to provide complex applications to its users?

[Ian Jackson]

Sean Whitton writes ("Re: What can Debian do to provide complex applications to 
its users?"):
> On Tue, Feb 27 2018, Ian Jackson wrote:
> > I would like to suggest a radical approach to the source code
> > management for your system: abandon source *packages* in favour of git
> > trees.
> 
> Why do you think Didier's proposal, in particular, represents an
> opportunity to do this?  Is it simply that it will require whole new
> tools to prepare and distribute the .vdebs, so we might as well not use
> source packages from the start?

Yes.

In particular, Didier's proposal is mostly a change to _source code_
management.

Most of Didier's features require no changes to the binary package
format; none require changes to the binary package distribution
mechanisms, nor the development of significant amounts of new software
for managing the construction and publication of (repositories of)
binary packages.

However, Didier's suggestion involves a radical departure in source
code management: specifically, automatic bulk conversion of upstream
sources with very lightweight human approval.  And, a new approach to
build scheduling and source->binary (build) management.  That involves
writing new software.

That new software should be as small as possible.

In Didier's proposal, there is no actual need to publish source
*packages*.  All the upstreams are all using the same VCS (git) in
roughly the same way, and developers who are used to working with this
software in other contexts all expect to be working with VCS trees
(git branches) rather than tarballs.  Ie, the reasons why Debian uses
.dscs rather than git branches do not apply.

Didier's approach *does* involve creating a new tool to do the
automatic conversion (and merging of any delta).  That tool will have
to consume upstream git branches because that's how the upstreams
publish things.  It would be daft not to publish the resulting Debian
source packages as git branches.

So the tool has to consume and produce git branches.  Having it
produce source packages too is more work.  Work that is not
neceessary.  For the success of Didier's suggested project,
nonessential work should be postponed until (i) the project has become
a success enough to justify adding bells and whistles (ii) someone
actually cares enough to do it.

(It is possible that building source packages rather than git branches
would enable the reuse, rather than replacement, of tools like
wanna-build.  But wanna-build is much more complicated than is needed
for Didier's suggestion, and not easy to set up - in part because of
its need to support multiple architectures, which Didier's proposal
rules out.)

> > Furthermore, abandon the patch queue approach to Debian package
> > management.  We will not be able to maintain a big delta to any of
> > these packages anyway.
> 
> No, but we might often have reason to maintain a small delta.  We patch
> upstream source for all sorts of reasons; it is hard to believe it
> wouldn't come up.

Yes.  But the changes will be very small.  Partly because the upstream
packages are usually very small, and partly because rebasing of a
large delta is inherently difficult and will therefore be impractical
to do automatically, frequently and reliably.

This means that there is a much reduced need to represent the delta
(where there is one) in a rich patch-queue form.  Using plain "git
merge" is good for small deltas.  (As you recognise yourself in your
dgit-maint-merge(7) tutorial manpage...)

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: What can Debian do to provide complex applications to its users?

[Sean Whitton]

Hello Ian,

On Tue, Feb 27 2018, Ian Jackson wrote:

> I would like to suggest a radical approach to the source code
> management for your system: abandon source *packages* in favour of git
> trees.

Why do you think Didier's proposal, in particular, represents an
opportunity to do this?  Is it simply that it will require whole new
tools to prepare and distribute the .vdebs, so we might as well not use
source packages from the start?

> Furthermore, abandon the patch queue approach to Debian package
> management.  We will not be able to maintain a big delta to any of
> these packages anyway.

No, but we might often have reason to maintain a small delta.  We patch
upstream source for all sorts of reasons; it is hard to believe it
wouldn't come up.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Sean Whitton]

Hello Didier,

Thanks for sharing this.

On Tue, Feb 27 2018, Didier 'OdyX' Raboud wrote:

> ** is restricted to be arch:all (~ shipping interpreter scripts)

There are compiled binary ecosystems that would benefit from your
proposal, such as Haskell, so could you say more about why you want this
restriction?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Russell Stuart]

On Tue, 2018-02-27 at 14:13 +0100, Didier 'OdyX' Raboud wrote:
> > - we could ship those applications not as .deb but as container
> >   and let them have their own lifecycle
> 
> tl;dr: a new package format is needed, with a new non-suite-specific 
> repository is needed to bring the Debian added-value to these
> ecosystems.

To me 'container' is the right solution.  The problem is Debian doesn't
support building light weight containers well.  In fact nobody does. 
Docker makes an attempt, but distributing static file system images
that have to get their security updates installed by replacing the
entire image is ick.

If I were do the entire thing over again, I would break Debian up into
a series of rings.  In the inner most ring is like the inner most ring of 
Linux.  It's filesystem(s) is readonly to all other rings.  In it sits the code 
for dpkg,  But dpkg wouldn't do much beyond pulling down packages and their 
security upgrades into a /debian directory, which would look rather like /pool 
on the mirrors now, but the .deb's and .dsc's would being directories rather 
than tar archives.

Other containers would run above this.  They create their /usr file
systems by linking into dpkg's /debian directory (which is readonly to
them).  Maintainer scripts would run when these are containers are
built.  This means dpkg is no longer running maintainer scripts, so
just like an Android application a malicious package is limited in the
harm it could cause and in particular uninstall would always work.
These containers would be notified when packages they are running have
security upgrades installed, so they can swap to the new versions at a
convenient time.  We still get to keep the "one copy of each library so
we only have to fix a vulnerability once" advantage Debian has now (and
other current solutions notably lack).

Anybody who has fiddled with containers will have no trouble filling in
the rest of the picture.  It gives us two things: much better security
and a faster way to build containers (because the unpacking step has
already been done).

I realise it sounds grandiose and far fetched, however it can be broken
down into small(ish) steps.  Step 1 would be having dpkg unpack
everything to the /debian directory (including the state it currently
stores under /var) rather than installing it in place, and just placing
links in /usr, /etc and so on.   (I'm am optimist in that I think you
could pull that off without too many things noticing.)  Step 2 would be
to isolate /debian, so the rest of the world sits in its own container
and run the maintainer scripts from within that container.  (I'm such a
optimist that even I think doing this wouldn't require many changes
beyond dpkg.)  The next steps would be moving each application into
it's own container.  They would be much harder, but I suspect once
you've done the refactoring to make dpkg maintained containers
possible, the thing would take on a life of it's own.

In this world, vdeb's are just packages that apt will only permit to be
installed in a container the user has somehow marked as insecure (means
no Debian QA, ie no security patches).  Anybody thinking "yeah, but not
that insecure" should probably read this bug report:

https://github.com/npm/npm/issues/19883

The idea Debian would by default prevent that from trashing my laptop
is real appealing.

signature.asc
Description: This is a digitally signed message part

Re: What can Debian do to provide complex applications to its users?

[Simon McVittie]

Here is a different straw man, which I think might be similarly effective
and a lot less work:

On Tue, 27 Feb 2018 at 14:13:41 +0100, Didier 'OdyX' Raboud wrote:
> As Debian, we
> are insisting that our releases ideally only contain a single version of a
> software, that we insist is made available at system-level.
...
> In other words, vendorization is the tool that allows developers to get rid 
> of 
> distribution constraints and get on with their development through installing 
> the dependencies from their ecosystem as they see fit (non-root), in the 
> (eventually precise) version they need.

I can't help wondering whether vendorizing dependencies (embedded code
copies) would be a better route for ecosystems like npm that we might
categorise as "aggressively modular" - package the app, but not the
dependencies, and treat the vendorized/embedded dependencies as part
of the app. Not embedding or duplicating libraries is already an ideal
that we have to compromise on for pragmatic reasons - browsers use
system libraries in testing/unstable but gradually make increasing use
of embedded code copies in stable[1], drifting further away from the
"no embedded code copies" ideal as time goes on and the latest browser
version becomes increasingly unbuildable against the increasingly old
libraries in stable. It's also how many Rust dependencies are already
managed, as far as I'm aware.

Similarly, we are willing to tolerate embedded code copies for C libraries
that are specifically designed to be "copylibs", such as GNU's gnulib[2],
GNOME's libglnx[3], libgd[4] and (historically) libegg, and the stb_image
micro-library used in multiple game engines, as well as many Autoconf
macros, CMake modules and other supporting files. We do that because these
copylibs are explicitly unstable (updating may require code changes in
the consuming code), or because consumers require a very new version,
or both. Is that really a million miles away from the npm ecosystem?

The obvious retort is "but what about security updates?". Well, what
about security updates? What would happen if there was a security
vulnerability in gnulib code? I'm fairly sure the answer is that the
upstream maintainers of packages that had imported the relevant code,
or the downstream maintainers of those packages, would be expected to
patch the vulnerability - just like what would happen if they had
pasted individual functions into their codebase. (Some nodejs modules
*are* individual functions, for that matter.)

Also, the security team specifically don't provide security
support for libv8, which apparently extends to node-* packages like
, so it's
hard to see how tolerating embedded code copies of nodejs modules in
particular would make their security support situation a whole lot worse:
it's already the case that the upstream and downstream maintainers of
these modules (or the applications that bundle them, or both) provide
the only security maintenance they'll get. In practice, this isn't as
awful as it first appears, because nodejs modules are often very small,
so an individual nodejs module is relatively unlikely to contain security
vulnerabilities even if its defect density is high, simply because there
isn't very much code to be vulnerable.

There is nothing to stop us from enforcing our quality and freeness
standards equally thoroughly for embedded/vendorized code copies -
they're just code, after all. If the process works for the main package,
then extending it to a bundle of vendorized dependencies that share
an orig tarball (or cluster of multiple orig tarballs in 3.0 source
formats) is a matter of scale rather than an entirely separate process,
and if a maintainer or maintainer team can't cope with that scale,
then the same actions need to be taken as for any other package that
can't be maintained to our community standards.

I'm not saying that we should vendorize everything: for large codebases
that are an identifiably separate module with an API (like libjpeg
and zlib) and/or have a history of security vulnerabilities that can
be fixed centrally without breaking ABI (like libjpeg and zlib), it is
of course often absolutely the right course of action to use a system
copy. Similarly, if a maintainer feels that "unvendorizing" a particular
part of the code and using a shared system-wide copy would make their
life easier, they should be welcome to do so.

I'm just not sure that taking the rules we follow for C/C++ shared
libraries and applying them to every other language's ecosystem is either
feasible or desirable - nodejs libraries are not the same as C libraries,
and their tradeoffs are not the same tradeoffs.

This approach is also not a million miles away from the approach taken
in Flatpak, where every dependency is either the runtime vendor's
responsibility (system libraries like libjpeg, zlib, SDL, GTK+) or the
app vendor's responsibility (embedded/bundled/vendorized 

Re: What can Debian do to provide complex applications to its users?

[Ian Jackson]

Ian Jackson writes ("Re: What can Debian do to provide complex applications to 
its users?"):
> The primary difficulty we have with Red Queen's Race [1] ecosystems is
> the lack of stable ABI/APIs, the tight binding of versions, and the
> rapid update cycle.

Missing footnote error.

[1] https://en.wikipedia.org/wiki/Red_Queen%27s_Race

Ian.

Re: What can Debian do to provide complex applications to its users?

[Ian Jackson]

Didier 'OdyX' Raboud writes ("Re: What can Debian do to provide complex 
applications to its users?"):
> Now, as a strawman proposition, here's what I fiddled with in my mind for 
> some 
> days now:
> 
> Imagine
> * a new .vdeb format variant that:
> ** enables for multiple versions to be installed in parallel, where files are
>unpacked in a version-specific paths
> ** forbids any kind of maintainer scripts
> ** is not bound to a specific suite
> ** is restricted to be arch:all (~ shipping interpreter scripts)
> ** can be built mostly automatically (from existing npm/composer/pypi
>packages)
> ** (ideally) can be user-installed
> * a repository of these .vdeb
> ** whose DFSG-freeness is checked
> ** which version set is known, and tracked
> ** that can provide new versions "on-demand"

I think this is an interesting idea and I look forward to seeing a
prototype and maybe using it.  It seems like a better plan than using
something like npm directly.

I have some specific comments:

> Imagine
> * a new .vdeb format variant that:

I think this is more of a "profile" than a new format.

> ** enables for multiple versions to be installed in parallel, where files are
>unpacked in a version-specific paths

Instead, establish a formal convention about embedding the (stable
part of) the version number in the package name.  This is important
so that it is possible to replace a specific "version" of one of these
packages with a modified package which is the "same version".

> ** forbids any kind of maintainer scripts
> ** is not bound to a specific suite
> ** is restricted to be arch:all (~ shipping interpreter scripts)

These can be verified either the archive server as part of package
acceptance, or by apt as an annotation in the sources.list.

> ** can be built mostly automatically (from existing npm/composer/pypi
>packages)

This is, effectively, a proposal for new kind of source code handlign.
More on that below.

> ** (ideally) can be user-installed

This one would require some thought.  Do the target ecosystems support
transplantation of installed directory trees, without modification ?

Perhaps the experience of other projects doing "prefixed" uses of dpkg
might be relevant.  Eg
  https://wiki.termux.com/wiki/FAQ#Is_Termux_a_complete_Linux_Environment.3F
(But the prefix there is baked into the .deb.)

How is this going to work for build-dependencies ?  If one says
"build-depends: npm-vdeb-foo", will the build system know to look in
$HOME as well as /usr ?

> * a repository of these .vdeb
> ** whose DFSG-freeness is checked
> ** which version set is known, and tracked
> ** that can provide new versions "on-demand"

It is important to appreciate what we are *discarding* as too hard.

The primary difficulty we have with Red Queen's Race [1] ecosystems is
the lack of stable ABI/APIs, the tight binding of versions, and the
rapid update cycle.

The downsides compared to our traditional model are:

  * No coherent security support.  Programs in these ecosystems are
typically full of unnoticed security bugs and changing too rapidly
to fix them.  When serious bugs occur they may or may not be
fixable, let alone actually fixed.  It will not in general be
possible to even discover whether the software is vulnerable.

This is a very big downside.  It means that systems running this
software should not do other tasks as well; that this software
should not be given direct access to personal data (certainly not
sensitive personal data); that arrangements need to be made for
recovery from the expected compromises.

These are all operational matters which we should draw to the
attention of users.  And we need to make sure that we aren't
making things worse than the upstream (which is why automatic
updates are important).

  * No useful ABI/API stability guarantees for local-user-supplied
programs building on top of this.  Ie, if you build on top of this
stuff you have to run to keep up just like everyone else.

  * No stability over the length of a Debian release (ie, users of
these packages will have to expect and cope with constant churn)

  * Support for whole application suites might randomly go away with
very little notice.  If they stop being updated upstream, we won't
be able to fix it.  For a while users can continue to use the
frozen version, probably.  But eventually the bugs in it will
become too serious and it will become impossible to build, or
impossible to run reliably.

> How does that sound?

Your proposal depends on continuous and almost-automatic
incorporation of upstream code.  Our current source package workflows
are not suited to this.

OTOH we do not want to abandon the Debian source package format
completely because we h

Re: What can Debian do to provide complex applications to its users?

[Didier 'OdyX' Raboud]

Le vendredi, 16 février 2018, 16.11:29 h CET Raphael Hertzog a écrit :
> I don't have any definite answers although there are ideas to explore:
> 
> - we could relax our requirements and have a way to document the
>   limitations of those packages (wrt our usual policies)
> 
> - we could ship those applications not as .deb but as container
>   and let them have their own lifecycle

tl;dr: a new package format is needed, with a new non-suite-specific 
repository is needed to bring the Debian added-value to these ecosystems.

My current understanding of the problem is that there are two problems 
entangled with eachothers, for which Debian is currently not providing 
solutions

* version spread
* layer-specific expectations

# Version spread

Certain software will need to be available in multiple versions to satisfy 
reverse dependencies constraints, be it for convenience reasons ("the 
only version of that library my application has been tested with"), 
willingness to use the shiniest and latest, or any other reason.

We have that problem already with jQuery and others, but it seems to be mostly 
contained thanks to reasonable back- and forward- compatibility. As Debian, we 
are insisting that our releases ideally only contain a single version of a 
software, that we insist is made available at system-level. The reasons for 
that are multiple, but the main one is the facilitated tracking and fixing of 
security issues, that we then only have to fix once, thereby protecting all 
its reverse dependencies. As has been mentioned with the example of Django, 
some software change too rapidly and too much for that single-version 
constraint to hold [0], and providing only one version is bound to not satisfy 
most reverse dependencies' needs; in other words, it's not going to be used.

Another aspect of the version spread is that it limits the set of software we 
check DFSG-freeness of. There's an initial "seal-of-approval" from the FTP-
Masters when going through NEW, but then it is currently left to the 
maintainer and the community to do this ongoing check. The cristallization 
towards the stable release will also encourage checking for eventual 
violations in only one version per software.

# Layer-specific expectations

Debian has long insisted that all software shipped in 'main' is constrained by 
the same standards (DFSG, Debian Policy, minimal code duplication, 
buildability, portability, maintainability over the course of a stable release 
cycle, etc), and gets the same support "in return" (security support, common 
bugtracker, etc). This is a very good model for "lower layers"; the plumbing 
(outside of new hardware support): I don't really care what version of CUPS I 
get, as long as I can print; I don't really care what minor version of Apache 
I get as long as it works, etc etc.

On the other hand, when developing a  application, with a rapidly-moving ecosystem, developer will insist on 
having the latest version of plenty of my direct dependencies, because working 
on whatever was released in Debian stable will just be a hassle more than a 
help. Also, the friction for them to fix a bug they have experienced in one of 
these dependencies is much lower than getting it fixed in Debian stable: given 
a responsive maintainer, one can get a new release with a pull request merged 
in a matter of days; and if that doesn't work, the tools allow to get a 
precise git hash for a given project. They can fix the bug, use the fixed 
version, and go on with my development.

The point I'm trying to make, which I hope is obvious, is that the 
expectations upon software in different layers are vastly different: I want 
rock-solid kernel, libc and language interpreter, I want reasonably recent 
intermediate layers, but I want bleeding edge direct dependencies. Of course, 
fast-moving layers will not provide the same warranties as slow-moving layers.
In a Debian-idealistic world, we could hold all layers up to the same 
standards, stabilize them up to a point where they can become part of stable, 
and all application developers would use Debian's set of libraries, from 
stable. Point is, that's not what is expected from Debian either: a Python 
application will only use Debian's python interpreter and virtualenv, all the 
rest will be taken from somewhere else than Debian, because it's way closer to 
how it'll be developped, and in many cases (just one missing library in 
stable), the only reasonable way to deploy said application. Then, instead of 
having only Debian as 'security' counterpart (getting the host to a secure 
state through a `apt upgrade`), the application developer now has to (do 
they?) care about the closer layer's security hirself.

# Now, onto a solution

In other words, vendorization is the tool that allows developers to get rid of 
distribution constraints and get on with their development through installing 
the dependencies from their ecosystem as they see fit (non-root), in the 
(eventually 

Re: What can Debian do to provide complex applications to its users?

[Wouter Verhelst]

On Fri, Feb 16, 2018 at 04:11:29PM +0100, Raphael Hertzog wrote:
> Hello everybody,
> 
> the fact that I had to request the removal of dolibarr from Debian makes
> me sad (see below for the reasons) and I believe that we should be able
> to do better to provide complex applications to our end users.

So, I think the problem here is not "complex applications" -- we have
many complex applications packaged for Debian -- but instead,
"applications written in languages that have an ecosystem which does not
align with Debian's procedures and technology very well".

I think the proper answer is not something along the lines of "they
suck, they should clean up their act". It is *also* not something along
the lines of "we suck, we should change our whole infrastructure".

Instead, I think the proper way to fix this issue is to engage with the
community behind the language that we currently can't package very well,
and see how we can improve that situation. I'm sure that having minified
javascript be part of a build system isn't necessarily something they'd
be opposed to -- after all, if you do that then you can limit the size
of the resulting .min.js files that you ship even more, which is good
for performance -- and then you end up with a situation of "static
libraries", which while not ideal is better than the status quo (we'd
just need to make sure that the security team's tools track
build-depends on javascript libraries, and they'd know which packages to
rebuild when a security update is necessary).

That's just an example; I'm sure there are other situations where we can
improve the situation on both sides and result in something that makes
everyone a little happier.

> Dolibarr is not alone:
> 
> - while gitlab is packaged in Debian, its packaging took years and the
>   result is brittle because it can break in many ways whenever one the 
>   dozens of dependencies gets updated to some new upstream version
>   (BTW salsa.debian.org does not use the official package)

Which I think is a shame, but given that gitlab doesn't really have any
"long term support" versions, not unsurprising either.

> - I have the Debian packaging of distro-tracker (the code behind
>   tracker.debian.org) available in the git repository for years
>   but I never released it into Debian because it embeds a few javascript
>   libraries (bootstrap, jquery) and I don't want to validate that
>   it does work with the versions currently in Debian

To do so, you can use something like the libjs-qunit package which
should allow you to run unit tests on the files in question. Given
enough unit tests, changing out a javascript library from under the rest
of your code shouldn't be too problematic (barring any bugs, which may
always appear and for which we have a very good bug tracker).

In addition, I think that the two specific things you mention here have
a static API, and thus that you shouldn't be afraid to use a different
version in production as compared to the one you used during
development.

After all, if even Debian Developers, while writing code specifically
for Debian, are not doing the things that we expect external developers
to do in order for them to see their code packaged in Debian, then who
are we to request that they do those things?

(this is why I chose to use the packaged versions of jQuery and
bootstrap for SReview rather than embedding them, FWIW)

-- 
Could you people please use IRC like normal people?!?

  -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
 Hacklab

Re: What can Debian do to provide complex applications to its users?

[Alexander Wirt]

On Wed, 21 Feb 2018, Vincent Bernat wrote:

>  ❦ 21 février 2018 07:07 +0100, Alexander Wirt  :
> 
> > No, backports doesn't have official security support in the meaning that
> > the team is tracking and looking after security issues in backports.
> > Nevertheless every backporter has to care about security, we do expect that
> > uploaders care about their packages - this does of course include security
> > support.
> 
> The net result for our users is that backports should not be expected to
> be up-to-date with security. It took me approximately one minute to go
> through latest DSA to find an example: Exim in backports is
> 4.89-2+deb9u1~bpo8+1. 4.89-2+deb9u2 has been uploaded in
> December. 4.89-2+deb9u3 has been uploaded in February.
yes, you are completely right. The maintainers responsibility was to upload
this package which he didn't. I just wanted to make the parameters of the
"best effort approach" clear. 

Alex


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Vincent Bernat]

 ❦ 21 février 2018 07:07 +0100, Alexander Wirt  :

> No, backports doesn't have official security support in the meaning that
> the team is tracking and looking after security issues in backports.
> Nevertheless every backporter has to care about security, we do expect that
> uploaders care about their packages - this does of course include security
> support.

The net result for our users is that backports should not be expected to
be up-to-date with security. It took me approximately one minute to go
through latest DSA to find an example: Exim in backports is
4.89-2+deb9u1~bpo8+1. 4.89-2+deb9u2 has been uploaded in
December. 4.89-2+deb9u3 has been uploaded in February.

I think backports are a great asset for Debian and a clear advantage
over other stable distributions. But we shouldn't lie to our users by
telling it is security supported (and, as a matter of fact, we don't).

I am sorry if it sounds like criticism, it shouldn't. I am only trying
to show we already have a non-security-supported archive in Debian (or
best-effort-security-supported archive if it sounds better).
-- 
Hain't we got all the fools in town on our side?  And hain't that a big
enough majority in any town?
-- Mark Twain, "Huckleberry Finn"


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Alexander Wirt]

On Tue, 20 Feb 2018, Vincent Bernat wrote:

>  ❦ 20 février 2018 09:05 +0200, Arto Jantunen  :
> 
> >> Moreover, backports do not accept security patches. You can only push a
> >> version in testing (or unstable). Notably, if the version in testing is
> >> not easily backportable (because of new dependencies), you may wait
> >> quite some time before you get a security update.
> >
> > Also not true. You can request an exception to this for your security
> > update, but you do need to communicate about this with the backports
> > team before uploading.
> 
> Also? What was not true? The Debian Backports FAQ?
> 
> The exception you mention is not documented. It is also likely to just be
> rejected:
>  
>  
> http://lists.alioth.debian.org/pipermail/pkg-roundcube-maintainers/2017-November/002070.html
> 
> And the backport team has been pretty clear this is not the right way to
> maintain backports:
> 
>  https://lists.debian.org/debian-backports/2017/05/msg00059.html
That does mean we don't want that packages are "maintained" that way in
backports. For a one time security patch, you can always ask for an
exception. But this is just to give the maintainer more time to update the
backport with the new version from testing/unstable. 

So speaking as one of the backports ftpmasters:

No, backports doesn't have official security support in the meaning that
the team is tracking and looking after security issues in backports.
Nevertheless every backporter has to care about security, we do expect that
uploaders care about their packages - this does of course include security
support.

For some specific security problem, you can always talk with us about an
(short living) exception to give the maintainer more time and keep our users
save. 

What we don't want is people maintaining packages in that way in backports. 
Source of backports are testing/stable (for old-stable-backports) and in some
times (security) unstable. We do expect that every package follows those
suites. If a maintainer doesn't want/can this, backports is the wrong
place for maintaing that package.

Hope that helps

Alex - Backports ftp-master



signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Paul Wise]

On Wed, Feb 21, 2018 at 4:10 AM, Gunnar Wolf wrote:

> It's sometimes hard to explain why we need updated software...

Perhaps it helps to point out where Debian and users are placed in the
ecosystem of software, hardware, technology and society and the
pressures that each actor places on other actors. There is a diagram
of some of the Debian side of it here:

http://cfnarede.com.br/en/infographic-of-debian
https://github.com/filhocf/infographics

PS: more diagrams:

https://wiki.debian.org/Diagrams

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What can Debian do to provide complex applications to its users?

[Paul Wise]

On Tue, Feb 20, 2018 at 11:27 PM, Adam Borowski wrote:

> And without security support for its dependencies, no reproducible build
> system, etc.

That isn't necessarily the case for Flatpak, it all depends on who is
doing the build and what their policies and procedures are.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What can Debian do to provide complex applications to its users?

[Gunnar Wolf]

Philipp Kern dijo [Mon, Feb 19, 2018 at 09:18:13AM +0100]:
> Putting security support over all else is surely how some people see it. But
> some upstreams also complain if you are going to ship ancient versions
> because the most recent ones contain all of the fixes. It's certainly more
> work to validate security fixes when backporting them to older versions. So
> it's also the "stable" guarantee (whatever it is seen as) that might need
> some re-adjustment.

Maybe we abuse the "security support" term - It's also "stability
support" or "bugfix commitment". We can commit to fixing the one
version of a library we ship that's producing erroneous results on
given inputs; that's not necessarily security-related (although it
might be). If libpng gets confused¹, it's not a security issue until
its input is treated on a sensitive way.

¹ http://gwolf.org/node/1952 for quite an old report

> One of the values is that you get some set of software that works together
> as a base and doesn't change, but then people install software on top of it
> that provides their service and if it's actually the thing they want to
> provide it's most likely not packaged anymore at this point. Because you'd
> want the latest features of the product you're using. So there's already a
> disconnect of essentially two tracks: the system's base at a solid version
> and whatever it is you want to offer at a fast moving pace.

This is an oft-repeating issue. At some point, Debian was the ugly
duckling because we did two- or three-year release cycles; at some
point, basically all other important distributions switched to a
longer release cycle in some way or another, keeping a "stabilized
snapshot" of sorts (i.e. Fedora for RH, Ubuntu on its non-LTSes). Or,
OTOH, became a fully rolling release. And we have both - only that we
don't recommend testing to users who expect the rock-solid we are
known to produce.

Having large software ready as a Debian package allows me, as a casual
user, to take it for a spin and evaluate it. As a sysadmin, I have the
habit of preferring Debian-packaged software for everything I can,
except when there is a _real_ reason for needing somewhat
newer. Again, I know I'm probably more Debian-centric than most of our
users, but I don't think it is too hard a line to pick: Use Debian for
all of your needs, and if it does not completely fill the bill, you
can locally install what you need. Of course, if it's hard to install
or something like that... maybe you want to learn the technologies you
will be using instead!

> That's also a reality in 2018. And coming up with arbitrary
> deadlines of support are not all that helpful. Users don't care if
> the ancient version of the software they need in stable is security
> supported until mid-2022. If it doesn't satisfy their requirements
> anymore, they move to testing or to another distribution.

My users are most often irked when I do OS upgrades and modify their
stale, beloved webmail installation. Of course, I still do it whenever
it's needed. But I have been repeatedly, even publicly asked as to why
do we push a "consumerist" worldview where old things are not
good. It's sometimes hard to explain why we need updated software...

Re: What can Debian do to provide complex applications to its users?

[Gunnar Wolf]

Raphael Hertzog dijo [Mon, Feb 19, 2018 at 03:19:59PM +0100]:
> On Fri, 16 Feb 2018, Jonathan Carter (highvoltage) wrote:
> > > - we could relax our requirements and have a way to document the
> > >   limitations of those packages (wrt our usual policies)
> > 
> > Which requirements are you referring to? If it's relaxing the need for
> > source for minified javascript, then no thanks.
> 
> Instead of requiring the source to be provided in the source package as a
> non-minified file, we could require the packager to document in
> debian/README.source where the upstream sources actually are.

Pointing to sources outside our system might go away, and that would
make us instant-buggy. That's why we have introduced the
debian/missing-sources mechanism; your source package bloats, but is
source-complete. Further, you can parse them with the "correct"
minifiers and compare the results are identical (or provide our
minified versions if they are not).

> When I was maintaining wordpress, I introduced the idea of providing
> debian/missing-sources/ to comply with the Debian policy.

Yay, so it's not you who I have to lecture on this ;-)

> I would just dump there the upstream tarball of the bundled
> libraries to be sure that we have the source for the correct
> version. The Debian/ftpmaster rules are respected but it's not
> really better than the above because you still don't have a simple
> way to rebuild a modified version of the javascript library shipped
> in the package.

build-depend on yui-compressor, node-uglifyjs-webpack-plugin,
libjavascript-minifier-perl, or whatever minifier suits you; add them
to the binary package in debian/rules instead of just copying over the
upstream-provided minified versions. That is, think of minification as
you would think of compilation.

> So instead of ugly work-arounds, it might be better to just acknowledge
> that we can't have the same level of support for all applications.
> (...)
> Those applications could rely on the package manager of their ecosystem to
> setup the dependencies as they need them without polluting the host
> system.

...But that pollutes the host system for all of their ecosystem. And
that's what I suggest - Paraphrasing the SC, «We acknowledge that some
of our users require the use of works that do not conform to the
Debian Free Software Guidelines». We might come up with an area
similar to contrib or non-free (or even decide to ship such systems
there, as they kind-of-belong there!)

Maybe Drupal, WordPress, NextCloud or WhatEver are not non-free by
themselves. But they are not allowing for a practical source
distribution; they force our hands into trusting software we cannot
vouch for or give warranties on. So, maybe they belong in non-free.

Re: What can Debian do to provide complex applications to its users?

[Adam Borowski]

On Tue, Feb 20, 2018 at 04:02:00PM +0200, Adrian Bunk wrote:
> You were talking about flatpak.
> 
> The whole point of flatpak is that the same app is equally integrated
> in all Linux distributions.

And without security support for its dependencies, no reproducible build
system, etc.

Dᴏ ɴᴏᴛ ᴡᴀɴᴛ.


-- 
An imaginary friend squared is a real enemy.

Re: What can Debian do to provide complex applications to its users?

[Paul Gevers]

Hi all,

This e-mail isn't in reply to any specific e-mail in this thread but I
like to add some words that may or may not inspire others for ideas.

It is my intent to soon start working¹ on the Debian bikesheds (or
Debian's PPA). Depending on requirements and use cases we may be able to
use those for some of the ideas mentioned in this thread.

Paul
PS: I am slightly nervous about announcing this intent so broadly, but I
thought it may be of interest in the discussion.

¹ As I haven't been involved in the discussions in the past it will take
a while before they materialize (if I am able to pull it off at all).
I'll first have to collect the outcome of previous discussions and
previous work. I'll contact the parties involved and determine which
connections need to be made and what code needs to be made where.



signature.asc
Description: OpenPGP digital signature

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Tue, Feb 20, 2018 at 11:56:04AM +0100, Michael Meskes wrote:
> > > Right, and that's why we were talking about stuff like flatpak that
> > > bring the application with its dependencies, more or less like a
> > > container.
> > 
> > That's a better solution for such cases than shipping the software
> > in Debian.
> > 
> > And it's distribution-agnostic, meaning it can be provided once by 
> > upstream for all distributions instead of duplicating work in every 
> > Linux distribution.
> 
> This is not exactly what I meant. What we are talking about is adding
> another section and/or another format to the Debian archive so we can
> publish packages for these applications that come with their
> dependencies. With things being integrated in a Debian system they are
> kind of Debian specific.

You were talking about flatpak.

The whole point of flatpak is that the same app is equally integrated
in all Linux distributions.

> Michael

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Michael Meskes]

> > Right, and that's why we were talking about stuff like flatpak that
> > bring the application with its dependencies, more or less like a
> > container.
> 
> That's a better solution for such cases than shipping the software
> in Debian.
> 
> And it's distribution-agnostic, meaning it can be provided once by 
> upstream for all distributions instead of duplicating work in every 
> Linux distribution.

This is not exactly what I meant. What we are talking about is adding
another section and/or another format to the Debian archive so we can
publish packages for these applications that come with their
dependencies. With things being integrated in a Debian system they are
kind of Debian specific.

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

Re: What can Debian do to provide complex applications to its users?

[Alastair McKinstry]

On 19/02/2018 20:42, Michael Meskes wrote:=
>> Various other packages in stable won't work with the latest Node.js
>> and will also require upgrading.
>>
>> In the Node.js ecosystem it is par for the course when upgrading
>> a package breaks countless reverse dependencies.
> Right, and that's why we were talking about stuff like flatpak that
> bring the application with its dependencies, more or less like a
> container.
>
> Michael

We need to be clear: is the problem we are facing one of timeliness
(stable being out of date), or systems integration ?

If its dependencies getting out of hand, containerisation is only a
temporary fix: it will become increasingly impossible to get  a complex
application working if the dependencies are conflicting. This then is an
upstream problem, not just a Debian one.

I think we also need to pay attention to the fact the DevOps approach
suits big organisations better than small: if I am managing a technical
portal with ~50-100 JAR files as deps, but 1-5 developers, I'm facing
much bigger issues than Google or Facebook with x5 the number of deps,
but x1000 the number of developers to handle the vendorization. I
suspect we're seeing a managed breakdown in standards to help
commercialize the free software world, much as it becomes much harder to
run your own mail / IM / Calendaring services.

If the problem is one of timeliness perhaps we need to reconsider our
cadence, especially with the advent of LTS releases: e.g. a 6-month
cadence for unstable with much longer LTS releases. If its integration,
we should be looking at changing the whole F/OSS landscape (hey, its
been done before), and leading the integration.

Alastair


-- 
Alastair McKinstry, , , 
https://diaspora.sceal.ie/u/amckinstry
Commander Vimes didn’t like the phrase “The innocent have nothing to fear,”
 believing the innocent had everything to fear, mostly from the guilty but in 
the longer term
 even more from those who say things like “The innocent have nothing to fear.”
 - T. Pratchett, Snuff

Re: What can Debian do to provide complex applications to its users?

[Vincent Bernat]

 ❦ 20 février 2018 09:05 +0200, Arto Jantunen  :

>> Moreover, backports do not accept security patches. You can only push a
>> version in testing (or unstable). Notably, if the version in testing is
>> not easily backportable (because of new dependencies), you may wait
>> quite some time before you get a security update.
>
> Also not true. You can request an exception to this for your security
> update, but you do need to communicate about this with the backports
> team before uploading.

Also? What was not true? The Debian Backports FAQ?

The exception you mention is not documented. It is also likely to just be
rejected:
 
 
http://lists.alioth.debian.org/pipermail/pkg-roundcube-maintainers/2017-November/002070.html

And the backport team has been pretty clear this is not the right way to
maintain backports:

 https://lists.debian.org/debian-backports/2017/05/msg00059.html
-- 
Choose a data representation that makes the program simple.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Arto Jantunen]

Vincent Bernat  writes:
> Moreover, backports do not accept security patches. You can only push a
> version in testing (or unstable). Notably, if the version in testing is
> not easily backportable (because of new dependencies), you may wait
> quite some time before you get a security update.

Also not true. You can request an exception to this for your security
update, but you do need to communicate about this with the backports
team before uploading.

-- 
Arto Jantunen

Re: What can Debian do to provide complex applications to its users?

[Vincent Bernat]

 ❦ 19 février 2018 22:59 GMT, Craig Small  :

>> >> a bit like backports that are not security supported
>> >> either.
>> >
>> > this is now the 2nd mail within 24h were you claim this *wrongly*.
>> >
>> > backports are (supposed to be) getting security support. if you dont do
>> > this for your backports, you shouldnt have upload rights.
>>
>> From https://backports.debian.org/FAQ/:
>>
>> Q: Is there security support for packages from backports.debian.org?
>>
>> A: Unfortunately not. This is done on a best effort basis by the people
>> who track the package
>
>
> This looks like a language confusion. There is no security *team* support
> but there is security *package updates* by the individual maintainers.

I don't see where is the confusion. There is no security support. It's
written verbatim. This doesn't mean there is no security update, but
these updates are pushed on a best effort basis.

Moreover, backports do not accept security patches. You can only push a
version in testing (or unstable). Notably, if the version in testing is
not easily backportable (because of new dependencies), you may wait
quite some time before you get a security update.
-- 
How apt the poor are to be proud.
-- William Shakespeare, "Twelfth-Night"


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Daniel Dehennin]

Michael Meskes  writes:


[...]

> Maybe you answered your question yourself. How about we tie our
> security support to upstream's? Instead of fixing and backporting
> ourselves we promise our users that this section of the archive will
> get upstream's latest fixes even if that means the version number
> changes.
>
> This way the users would get a lot of benefits from using Debian but no
>  drawback compared to the self-installed alternative.

Hello, as a sysadmin and Ubuntu derivatives[1] at work, I remembered
having the nice surprise of some incompatible changes in MySQL some time
ago.

Backporting the fix was not possible/too complex so new upstream patch
level was integrated, modifying something around comments handling in
.sql files IIRC.

Finding the erratic problem, fixing it and distributing it was quite
“intensive”.

We monitor the -proposed repository but the change passed unseen by our
jenkins.

So please, before considering following upstream, consider what a
sysadmin needs to do to upgrade/test/deploy the configuration.

I'm dreaming the day every configuration file will be managed by
Config::Model but even this is not bullet proof ;-)

my 2¢.

Footnotes: 
[1]  around 25k servers deployed

-- 
Daniel Dehennin
Récupérer ma clef GPG: gpg --recv-keys 0xCC1E9E5B7A6FE2DF
Fingerprint: 3E69 014E 5C23 50E8 9ED6  2AAD CC1E 9E5B 7A6F E2DF


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Craig Small]

On Tue, 20 Feb. 2018, 09:44 Vincent Bernat,  wrote:

>  ❦ 19 février 2018 22:33 GMT, Holger Levsen  :
>
> >> a bit like backports that are not security supported
> >> either.
> >
> > this is now the 2nd mail within 24h were you claim this *wrongly*.
> >
> > backports are (supposed to be) getting security support. if you dont do
> > this for your backports, you shouldnt have upload rights.
>
> From https://backports.debian.org/FAQ/:
>
> Q: Is there security support for packages from backports.debian.org?
>
> A: Unfortunately not. This is done on a best effort basis by the people
> who track the package


This looks like a language confusion. There is no security *team* support
but there is security *package updates* by the individual maintainers.

The individual contribution sounds optional here in the FAQ but from my
experience with backports it's strongly encouraged.

 - Craig

>
>
-- 
Craig Small https://dropbear.xyz/ csmall at : dropbear.xyz
Debian GNU/Linuxhttps://www.debian.org/   csmall at : debian.org
Mastodon: @smalls...@social.dropbear.xyz Twitter: @smallsees
GPG fingerprint:  5D2F B320 B825 D939 04D2  0519 3938 F96B DF50 FEA5

Re: What can Debian do to provide complex applications to its users?

[Vincent Bernat]

 ❦ 19 février 2018 22:33 GMT, Holger Levsen  :

>> a bit like backports that are not security supported
>> either.
>
> this is now the 2nd mail within 24h were you claim this *wrongly*.
>
> backports are (supposed to be) getting security support. if you dont do
> this for your backports, you shouldnt have upload rights.

From https://backports.debian.org/FAQ/:

Q: Is there security support for packages from backports.debian.org?

A: Unfortunately not. This is done on a best effort basis by the people
who track the package, usually the ones who originally did upload the
package into backports. When security related bugs are fixed in Debian
unstable the backporter is permitted to upload the package from directly
there instead of having to wait until the fix hits testing. You can see
the open issues for jessie-backports in the security tracker (though
there may be false positives too, the version compare isn't perfect yet)
-- 
Many a writer seems to think he is never profound except when he can't
understand his own meaning.
-- George D. Prentice


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Holger Levsen]

On Mon, Feb 19, 2018 at 08:44:58PM +0100, Vincent Bernat wrote:
> a bit like backports that are not security supported
> either.

this is now the 2nd mail within 24h were you claim this *wrongly*.

backports are (supposed to be) getting security support. if you dont do
this for your backports, you shouldnt have upload rights.


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[The Wanderer]

On 2018-02-19 at 16:03, Adrian Bunk wrote:

> On Mon, Feb 19, 2018 at 03:52:30PM -0500, Roberto C. Sánchez wrote:
> 
>> On Mon, Feb 19, 2018 at 10:16:56PM +0200, Adrian Bunk wrote:

>>> Debian already does "security by upstream releases" for Firefox, 
>>> and this clearly shows why this is problematic:
>> 
>> Also PostgreSQL, formerly MySQL, OpenJDK, etc. Some go smoothly (I
>> think PostgreSQL upstream is very good here), and some do not.
> 
> These (and also the kernel) are "following an upstream LTS branch", 
> not "security by upstream releases".
> 
> Also for Firefox the new releases on an upstrem LTS branch (currently
> 52) are usually not a problem.
> 
> The problem with Firefox is the once per year switch to a new LTS
> branch, like this year Firefox 52 -> 59. We aren't doing that for
> PostgreSQL.

Nit: the new Firefox ESR this year will apparently be version 60, not
59. They've postponed it by one release this time around, for reasons I
haven't bothered to retain.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw



signature.asc
Description: OpenPGP digital signature

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Mon, Feb 19, 2018 at 03:52:30PM -0500, Roberto C. Sánchez wrote:
> On Mon, Feb 19, 2018 at 10:16:56PM +0200, Adrian Bunk wrote:
> > On Mon, Feb 19, 2018 at 08:40:12PM +0100, Michael Meskes wrote:
> > >...
> > > > An example what "no security support" means in practice:
> > > 
> > > I don't think anyone suggest "no security", but something like
> > > "security by upstream releases".
> > 
> > How can you guarantee that to our users for buster until mid-2022?
> > 
> > This only works when upstream provides an LTS branch covering the
> > lifetime of the Debian release.
> > 
> > Debian already does "security by upstream releases" for Firefox,
> > and this clearly shows why this is problematic:
> 
> Also PostgreSQL, formerly MySQL, OpenJDK, etc. Some go smoothly (I think
> PostgreSQL upstream is very good here), and some do not.

These (and also the kernel) are "following an upstream LTS branch",
not "security by upstream releases".

Also for Firefox the new releases on an upstrem LTS branch
(currently 52) are usually not a problem.

The problem with Firefox is the once per year switch to a new
LTS branch, like this year Firefox 52 -> 59.
We aren't doing that for PostgreSQL.

> Regards,
> 
> -Roberto

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Mon, Feb 19, 2018 at 09:42:28PM +0100, Michael Meskes wrote:
> > > And why wouldn't we offer said upstream version instead of the
> > > unsupported older one?
> > 
> > In some cases this might require changing literally thousands of 
> > packages in stable.
> > 
> > Imagine said upstream version requires the latest Node.js.
> > 
> > Various other packages in stable won't work with the latest Node.js
> > and will also require upgrading.
> > 
> > In the Node.js ecosystem it is par for the course when upgrading
> > a package breaks countless reverse dependencies.
> 
> Right, and that's why we were talking about stuff like flatpak that
> bring the application with its dependencies, more or less like a
> container.

That's a better solution for such cases than shipping the software
in Debian.

And it's distribution-agnostic, meaning it can be provided once by 
upstream for all distributions instead of duplicating work in every 
Linux distribution.

> Michael

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Roberto C . Sánchez]

On Mon, Feb 19, 2018 at 09:42:28PM +0100, Michael Meskes wrote:
> 
> Right, and that's why we were talking about stuff like flatpak that
> bring the application with its dependencies, more or less like a
> container.
> 
Which happens to bring with an entirely different set of problems. That
said, the problems are not really any different than the problems
introduced by installing components with cpan or pip and then forgetting
about them.

Regards,

-Roberto

-- 
Roberto C. Sánchez

Re: What can Debian do to provide complex applications to its users?

[Roberto C . Sánchez]

On Mon, Feb 19, 2018 at 10:16:56PM +0200, Adrian Bunk wrote:
> On Mon, Feb 19, 2018 at 08:40:12PM +0100, Michael Meskes wrote:
> >...
> > > An example what "no security support" means in practice:
> > 
> > I don't think anyone suggest "no security", but something like
> > "security by upstream releases".
> 
> How can you guarantee that to our users for buster until mid-2022?
> 
> This only works when upstream provides an LTS branch covering the
> lifetime of the Debian release.
> 
> Debian already does "security by upstream releases" for Firefox,
> and this clearly shows why this is problematic:

Also PostgreSQL, formerly MySQL, OpenJDK, etc. Some go smoothly (I think
PostgreSQL upstream is very good here), and some do not.

Regards,

-Roberto

-- 
Roberto C. Sánchez

Re: What can Debian do to provide complex applications to its users?

[Michael Meskes]

> > And why wouldn't we offer said upstream version instead of the
> > unsupported older one?
> 
> In some cases this might require changing literally thousands of 
> packages in stable.
> 
> Imagine said upstream version requires the latest Node.js.
> 
> Various other packages in stable won't work with the latest Node.js
> and will also require upgrading.
> 
> In the Node.js ecosystem it is par for the course when upgrading
> a package breaks countless reverse dependencies.

Right, and that's why we were talking about stuff like flatpak that
bring the application with its dependencies, more or less like a
container.

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Mon, Feb 19, 2018 at 08:44:58PM +0100, Vincent Bernat wrote:
>...
> Or we could put those software in a special repository (called "unsupported")
>...

What about calling it "nsa-enablement"?
Cause that's what it is.

But to be fair, no longer installing packages without security support 
in the default install would already be an improvement compared to 
stretch...

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Mon, Feb 19, 2018 at 08:40:12PM +0100, Michael Meskes wrote:
>...
> > An example what "no security support" means in practice:
> 
> I don't think anyone suggest "no security", but something like
> "security by upstream releases".

How can you guarantee that to our users for buster until mid-2022?

This only works when upstream provides an LTS branch covering the
lifetime of the Debian release.

Debian already does "security by upstream releases" for Firefox,
and this clearly shows why this is problematic:
- Firefox is very picky about the versions of two compilers
  (gcc and rust) being used. That's why wheezy contains a
  more recent gcc for Firefox, and that's why soon there will
  have to be a bootstrap of a more recent rust in stretch.
- Firefox upstream decided to break all extensions, including the
  ones packaged in stable, in the next ESR.

Except for extensions Firefox is a leaf package, imagining
"security by upstream releases" for some core part of the
system like OpenSSL sounds hilarious.

Node.js is the core of an ecosystem with > 1k packages in Debian.

gitlab is not the core of an ecosystem, but it uses both uses Node.js 
and Rails. How can you guarantee to provide "security by upstream 
releases" for gitlab until mid-2022 if a new gitlab might require
more recent versions of many dependencies?

> Michael

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Mon, Feb 19, 2018 at 08:35:29PM +0100, Michael Meskes wrote:
> > What is the user supposed to do when Debian announces that some 
> > software essential for that user is no longer supported in the
> > stable release the user is using?
> 
> Again, where does this differ from the user realizing their own self-
> baked installation cannot be upgraded anymore?

Upstream often has ways and schedules for their software that just 
happen to differ from how and when Debian does things.

"self-baked" might just be the normal upstream way of installing the 
software.

In the best case uptream provides a container containing everything the 
software needs.

> > At that point the options for the user are either to continue using 
> > software that might already have known grave vulnerabilities, or to
> > do a rushed attempt trying to find some way to self-install an
> > upstream
> > version that is still supported.
> 
> And why wouldn't we offer said upstream version instead of the
> unsupported older one?

In some cases this might require changing literally thousands of 
packages in stable.

Imagine said upstream version requires the latest Node.js.

Various other packages in stable won't work with the latest Node.js
and will also require upgrading.

In the Node.js ecosystem it is par for the course when upgrading
a package breaks countless reverse dependencies.

And all this automatically deployed via unattended-upgrades to
millions of machines running Debian stable.

> Michael

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Vincent Bernat]

 ❦ 19 février 2018 20:36 +0200, Adrian Bunk  :

>> Debian is not only about security support. We provide packages without
>> security support. We also have backports that come without security
>> support either. This is still better than installing random packages
>> made by random people which may or may not integrate properly into
>> Debian.
>
> The software might integrate properly into Debian - and allow everyone 
> on the internet to take control of your computer.
>
> On the server side we are talking about software like Node.js or gitlab.
>
> On the desktop side the MUA that comes with our default desktop renders
> HTML emails using a web engine that is not security supported by Debian.
>
> An example what "no security support" means in practice:
>
> If you are running Debian stable and haven't yet installed the
> LibreOffice security updates Debian published a few days ago,
> then opening a document is sufficient to make LibreOffice silently
> send your gpg and ssh private keys to whatever server the author
> of that document wants - no dialog, no warnings, you won't notice.
>
> If Debian would provide LibreOffice without security support,
> how many of our users would have been aware of this problem?

I don't know what's your point. You acknowledge we already ship not
security supported software. We could put a large dialog box when
installing/upgrading LibreOffice, like for other not security supported
software. Or we could put those software in a special repository (called
"unsupported"), a bit like backports that are not security supported
either.

Ubuntu does that since many years (everything in universe/multiverse is
unsupported) and nobody cares.
-- 
Use variable names that mean something.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Michael Meskes]

> The software might integrate properly into Debian - and allow
> everyone 
> on the internet to take control of your computer.

Which is of course independent of the way you install the software.

> An example what "no security support" means in practice:

I don't think anyone suggest "no security", but something like
"security by upstream releases".

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

Re: What can Debian do to provide complex applications to its users?

[Michael Meskes]

> What is the user supposed to do when Debian announces that some 
> software essential for that user is no longer supported in the
> stable release the user is using?

Again, where does this differ from the user realizing their own self-
baked installation cannot be upgraded anymore?

> At that point the options for the user are either to continue using 
> software that might already have known grave vulnerabilities, or to
> do a rushed attempt trying to find some way to self-install an
> upstream
> version that is still supported.

And why wouldn't we offer said upstream version instead of the
unsupported older one?

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

Re: What can Debian do to provide complex applications to its users?

[Michael Meskes]

> > Let's agree to disagree. I find it perfectly fine if we told people
> > up
> > front that we support it as long as upstream has a version that
> > works
> > with the stable base. They are still better or at least not worse
> > of
> > with that than with a self-installed one.
> 
> Why? With the self-installed one they know up front that they need
> to 
> set up some kind of infrastructure to maintain it and can make an 
> informed decision about whether they want to take that on. How is it 
> better to think you have a debian supported install but in fact have
> to 
> come up with the very infrastructure you avoided on an emergency
> basis 
> at some point in the future?

I don't understand how this connects to what I, and others, were
saying. Nobody ever suggested to leave users in the dark, we talked
about documenting very clearly what they get and what they don't.
Besides my personal experience is that most people do not set up the
infrastructure you mentioned so they get caught unaware no matter what.

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Mon, Feb 19, 2018 at 09:18:13AM +0100, Philipp Kern wrote:
> On 2018-02-18 22:53, Adrian Bunk wrote:
> > In the year 2018, any kind of "properly maintain" includes security
> > support.
> > 
> > Please elaborate how Debian can provide security support for packages
> > like gitlab and all their dependencies in buster until mid-2022.
> > 
> > If Debian cannot provide security support for the lifetime of a stable
> > Debian release, it is better for our users when they are installing the
> > software from upstream with the security support provided by upstream.
> 
> Putting security support over all else is surely how some people see it. But
> some upstreams also complain if you are going to ship ancient versions
> because the most recent ones contain all of the fixes. It's certainly more
> work to validate security fixes when backporting them to older versions. So
> it's also the "stable" guarantee (whatever it is seen as) that might need
> some re-adjustment.

Every change that gets published as DSA or part of a stable update gets 
automatically installed on millions of machines.

> One of the values is that you get some set of software that works together
> as a base and doesn't change, but then people install software on top of it
> that provides their service and if it's actually the thing they want to
> provide it's most likely not packaged anymore at this point. Because you'd
> want the latest features of the product you're using.

Sometimes that is true and sometimes it is not.
And what are the latest features today will likely be included in the 
version in the next Debian stable.

The main question is what Debian can offer throughout the distribution.

Installing some or few (open source or proprietary) products on top of 
the distribution is often required for various reasons.
If Debian ships an older version of one of these products that is not
a problem.

What is a problem is if such a 3rd party product suddenly breaks due
to an update in the stable distribution, or becomes vulnerable due to
unfixed CVEs in the stable distribution.

> So there's already a
> disconnect of essentially two tracks: the system's base at a solid version
> and whatever it is you want to offer at a fast moving pace. That's also a
> reality in 2018. And coming up with arbitrary deadlines of support are not
> all that helpful. Users don't care if the ancient version of the software
> they need in stable is security supported until mid-2022. If it doesn't
> satisfy their requirements anymore, they move to testing or to another
> distribution.

Let's make a real-life example:
salsa.debian.org and gitlab.

salsa currently runs a manually installed gitlab.
At some point after the release of buster salsa is expected to be
upgraded to buster.

If buster ships a gitlab package with no security support from Debian, 
would you recommend that Debian uses this package on salsa until 
bullseye gets released?

If buster ships a gitlab package that is supported by Debian by every 
once in a while upgrading to the latest upstream gitlab, would you 
recommend that Debian uses this package on salsa instead of continuing
to use a manually installed gitlab?

Even if gitlab would have been part of stretch it is clear that salsa 
wouldn't use the version in stretch. The "ancient version of the 
software" in buster will likely be recent enough for salsa, but for
such a central and exposed service security support and no regressions
are also important.

> Kind regards
> Philipp Kern

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Sun, Feb 18, 2018 at 11:47:52PM +0100, Vincent Bernat wrote:
>  ❦ 18 février 2018 23:53 +0200, Adrian Bunk  :
> 
> >> Who said we cannot properly maintain this stuff? And where do you
> >> think our expected level of quality (whatever that is) will not be
> >> reached?
> >
> > In the year 2018, any kind of "properly maintain" includes security support.
> >
> > Please elaborate how Debian can provide security support for packages 
> > like gitlab and all their dependencies in buster until mid-2022.
> >
> > If Debian cannot provide security support for the lifetime of a stable 
> > Debian release, it is better for our users when they are installing the
> > software from upstream with the security support provided by upstream.
> 
> Debian is not only about security support. We provide packages without
> security support. We also have backports that come without security
> support either. This is still better than installing random packages
> made by random people which may or may not integrate properly into
> Debian.

The software might integrate properly into Debian - and allow everyone 
on the internet to take control of your computer.

On the server side we are talking about software like Node.js or gitlab.

On the desktop side the MUA that comes with our default desktop renders
HTML emails using a web engine that is not security supported by Debian.

An example what "no security support" means in practice:

If you are running Debian stable and haven't yet installed the
LibreOffice security updates Debian published a few days ago,
then opening a document is sufficient to make LibreOffice silently
send your gpg and ssh private keys to whatever server the author
of that document wants - no dialog, no warnings, you won't notice.

If Debian would provide LibreOffice without security support,
how many of our users would have been aware of this problem?

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Mon, Feb 19, 2018 at 07:03:04PM +0100, Michael Meskes wrote:
> > Because eventually a future version will come out that doesn't work
> > with 
> > the stable base, at which point we suddenly stop supporting the
> > package. 
> > That's much worse than just admitting up front that we can't support
> > the 
> > package for the next 4 years.
> 
> Let's agree to disagree. I find it perfectly fine if we told people up
> front that we support it as long as upstream has a version that works
> with the stable base. They are still better or at least not worse of
> with that than with a self-installed one.

They are not.

Ideally Debian should provide and support the software, but support that 
might suddenly go away without any clear option forward is the worst
option Debian could offer.

What is the user supposed to do when Debian announces that some 
software essential for that user is no longer supported in the
stable release the user is using?

At that point the options for the user are either to continue using 
software that might already have known grave vulnerabilities, or to
do a rushed attempt trying to find some way to self-install an upstream
version that is still supported.

This is clearly worse than using a self-installed version from the 
start, which gives the user the knowledge how to use the self-installed 
version (that might differ from how Debian provides the software) and 
gives a clear picture from the start when upgrades to new major versions 
will have to be planned.

> Michael

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Fri, Feb 16, 2018 at 08:18:13PM +0100, Samuel Thibault wrote:
> W. Martin Borgert, on ven. 16 févr. 2018 18:59:21 +0100, wrote:
>...
> > This is very much a web application problem. Other software is
> > less affected in my experience.
> 
> Sure. But the current world is more and more focused on web
> applications.

That's not a good justification for doing something if it doesn't fit 
into Debian.

> If we disconnect from the current world, we'll have a hard
> time attracting new people to maintain Debian on the long run.
>...

In the current world Debian[1] has an 8 digit userbase on the
Raspberry Pi.

These are existing Debian users, and the Raspberry Pi ecosystem
is culturally much closer to Debian than the JS universe.

If you (or anyone else) wants to spend effort on attracting new people 
to maintain Debian, the Raspberry Pi ecosystem would be the more
logical choice here.

> Samuel

cu
Adrian

[1] and the Debian derivative Raspbian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Michael Stone]

On Mon, Feb 19, 2018 at 07:03:04PM +0100, Michael Meskes wrote:

Because eventually a future version will come out that doesn't work
with
the stable base, at which point we suddenly stop supporting the
package.
That's much worse than just admitting up front that we can't support
the
package for the next 4 years.


Let's agree to disagree. I find it perfectly fine if we told people up
front that we support it as long as upstream has a version that works
with the stable base. They are still better or at least not worse of
with that than with a self-installed one.


Why? With the self-installed one they know up front that they need to 
set up some kind of infrastructure to maintain it and can make an 
informed decision about whether they want to take that on. How is it 
better to think you have a debian supported install but in fact have to 
come up with the very infrastructure you avoided on an emergency basis 
at some point in the future?


Mike Stone

Re: What can Debian do to provide complex applications to its users?

[Michael Meskes]

> Because eventually a future version will come out that doesn't work
> with 
> the stable base, at which point we suddenly stop supporting the
> package. 
> That's much worse than just admitting up front that we can't support
> the 
> package for the next 4 years.

Let's agree to disagree. I find it perfectly fine if we told people up
front that we support it as long as upstream has a version that works
with the stable base. They are still better or at least not worse of
with that than with a self-installed one.

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

Re: What can Debian do to provide complex applications to its users?

[Pirate Praveen]

On വെള്ളി 16 ഫെബ്രുവരി 2018 08:41 വൈകു, Raphael Hertzog wrote:
> - while gitlab is packaged in Debian, its packaging took years and the
>   result is brittle because it can break in many ways whenever one the 
>   dozens of dependencies gets updated to some new upstream version
>   (BTW salsa.debian.org does not use the official package)

I maintain complex packages like gitlab and diaspora and I agree about
the amount of work required. But the reason why library updates break
can be improved by two steps.

1. Enable functionality tests for all dependencies (we already have
pretty good test coverage and that should be increased).
2. Find out about possible breakages before the upload.
This can be achieved by using scripts like build-and-upload which runs
autopkgtests of all reverse dependencies and rebuilds all reverse
dependencies. Unfortunately use of this script is not wide spread yet
and we need to include it in a commonly used package like devscripts.
Many if the recent breakages could have been avoided if the uploaders
used this option.
3. When introducing a breaking change, we need to help upstreams to move
to the new version along with us. This is not different from how we
handle transitions, but the number of transitions will increase.

> I'm sure we are missing lots of good applications due to our requirements.
> What can we do to avoid this?

I think the situation regarding javascript have improved largely in last
two years. It took me more than 2 years to get handlebars in debian
because none of the build tools were packaged. Now with many common
build tools like grunt, gulp, webpack, rollup, babel etc packaged, the
effort required to package a new library has significantly reduced (a
complex library like react took only a fraction of time required for
handlebars). I believe this will encourage more people to start
maintaining javascript libraries as the barrier of entry has been
reduced significantly. Case before two years was reverse engineering the
build system using tools like sed, which is not the case now.

As for the number of packages to maintain is very large, I realized it
would be hard for me to maintain these long dependencies alone, so I
have been aggressively seeking and mentoring more people to maintain
packages. I have organized countless offline and online packaging
sessions. I have reasonable success (compared to the effort I put in)
and hope to continue those efforts.

We not only have to get more people to package, but since upstream is
not listening to us, we need to get more people to help with fixing
upstream compatibility issues too.

I have got some very good help from people like Shanavas, Jishnu, Aruna
and Harish to make upstream compatible with the library versions we have
in debian (moving handlebars to babel 6 and webpack 3, moving many
libraries to chai 4, adding global library support to grunt, etc to name
a few). They have been helping with node specific issue and got upstream
to accept pull requests. In other pending cases, even though they don't
help us, they are willing to take pull requests.

Also I have decided to package a module only if a at least two packages
depend on it, which will speed up the process and also reduce the load
on ftp masters (please help ftp masters review the long queue if you can).

See
https://salsa.debian.org/js-team/node-ava/tree/master/debian/node_modules

(some of those currently embedded there will need to be packaged
separately because they are generated files. But if a module is pure es5
and required only for one module it could be embedded. Also in newer
nodejs versions we may be able to do this with es6 modules too).



signature.asc
Description: OpenPGP digital signature

Re: What can Debian do to provide complex applications to its users?

[Samuel Thibault]

Raphael Hertzog, on lun. 19 févr. 2018 15:52:14 +0100, wrote:
> On Mon, 19 Feb 2018, Samuel Thibault wrote:
> > But what if that upstream website goes down? We don't have the source
> > any more. Better at least keep a copy of the tarball.
> 
> Sure. But as a packager, I don't want to have to do this manually. So
> one possible idea might be to extend our copyright file format. We should
> be able to put there the URL for the sources of something that has been
> embedded in the application

Or simply some debian/rules lines, so that it's at the time of the
packaging? (like the common get-orig rule)

Samuel

Re: What can Debian do to provide complex applications to its users?

[Holger Levsen]

On Mon, Feb 19, 2018 at 09:52:57AM -0500, Michael Stone wrote:
> I'd argue that what people should stop being afraid of is just using a third
> party package if that's the optimal solution.

+1


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Michael Stone]

On Mon, Feb 19, 2018 at 02:51:31PM +0100, Raphael Hertzog wrote:

Our core value is here and we can still provide value to our users in
the new world that is emerging around us. We should just stop to be afraid
of it.


I'd argue that what people should stop being afraid of is just using a 
third party package if that's the optimal solution. 


Mike Stone

Re: What can Debian do to provide complex applications to its users?

[Raphael Hertzog]

On Mon, 19 Feb 2018, Samuel Thibault wrote:
> Raphael Hertzog, on lun. 19 févr. 2018 15:19:59 +0100, wrote:
> > On Fri, 16 Feb 2018, Jonathan Carter (highvoltage) wrote:
> > > > - we could relax our requirements and have a way to document the
> > > >   limitations of those packages (wrt our usual policies)
> > > 
> > > Which requirements are you referring to? If it's relaxing the need for
> > > source for minified javascript, then no thanks.
> > 
> > Instead of requiring the source to be provided in the source package as a
> > non-minified file, we could require the packager to document in
> > debian/README.source where the upstream sources actually are.
> 
> But what if that upstream website goes down? We don't have the source
> any more. Better at least keep a copy of the tarball.

Sure. But as a packager, I don't want to have to do this manually. So
one possible idea might be to extend our copyright file format. We should
be able to put there the URL for the sources of something that has been
embedded in the application and some debian.org service would ensure that
we keep a publicly-accessible copy of all those sources for as long as we
want them.

BTW we could also rely on our copyright file to document the fact that we
have vendored copies of some software.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: What can Debian do to provide complex applications to its users?

[Alastair McKinstry]

On 19/02/2018 14:28, Samuel Thibault wrote:
> Raphael Hertzog, on lun. 19 févr. 2018 15:19:59 +0100, wrote:
>> On Fri, 16 Feb 2018, Jonathan Carter (highvoltage) wrote:
 - we could relax our requirements and have a way to document the
   limitations of those packages (wrt our usual policies)
>>> Which requirements are you referring to? If it's relaxing the need for
>>> source for minified javascript, then no thanks.
>> Instead of requiring the source to be provided in the source package as a
>> non-minified file, we could require the packager to document in
>> debian/README.source where the upstream sources actually are.
> But what if that upstream website goes down? We don't have the source
> any more. Better at least keep a copy of the tarball.

I second this, for multiple reasons. If we go to a 'container capturing
a non-Debian build' approach we should always capture the sources
involved - for security tracking at least. e.g. Maven for Java seems to
be particularly bad at pulling JARs and other components randomly, often
over HTTP not HTTPS and without signatures, etc.  Gradle even does  which is scary.

One step towards a fully proper Debian build and packaging is an 
infrastructure to at least capture all the sources involved in
containers and track them.

> Samuel

Alastair

-- 
Alastair McKinstry, , , 
https://diaspora.sceal.ie/u/amckinstry
Commander Vimes didn’t like the phrase “The innocent have nothing to fear,”
 believing the innocent had everything to fear, mostly from the guilty but in 
the longer term
 even more from those who say things like “The innocent have nothing to fear.”
 - T. Pratchett, Snuff

Re: What can Debian do to provide complex applications to its users?

[Adam Borowski]

On Mon, Feb 19, 2018 at 03:19:59PM +0100, Raphael Hertzog wrote:
> Instead of requiring the source to be provided in the source package as a
> non-minified file, we could require the packager to document in
> debian/README.source where the upstream sources actually are.

Ie, it'd be fine to ship pre-built C libraries as long as some non-buildable
sources were once placed at website X?


-- 
An imaginary friend squared is a real enemy.

Re: What can Debian do to provide complex applications to its users?

[Raphael Hertzog]

On Mon, 19 Feb 2018, Paul Wise wrote:
> On Mon, Feb 19, 2018 at 9:51 PM, Raphael Hertzog wrote:
> 
> > I don't want to lower the quality of what we have built so far, so while
> > it's technically possible to build .deb and include a bundle of libraries
> > pinned at the correct version, I don't think that this should allowed into
> > the main archive.
> 
> The "not allowing embedded code copies in the main archive" ship has sailed:
> https://salsa.debian.org/security-tracker-team/security-tracker/raw/master/data/embedded-code-copies
> https://wiki.debian.org/EmbeddedCodeCopies

I know this but at least you have the sources in the source package. I was
referring to things like what I did in Kali for metasploit, it's a ruby
application. The .deb ships the metasploit code (available in the source
package) but also all the ruby libraries in a directory created with
"bundle install --vendor" and this includes lots of stuff... plain ruby
code but also binary extensions compiled on other systems and made
available as ready-to-download gems.

Sources:
http://git.kali.org/gitweb/?p=packages/metasploit-framework.git;a=shortlog;h=refs/heads/kali/master
Binary packages:
https://http.kali.org/pool/main/m/metasploit-framework/

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: What can Debian do to provide complex applications to its users?

[Alastair McKinstry]

> I think Debian has never been afraid of tackling hard problems and we
> should find a third way.
>
> I don't want to lower the quality of what we have built so far, so while
> it's technically possible to build .deb and include a bundle of libraries
> pinned at the correct version, I don't think that this should allowed into
> the main archive.
>
> However I also think that Debian has to provide all those hard-to-package
> applications to end users. Right now, my gut feeling is that the best
> approach is probably to rely on containers built out of Debian
> binary packages for the plumbing and with the language-specific package
> management tool for the application libraries. The role of the Debian
> developer is then to maintain a recipe file that is used to build the
> container (and future updated versions) and to provide some integration
> with the host to export/import data out of the application (think
> backup/restore). Since Debian would start to provide many containers like
> those, we would likely also start to build infrastructure to manage those
> containers, including some way to identify security vulnerabilities
> present in the container and a lintian for containers. And we would draft
> a policy for how to manage an application in a container, etc.
If we do this, we need to security-track the application libraries in
language-specific tools (e.g. pip), what are we dropping in quality
/work compared to auto-packaging from the language? while we don't
typically want to package 10,000 python packages for the sake of it, if
we need pyfoo (1.2) and are tracking it for security anyway, perhaps we
should write script python auto-packaging scripts,etc.

(Presuming licenses are automatically managed - taking license info from
pip / github, etc).

> Our core value is here and we can still provide value to our users in
> the new world that is emerging around us. We should just stop to be afraid
> of it.



-- 
Alastair McKinstry, , , 
https://diaspora.sceal.ie/u/amckinstry
Commander Vimes didn’t like the phrase “The innocent have nothing to fear,”
 believing the innocent had everything to fear, mostly from the guilty but in 
the longer term
 even more from those who say things like “The innocent have nothing to fear.”
 - T. Pratchett, Snuff

Re: What can Debian do to provide complex applications to its users?

[Samuel Thibault]

Raphael Hertzog, on lun. 19 févr. 2018 15:19:59 +0100, wrote:
> On Fri, 16 Feb 2018, Jonathan Carter (highvoltage) wrote:
> > > - we could relax our requirements and have a way to document the
> > >   limitations of those packages (wrt our usual policies)
> > 
> > Which requirements are you referring to? If it's relaxing the need for
> > source for minified javascript, then no thanks.
> 
> Instead of requiring the source to be provided in the source package as a
> non-minified file, we could require the packager to document in
> debian/README.source where the upstream sources actually are.

But what if that upstream website goes down? We don't have the source
any more. Better at least keep a copy of the tarball.

Samuel

Re: What can Debian do to provide complex applications to its users?

[Raphael Hertzog]

On Fri, 16 Feb 2018, Jonathan Carter (highvoltage) wrote:
> > - we could relax our requirements and have a way to document the
> >   limitations of those packages (wrt our usual policies)
> 
> Which requirements are you referring to? If it's relaxing the need for
> source for minified javascript, then no thanks.

Instead of requiring the source to be provided in the source package as a
non-minified file, we could require the packager to document in
debian/README.source where the upstream sources actually are.

When I was maintaining wordpress, I introduced the idea of providing
debian/missing-sources/ to comply with the Debian policy. I would just dump
there the upstream tarball of the bundled libraries to be sure that we
have the source for the correct version. The Debian/ftpmaster rules are
respected but it's not really better than the above because you still
don't have a simple way to rebuild a modified version of the javascript
library shipped in the package.

So instead of ugly work-arounds, it might be better to just acknowledge
that we can't have the same level of support for all applications.

> > - we could ship those applications not as .deb but as container
> >   and let them have their own lifecycle
> 
> What would this solve and how will it solve it?

Those applications could rely on the package manager of their ecosystem to
setup the dependencies as they need them without polluting the host
system.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: What can Debian do to provide complex applications to its users?

[Paul Wise]

On Mon, Feb 19, 2018 at 9:51 PM, Raphael Hertzog wrote:

> I don't want to lower the quality of what we have built so far, so while
> it's technically possible to build .deb and include a bundle of libraries
> pinned at the correct version, I don't think that this should allowed into
> the main archive.

The "not allowing embedded code copies in the main archive" ship has sailed:

https://salsa.debian.org/security-tracker-team/security-tracker/raw/master/data/embedded-code-copies
https://wiki.debian.org/EmbeddedCodeCopies

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What can Debian do to provide complex applications to its users?

[Raphael Hertzog]

On Sat, 17 Feb 2018, Russ Allbery wrote:
> The reason why Debian in general doesn't like to support vendored source
> is because of the security implications: when there's a security
> vulnerability in one of the vendored libraries, updating the relevant
> packages becomes a nightmare.  It's a logistical challenge even if the
> vendored source can be safely upgraded, but of course it usually can't
> since that's the whole point of vendoring the source.  So we would be
> faced with backporting security fixes to every vendored version of the
> package, and we don't have the resources to do this.

We might not have the resources to do this but we should have the
infrastructure to track those problems, make them available to upstream,
and get them to fix their vendored libraries. And let users decide
whether it's safe to install or not, and track the security status over
time.

I don't agree that "vendored sources" cannot be safely upgraded. It really
depends on the reason why the source has been vendored. When it's a
deliberate fork, then yes the upgrade might be hard. But more often than
not, it's just a convenience to avoid system dependencies or a simple way
to ensure that the project has the version that they expect.

Furthermore many projects have continuous integration tools that let them
know whether things are still working after the update (be it because they
switched to latest upstream of all their dependencies, or because they
upgraded a library that they had vendored).

> It's hard to avoid the feeling that we have two choices with these sorts
> of applications:

I think Debian has never been afraid of tackling hard problems and we
should find a third way.

I don't want to lower the quality of what we have built so far, so while
it's technically possible to build .deb and include a bundle of libraries
pinned at the correct version, I don't think that this should allowed into
the main archive.

However I also think that Debian has to provide all those hard-to-package
applications to end users. Right now, my gut feeling is that the best
approach is probably to rely on containers built out of Debian
binary packages for the plumbing and with the language-specific package
management tool for the application libraries. The role of the Debian
developer is then to maintain a recipe file that is used to build the
container (and future updated versions) and to provide some integration
with the host to export/import data out of the application (think
backup/restore). Since Debian would start to provide many containers like
those, we would likely also start to build infrastructure to manage those
containers, including some way to identify security vulnerabilities
present in the container and a lintian for containers. And we would draft
a policy for how to manage an application in a container, etc.

Our core value is here and we can still provide value to our users in
the new world that is emerging around us. We should just stop to be afraid
of it.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: What can Debian do to provide complex applications to its users?

[Steve McIntyre]

Holger wrote:
>On Sat, Feb 17, 2018 at 11:14:51PM +, Colin Watson wrote:
>>  * Constrained to the sort of server-side applications that might
>>reasonably be run in a container on their own, just to keep the
>>problem size down a bit.
>
>why this contraint, there are more and more desktop application written in 
>node...

Oh dear $deity, why?

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"Every time you use Tcl, God kills a kitten." -- Malcolm Ray

Re: What can Debian do to provide complex applications to its users?

[Raphael Hertzog]

On Fri, 16 Feb 2018, W. Martin Borgert wrote:
> Is was a relevant part of the problem mentioned in Raphaels bug
> report: Minified JS libraries without source code. this was one
> of the starting points of this discussion. (#890598)

It's not "without source code", it's just that the source code of a single
.min.js is not another non-minified javascript file but a whole git
repository that you can't just drop near the .min.js file to please lintian.

And building ourselves the non-minified file together with the corresponding
minified file is just too time-consuming on the long term when you rely on
dozens of libraries.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: What can Debian do to provide complex applications to its users?

[Michael Stone]

On Mon, Feb 19, 2018 at 10:21:18AM +0100, Michael Meskes wrote:

Maybe you answered your question yourself. How about we tie our
security support to upstream's? Instead of fixing and backporting
ourselves we promise our users that this section of the archive will
get upstream's latest fixes even if that means the version number
changes.


Because eventually a future version will come out that doesn't work with 
the stable base, at which point we suddenly stop supporting the package. 
That's much worse than just admitting up front that we can't support the 
package for the next 4 years.


Mike Stone

Re: What can Debian do to provide complex applications to its users?

[Michael Meskes]

> > Who said we cannot properly maintain this stuff? And where do you
> > think our expected level of quality (whatever that is) will not be
> > reached?
> 
> In the year 2018, any kind of "properly maintain" includes security
> support.

Indeed it does, but not necessarily the way we handle it now.

> Please elaborate how Debian can provide security support for
> packages 
> like gitlab and all their dependencies in buster until mid-2022.
> 
> If Debian cannot provide security support for the lifetime of a
> stable 
> Debian release, it is better for our users when they are installing
> the
> software from upstream with the security support provided by
> upstream.

Maybe you answered your question yourself. How about we tie our
security support to upstream's? Instead of fixing and backporting
ourselves we promise our users that this section of the archive will
get upstream's latest fixes even if that means the version number
changes.

This way the users would get a lot of benefits from using Debian but no
 drawback compared to the self-installed alternative.

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

Re: What can Debian do to provide complex applications to its users?

[Craig Small]

On Sat, 17 Feb 2018 at 02:11 Raphael Hertzog  wrote:

> I'm sure we are missing lots of good applications due to our requirements.
> What can we do to avoid this?
>
[...]

> What do you think? Do you have other ideas? Are there other persons
> who are annoyed by the current situation?
>
I'd like to start this with, I don't have an answer. However it is annoying!

I have wordpress that ships some javascript libraries purely because the
ones in Debian are ancient or not packaged.  Ideally I'd like to use to
just depend on some other packages but the problem is, is the javascript
library shipped by wordpress the same as the upstream?  Certainly
dh_linktree (thanks Raphael!) help here, but its a problem I don't want to
encounter.

Then I have another program. It's written in C++ so most of that side of
things is fine, but it uses Lua plugins and this is where the drama starts.
I just ship with the embedded ones, because version control of the
(hypothetical) lua library packages and syncing it with whatever arbitary
version of the library the program needs is hard.

Finally, for programs I write and use myself, if its a C program I'm pretty
sure I'll find what I need for libraries, but in other languages not so
much.

In the ancient days, we used to have some of these problems with binaries
and shared libraries. Then newer, for their time, distributions such as
Debian came in with versioned dependencies between the binary and the
library. That didn't help when upstream used their own hacky un-maintained
for years version of some library but it caught most of the cases.

So, perhaps there needs to be a new way for some of these newer packaging
methods. I don't think we can say package all the things, even Debian has
its limits.

 - Craig
-- 
Craig Small https://dropbear.xyz/ csmall at : dropbear.xyz
Debian GNU/Linuxhttps://www.debian.org/   csmall at : debian.org
Mastodon: @smalls...@social.dropbear.xyz Twitter: @smallsees
GPG fingerprint:  5D2F B320 B825 D939 04D2  0519 3938 F96B DF50 FEA5

Re: What can Debian do to provide complex applications to its users?

[Philipp Kern]

On 2018-02-18 22:53, Adrian Bunk wrote:
In the year 2018, any kind of "properly maintain" includes security 
support.


Please elaborate how Debian can provide security support for packages
like gitlab and all their dependencies in buster until mid-2022.

If Debian cannot provide security support for the lifetime of a stable
Debian release, it is better for our users when they are installing the
software from upstream with the security support provided by upstream.


Putting security support over all else is surely how some people see it. 
But some upstreams also complain if you are going to ship ancient 
versions because the most recent ones contain all of the fixes. It's 
certainly more work to validate security fixes when backporting them to 
older versions. So it's also the "stable" guarantee (whatever it is seen 
as) that might need some re-adjustment.


One of the values is that you get some set of software that works 
together as a base and doesn't change, but then people install software 
on top of it that provides their service and if it's actually the thing 
they want to provide it's most likely not packaged anymore at this 
point. Because you'd want the latest features of the product you're 
using. So there's already a disconnect of essentially two tracks: the 
system's base at a solid version and whatever it is you want to offer at 
a fast moving pace. That's also a reality in 2018. And coming up with 
arbitrary deadlines of support are not all that helpful. Users don't 
care if the ancient version of the software they need in stable is 
security supported until mid-2022. If it doesn't satisfy their 
requirements anymore, they move to testing or to another distribution.


Kind regards
Philipp Kern

Re: What can Debian do to provide complex applications to its users?

[Robert Collins]

On 18 February 2018 at 12:14, Colin Watson  wrote:
...
>  * Maybe truncate the frozen dependency tree at C extensions, in order
>that we can make sure those are built for all architectures, so you'd
>still have to care about compatibility with those.  It'd be a much
>smaller problem, though.

One case where this truncation would be a challenge is the Numpy and
scipy ecosystem, where the extensions are tightly coupled to blas
library used and the numpy ABI - because extensions built on top of
numpy depend on that ABI. pip has had difficulty shipping binaries of
these stacks because it doesn't represent the ABI - it assumed that
anything with a matching version number was compatible, which isn't
the case. Arguably numpy should do something different there, but it
doesn't :)

>  * Every frozen dependency must be installed in a particular defined
>path based on (application name, dependency name, dependency
>version), and must be installed using blessed tools that generate
>appropriate packaging metadata that can be scanned centrally without
>having to download lots of source packages.
>
>  * Updating a single frozen dependency (assuming no API changes to the
>application's source code) must be no harder than updating a single
>line in a single file and bumping the Debian package version.
>
>  * Where applicable, deprecation warnings must be configured as
>warnings, not errors, to maximise compatibility.

Perahps its worth adding 'deprecations should be enabled if off by
default - at least during build-time' - I'm thinking of Python where
we (upstream) disabled deprecation warnings and many folk get
surprised as a result.

>  * Dependencies with a non-trivial CVE history would have an acceptable
>range of versions, so we could drop applications that persistently
>cause trouble while still allowing a bit of slack.  For some
>dependencies it might not be a problem at all; e.g. we probably don't
>care what version of python-testtools something uses.

:P

-Rob

Re: What can Debian do to provide complex applications to its users?

[Vincent Bernat]

 ❦ 18 février 2018 23:53 +0200, Adrian Bunk  :

>> Who said we cannot properly maintain this stuff? And where do you
>> think our expected level of quality (whatever that is) will not be
>> reached?
>
> In the year 2018, any kind of "properly maintain" includes security support.
>
> Please elaborate how Debian can provide security support for packages 
> like gitlab and all their dependencies in buster until mid-2022.
>
> If Debian cannot provide security support for the lifetime of a stable 
> Debian release, it is better for our users when they are installing the
> software from upstream with the security support provided by upstream.

Debian is not only about security support. We provide packages without
security support. We also have backports that come without security
support either. This is still better than installing random packages
made by random people which may or may not integrate properly into
Debian.
-- 
Watch out for off-by-one errors.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Adrian Bunk]

On Fri, Feb 16, 2018 at 06:12:04PM +0100, Michael Meskes wrote:
> On Fri, Feb 16, 2018 at 11:12:51AM -0500, Michael Stone wrote:
> > On Fri, Feb 16, 2018 at 04:58:04PM +0100, Michael Meskes wrote:
> > > I know that this does create some problems for us, e.g. on the security
> > > side, but the alternative is, as you mentioned, manually installed
> > > software and that surely is worse.
> > 
> > No, I think it's better if people know they're on their own for maintaining
> > something. What's surely worse is when we ship stuff that we know we can't
> > properly maintain in the long term. Better to be out of the archive than in
> > without the expected level of quality.
> 
> Who said we cannot properly maintain this stuff? And where do you think our 
> expected level of quality (whatever that is) will not be reached?

In the year 2018, any kind of "properly maintain" includes security support.

Please elaborate how Debian can provide security support for packages 
like gitlab and all their dependencies in buster until mid-2022.

If Debian cannot provide security support for the lifetime of a stable 
Debian release, it is better for our users when they are installing the
software from upstream with the security support provided by upstream.

> Michael

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: What can Debian do to provide complex applications to its users?

[Colin Watson]

On Sun, Feb 18, 2018 at 12:48:49PM +, Holger Levsen wrote:
> On Sat, Feb 17, 2018 at 11:14:51PM +, Colin Watson wrote:
> >  * Constrained to the sort of server-side applications that might
> >reasonably be run in a container on their own, just to keep the
> >problem size down a bit.
> 
> why this contraint, there are more and more desktop application written in 
> node...

I did say it was a strawman.  I picked it mostly because trying to solve
every problem at once seems like a recipe for failure, though; better to
start small and expand once it's working.

-- 
Colin Watson   [cjwat...@debian.org]

Re: What can Debian do to provide complex applications to its users?

[Holger Levsen]

On Sat, Feb 17, 2018 at 11:14:51PM +, Colin Watson wrote:
>  * Constrained to the sort of server-side applications that might
>reasonably be run in a container on their own, just to keep the
>problem size down a bit.

why this contraint, there are more and more desktop application written in 
node...


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Michael Meskes]

> Michael Meskes dijo [Sat, Feb 17, 2018 at 01:57:53PM +0100]:
> > I disagree, it is not maintainable source code, yes, but source
> > code
> > nonetheless. According to wikipedia source code is:
> > ...
> 
> Some others have answered to this claim. As I understand it, source
> code should ideally be the author's preferred form of
> modification. That is clearly different from what a minified
> representation is.
> ...

It's amazing how my email seems to have been understood differs from
what I wanted to say. I thought it was clear from the "not
maintainable" part that I did not suggest we use minified source as
source code. I was merely pointing out that it technically still was
source code in the sense that the same compiler/interpreter yields the
same result with the minified code as with the original one.

For all practical manners this is irrelevant.


> > I get your point, I just don't accept the consequence that we
> > should
> > not package these applications.
> 
> Well, try to find somebody with time and motivation to _properly_ do
> the packaging, and to keep it up for several years...

I absolutely agree with you that it does not work with our current way
of packaging, but that's what this discussion about. IMO it would be
great if we found a way to offer these applications to our users.

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

signature.asc
Description: This is a digitally signed message part

Re: What can Debian do to provide complex applications to its users?

[Paul Wise]

On Sun, Feb 18, 2018 at 2:59 AM, Thorsten Alteholz wrote:

> Other javascript libraries like libjs-* and *.js even don't get a CVE. So
> either they are secure or nobody cares.

We also miss out on some JS vulnerabilities because NodeSecurity don't
systematically participate in the CVE system and we don't
systematically import their issues.

https://nodesecurity.io/advisories

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What can Debian do to provide complex applications to its users?

[Paul Wise]

On Sat, Feb 17, 2018 at 9:20 PM, Paul Wise wrote:

> It may be code but it is definitely not source in the sense of DFSG
> item 2 or the GPL.

Also, non-minified JavaScript can also not be source code, for example
in the case it was generated from CoffeeScript or some other language.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What can Debian do to provide complex applications to its users?

[Colin Watson]

On Sat, Feb 17, 2018 at 07:22:05PM +0100, Tollef Fog Heen wrote:
> I think there's at least two types of vendoring you're referring to
> here, and they're substantially different.
> 
> One is how Go currently does (but my understanding is that this is
> changing in newer versions).  Here, the source code of dependencies are
> shipped together with the source code for the application.  This leads
> to trees like
> https://github.com/kubernetes/kubernetes/tree/master/vendor where any
> one of those dependencies might be a released version or tag, or it
> might just be a random git snapshot, and there's not really any way to
> know.
> 
> The other (you refer to this as freezing dependencies) is how
> Node.js/npm/yarn, Ruby/gem, (to some extent) Python/pip, and Rust/cargo
> does it.  In those cases, you have some file specifying the versions of
> libraries the application needs, usually as «this version of this
> gem/crate/package» and there is somewhere those packages live by
> default.  Quite often, there's also a lock file of some sort which lists
> out the exact versions used, recursively, which ensures you can deploy
> the exact same code multiple times.
> 
> The second method means you can reason about what versions of code are
> included where.

This is an excellent point.  (In fact, I think you can do some of that
reasoning for the first method too; it's just going to take more effort.
For example, you could run "govendor sync" or whatever and check that
the output tree actually matches what's promised.  In some ways this is
similar to the question of "is this package's .orig.tar actually the
version it claims to be?".)

It's worth remembering that this isn't just some lazy upstreams not
getting round to keeping their dependencies up to date:

 * Often they need newer versions of dependencies than are in Debian
   stable, because retaining compatibility back to whatever version we
   shipped for a half-million-line web application and actually testing
   the result is a combinatorially-hard problem.

 * Often the existence of packaged web applications in Debian itself
   makes it harder to upgrade the libraries they depend on.  Take
   Django, for instance: it has really remarkably good documentation of
   what you need to do to upgrade an application to a newer version and
   a very clear deprecation policy, but you still have to do it and
   keeping your application compatible with more than two or three
   Django versions (by which I mean 1.8, 1.9, 1.10, etc.) before the
   newest one you support is likely to be pretty painful.  Now package
   half a dozen Django applications and suddenly the library package
   maintainers have a heck of a time keeping everything working. [1]

 * And yes, sometimes an upstream can end up with out-of-date
   dependencies that are going to take significant effort to resolve,
   but there's something to be said for making it possible to shift the
   upgrade effort to a time when it's convenient to do so, even though
   there's obviously a trade-off with the possibility that you might end
   up putting it off indefinitely.  Not to mention that if they did
   upgrade their dependencies, they might end up being incompatible with
   the versions in Debian stable, and as it is that doesn't help our
   users of packaged applications very much either.  The lockstep
   problem is really nasty.

 * Packaging individual libraries that aren't especially useful on the
   client side can be pretty boring work and is often not very socially
   valued (the people trying to deal with JavaScript dependencies in
   Debian have taken a lot of flak, for instance).

And yet ... it can be really nice to have some of these things packaged
in a common format.  I know roughly nothing about PHP, and I don't want
to have to learn yet another packaging stack or another upstream's
favourite container technology, so it's great that Icinga Web 2 is
packaged so that I can just install it without having to learn that.  I
want it to be as easy as possible for the security team and the people
who care about it to keep it up to date.  For the case where something
depends on OpenSSL, it's mostly pretty obvious where the trade-offs lie.
For the case where something depends on 80 npm libraries and a security
update to the application might require updating 10 of those and maybe
breaking some other packaged application in the process, it's not so
clear that it even serves our developers let alone our users.

[1] Disclaimer: I only do in-house development at work of Django
applications that aren't packaged in Debian, and I'm not involved in
Django's Debian maintenance, so this is an outside view and I may
have some details wrong.

> You might still have to patch five versions of libvulnerable, but at
> least it's possible to find which five versions are in use, and get
> those fixed in a central place and then rebuild everything that's
> vulnerable, recursively.  Not the most fun job in the 

Re: What can Debian do to provide complex applications to its users?

[Andreas Tille]

On Sat, Feb 17, 2018 at 10:32:09AM -0700, Sean Whitton wrote:
> > So if you are claiming we have no manpower that's pretty much our own
> > fault since we do not actively care.  There are ways to attract gifted
> > and interested people if we would only try.
> 
> I was making a more specific claim -- we don't and will never have the
> manpower to provide security support for multiple different versions of
> hundreds of little JavaScript libraries.

I agree that guessing from past experience chances are good that your
statement is true.  But my claim was that we do not even try to find the
according manpower (for several urgent jobs that need to be done).
 
> I don't doubt that we could have far more volunteers for other work if
> we made it easier to get involved.

So it would be good to try. :-)

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: What can Debian do to provide complex applications to its users?

[Colin Watson]

On Sat, Feb 17, 2018 at 08:42:44PM +0100, Adam Borowski wrote:
> Binary code that has debug symbols isn't that bad.  I don't think you can
> have those for minified JS.

JavaScript source maps have been around for a few years and AIUI are
pretty much exactly that.

-- 
Colin Watson   [cjwat...@debian.org]

Re: What can Debian do to provide complex applications to its users?

[Gunnar Wolf]

Michael Meskes dijo [Sat, Feb 17, 2018 at 01:57:53PM +0100]:
> > Minification is quite comparable to compilation. I will give you some
> > examples from my frustration with Drupal8 in this answer. This can no
> > longer be seen as source code:
> > ...
> 
> I disagree, it is not maintainable source code, yes, but source code
> nonetheless. According to wikipedia source code is:
> 
> In computing, source code is any collection of computer instructions,
> possibly with comments, written using[1] a human-readable programming
> language, usually as plain text.
> 
> I guess minified source code does qualify. However, this discussion is
> mood since the bigger lies in the modules that get included without any
> real documentation.

Some others have answered to this claim. As I understand it, source
code should ideally be the author's preferred form of
modification. That is clearly different from what a minified
representation is.

Even adjusting this a bit... Maybe not a preferred form of
modification, but a plausible form for performing support? Well, even
leaving the obvious change from a readable, indented set of
instructions to a single line with no comments nor anything to aid
humans to understand it, it also does not cut the bill. Minification
preserves function (and thus, calls to the stdlib and such are
discernible), but function and variable names are mangled. That is a
_huge_ obstacle for being able to fix anything but the most trivial
details. Of course, we cannot push our patches upstream.

> > But packaging the precise version that is required in each little
> > bump
> > is just impossible.
> 
> I get your point, I just don't accept the consequence that we should
> not package these applications.

Well, try to find somebody with time and motivation to _properly_ do
the packaging, and to keep it up for several years...


signature.asc
Description: PGP signature

Re: What can Debian do to provide complex applications to its users?

[Adam Borowski]

On Sat, Feb 17, 2018 at 01:24:30PM -0500, Sergio Durigan Junior wrote:
> On Saturday, February 17 2018, Michael Meskes wrote:
> > I disagree, it is not maintainable source code, yes, but source code
> > nonetheless. According to wikipedia source code is:
> >
> > In computing, source code is any collection of computer instructions,
> > possibly with comments, written using[1] a human-readable programming
> > language, usually as plain text.
> 
> You can disagree, but you still can't edit and maintain the minified JS.
> This is basically what is expected from a source code (and it doesn't
> really matter whether Wikipedia's definition is different).  If you've
> had to deal with minified JS, then you know it's the same as dealing
> with binary code.

Binary code that has debug symbols isn't that bad.  I don't think you can
have those for minified JS.

(I can read adequately only x86 assembly, but reading other archs isn't
a rare skill.)


-- 
⢀⣴⠾⠻⢶⣦⠀ The bill with 3 years prison for mentioning Polish concentration
⣾⠁⢰⠒⠀⣿⡁ camps is back.  What about KL Warschau (operating until 1956)?
⢿⡄⠘⠷⠚⠋⠀ Zgoda?  Łambinowice?  Most ex-German KLs?  If those were "soviet
⠈⠳⣄ puppets", Bereza Kartuska?  Sikorski's camps in UK (thanks Brits!)?