subject:"Re\: Multiarch file overlap summary and proposal"

On Wed, 2012-02-15 at 16:41:21 +, Ian Jackson wrote:
 Guillem Jover writes (Re: Multiarch file overlap summary and proposal (was: 
 Summary: dpkg shared / reference counted files and version match)):
   [...]  But trying to workaround this by coming
  up with stacks of hacked up solutions  [...]
 
 I disagree with your tendentious phrasing.  The refcnt feature is not
 a hacked up solution (nor a stack of them).  It is entirely normal
 in Debian core tools (as in any substantial piece of software serving
 a lot of diverse needs) to have extra code to make it easier to deploy
 or use in common cases simpler.

All along this thread, when referring to the additional complexity and
the additional hacks, I've not been talking about the refcnt'ing at
all, but to all the other fixes needed to make it a workable solution.

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120229195152.ga4...@gaara.hadrons.org

Re: Multiarch file overlap summary and proposal

On Wed, 2012-02-15 at 19:31:10 -0800, Russ Allbery wrote:
 I agree that it's asymmetric.  apt-get install libfoo means libfoo:native,
 but apt-get remove libfoo means libfoo:*.  And asymmetric is bad, all
 things being equal.  But I think this may be one place where asymmetric is
 still the right thing to do; I would argue that it means you're
 implementing the most common operation in both cases.  apt-get install
 libfoo generally means give me a native libfoo since non-native libfoo
 is going to be an unusual case, and apt-get remove libfoo generally means
 I have no more interest in libfoo, make it go away.  I think that people
 who want to get rid of one architecture of libfoo but keep the other are
 already going to be thinking about architectures, and it's natural to ask
 them to qualify their request.
 
 If removing the non-native architecture has cascading effects, apt is
 obviously going to warn them about that already and they'll realize what's
 going on.

This was already contemplated at least as part of one of the threads David
referenced:

  http://lists.debian.org/debian-dpkg/2011/12/msg00068.html

 David Kalnischkies kalnischk...@gmail.com writes:
  (Note though that e.g. APT is not able to handle installed architectures
  as an 'attribute'. It not only has to handle them as 'different'
  packages (and more specific different versions) to keep
  backward-compatibility, also different dependencies on different
  architectures would make it pretty messy in practice. But double-think
  is a requirement for APT development anyway. ;) )
 
 Yes, definitely the internals of our package management software can't
 fully compress the packages together; at the least, the dependencies are
 going to be different between architectures and have to be stored
 separately. [...]

 But I think what we should be telling the *user*, regardless of our
 internals, is don't think of libfoo:i386 and libfoo:amd64 as two separate
 packages that you can maintain independently; think of libfoo as being
 installed for one or more architectures.

The thing is, in practice they cannot share much at all, because even
if they might end up being at the same version, they need to go
through different versions inbetween. For dpkg, only the package name
and the reverse dependencies are shared, assuming any other field is
equal will only come down to lost metadata.

And while I have initially actually been working with the mental model
of pkgname with multiple arch instances as an internal detail, the fact
is that this abstraction just leaks everywhere, and trying to shield
the users and maintainers from that reality will only cause pain. It's
just a nice illusion coming from the fact that those packages share the
package name. But considering pkgname == pkgname:* means for example that
all query commands have to output information for multiple instances, so
packages/scripts/etc have to be adapted anyway to handle those, and
while I don't consider that a problem, just another side of the changes
needed for multiarch, it shows how the interface can only possibly be
transparent on one side of the interface, if at all.

Finally, the thing is, those packages are really independent, they just
happen to share a name and a common source ancestor, but they can contain
different files per arch instance, different metadata, even different
maintainer script behaviour per arch, etc. And only packages from the
same arch can depend on them.

  Mhh. The current spec just forbids binNMU for M-A:same packages -
  the 'sync' happens on the exact binary version.
  Somewhere else in this multiarch-discussion was hinted that we could
  sync on the version in (optional) Source tag instead to allow binNMU.
 
 I think that the best long-term way to handle binNMUs may be to move the
 build number into a different piece of package metadata from the version.
 So a binNMU of a package with version 1.4-1 would still have version 1.4-1
 but would have a build number of 2 instead of 1.  I think this would be
 way cleaner in the long run, and not just for multiarch.

That means then we cannot state a relationship based on the binNMU
version. And while that might be desirable most of the times, it makes
it impossible when it might be desirable. Without considering this
deeper, it also reminds me of when Revision was a distinct field. In
any case how to handle binNMUs is something that should be carefully
considered and not be rushed out now, just because suddently they cannot
be used...

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120229215831.gb4...@gaara.hadrons.org

Re: Multiarch file overlap summary and proposal

2012-02-29 Thread Russ Allbery

Guillem Jover guil...@debian.org writes:
 On Wed, 2012-02-15 at 19:31:10 -0800, Russ Allbery wrote:

 I think that the best long-term way to handle binNMUs may be to move
 the build number into a different piece of package metadata from the
 version.  So a binNMU of a package with version 1.4-1 would still have
 version 1.4-1 but would have a build number of 2 instead of 1.  I think
 this would be way cleaner in the long run, and not just for multiarch.

 That means then we cannot state a relationship based on the binNMU
 version. And while that might be desirable most of the times, it makes
 it impossible when it might be desirable.

Good point.

 Without considering this deeper, it also reminds me of when Revision was
 a distinct field. In any case how to handle binNMUs is something that
 should be carefully considered and not be rushed out now, just because
 suddently they cannot be used...

I agree with this sentiment.  Personally, I'm fine with moving forward
with a multiarch approach that doesn't allow for binNMUs on a subset of
arches as the first cut, and then go back and figure out what we're doing
with binNMUs later.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87boohxpa8@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal

On Thu, 2012-02-16 at 10:43:53 -0800, Russ Allbery wrote:
 I was thinking more about this, and I was finally able to put a finger on
 why I don't like package splitting as a solution.
 
 We know from prior experience with splitting packages for large
 arch-independent data that one of the more common mistakes that we'll make
 is to move the wrong files: to put into the arch-independent package a
 file that's actually arch-dependent.

This was brought up by Steve in the thread, my reply:

  http://lists.debian.org/debian-devel/2012/02/msg00497.html

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120301020029.ga8...@gaara.hadrons.org

Re: Multiarch file overlap summary and proposal

2012-02-29 Thread Russ Allbery

Guillem Jover guil...@debian.org writes:
 On Thu, 2012-02-16 at 10:43:53 -0800, Russ Allbery wrote:

 I was thinking more about this, and I was finally able to put a finger
 on why I don't like package splitting as a solution.

 We know from prior experience with splitting packages for large
 arch-independent data that one of the more common mistakes that we'll
 make is to move the wrong files: to put into the arch-independent
 package a file that's actually arch-dependent.

 This was brought up by Steve in the thread, my reply:

   http://lists.debian.org/debian-devel/2012/02/msg00497.html

Thanks for the pointer, Guillem, but I'm afraid I don't think this reply
addresses my concerns.  See the specific enumeration of things that we
would have to split, and the ways in which they can break.  I think the
issue with C headers is particularly severe.

I don't think this mirrors an existing problem.  The sorts of things we
split into arch: all packages are nowhere near as intrusive or as tightly
coupled as the things we're talking about splitting to avoid refcounting;
for example, right now, splitting out C headers into arch: all packages is
very rare.  The sort of package splitting that we would do to avoid
refcounting would run a serious risk of introducing substantial new
problems that we don't currently have.

The situation with refcounting seems much less fragile than the situation
without refcounting to me.  Refcounting puts the chance of error in the
right place (on people who want to use the new feature, since the
situation will not change for users who continue using packages the way
they do today), provides a clear error message rather than silent
corruption, and fails safely (on any inconsistency) rather than appearing
to succeed in situations that are not consistent.  Those are all good
design principles to have.

I think the principle of not changing things for people who are not using
multiarch is particularly important, and is inconsistent with either
package splitting or with moving files into arch-qualified paths.  We
should attempt to adopt new features in a way that puts most of the risk
on the people who are making use of the new features, and tries to be as
safe as possible for existing users.  I agree that we should not pursue
that to an extreme that leads to an unmaintainable system, but I don't
believe refcounting has that problem.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874nu9vzbb@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal

On Wed, 2012-02-15 at 16:32:38 -0800, Russ Allbery wrote:
 Guillem Jover guil...@debian.org writes:
  If packages have to be split anyway to cope with the other cases, then
  the number of new packages which might not be needed otherwise will be
  even smaller than the predicted amount, at which point it makes even
  less sense to support refcnt'ing.
 
 I don't think the package count is really the interesting metric here,
 unless the number of introduced packages is very large (see below about
 -dev packages).  I'm more concerned with maintainer time and with
 dependency complexity, and with the known problems that we introduce
 whenever we take tightly-coupled files and separate them into independent
 packages.

Well, people have been using the amount of packages as a metric, I've
just been trying to counter it. It also in a way represents the amount
of work needed.

About tightly-coupled files, they can cause serious issues also with
refcounting, consider that there's always going to be a point when
unpacking one of the new instances will have a completely different
vesion than the other already unpacked instance(s). So packages could
stop working for a long time if say unpacked libfoo0:i386 1.0 has
file-format-0, but new libfoo0:amd64 4.0 has file-format-2, and the
file didn't change name (arguably this could be considered an upstream
problem, depending on the situation), this would be particularly
problematic for pseudo-essential packages.

 I just posted separately about version lockstep: I think this is a
 feature, not a bug, in our multiarch implementation.  I think this is the
 direction we *should* go, because it reduces the overall complexity of the
 system. [...]

I've replied to that separately, in any case I think the best compromise
would be to add version lockstep to dpkg, but not refcounting. Because
the first is a restriction that can always be lifted if it's confirmed
to cause issues (which I think it will), and the second can always be
added later because it's something that allows things not permitted
previously.

But at this point it seems I'm alone in thinking that refcounting has
more negative implications than positive ones, and I cannot get myself
to care enough any longer to push for this. So some weeks ago I added
back both those things to my local repo.

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120301030201.gb8...@gaara.hadrons.org

Re: Multiarch file overlap summary and proposal

2012-02-29 Thread Russ Allbery

Guillem Jover guil...@debian.org writes:

 About tightly-coupled files, they can cause serious issues also with
 refcounting, consider that there's always going to be a point when
 unpacking one of the new instances will have a completely different
 vesion than the other already unpacked instance(s). So packages could
 stop working for a long time if say unpacked libfoo0:i386 1.0 has
 file-format-0, but new libfoo0:amd64 4.0 has file-format-2, and the file
 didn't change name (arguably this could be considered an upstream
 problem, depending on the situation), this would be particularly
 problematic for pseudo-essential packages.

Yes, I agree.  Refcounting does complicate the upgrade situation, since
you really want to upgrade all installed architectures in lockstep to
ensure that we maintain as many of the guarantees of file consistency as
we do now with single-arch upgrades.

 I've replied to that separately, in any case I think the best compromise
 would be to add version lockstep to dpkg, but not refcounting. Because
 the first is a restriction that can always be lifted if it's confirmed
 to cause issues (which I think it will), and the second can always be
 added later because it's something that allows things not permitted
 previously.

I definitely understand where you're coming from, and I would be lying if
I said that introducing refcounting doesn't make me nervous.  You're
right, it's something that's going to be very difficult to back out of if
we decide it's a mistake.

I do think it's the best solution to a complex set of issues, but we're
going to have to use it in conjunction with pretty tight version lockstep
to avoid problems with file inconsistency.

 But at this point it seems I'm alone in thinking that refcounting has
 more negative implications than positive ones, and I cannot get myself
 to care enough any longer to push for this. So some weeks ago I added
 back both those things to my local repo.

Well... no one likes to win an argument under those terms.  I'd much
rather have us all agree.  But I do want to wholeheartedly second
Christian's thanks for all your work on dpkg in the middle of a really
difficult situation, and your willingness to make compromises like this
even when you think they're the wrong technical decision.  That's really
hard to do, and I think it's also very admirable.

If this all turns out to be a horrible mistake, I for one will try to help
us back out of it as needed, to put my resources where my advocacy has
been.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87sjhtuidr@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal

Russ Allbery r...@debian.org writes:

 If this is comprehensive, then I propose the following path forward, which
 is a mix of the various solutions that have been discussed:

 * dpkg re-adds the refcounting implementation for multiarch, but along
   with a Policy requirement that packages that are multiarch must only
   contain files in classes 1 and 2 above.

 * All packages that want to be multiarch: same have to move all generated
   documentation into a separate package unless the maintainer has very
   carefully checked that the generated documentation will be byte-for-byte
   identical even across minor updates of the documentation generation
   tools and when run at different times.

 * Lintian should recognize arch-qualified override files, and multiarch:
   same packages must arch-qualify their override files.  debhelper
   assistance is desired for this.

I think that, provided the files are byte for byte identical across
architectures they need not be arch qualified. So they should be
refcounted and having non-identical files across archs should be
forbidden by policy. The maintainer must then resolve this by 1) making
the file identical across archs or 2) arch qualifying the name.

So lintian should support arch qualified names but policy should not
needlessly require them.

 * Policy prohibits arch-varying data files in multiarch: same packages
   except in arch-qualified paths.

 * The binNMU process is changed to add the binNMU changelog entry to an
   arch-qualified file (changelog.Debian.arch, probably).  We need to
   figure out what this means if the package being binNMU'd has a
   /usr/share/doc/package symlink to another package, though; it's not
   obvious what to do here.

 Please note that this is a bunch of work.  I think the Lintian work is a
 good idea regardless, and it can start independently.  I think the same is
 true of the binNMU changelog work, since this will address some
 long-standing issues with changelog handling in some situations, including
 resolving just how we're supposed to handle /usr/share/doc symlinks.  But
 even with those aside, this is a lot of stuff that we need to agree on,
 and in some cases implement, in a fairly short timeline if this is going
 to make wheezy.

In case /usr/share/doc/pkg is a symlink the binNMU changelog should be
stored in the destination of the symlink. For this to work the (binNMU)
changelog should be both arch and package qualified
(changelog.Debain.bin-pkg.arch). This would allow libfoo:any,
foo-bin:any + foo-common:all to be binNMUed. Without the package
qualifier the libfoo and foo-bin package would both contain
/usr/share/doc/foo-common/changelog.Debian.arch and produce a file
overwrite error.

After a binNMU (on amd64) the following files would exist:

foo-bin:/usr/share/doc/foo-bin - foo-common
foo-bin:/usr/share/doc/foo-common/changelog.Debian.foo-bin.amd64
foo-common: /usr/share/doc/foo-common/changelog.Debian
libfoo: /usr/share/doc/foo-common/changelog.Debian.libfoo.amd64
libfoo: /usr/share/doc/libfoo - foo-common

The binNMU changelogs would be identical wasting a little disk and
mirror space and bandwidth but that is probably unavoidable.

One way to reduce the overhead would be to split the binNMU entry into a
separate changelog:

foo-bin:/usr/share/doc/foo-bin - foo-common
foo-bin:/usr/share/doc/foo-common/changelog.binNMU.foo-bin.amd64
foo-common: /usr/share/doc/foo-common/changelog.Debian
libfoo: /usr/share/doc/foo-common/changelog.binNMU.libfoo.amd64
libfoo: /usr/share/doc/libfoo - foo-common

The binNMU changelogs would be just one entry each, the reason for the
binNMU.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d395zn6m.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

Russ Allbery r...@debian.org writes:

 Carsten Hey cars...@debian.org writes:
 * Russ Allbery [2012-02-16 14:55 -0800]:

 Every file that differs has to be fixed in the current multi-arch plan.
 Documentation that contains its build date is going to need to be split
 out into a separate -docs package.

 I doubt that ftpmaster would be happy about -doc packages that contain
 just a few small man pages.

 I think they'll be okay with it when it's the cost of reasonable
 multiarch.

 I feel fairly strongly that it isn't sane to have a file overlap when the
 file doesn't match.  You then lose the error detection when there are real
 problems, and I don't trust any of us, myself included, to correctly tag
 files where it doesn't matter.

 On this front, I agree with Guillem: some amount of package splitting is
 fine, and package splitting instead of additional complexity, like tagging
 files that are allowed to vary, looks like a good tradeoff to me.  The
 splitting that I'm worried about is where the files are tightly coupled,
 which is not the case for development man pages that are now in -dev
 packages.

+1.

I find the argument about past experience with splitting packages and
upstream later changing files so they become arch dependent convincing
in this regard. Refcounting and no exception for file differences seems
to be the best way.

 debianutils uses a special make target 'prebuild' in debian/rules to
 update build system related files and PO files before the actual source
 package is built.

 This basic idea also could be used to build problematic documentation
 files on the maintainers computer before he/she builds the package.  The
 other targets would then install the prebuilt documentation into the
 package without the need to build it first.  A proper dependency on
 debian/$prebuilt_doc could ensure that maintainers do not forget to run
 debian/rules prebuild.

 If maintainers choose to use such a target, suggesting a common name for
 it in the developers reference could be reasonable.

 That's an interesting idea.  That's very similar to what I already do as
 upstream (I build POD-generated man pages from my autogen script, and in
 Debian packaging don't bother to regenerate them).

Indeed. Another +1.

You are probably not the only one doing something like this. Who else
does this? What automatism do you have in your debian/rules to help?
Lets see if we can get a good set of examples to work out some
recommendations.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/878vjtzmjj.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

Josselin Mouette j...@debian.org writes:

 Le lundi 13 fÃ©vrier 2012 Ã  22:43 -0800, Russ Allbery a Ã©crit : 
 There's been a lot of discussion of this, but it seems to have been fairly
 inconclusive.  We need to decide what we're doing, if anything, for wheezy
 fairly soon, so I think we need to try to drive this discussion to some
 concrete conclusions.

 Thank you very much for your constructive work.

 3. Generated documentation.  Here's where I think refcounting starts
failing.

 So we need to move a lot of documentation generated with gtk-doc or
 doxygen from -dev packages to -doc packages. But it really seems an
 acceptable tradeoff between the amount of work required and the
 cleanness of the solution.

 Does this seem comprehensive to everyone?  Am I missing any cases?

 Are there any cases of configuration files in /etc that vary across
 architectures? Think of stuff like ld.so.conf, where some plugins or
 library path is coded in a configuration file.

Generally conffiles in library packages is a violation of policy
8.2. They would create a file overwrite conflict if the SONAME
changes. Putting the conffile into the -common package is a good
solution.

There are some exceptions where the conffiles are version qualified.
E.g.: libgtk2.0-0: /etc/gtk-2.0/im-multipress.conf

For conffiles that vary across architectures the path or name must
include the multiarch triplet.
E.g: libc6: /etc/ld.so.conf.d/x86_64-linux-gnu.conf

(Note: this is actualy a violation of policy 8.2 and needs to be fixed
when we get a libc7).

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874nuhzm2m.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

Joey Hess jo...@debian.org writes:

 Goswin von Brederlow wrote:
 pkg:arch will still be unique and the dpkg/apt output will use the
 architecture where required for uniqueness. So I think that after some
 getting used to it it will be clear enough again.

 Here are a few examples of the problems I worry about. I have not
 verified any of them, and they're clearly biased toward code I am
 familiar with, which suggests there are many other similar problems.

 * Puppet not only installs packages, it may remove them. A puppet config
   that does dpkg --purge foo will fail if multiarch is enabled, now
   it needs to find and remove foo:*

 * dpkg-repack pkg:arch will create a package with that literal name (or fail)

 * dpkg-reconfigure probably can't be used with M-A same packages.
   debconf probably generally needs porting to multiarch.

 * tasksel uses dpkg --query to work out if a task's dependencies are
   installed. In the event that a library is directly part of a task,
   this will fail when multiarch is enabled.

 * Every piece of documentation that gives commands lines manipulating
   library packages is potentially broken.

 Seems like we need a release where multiarch is classed as an
 experimental feature, which when enabled can break the system. But the
 sort of problems above are the easy to anticipate ones; my real worry is
 the unanticipated classes of problems. Especially if we find intractable
 problems or levels of complexity introduced by dropping the unique
 package name invariant.

 My nightmare scenario is that we release with multiarch, discover that
 it's a net negative for our users (proprietary software on amd64 aside,
 nearly all the benefits are to developers AFAICS), and are stuck with it.

The specs were initialy written in such a way that single arch systems
would not change, that multiarch packages would keep functioning with a
mono-arch apt/dpkg and I think this was preserved so far.

If all interface changes foolow that idea then worst case tools will not
work in a multiarch configuration but still work in a monoarch
configuration. So let multiarch be experimental and only for developers
and risk takers. That is already a huge number of people.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zkc9y6p4.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

Russ Allbery r...@debian.org writes:

 I think it would be better to have a world in which all the architectures
 of the foo package on the system have to have the same version, because
 then you don't have to treat foo:i386 and foo:amd64 like they're separate
 packages.  The list of installed architectures is an *attribute* of the
 package.  A package is still either installed or not installed, but when
 it's installed, it can be installed for one or more architectures.  But if
 it's installed for multiple architectures, those architectures are always
 upgraded together and always remain consistent.  That avoids all weirdness
 of having a package work differently because the version varies depending
 on the ABI, and it significantly simplifies the mental model behind the
 package.

In such a world architecture all could also be considered another
architecture. And then foo:i386, foo:amd64 and foo:all could be
coinstallable. That would mean that files shared between architectures
could be moved into foo:all and foo:any could implicitly depend on
foo:all. The benefit of this over foo-common would be that apt-cache
search, apt-cache policy, aptitude, dpkg --remove, ... would only have
one package (foo) instead of 2 (foo + foo-common).

This has been previously suggested too but has been droped because it
would be incompatible with existing systems (i.e. monoarch dpkg couldn't
install packages from such a world).

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87vcmxy6dk.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

David Kalnischkies kalnischk...@gmail.com writes:

 On Thu, Feb 16, 2012 at 23:10, Carsten Hey cars...@debian.org wrote:
 * David Kalnischkies [2012-02-16 03:59 +0100]:
 On Thu, Feb 16, 2012 at 00:39, Russ Allbery r...@debian.org wrote:
 (the only problem i see is that i don't have ${source:Version} available
  currently in the version structure, but we haven't even tried pushing
  apt's abibreak to sid specifically as i feared last-minute changes)

 I'm not sure if you meant this with Source tag, anyway, the 'Packages'
 files miss the source version too, but this could be added as optional
 field that would be used if it differs from the 'Version:' field.

 It's already in for quiet some time ('current' sid amd64, first hit):
 Package: 3depict
 Source: 3depict (0.0.9-1)
 Version: 0.0.9-1+b1
 []

 It's used in other places in APT, e.g. 'apt-get source', which just looks
 at the Packages file stanza. That's fine as this isn't a speed critical
 operation - but if we want it for the lock-step operation apt needs that
 piece of information in its internal structures for fast access to it and
 adding new fields in these structures will require an abibreak.
 That's the intended meaning of the quoted sentence.

Except that doesn't have to work (sorry for the ubuntu part):

Package: gcc
Source: gcc-defaults (1.93ubuntu1)
Version: 4:4.4.3-1ubuntu1

What would the version be for a binNMU of gcc-defaults? I think it would
be

Package: gcc
Source: gcc-defaults (1.93ubuntu1)
Version: 4:4.4.3-1ubuntu1+b1

What we want is for apt/dpkg to consider this to be compatible with
4:4.4.3-1ubuntu1.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r4xly64e.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

2012-02-23 Thread Raphael Hertzog

On Thu, 23 Feb 2012, Goswin von Brederlow wrote:
  Package: 3depict
  Source: 3depict (0.0.9-1)
  Version: 0.0.9-1+b1
 
 Except that doesn't have to work (sorry for the ubuntu part):
 
 Package: gcc
 Source: gcc-defaults (1.93ubuntu1)
 Version: 4:4.4.3-1ubuntu1
 
 What would the version be for a binNMU of gcc-defaults?

What about trying it?

$ head -n 1 debian/changelog 
gcc-defaults (1.112+b1) unstable; urgency=low
$ debuild -us -uc
[...]
$ dpkg -I ../gcc_4.6.2-4+b1_i386.deb 
[...]
 Package: gcc
 Source: gcc-defaults (1.112)
 Version: 4:4.6.2-4+b1

In any case, the fact that the source version is unrelated to the binary
version doesn't change anything to the requirement. We just want to ensure
that all (M-A: same) co-installable packages are synchronized at the same
version (either the source version, or the binary version but stripped
from its bin-NMU suffix).

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120223145305.gd5...@rivendell.home.ouaza.com

Re: Multiarch file overlap summary and proposal

2012-02-22 Thread Goswin von Brederlow

Jonathan Nieder jrnie...@gmail.com writes:

 Jonathan Nieder wrote:
 David Kalnischkies wrote:

 Why would it be intuitive to add a specific value for the arch attribute 
 with
 apt-get install foo   # arch |= native
 but remove all values of the attribute with
 apt-get remove foo# arch = ~all-architectures
 ?
 [...]
 But I really think this is something anyone can get used to.  In the
 examples you listed above:

  apt-get install foo;# install foo with default arch-list (native)
  apt-get remove foo; # remove foo

 If foo is installed for no architectures, that does not mean it is
 installed with an empty architecture list.  It means it is simply not
 installed.

 Ok, now I think I figured out the inconsistency you are pointing to.
 If i386 is the native architecture, what would you expect the
 following sequence of commands to do?

   apt-get install linux-image-3.2.0-1-amd64:amd64

   ... wait a few weeks ...

   apt-get install linux-image-3.2.0-1-amd64

 I would expect it to install the kernel with 'Architecture: amd64' and
 then to upgrade it.

 So the proposed semantics are not quite 'arch |= native'.  They are
 more like 'arch defaults to native for non-installed packages'.

 Jonathan

Assuming linux-image-3.2.0-1-amd64:i386 still exists I would expect apt
to install that if it has a equal or greater version than the installed
linux-image-3.2.0-1-amd64:amd64.

Current apt behaviour is a bit strange there though:

mrvn@frosties:~% apt-cache policy acl
acl:
  Installed: 2.2.51-4
  Candidate: 2.2.51-5
  Version table:
 2.2.51-5 0
500 http://ftp.de.debian.org/debian/ sid/main amd64 Packages
 *** 2.2.51-4 0
100 /var/lib/dpkg/status

mrvn@frosties:~% apt-cache policy acl:i386
acl:i386:
  Installed: (none)
  Candidate: 2.2.51-5
  Version table:
 2.2.51-5 0
500 http://ftp.de.debian.org/debian/ sid/main i386 Packages

I would expect something like:

mrvn@frosties:~% apt-cache policy acl
acl:
  Installed: 2.2.51-4 amd64
  Candidate: 2.2.51-5 amd64
  Version table:
 2.2.51-5 0
500 http://ftp.de.debian.org/debian/ sid/main amd64 Packages
499 http://ftp.de.debian.org/debian/ sid/main i386 Packages
 *** 2.2.51-4 0
100 /var/lib/dpkg/status amd64


But it seems my patch to reduce the pin of non-native architectures is
not in current apt and policy doesn't list all archs for M-A:foreign
packages.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r4xn4171.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread Carsten Hey

* Russ Allbery [2012-02-16 14:55 -0800]:
 Carsten Hey cars...@debian.org writes:
  There are still files that differ that do not need to be fixed, for
  example documentation that contains it's build date.

 Every file that differs has to be fixed in the current multi-arch plan.
 Documentation that contains its build date is going to need to be split
 out into a separate -docs package.

I doubt that ftpmaster would be happy about -doc packages that contain
just a few small man pages.

 I'm fine with splitting documentation; that has far fewer problems than
 splitting other types of files, since documentation isn't tightly coupled
 at a level that breaks software.

debianutils uses a special make target 'prebuild' in debian/rules to
update build system related files and PO files before the actual source
package is built.

This basic idea also could be used to build problematic documentation
files on the maintainers computer before he/she builds the package.  The
other targets would then install the prebuilt documentation into the
package without the need to build it first.  A proper dependency on
debian/$prebuilt_doc could ensure that maintainers do not forget to run
debian/rules prebuild.

If maintainers choose to use such a target, suggesting a common name for
it in the developers reference could be reasonable.


Regards
Carsten


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120217082256.ga19...@furrball.stateful.de

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread Russ Allbery

Carsten Hey cars...@debian.org writes:
 * Russ Allbery [2012-02-16 14:55 -0800]:

 Every file that differs has to be fixed in the current multi-arch plan.
 Documentation that contains its build date is going to need to be split
 out into a separate -docs package.

 I doubt that ftpmaster would be happy about -doc packages that contain
 just a few small man pages.

I think they'll be okay with it when it's the cost of reasonable
multiarch.

I feel fairly strongly that it isn't sane to have a file overlap when the
file doesn't match.  You then lose the error detection when there are real
problems, and I don't trust any of us, myself included, to correctly tag
files where it doesn't matter.

On this front, I agree with Guillem: some amount of package splitting is
fine, and package splitting instead of additional complexity, like tagging
files that are allowed to vary, looks like a good tradeoff to me.  The
splitting that I'm worried about is where the files are tightly coupled,
which is not the case for development man pages that are now in -dev
packages.

 debianutils uses a special make target 'prebuild' in debian/rules to
 update build system related files and PO files before the actual source
 package is built.

 This basic idea also could be used to build problematic documentation
 files on the maintainers computer before he/she builds the package.  The
 other targets would then install the prebuilt documentation into the
 package without the need to build it first.  A proper dependency on
 debian/$prebuilt_doc could ensure that maintainers do not forget to run
 debian/rules prebuild.

 If maintainers choose to use such a target, suggesting a common name for
 it in the developers reference could be reasonable.

That's an interesting idea.  That's very similar to what I already do as
upstream (I build POD-generated man pages from my autogen script, and in
Debian packaging don't bother to regenerate them).

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87vcn5hml3@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread David Kalnischkies

On Thu, Feb 16, 2012 at 23:10, Carsten Hey cars...@debian.org wrote:
* David Kalnischkies [2012-02-16 03:59 +0100]:
On Thu, Feb 16, 2012 at 00:39, Russ Allbery r...@debian.org wrote:
it needs to find and remove foo:*

foo:all (or foo:any) instead of foo:* would save the need to quote it.

:all is already an architecture and currently it seems like dpkg accepts
only that while APT will accept :amd64 (or whatever is native), too,
partly for internal reasons, but also because the difference for an apt-get
user is not really important. Either way, overloading wouldn't be nice.

:any has a special meaning in build-dependencies already meaning - surprise
surprise - give me any package. Overloading would be uncool, beside that
its the wrong word for it anyway if you want all, and not just any package.

Actually, why would that be the behavior? Why would dpkg --purge foo not
just remove foo for all architectures for which it's installed, and
require that if you want to remove only a specific architecture you then
use the expanded syntax?

We (as in APT team and dpkg team) had a lot of discussions about that,
see for starters (there a properly more in between the 10 months…)
[0] http://lists.debian.org/debian-dpkg/2011/01/msg00046.html
[1] http://lists.debian.org/debian-dpkg/2011/12/msg5.html

In short, i think the biggest counter is that it feels unintuitive to
install a library (in native arch) with e.g. apt-get install libfoo
while you have to be specific at removal to avoid nuking 'unrelated' packages
with apt-get remove libfoo.

I would expect this (especially if the package foo is not a library, but
I would also expect this for libraries):

You generously left out the paragraph describing how APT should
detect that the package foo is in fact a library and not, say, a
plugin, a dev-package, a dbg-package or a future-coinstallable binary.
And the foo:* default would be okay and intuitive for all of those?

You also skipped the part of backward-compatibility with tools which
expect a single line/stanza of response and commands like
dpkg --{get,set}-selection which by definition work only with a single
package.

And backward-compatibility means in this context also to support
a dist-upgrade from squeeze to wheezy. If a new version of dpkg
doesn't accept the 'old' commands APT uses the upgrade will
be a pain.

The two threads i mentioned contain a lot more of these
considerations, so it might be in order to read these before
coming up with 'new' ideas.

* apt-get remove foo removes all installed foo packages (on all
architectures).

More of a theoretical nitpick, but this was never the case.
apt-get pre-multi-arch handled only packages from native arch (and :all),
so if you had installed foreign packages with dpkg --force-architecture
apt-get would have ignored it, not removed it.

This summarises how apt without multi-arch handles this, the above would
make apt with multi-arch also behave so:

apt-get install foo
---
foo is not installed foo is installed
apt-get remove foo
---

Is 'apt-get remove foo+' then going to install all foo's or just one?

The current implementation of always foo == foo:native doesn't fail
your diagram, too, so what is this going to show us?

(the only problem i see is that i don't have ${source:Version} available
currently in the version structure, but we haven't even tried pushing
apt's abibreak to sid specifically as i feared last-minute changes…)

I'm not sure if you meant this with Source tag, anyway, the 'Packages'
files miss the source version too, but this could be added as optional
field that would be used if it differs from the 'Version:' field.

It's already in for quiet some time ('current' sid amd64, first hit):
Package: 3depict
Source: 3depict (0.0.9-1)
Version: 0.0.9-1+b1
[…]

It's used in other places in APT, e.g. 'apt-get source', which just looks
at the Packages file stanza. That's fine as this isn't a speed critical
operation - but if we want it for the lock-step operation apt needs that
piece of information in its internal structures for fast access to it and
adding new fields in these structures will require an abibreak.
That's the intended meaning of the quoted sentence.

Best regards

David Kalnischkies

--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/CAAZ6_fB3CeUcFuue1CjPbkqoHNaSvaKt8Q=imgfx4uqmw_m...@mail.gmail.com

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread Jonathan Nieder

David Kalnischkies wrote:

 You generously left out the paragraph describing how APT should
 detect that the package foo is in fact a library and not, say, a
 plugin, a dev-package, a dbg-package or a future-coinstallable binary.
 And the foo:* default would be okay and intuitive for all of those?

Yes, the foo:native default for installation and foo:* default for
removal would be intuitive for all of those.

See [1] for a mental model.

Hope that helps,
Jonathan

[1] 
http://thread.gmane.org/gmane.linux.debian.devel.dpkg.general/14028/focus=14031


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120217144631.GA5039@burratino

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread David Kalnischkies

On Fri, Feb 17, 2012 at 15:46, Jonathan Nieder jrnie...@gmail.com wrote:
 David Kalnischkies wrote:

 You generously left out the paragraph describing how APT should
 detect that the package foo is in fact a library and not, say, a
 plugin, a dev-package, a dbg-package or a future-coinstallable binary.
 And the foo:* default would be okay and intuitive for all of those?

 Yes, the foo:native default for installation and foo:* default for
 removal would be intuitive for all of those.

 See [1] for a mental model.

 Hope that helps,
 Jonathan

 [1] 
 http://thread.gmane.org/gmane.linux.debian.devel.dpkg.general/14028/focus=14031

Why would it be intuitive to add a specific value for the arch attribute with
apt-get install foo   # arch |= native
but remove all values of the attribute with
apt-get remove foo# arch = ~all-architectures
?

Isn't it more intuitive to have it this way:
apt-get remove foo# arch = ~native
?

Maybe we just have different definitions of intuitive.


Best regards

David Kalnischkies


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caaz6_fbq6z7tj1slohgs0kwh+xocsa_xjcvq+zs2omg3_y1...@mail.gmail.com

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread Jonathan Nieder

David Kalnischkies wrote:

 Why would it be intuitive to add a specific value for the arch attribute with
 apt-get install foo   # arch |= native
 but remove all values of the attribute with
 apt-get remove foo# arch = ~all-architectures
 ?

 Isn't it more intuitive to have it this way:
 apt-get remove foo# arch = ~native
 ?

 Maybe we just have different definitions of intuitive.

Intuitions vary from person to person; that's definitely not news.

But I really think this is something anyone can get used to.  In the
examples you listed above:

 apt-get install foo;   # install foo with default arch-list (native)
 apt-get remove foo;# remove foo

If foo is installed for no architectures, that does not mean it is
installed with an empty architecture list.  It means it is simply not
installed.

In practice, that would match what I want to do, too.

 * There is a web browser I would like to use.  I don't care which
   arch --- that's an implementation detail.

apt-get install iceweasel

 * Oops, never mind --- not interested in using that web browser any
   more.

apt-get --purge remove iceweasel

 * I've never heard of this multiarch stuff, but the unpackaged
   software I am trying to install is giving complaints about missing
   libfoo.so.1

apt-get install libfoo1

 * Ok, now I've learned about multiarch, and I want to install libfoo
   to satisfy a dependency for a binary on a foreign architecture.

apt-get install libfoo1:amd64

 * I don't want libfoo any more --- remove it completely from the
   system.

apt-get --purge remove libfoo1

Wait! you might protest.  Isn't that last command too aggressive?
After all, I did not specify which architecture motivated the removal
of libfoo1.  Maybe I was removing libfoo1 for the sake of my
unpackaged i386 software but I still need it for unpackaged amd64
software, and apt could help me out by picking the architecture I
intended and not removing it elsewhere, right?

But no, that would not be helpful at all.  It's true that libfoo1
might be installed for more than one reason and I might have forgotten
about some and therefore remove it when that is not warranted, but
that's true whether multiarch is involved or not.  This safety feature
does not add any real consistent safety.

I can think of only one advantage to making apt-get remove libfoo1
remove libfoo1:native, though it's a good one.  That's to support
muscle memory and scripts that rely on the libfoo1 always means
libfoo1:native semantics that have been present in Ubuntu for a
little while.  I think it's worth losing that, since as we've seen,
most scripts dealing with cases like this are going to need changes to
work with multiarch: same packages anyway (and humans can grow to
appreciate the simple mental model Russ suggested).

Jonathan


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120217165937.GA9360@burratino

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread Jonathan Nieder

Jonathan Nieder wrote:
 David Kalnischkies wrote:

 Why would it be intuitive to add a specific value for the arch attribute with
 apt-get install foo   # arch |= native
 but remove all values of the attribute with
 apt-get remove foo# arch = ~all-architectures
 ?
[...]
 But I really think this is something anyone can get used to.  In the
 examples you listed above:

  apt-get install foo; # install foo with default arch-list (native)
  apt-get remove foo;  # remove foo

 If foo is installed for no architectures, that does not mean it is
 installed with an empty architecture list.  It means it is simply not
 installed.

Ok, now I think I figured out the inconsistency you are pointing to.
If i386 is the native architecture, what would you expect the
following sequence of commands to do?

apt-get install linux-image-3.2.0-1-amd64:amd64

... wait a few weeks ...

apt-get install linux-image-3.2.0-1-amd64

I would expect it to install the kernel with 'Architecture: amd64' and
then to upgrade it.

So the proposed semantics are not quite 'arch |= native'.  They are
more like 'arch defaults to native for non-installed packages'.

Jonathan


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120217171734.GB9360@burratino

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread Carsten Hey

* David Kalnischkies [2012-02-17 14:15 +0100]:
 You generously left out the paragraph describing how APT should
 detect that the package foo is in fact a library ...

My impression was that you think very library centric.  All I wrote was
(in other words), that we should consider non-library packages as much
as library packages, and I did not write nor implied that libraries
should be handled in a different way.


 Is 'apt-get remove foo+' then going to install all foo's or just one?

apt-get install g+++ is a weird syntax.


 The current implementation of always foo == foo:native doesn't fail
 your diagram, too, so what is this going to show us?

It depends on how one reads it, anyway, examples I consider to be
inconsistent are more helpful than a diagram without clear semantic.


  # dpkg --print-architecture
  amd64

  # perl -00 -lne 'print if /^Package: (clang|tendra)$/m 
  /^Status: install ok installed$/m' /var/lib/dpkg/status \
  | awk '/^Package:/ {printf %s:, $2} /^Architecture:/ {print $2}'
  tendra:i386
  clang:i386

I was not able to find a command that shows this information.


  # apt-cache policy tendra | sed -n 1p
  tendra:i386:

  # apt-cache policy clang | sed -n 1p
  clang:

  # apt-get remove tendra
  The following packages will be REMOVED:
tendra:i386

  # apt-get remove clang
  Package clang is not installed, so not removed

The above shows that it seems to depend on the availability of
foo:native if apt-get remove foo removes foo:foreign.


  # dpkg -l | awk '$2==clang{print}'
  ii  clang   3.0-5 Low-Level ...

  # dpkg -S bin/clang
  clang: /usr/bin/clang
  clang: /usr/bin/clang++

  # dpkg -r clang
  dpkg: warning: there's no installed package matching clang

  # apt-get remove clang
  Package clang is not installed, so not removed

According to dpkg's command line interface, the file /usr/bin/clang is
in the package clang and dpkg -l shows it as installed, but it can not
be removed using this name, neither by apt nor by dpkg.


  # dpkg -l | grep libzookeeper-st2
  ii  libzookeeper-st2:amd64  3.3.4+dfsg1-3 Single ...
  ii  libzookeeper-st2:i386   3.3.4+dfsg1-3 Single ...

Unlike the above dpkg -l output showing the foreign clang package,
libzookeeper-st2 is shown with the architecture appended.


Regards
Carsten


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120217185319.ga29...@furrball.stateful.de

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread Carsten Hey

* David Kalnischkies [2012-02-17 17:20 +0100]:
 Why would it be intuitive to add a specific value for the arch attribute with
 apt-get install foo   # arch |= native
 but remove all values of the attribute with
 apt-get remove foo# arch = ~all-architectures
 ?

We had a similar discussion years ago.

Package: foo
Depends: a, b, c
Conflicts: x, y, z

To be able to install foo, you need to:

  * ensure that a  b  c is true.
The command to do so is apt-get install a b c

  * ensure that x || y || z is false.
The command to do so is (or rather should in my opinion be)
apt-get remove x y z


To satisfy the dependency line, *all* packages in it need to be installed.

To satisfy a conflicts field (that is, there is a conflict), *any* of
the packages in it needs to be installed.


Carsten


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120217191404.gb29...@furrball.stateful.de

Re: Multiarch file overlap summary and proposal

2012-02-17 Thread David Kalnischkies

On Fri, Feb 17, 2012 at 19:53, Carsten Hey cars...@debian.org wrote:
* David Kalnischkies [2012-02-17 14:15 +0100]:
You generously left out the paragraph describing how APT should
detect that the package foo is in fact a library ...

My impression was that you think very library centric. All I wrote was
(in other words), that we should consider non-library packages as much
as library packages, and I did not write nor implied that libraries
should be handled in a different way.

Is 'apt-get remove foo+' then going to install all foo's or just one?

apt-get install g+++ is a weird syntax.

But its a syntax we support since basically ever and comes in handy
if you want to tell APT to not choose a specific alternative
(or disable specific recommends, …) without holds.
aptitude supports a few more of these modifiers (e.g. ) btw.

The above shows that it seems to depend on the availability of
foo:native if apt-get remove foo removes foo:foreign.

It's the availability of the native (or for that matter: most preferred arch)
which 'changes' this behavior. As tendra is not available for amd64 its
a pretty fair guess that i386 was meant and - as the result would be an
error otherwise - install it.
It's removed with the same command s/install/remove/.

A similar guess isn't done for removes in case the native is not installed
(but available and foreigns are installed) as it is a destructive command
(beside that it would fail the s/remove/install/ test).
See also the arguments against the
foo == foo:whatever provided that whatever is unique
in Raphaels mail in thread [0] mentioned above.

And as i said, its not only about apt-get install/remove.
It would be nice to have an approach usable for the various
commands of apt-mark and apt-cache, too.
(bonus points if it doesn't break usage with dpkg completely)

# dpkg -l | awk '$2==clang{print}'
ii clang 3.0-5 Low-Level ...

# dpkg -S bin/clang
clang: /usr/bin/clang
clang: /usr/bin/clang++

# dpkg -r clang
dpkg: warning: there's no installed package matching clang

The last one is an inconsistency in what dpkg should do.
If i understood the outcome of thread [1] above right dpkg
doesn't want to arch qualify packages for which only one package
can be meant. I personally think this is misfortune, but so it be.
APT can't do this as while you might have only one installed at a
time you can still jungle with different archs in one apt command
so we need differentiate here.

Long story short, 'apt-get remove clang' fails, yes, but you have
it installed with 'apt-get install clang:i386' so we are at least
consistent (see foo == foo:whatever).
I could have a look at printing a notice through to be nice…

# dpkg -l | grep libzookeeper-st2
ii libzookeeper-st2:amd64 3.3.4+dfsg1-3 Single ...
ii libzookeeper-st2:i386 3.3.4+dfsg1-3 Single ...

Unlike the above dpkg -l output showing the foreign clang package,
libzookeeper-st2 is shown with the architecture appended.

dpkg prefers users which are specific if they refer to a M-A:same package.
It allows no arch as ~ 'installed' arch, but mostly only for backward-
compatibility as apt/squeeze obviously doesn't know about that new
requirement.

Best regards

David Kalnischkies

--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/caaz6_fdgs4zegelof9sb8-r-nfur2zo5bcjy0_8zpqfrfyz...@mail.gmail.com

Re: Multiarch file overlap summary and proposal

2012-02-16 Thread Goswin von Brederlow

Russ Allbery r...@debian.org writes:

 David Kalnischkies kalnischk...@gmail.com writes:
 On Thu, Feb 16, 2012 at 00:39, Russ Allbery r...@debian.org wrote:

 Actually, why would that be the behavior? Â Why would dpkg --purge foo
 not just remove foo for all architectures for which it's installed, and
 require that if you want to remove only a specific architecture you
 then use the expanded syntax?

 We (as in APT team and dpkg team) had a lot of discussions about that,
 see for starters (there a properly more in between the 10 monthsâ¦)
 [0] http://lists.debian.org/debian-dpkg/2011/01/msg00046.html
 [1] http://lists.debian.org/debian-dpkg/2011/12/msg5.html

 In short, i think the biggest counter is that it feels unintuitive to
 install a library (in native arch) with e.g. apt-get install libfoo
 while you have to be specific at removal to avoid nuking 'unrelated'
 packages with apt-get remove libfoo.

 Ah, hm... I suppose that's a good point, although honestly I wouldn't mind
 having apt-get remove libfoo remove all instances of libfoo that are
 installed.  I think that would be quite reasonable behavior, and don't
 find it particularly unintuitive.

 I agree that it's asymmetric.  apt-get install libfoo means libfoo:native,
 but apt-get remove libfoo means libfoo:*.  And asymmetric is bad, all
 things being equal.  But I think this may be one place where asymmetric is
 still the right thing to do; I would argue that it means you're
 implementing the most common operation in both cases.  apt-get install
 libfoo generally means give me a native libfoo since non-native libfoo
 is going to be an unusual case, and apt-get remove libfoo generally means
 I have no more interest in libfoo, make it go away.  I think that people
 who want to get rid of one architecture of libfoo but keep the other are
 already going to be thinking about architectures, and it's natural to ask
 them to qualify their request.

In another thread we discussed the problem with plugins (e.g. input
methods for chinese/japanese) and LD_PRELOAD (e.g. fakeroot) using
stuff. For those packages it would be great if

apt-get install plugin

would install all architectures of the package (for various values of
all :). This would add asymetry in that apt-get install libfoo would
sometimes mean libfoo:native and sometimes libfoo:*. Having apt-get
install libfoo:* for anything M-A:same would make it more symetric in
that case.

apt-get install libfoo generaly means please upgrade libfoo to the
latest version. That should be apt-get upgrade libfoo which doesn't
yet exists. Libraries should be pulled in from binaries and not
installed manually so I wouldn't give that case much weight.

Instead concentrate on the more usefull cases:

apt-get install plugin binary libfoo-dev bindings-for-some-interpreter

Plugins will be M-A:same and depend on something M-A:same. They will
have some other indication (to be implemented) that they are
plugins. Libfoo-dev will be M-A:same. Binaries will be M-A:foreign.
Bindings will be M-A:same but depends on something M-A:allowed.

Now think what would be most usefull for those cases.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8739abnpz4.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

2012-02-16 Thread David Kalnischkies

On Thu, Feb 16, 2012 at 09:26, Goswin von Brederlow goswin-...@web.de wrote:
 Russ Allbery r...@debian.org writes:
 David Kalnischkies kalnischk...@gmail.com writes:
 On Thu, Feb 16, 2012 at 00:39, Russ Allbery r...@debian.org wrote:

 Actually, why would that be the behavior?  Why would dpkg --purge foo
 not just remove foo for all architectures for which it's installed, and
 require that if you want to remove only a specific architecture you
 then use the expanded syntax?

 We (as in APT team and dpkg team) had a lot of discussions about that,
 see for starters (there a properly more in between the 10 months…)
 [0] http://lists.debian.org/debian-dpkg/2011/01/msg00046.html
 [1] http://lists.debian.org/debian-dpkg/2011/12/msg5.html

 In short, i think the biggest counter is that it feels unintuitive to
 install a library (in native arch) with e.g. apt-get install libfoo
 while you have to be specific at removal to avoid nuking 'unrelated'
 packages with apt-get remove libfoo.

 Ah, hm... I suppose that's a good point, although honestly I wouldn't mind
 having apt-get remove libfoo remove all instances of libfoo that are
 installed.  I think that would be quite reasonable behavior, and don't
 find it particularly unintuitive.

 I agree that it's asymmetric.  apt-get install libfoo means libfoo:native,
 but apt-get remove libfoo means libfoo:*.  And asymmetric is bad, all
 things being equal.  But I think this may be one place where asymmetric is
 still the right thing to do; I would argue that it means you're
 implementing the most common operation in both cases.  apt-get install
 libfoo generally means give me a native libfoo since non-native libfoo
 is going to be an unusual case, and apt-get remove libfoo generally means
 I have no more interest in libfoo, make it go away.  I think that people
 who want to get rid of one architecture of libfoo but keep the other are
 already going to be thinking about architectures, and it's natural to ask
 them to qualify their request.

 In another thread we discussed the problem with plugins (e.g. input
 methods for chinese/japanese) and LD_PRELOAD (e.g. fakeroot) using
 stuff. For those packages it would be great if

    apt-get install plugin

 would install all architectures of the package (for various values of
 all :). This would add asymetry in that apt-get install libfoo would
 sometimes mean libfoo:native and sometimes libfoo:*. Having apt-get
 install libfoo:* for anything M-A:same would make it more symetric in
 that case.

 apt-get install libfoo generaly means please upgrade libfoo to the
 latest version. That should be apt-get upgrade libfoo which doesn't
 yet exists. Libraries should be pulled in from binaries and not
 installed manually so I wouldn't give that case much weight.

But M-A:same will end-up on dev-packages as well, and these are quiet
likely to be installed manually. And in the end are libraries more often
installed by hand then they are removed - think e.g. of the binary
distribution of certain applications (aka mostly games).
I need libfoo for my new amd64 game so i install it. Later i remove the
game and remember to remove libfoo with it also. I just forgot that i have
a i386 game i play from time to time which requires libfoo:i386 which is
killed by that, too. That i haven't packaged my games is misfortune, but
we are talking about real-world usage here…

Also, in some distant future we might be able to co-install binaries.
It's easy to think of M-A:same just as libraries but i personally think that
this is an unnecessary mental limitation and just exists because it is
currently the (mostly imaginative) case.

And it seems like you assume apt-get and co are only used by humans.
In fact i think it is at least equally common used in scripts, usually with -y
to e.g. remove obsolete packages. I can't wait for the resulting shitstorm…

(btw, you know that 'apt-get purge foo+' is possible, right?
 Which behavior would you expect? The same as 'apt-get install foo' ?)


The same-as thing in the plugin thread just smells like poor-man's
conditional dependency - and it's trying to solve something which isn't
solvable on that level: Just because i have armel packages installed on my
system doesn't mean that i am going to execute an armel binary.
Cross-building for example will install libc6:armel for me, but i am still
not even close to be interested in libfakeroot:armel.
To get libfakeroot:armel on the system is the responsibility of whatever tool
helps the administrator to setup an foreign armel system on his host,
which is his brain if (s)he chooses to setup it by hand with apt-get.

It's comparable with the dependency grouping for xul-applications:
The user has a variety of usecases to choose from but all these usecases
include the same apt-get command. Which usecase is the most popular
is not really measurable and even if it would it changes over time, but the
behavior of the apt-* tools is expected to be stable at the same time.

I was

Re: Multiarch file overlap summary and proposal

2012-02-16 Thread Russ Allbery

I was thinking more about this, and I was finally able to put a finger on
why I don't like package splitting as a solution.

We know from prior experience with splitting packages for large
arch-independent data that one of the more common mistakes that we'll make
is to move the wrong files: to put into the arch-independent package a
file that's actually arch-dependent.

Look at the failure mode when that happens with the sort of package that
we're talking about splitting out of m-a:same packages:

* The arch-independent package gets arch-dependent content that happens to
  match the architecture of the maintainer's build machine, since that's
  the only place the arch-independent package is built.  The maintainer
  will by definition not notice, since the content is right for their
  system.

* The maintainer is probably using a popular system type (usually either
  i386 or amd64), and everyone else on that system type will also not
  notice, so the bug can be latent for some time.

* Systems with the wrong architecture will get data files that have the
  wrong format or the wrong information.  This is usually not a case that
  the software is designed to detect, so the result is normally random
  segfaults or similar sorts of major bugs.  The failure case for header
  files is *particularly* bad: C software will generally compile fine with
  the wrong-sized data types and then, at *runtime*, happily pass the
  wrong data into the library, resulting in random segfaults and possibly
  even data corruption.  This won't happen until runtime, so could go
  undetected for long periods of time.

This is a particularly nasty failure mode due to how long it can stay
undetected and how much havoc it causes.

Now, compare to the failure mode with refcounting if the maintainer
doesn't realize that an arch-specific file can't be shared:

* Each arch-specific package will continue to get the appropriate files
  for that architecture.  Each package will still be usable and consistent
  independently, so users who don't care about multiarch won't ever see a
  problem.

* Users who want to co-install separate architectures will immediately
  encounter a dpkg error saying that the files aren't consistent.  This
  means they won't be able to co-install the packages, but dpkg will
  prevent any actual harm from happening.  The user will then report a bug
  and the maintainer will realize what happened and be able to find some
  way to fix it.

* Even better, we can automatically detect this error case by scanning the
  archive for architecture pairs that have non-matching overlapping files
  and deal with it proactively.

The refcounting failure mode behavior is just completely superior here.
And this *is* a mistake that we're going to make frequently; we know that
from past experience with splitting packages.  Note that this problem
often happens because, when the maintainer originally split the package,
there was nothing arch-specific in the file, but upstream made it
arch-specific later on and the maintainer didn't notice.  (It's very easy
to miss.)  This is particularly common with header files.

Note that arch-qualifying all of the files does not have the problems of
package splitting, but it's also a much more intrusive fix.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqdeobyu@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal

2012-02-16 Thread Carsten Hey

* David Kalnischkies [2012-02-16 03:59 +0100]:
 On Thu, Feb 16, 2012 at 00:39, Russ Allbery r...@debian.org wrote:
    it needs to find and remove foo:*

foo:all (or foo:any) instead of foo:* would save the need to quote it.

  Actually, why would that be the behavior?  Why would dpkg --purge foo not
  just remove foo for all architectures for which it's installed, and
  require that if you want to remove only a specific architecture you then
  use the expanded syntax?

 We (as in APT team and dpkg team) had a lot of discussions about that,
 see for starters (there a properly more in between the 10 months…)
 [0] http://lists.debian.org/debian-dpkg/2011/01/msg00046.html
 [1] http://lists.debian.org/debian-dpkg/2011/12/msg5.html

 In short, i think the biggest counter is that it feels unintuitive to
 install a library (in native arch) with e.g. apt-get install libfoo
 while you have to be specific at removal to avoid nuking 'unrelated' packages
 with apt-get remove libfoo.

I would expect this (especially if the package foo is not a library, but
I would also expect this for libraries):

 * apt-get install foo tries to install foo:native if possible, if it is
   not possible, it tries to install the package foo from an other
   architecture but ask before proceeding (as if additional dependencies
   are required to install a package).
 * apt-get remove foo removes all installed foo packages (on all
   architectures).


This summarises how apt without multi-arch handles this, the above would
make apt with multi-arch also behave so:

apt-get install foo
---
  foo is not installed  foo is installed
apt-get remove foo
   ---

  Note that this obviously requires that a binNMU not be considered a
  different version of the package for this purpose.  But I think that too
  makes sense.  A binNMU *isn't* a full new version of the package; it's a
  new build of the same version.  We've historically been a bit sloppy about
  this distinction, but I think it's a real distinction and a meaningful
  one.

 Mhh. The current spec just forbids binNMU for M-A:same packages -
 the 'sync' happens on the exact binary version.
 Somewhere else in this multiarch-discussion was hinted that we could
 sync on the version in (optional) Source tag instead to allow binNMU.
 It's a bit too late (in my timezone) for me to do serious predictions on
 difficult-levels on changing this in APT but i guess its relatively easy.

 (the only problem i see is that i don't have ${source:Version} available
  currently in the version structure, but we haven't even tried pushing
  apt's abibreak to sid specifically as i feared last-minute changes…)

I'm not sure if you meant this with Source tag, anyway, the 'Packages'
files miss the source version too, but this could be added as optional
field that would be used if it differs from the 'Version:' field.


Regards
Carsten


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120216221059.ga8...@furrball.stateful.de

Re: Multiarch file overlap summary and proposal

2012-02-16 Thread Carsten Hey

* Russ Allbery [2012-02-16 10:43 -0800]:
 * Users who want to co-install separate architectures will immediately
   encounter a dpkg error saying that the files aren't consistent.  This
   means they won't be able to co-install the packages, but dpkg will
   prevent any actual harm from happening.  The user will then report a bug
   and the maintainer will realize what happened and be able to find some
   way to fix it.

 * Even better, we can automatically detect this error case by scanning the
   archive for architecture pairs that have non-matching overlapping files
   and deal with it proactively.

There are still files that differ that do not need to be fixed, for
example documentation that contains it's build date.

One way to address this is to use a new dpkg control file (placed in
/var/lib/dpkg/info) that lists files that dpkg considers to be equal
even if they differ.


Carsten


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120216224340.gb8...@furrball.stateful.de

Re: Multiarch file overlap summary and proposal

2012-02-16 Thread Russ Allbery

Carsten Hey cars...@debian.org writes:
 * Russ Allbery [2012-02-16 10:43 -0800]:

 * Users who want to co-install separate architectures will immediately
   encounter a dpkg error saying that the files aren't consistent.  This
   means they won't be able to co-install the packages, but dpkg will
   prevent any actual harm from happening.  The user will then report a bug
   and the maintainer will realize what happened and be able to find some
   way to fix it.

 * Even better, we can automatically detect this error case by scanning the
   archive for architecture pairs that have non-matching overlapping files
   and deal with it proactively.

 There are still files that differ that do not need to be fixed, for
 example documentation that contains it's build date.

Every file that differs has to be fixed in the current multi-arch plan.
Documentation that contains its build date is going to need to be split
out into a separate -docs package.

I'm fine with splitting documentation; that has far fewer problems than
splitting other types of files, since documentation isn't tightly coupled
at a level that breaks software.

 One way to address this is to use a new dpkg control file (placed in
 /var/lib/dpkg/info) that lists files that dpkg considers to be equal
 even if they differ.

I don't think this is a good idea.  I don't think we should allow this
sort of inconsistency depending on what package is installed first.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d39eie26@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal

2012-02-15 Thread Goswin von Brederlow

Ian Jackson ijack...@chiark.greenend.org.uk writes:

 Russ Allbery writes (Multiarch file overlap summary and proposal (was: 
 Summary: dpkg shared / reference counted files and version match)):
 5. Data files that vary by architecture.  This includes big-endian
vs. little-endian issues.  These are simply incompatible with
multiarch as currently designed, and incompatible with the obvious
variations that I can think of, and will have to either be moved
into arch-qualified directories (with corresponding patches to the
paths from which the libraries load the data) or these packages
can't be made multiarch.

 Yes.  Of these, arch-qualifying the path seem to be to be obviously
 the right answer.  Of course eg if the data files just come in big-
 and little-endian, you can qualify the path with only the endianness
 and use refcounting to allow the equal-endianness packages to share.

 Ian.

Preferably -data-be:all and -data-le:all packages if they can be build
irespective of the buildds endianness.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zkckrwnl.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

2012-02-15 Thread Goswin von Brederlow

Ian Jackson ijack...@chiark.greenend.org.uk writes:

 Guillem Jover writes (Re: Multiarch file overlap summary and proposal (was: 
 Summary: dpkg shared / reference counted files and version match)):
 This still does not solve the other issues I listed, namely binNMUs
 have to be performed in lock-step, more complicated transitions /
 upgrades.

 I don't think I see where this is coming from.  Are you talking about
 variation in gzip output ?  Given the evidence we've seen here, in
 practice I think that is not going to be a problem.  Certainly it
 won't demand that binNMUs be performed in lock-step.

Note that splitting files (specifically changelog) into -common package
would require an explicit versioned dependency on the -common package and
produce the same (or similar) lock-step problem for upgrades and
binNMUs. Arch qualifying the files on the other hand would avoid that.

Splitting data files into -common packages will also often need a close
versioned dependency forcing a lock-step of packages. But probably not
so terse that binNMUs would have to be lock-steped.

Overall I think the lock-step being required for reference counted files
won't have such a large effect as you might think.



I think the idea of splitting the binNMU changelog into an extra file is
a great idea as that would allow putting the changelog into -common:all
and depend on the source version and then have the binNMU changelog in
the foo:any package in the symlinked directory. For this to work the
binNMU changelog should be arch and pkg qualified, e.g.

/usr/share/doc/foo-common/Changelog
/usr/share/doc/foo-common/Changelog.binNMU-foo-amd64
/usr/share/doc/foo-common/Changelog.binNMU-bar-i386
/usr/share/doc/foo - foo-common
/usr/share/doc/bar - foo-common 

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87vcn8rw1z.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

2012-02-15 Thread Goswin von Brederlow

Russ Allbery r...@debian.org writes:

 Joey Hess jo...@debian.org writes:

 Anyway, my worry about the refcounting approach (or perhaps M-A: same in
 general) is not the details of the implementation in dpkg, but the added
 mental complexity of dpkg now being able to have multiple distinct
 packages installed under the same name. I had a brief exposure to rpm,
 which can install multiple versions of the same package, and that was
 the main cause of much confusing behavior in rpm. While dpkg's invariant
 that all co-installable package names be unique (and have unique files)
 has certianly led to lots of ugly package names, it's kept the users'
 and developers' mental models quite simple.

 I worry that we have barely begun to scratch the surface of the added
 complexity of losing this invariant.

 This does seem to be more M-A: same in general, to me, since whether we
 have file overlaps or not we still have multiple packages with the same
 name.  Which will force changes in everything that deals with packages,
 like Puppet, to be able to specify packages with particular architectures.

 I definitely agree on the complexity this adds.  But I don't think there's
 an alternative to that complexity without using something like --sysroot
 or mini-chroots, and I don't think those are satisfying solutions to the
 set of problems we're trying to solve.

pkg:arch will still be unique and the dpkg/apt output will use the
architecture where required for uniqueness. So I think that after some
getting used to it it will be clear enough again.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r4xwrvw2.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

2012-02-15 Thread Joey Hess

Goswin von Brederlow wrote:
 pkg:arch will still be unique and the dpkg/apt output will use the
 architecture where required for uniqueness. So I think that after some
 getting used to it it will be clear enough again.

Here are a few examples of the problems I worry about. I have not
verified any of them, and they're clearly biased toward code I am
familiar with, which suggests there are many other similar problems.

* Puppet not only installs packages, it may remove them. A puppet config
  that does dpkg --purge foo will fail if multiarch is enabled, now
  it needs to find and remove foo:*

* dpkg-repack pkg:arch will create a package with that literal name (or fail)

* dpkg-reconfigure probably can't be used with M-A same packages.
  debconf probably generally needs porting to multiarch.

* tasksel uses dpkg --query to work out if a task's dependencies are
  installed. In the event that a library is directly part of a task,
  this will fail when multiarch is enabled.

* Every piece of documentation that gives commands lines manipulating
  library packages is potentially broken.

Seems like we need a release where multiarch is classed as an
experimental feature, which when enabled can break the system. But the
sort of problems above are the easy to anticipate ones; my real worry is
the unanticipated classes of problems. Especially if we find intractable
problems or levels of complexity introduced by dropping the unique
package name invariant.

My nightmare scenario is that we release with multiarch, discover that
it's a net negative for our users (proprietary software on amd64 aside,
nearly all the benefits are to developers AFAICS), and are stuck with it.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

2012-02-15 Thread Ian Jackson

Guillem Jover writes (Re: Multiarch file overlap summary and proposal (was: 
Summary: dpkg shared / reference counted files and version match)):
 On Tue, 2012-02-14 at 14:28:58 +, Ian Jackson wrote:
  I think the refcounting approach is very worthwhile because it
  eliminates unnecessary work (by human maintainers) in many simple
  cases.
 
 Aside from what I said on my other reply, I just wanted to note that
 this seems to be a recurring point of tension in the project when it
 comes to archive wide source package changes, where supposed short
 term convenience (with its usually long term harmful effects) appears
 to initially seduce people over what seems to be the cleaner although
 slightly a bit more laborious solution.

The refcnt doesn't just eliminate unnecessary multiarch
conversion work.  It also eliminates unnecessary maintenance effort.
Maintaining a split package will be more work than without.

I think that over the lifetime of the multiarch deployment this extra
packaging work will far outweigh the extra maintenance and
documentation burden of the refcnt feature.

  [...]  But trying to workaround this by coming
 up with stacks of hacked up solutions  [...]

I disagree with your tendentious phrasing.  The refcnt feature is not
a hacked up solution (nor a stack of them).  It is entirely normal
in Debian core tools (as in any substantial piece of software serving
a lot of diverse needs) to have extra code to make it easier to deploy
or use in common cases simpler.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20283.57393.237949.649...@chiark.greenend.org.uk

Re: Multiarch file overlap summary and proposal

2012-02-15 Thread Ian Jackson

Russ Allbery writes (Re: Multiarch file overlap summary and proposal):
 I definitely agree on the complexity this adds.  But I don't think there's
 an alternative to that complexity without using something like --sysroot
 or mini-chroots, and I don't think those are satisfying solutions to the
 set of problems we're trying to solve.

In principle we could have added the arch name to the package name of
every library, the same way we add (part of) the version number to the
package name.

I think the current multiarch proposal is a better idea than that.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20283.57580.777378.472...@chiark.greenend.org.uk

Re: Multiarch file overlap summary and proposal

2012-02-15 Thread Ian Jackson

Joey Hess writes (Re: Multiarch file overlap summary and proposal):
 Goswin von Brederlow wrote:
  pkg:arch will still be unique and the dpkg/apt output will use the
  architecture where required for uniqueness. So I think that after some
  getting used to it it will be clear enough again.
 
 Here are a few examples of the problems I worry about. I have not
 verified any of them, and they're clearly biased toward code I am
 familiar with, which suggests there are many other similar problems.
 
 * Puppet not only installs packages, it may remove them. A puppet config
   that does dpkg --purge foo will fail if multiarch is enabled, now
   it needs to find and remove foo:*

Yes.

 * dpkg-repack pkg:arch will create a package with that literal name (or fail)

This is of course also a (new) bug.  There will be a lot of bugs like
this while we deploy multiarch.

 * dpkg-reconfigure probably can't be used with M-A same packages.
   debconf probably generally needs porting to multiarch.

I think the non-arch-qualified nature of debconf question ids is
probably a feature rather than a bug.  These questions should be
arch-qualified by the package only if they need to be.

 [other examples]

Yes.

 Seems like we need a release where multiarch is classed as an
 experimental feature, which when enabled can break the system.

Certainly.  That's what wheezy will be.

  But the sort of problems above are the easy to anticipate ones; my
 real worry is the unanticipated classes of problems. Especially if
 we find intractable problems or levels of complexity introduced by
 dropping the unique package name invariant.

I think the fact that (package name, architecture) is still unique on
the system will be sufficient to make it possible to update all
existing software to deal with the new model.

 My nightmare scenario is that we release with multiarch, discover that
 it's a net negative for our users (proprietary software on amd64 aside,
 nearly all the benefits are to developers AFAICS), and are stuck with it.

I don't think this is likely.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20283.57768.769487.440...@chiark.greenend.org.uk

Re: Multiarch file overlap summary and proposal

Ian Jackson ijack...@chiark.greenend.org.uk writes:
 Joey Hess writes (Re: Multiarch file overlap summary and proposal):

 Here are a few examples of the problems I worry about. I have not
 verified any of them, and they're clearly biased toward code I am
 familiar with, which suggests there are many other similar problems.

 * Puppet not only installs packages, it may remove them. A puppet config
   that does dpkg --purge foo will fail if multiarch is enabled, now
   it needs to find and remove foo:*

 Yes.

Actually, why would that be the behavior?  Why would dpkg --purge foo not
just remove foo for all architectures for which it's installed, and
require that if you want to remove only a specific architecture you then
use the expanded syntax?

This comes back to something that I started thinking about after Guillem's
message, namely that I think the version lockstep is a feature, not a
limitation.  A lot of the complexity from biarch with RPM comes from the
fact that the packages are (as I recall) independent and can be of
different versions.  This sounds like a nice feature, but I think it's
actually a huge source of potential confusion and isn't as useful as it
may look.

I think it would be better to have a world in which all the architectures
of the foo package on the system have to have the same version, because
then you don't have to treat foo:i386 and foo:amd64 like they're separate
packages.  The list of installed architectures is an *attribute* of the
package.  A package is still either installed or not installed, but when
it's installed, it can be installed for one or more architectures.  But if
it's installed for multiple architectures, those architectures are always
upgraded together and always remain consistent.  That avoids all weirdness
of having a package work differently because the version varies depending
on the ABI, and it significantly simplifies the mental model behind the
package.

Note that this obviously requires that a binNMU not be considered a
different version of the package for this purpose.  But I think that too
makes sense.  A binNMU *isn't* a full new version of the package; it's a
new build of the same version.  We've historically been a bit sloppy about
this distinction, but I think it's a real distinction and a meaningful
one.

I have not looked in detail at the current multiarch implementations and I
have no idea how closely the above matches what is actually being done,
but I would advocate for it, even to the degree of embedding understanding
of binNMU versioning in the package tools so that they know a binNMU is a
new version from the perspective of needing to upgrade the package but not
a new version from the perspective of version mismatches between arches.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ehtv650n@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal

Guillem Jover guil...@debian.org writes:

 If packages have to be split anyway to cope with the other cases, then
 the number of new packages which might not be needed otherwise will be
 even smaller than the predicted amount, at which point it makes even
 less sense to support refcnt'ing.

I don't think the package count is really the interesting metric here,
unless the number of introduced packages is very large (see below about
-dev packages).  I'm more concerned with maintainer time and with
dependency complexity, and with the known problems that we introduce
whenever we take tightly-coupled files and separate them into independent
packages.

 It also requires maintainers to carefully consider if the (doc, etc)
 toolchains will generate predictible ouput.

Yes, I agree.  There's a tradeoff between having to think about this and
having to do the work to use arch-qualified directories.  But I think it's
worth having the tradeoff available as an option.

 This still does not solve the other issues I listed, namely binNMUs have
 to be performed in lock-step, more complicated transitions /
 upgrades. And introduces different solutions for different problems,
 while my proposal is generic for all cases.

I did review your message again (I read it when you originally posted it
as well), and I think that I addressed the cases that you pointed out in
the set of cases that I discussed in my message apart from the point about
version lockstep.

I just posted separately about version lockstep: I think this is a
feature, not a bug, in our multiarch implementation.  I think this is the
direction we *should* go, because it reduces the overall complexity of the
system.  Yes, that requires treating binNMUs as something different than a
sourceful version change, but I think that's a good idea *anyway*, even in
the absence of multiarch.  It's more semantically accurate, and it would
address other lingering problems we've had with binNMUs, or at least
reduce them.

As for the benefits of refcounting, there are three places where I think
this has substantial benefit, so let me talk through them:

1. If you look at the list of files that Steve gave in multiarch: same
   packages in Ubuntu, most of the cases that don't fall into the known
   documentation and package metadata areas are a bunch of separate
   special cases.  They don't fall easily into a handful of cases.  *But*,
   they are mostly all files in either textual or consistent binary
   formats that are installed directly from the package and are not going
   to change in a binNMU.  That means that refcounting provides a nice
   simplification: there are a bunch of random additional files of various
   different types that can all be handled the same way, without
   additional packaging complexity.  They can't all be arch-qualified in
   the same way as easily, plus arch-qualifying files that absolutely
   should not differ between architectures and where that difference would
   be a bug (such as with PAM configuration) seems wrong.

   They can also be split out into an arch: all package.  But here I think
   it's worth remembering that splitting tightly-coupled files into
   separate packages has real drawbacks and is something we should avoid
   doing if we can.  There are places where the advantages to doing so are
   overwhelming (-dev packages from shared libraries, for example), but we
   should be sure we're in that case.

   This is something that working on Lintian for a while really drove home
   for me.  People split binary packages with large data into an arch: any
   and arch: all package (often because Lintian recommends it to save
   archive space), and they do it wrong *all the time*.  They get the
   dependencies wrong, or don't think about what files belong in which
   package, or accidentally put an arch-specific file in the data package.
   I have a package that does this sort of split (gnubg), and from
   personal experience know that it's not an easy thing to maintain.
   We're not saving packaging complexity by asking people to do this
   instead of refcounting, IMO.

   Also, there are other drawbacks of splitting closely coupled files into
   separate packages even apart from packaging complexity.  For example,
   it's common for people to move the man pages and desktop files for
   binaries into that arch: all data package too, since hey they're
   arch-independent and in /usr/share and that's easy.  But this is
   usually the wrong thing to do.  Now you have created the possibility of
   having desktop files installed for binaries that don't exist, you've
   made it much harder for tools like Lintian to check that your man pages
   and desktop files are consistent with the binaries, and you have to be
   very careful about versioning dependencies.  You also create a separate
   package that's artificial from the user's perspective, may not get
   removed when the main package is removed, and shows up in apt-cache
   search and

Re: Multiarch file overlap summary and proposal

2012-02-15 Thread David Kalnischkies

On Thu, Feb 16, 2012 at 00:39, Russ Allbery r...@debian.org wrote:
 Ian Jackson ijack...@chiark.greenend.org.uk writes:
 Joey Hess writes (Re: Multiarch file overlap summary and proposal):

 Here are a few examples of the problems I worry about. I have not
 verified any of them, and they're clearly biased toward code I am
 familiar with, which suggests there are many other similar problems.

 * Puppet not only installs packages, it may remove them. A puppet config
   that does dpkg --purge foo will fail if multiarch is enabled, now
   it needs to find and remove foo:*

 Yes.

 Actually, why would that be the behavior?  Why would dpkg --purge foo not
 just remove foo for all architectures for which it's installed, and
 require that if you want to remove only a specific architecture you then
 use the expanded syntax?

We (as in APT team and dpkg team) had a lot of discussions about that,
see for starters (there a properly more in between the 10 months…)
[0] http://lists.debian.org/debian-dpkg/2011/01/msg00046.html
[1] http://lists.debian.org/debian-dpkg/2011/12/msg5.html

In short, i think the biggest counter is that it feels unintuitive to
install a library (in native arch) with e.g. apt-get install libfoo
while you have to be specific at removal to avoid nuking 'unrelated' packages
with apt-get remove libfoo.

You can counter this with: Then be specific while installing, but this not
only breaks backward-compatibility (especially dist-upgrades), it also adds
either unnecessary work for single-arch users or it feels strange to accept
a command in singlearch while it fails in multiarch…
(and yes, i know that i have 'lost' the 'battle' in [1], so the handling is
 different in APT vs dpkg unfortunately, but that's life it seems…)


Maybe it would help this kind of discussions if we would have a list of
interface changes in dpkg and how someone is supposed to use it in the
future in case this has changed - i personally lost track of that.
(In case someone wants to know this for APT: foo is interpreted as foo:native.
 If arch of foo is not native, the packagename is display as foo:arch.
 So at least in my eyes brain-death simple - and backward compatible.
 [and no, foo:* is currently not supported, but its on the todo])


 I think it would be better to have a world in which all the architectures
 of the foo package on the system have to have the same version, because
 then you don't have to treat foo:i386 and foo:amd64 like they're separate
 packages.  The list of installed architectures is an *attribute* of the
 package.  A package is still either installed or not installed, but when
 it's installed, it can be installed for one or more architectures.  But if
 it's installed for multiple architectures, those architectures are always
 upgraded together and always remain consistent.  That avoids all weirdness
 of having a package work differently because the version varies depending
 on the ABI, and it significantly simplifies the mental model behind the
 package.

This is the case for M-A:same packages currently.
libfoo:i386 v1 and libfoo:amd64 v2 are not supposed to be co-exist on a
system and as you describe it: its a feature in my eyes, too.
If you want co-installable different versions, call the packages libfoo1 and
libfoo2…

stable/testing users will not have a problem with this version-lock anyway
and unstable users should be able to deal with temporary uninstallable
packages caused by packages not built for all archs yet.
Its not exactly a new type of problem for them anyway…
(btw: doesn't britney do these lock-step upgrades all day long…)

(Note though that e.g. APT is not able to handle installed architectures as an
 'attribute'. It not only has to handle them as 'different' packages (and more
 specific different versions) to keep backward-compatibility, also different
 dependencies on different architectures would make it pretty messy in
 practice. But double-think is a requirement for APT development anyway. ;) )


 Note that this obviously requires that a binNMU not be considered a
 different version of the package for this purpose.  But I think that too
 makes sense.  A binNMU *isn't* a full new version of the package; it's a
 new build of the same version.  We've historically been a bit sloppy about
 this distinction, but I think it's a real distinction and a meaningful
 one.

Mhh. The current spec just forbids binNMU for M-A:same packages -
the 'sync' happens on the exact binary version.
Somewhere else in this multiarch-discussion was hinted that we could
sync on the version in (optional) Source tag instead to allow binNMU.
It's a bit too late (in my timezone) for me to do serious predictions on
difficult-levels on changing this in APT but i guess its relatively easy.
(the only problem i see is that i don't have ${source:Version} available
 currently in the version structure, but we haven't even tried pushing
 apt's abibreak to sid specifically as i feared last-minute changes…)
The question

Re: Multiarch file overlap summary and proposal

David Kalnischkies kalnischk...@gmail.com writes:
 On Thu, Feb 16, 2012 at 00:39, Russ Allbery r...@debian.org wrote:

 Actually, why would that be the behavior?  Why would dpkg --purge foo
 not just remove foo for all architectures for which it's installed, and
 require that if you want to remove only a specific architecture you
 then use the expanded syntax?

 We (as in APT team and dpkg team) had a lot of discussions about that,
 see for starters (there a properly more in between the 10 months…)
 [0] http://lists.debian.org/debian-dpkg/2011/01/msg00046.html
 [1] http://lists.debian.org/debian-dpkg/2011/12/msg5.html

 In short, i think the biggest counter is that it feels unintuitive to
 install a library (in native arch) with e.g. apt-get install libfoo
 while you have to be specific at removal to avoid nuking 'unrelated'
 packages with apt-get remove libfoo.

Ah, hm... I suppose that's a good point, although honestly I wouldn't mind
having apt-get remove libfoo remove all instances of libfoo that are
installed.  I think that would be quite reasonable behavior, and don't
find it particularly unintuitive.

I agree that it's asymmetric.  apt-get install libfoo means libfoo:native,
but apt-get remove libfoo means libfoo:*.  And asymmetric is bad, all
things being equal.  But I think this may be one place where asymmetric is
still the right thing to do; I would argue that it means you're
implementing the most common operation in both cases.  apt-get install
libfoo generally means give me a native libfoo since non-native libfoo
is going to be an unusual case, and apt-get remove libfoo generally means
I have no more interest in libfoo, make it go away.  I think that people
who want to get rid of one architecture of libfoo but keep the other are
already going to be thinking about architectures, and it's natural to ask
them to qualify their request.

If removing the non-native architecture has cascading effects, apt is
obviously going to warn them about that already and they'll realize what's
going on.

 You can counter this with: Then be specific while installing,

No, I agree that preserving apt-get install libfoo as install a native
libfoo is important.

 Maybe it would help this kind of discussions if we would have a list of
 interface changes in dpkg and how someone is supposed to use it in the
 future in case this has changed - i personally lost track of that.
 (In case someone wants to know this for APT: foo is interpreted as foo:native.
  If arch of foo is not native, the packagename is display as foo:arch.
  So at least in my eyes brain-death simple - and backward compatible.
  [and no, foo:* is currently not supported, but its on the todo])

Yes, one drawback of my approach is that you use that very simple rule.

 (Note though that e.g. APT is not able to handle installed architectures
 as an 'attribute'. It not only has to handle them as 'different'
 packages (and more specific different versions) to keep
 backward-compatibility, also different dependencies on different
 architectures would make it pretty messy in practice. But double-think
 is a requirement for APT development anyway. ;) )

Yes, definitely the internals of our package management software can't
fully compress the packages together; at the least, the dependencies are
going to be different between architectures and have to be stored
separately.  And I think we're going to want binNMUs in the long run as
well; they're just too helpful for library transitions.

But I think what we should be telling the *user*, regardless of our
internals, is don't think of libfoo:i386 and libfoo:amd64 as two separate
packages that you can maintain independently; think of libfoo as being
installed for one or more architectures.

 Mhh. The current spec just forbids binNMU for M-A:same packages -
 the 'sync' happens on the exact binary version.
 Somewhere else in this multiarch-discussion was hinted that we could
 sync on the version in (optional) Source tag instead to allow binNMU.

I think that the best long-term way to handle binNMUs may be to move the
build number into a different piece of package metadata from the version.
So a binNMU of a package with version 1.4-1 would still have version 1.4-1
but would have a build number of 2 instead of 1.  I think this would be
way cleaner in the long run, and not just for multiarch.

A lot of stuff would have to change to get to there from here, though.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87k43n1mkh@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal

Russ Allbery r...@debian.org writes:
 David Kalnischkies kalnischk...@gmail.com writes:

 Maybe it would help this kind of discussions if we would have a list of
 interface changes in dpkg and how someone is supposed to use it in the
 future in case this has changed - i personally lost track of that.  (In
 case someone wants to know this for APT: foo is interpreted as
 foo:native.  If arch of foo is not native, the packagename is display
 as foo:arch.  So at least in my eyes brain-death simple - and backward
 compatible.  [and no, foo:* is currently not supported, but its on the
 todo])

 Yes, one drawback of my approach is that you use that very simple rule.

Ack, sorry, that was supposed to be that you *lose* that very simple
rule.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d39f1mfj@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal

2012-02-14 Thread Andreas Beckmann

On 2012-02-14 07:43, Russ Allbery wrote:

 4. Lintian overrides.  I believe these should be qualified with the
architecture on any multiarch: same package so that the overrides can
vary by architecture, since this is a semi-frequent use case for
Lintian.

 * Lintian should recognize arch-qualified override files, and multiarch:
   same packages must arch-qualify their override files.  debhelper
   assistance is desired for this.

 Please note that this is a bunch of work.  I think the Lintian work is a
 good idea regardless, and it can start independently.  I think the same is

Lintian (2.5.1) already supports arch qualification inside the override file:

+ [NT] Allow simple architecture overrides such as [i386] or [!i386].
  Architecture wildcards are not supported yet.  The actual syntax
  for the architecture the same as the one used in the Build-Depends.
  Thanks to Andreas Beckmann for the bug report and the solution
  proposal.  (Closes: #622888)

This is already used for nvidia-graphics-drivers etc.


Andreas


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4f3a463e.9040...@abeckmann.de

Re: Multiarch file overlap summary and proposal

2012-02-14 Thread Niels Thykier

On 2012-02-14 07:43, Russ Allbery wrote:
 [...]
 
 * Lintian should recognize arch-qualified override files, and multiarch:
   same packages must arch-qualify their override files.  debhelper
   assistance is desired for this.
 
 [...]

I have no problem with Lintian accepting arch-qualified override files,
but I do not see the strict (i.e. must) requirement[1].  Lintian
already allows you to do arch-specific overrides and 2.5.5 will even
allow architecture wildcards as well[2].


~Niels

[1] Exception being compressed override files, but of the 161 override
files on my system not a single one of them are compressed.

[2] http://lintian.debian.org/manual/section-2.4.html#section-2.4.1

Admittedly the link above does not have an example with it, but Lintian
(git) has an entire section dedicated to architecture specific
overrules.  So there will be several examples with next release.  :)


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4f3a4888@thykier.net

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

On Tue, 14 Feb 2012, Philipp Kern wrote:
 On 2012-02-14, Raphael Hertzog hert...@debian.org wrote:
  Somehow my suggestion is then to extend dpkg-parsechangelog to provide
  the required logic to split the changelog in its bin-nmu part and its
  usual content.
 
  dpkg-parsechangelog --split-binnmu binnmu-part-file remaining-part-file
 
  Then dh_installchangelogs could try to use this (and if it fails, fallback
  to the standard changelog installation).
 
  Does that sound sane? If yes, I can have a look at implementing this.
 
 In theory sbuild could also offload this to dpkg-buildpackage by passing
 something like --binnmu-version 2 --binnmu-changelog 'Rebuild for libfoo
 transition'.  The only thing that would be annoying is checking if the old
 style or the new style must be used.  (I.e. there must be some sort of feature
 query first.)

Yes but that doesn't change anything to the fact that dpkg-dev should not
install files in the generated .deb. So we still need some interaction
with dh_installchangelogs... but your suggestion lead me to another
proposal.

dpkg-buildpackage --binary-version ver --binary-changelog 'foo'
could create debian/changelog.build with the given changelog version and
changelog entry.

dpkg-parsechangelog could be taught to read debian/changelog.build
before debian/changelog so that dpkg-parsechangelog continues to do the
right thing (when called from debian/rules).

And dh_installchangelogs can be taught to install debian/changelog.build
as /usr/share/doc/foo/changelog.Debian.build-$arch.

dpkg-buildpackage would clean up debian/changelog.build if it wasn't
passed the proper option. dpkg-source would learn to not include it in
generated source packages, too.

This looks like rather appealing to me. What do you think?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120214131720.gd11...@rivendell.home.ouaza.com

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

2012-02-14 Thread Guillem Jover

On Mon, 2012-02-13 at 22:43:04 -0800, Russ Allbery wrote:
 If this is comprehensive, then I propose the following path forward, which
 is a mix of the various solutions that have been discussed:

 * dpkg re-adds the refcounting implementation for multiarch, but along
   with a Policy requirement that packages that are multiarch must only
   contain files in classes 1 and 2 above.

 * All packages that want to be multiarch: same have to move all generated
   documentation into a separate package unless the maintainer has very
   carefully checked that the generated documentation will be byte-for-byte
   identical even across minor updates of the documentation generation
   tools and when run at different times.

If packages have to be split anyway to cope with the other cases, then
the number of new packages which might not be needed otherwise will be
even smaller than the predicted amount, at which point it makes even
less sense to support refcnt'ing.

It also requires maintainers to carefully consider if the (doc, etc)
toolchains will generate predictible ouput.

Your proposal still requires papering over the other corner-cases.

 * Policy prohibits arch-varying data files in multiarch: same packages
   except in arch-qualified paths.

Well, there's no escape from this any way you look at it, regardless of
refcnt'ing or not.

 * The binNMU process is changed to add the binNMU changelog entry to an
   arch-qualified file (changelog.Debian.arch, probably).  We need to
   figure out what this means if the package being binNMU'd has a
   /usr/share/doc/package symlink to another package, though; it's not
   obvious what to do here.

This requires IMO multitude of hacks when the simplest and obvious
arch-qualified pkgname solves this cleanly, and allows debhelper to
automatically deal with it. And for tools to just change where they
always look for those files in the M-A:same case regardless of the
package being binNMUed or not.

This still does not solve the other issues I listed, namely binNMUs
have to be performed in lock-step, more complicated transitions /
upgrades. And introduces different solutions for different problems,
while my proposal is generic for all cases.

So this is still pretty much unconvincing, and seems like clinging
into the refcnt'ing “solution” while it makes things overall more
complicated, will introduce inconsistency and incertainty to
maintainers, needs way more global changes to keep it going, etc.

What I'd change to my proposal in the summary mail, is that arch-indep
files might be considered for splitting at maintainers discretion,
when it actually seems worth it, in the same way we've handled
splitting arch-indep files from arch:any up to now. So for example a
couple of headers could be kept on the -dev package, or Ian's case on
essential and data files could also be kept on the same lib package,
as long as their paths are arch-qualified either trhough a pkgname:arch
or the multiarch triplet. This would reduce even more the amount of
newly split packages.

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120214140138.ga23...@gaara.hadrons.org

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

2012-02-14 Thread Ian Jackson

Russ Allbery writes (Multiarch file overlap summary and proposal (was: 
Summary: dpkg shared / reference counted files and version match)):
 There's been a lot of discussion of this, but it seems to have been fairly
 inconclusive.  We need to decide what we're doing, if anything, for wheezy
 fairly soon, so I think we need to try to drive this discussion to some
 concrete conclusions.

Yes.

 If this is comprehensive, then I propose the following path forward, which
 is a mix of the various solutions that have been discussed:

Thanks for this summary and analysis.  I agree with your conclusions.

 Does this seem comprehensive to everyone?  Am I missing any cases?

I think you have covered all of the cases that have been brought up on
this list, which I think are all of the important and frequent cases.

Thinking about other corner cases can be deferred for now because we
can put off converting affected packages until wheezy+1, or if we
really want to convert we can very probably add a common package.
So let us press on.

 * The binNMU process is changed to add the binNMU changelog entry to an
   arch-qualified file (changelog.Debian.arch, probably).  We need to
   figure out what this means if the package being binNMU'd has a
   /usr/share/doc/package symlink to another package, though; it's not
   obvious what to do here.

If we always put the binNMU changelog file in
 /usr/share/doc/package/changelog.Debian.package:arch
then in the symlink case we can put it file in 
 /usr/share/doc/symlink-target/changelog.Debian.original-package:arch
and everything will work (apart from the fact that some minority of
changelog-reading tools will need to be taught to look at the new path).

 Please note that this is a bunch of work.  I think the Lintian work is a
 good idea regardless, and it can start independently.  I think the same is
 true of the binNMU changelog work, since this will address some
 long-standing issues with changelog handling in some situations, including
 resolving just how we're supposed to handle /usr/share/doc symlinks.  But
 even with those aside, this is a lot of stuff that we need to agree on,
 and in some cases implement, in a fairly short timeline if this is going
 to make wheezy.

Yes.  The work that absolutely needs to be done ASAP seems to be:
 - put the refcounting back in dpkg
 - lintian support for arch-qualified overrides
 - update the binNMU machinery to write the new changelog file instead

Things that should be done but are not on the critical paths:
 - transpose the restrictions on use of refcounting into policy
   (for now they can go in a text file in dpkg-dev, or even just
   a reference to your email)
 - update changelog-reading tools to look for binNMU changelogs too

Things which we can do at our leisure:
 - convert individual libraries
 - think about whether to always arch-qualify the whole changelog
 - think other refcounting corner cases (see my comments above)

 5. Data files that vary by architecture.  This includes big-endian
vs. little-endian issues.  These are simply incompatible with
multiarch as currently designed, and incompatible with the obvious
variations that I can think of, and will have to either be moved
into arch-qualified directories (with corresponding patches to the
paths from which the libraries load the data) or these packages
can't be made multiarch.

Yes.  Of these, arch-qualifying the path seem to be to be obviously
the right answer.  Of course eg if the data files just come in big-
and little-endian, you can qualify the path with only the endianness
and use refcounting to allow the equal-endianness packages to share.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20282.26861.126288.496...@chiark.greenend.org.uk

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

2012-02-14 Thread Josselin Mouette

Le lundi 13 février 2012 à 22:43 -0800, Russ Allbery a écrit : 
 There's been a lot of discussion of this, but it seems to have been fairly
 inconclusive.  We need to decide what we're doing, if anything, for wheezy
 fairly soon, so I think we need to try to drive this discussion to some
 concrete conclusions.

Thank you very much for your constructive work.

 3. Generated documentation.  Here's where I think refcounting starts
failing.

So we need to move a lot of documentation generated with gtk-doc or
doxygen from -dev packages to -doc packages. But it really seems an
acceptable tradeoff between the amount of work required and the
cleanness of the solution.

 Does this seem comprehensive to everyone?  Am I missing any cases?

Are there any cases of configuration files in /etc that vary across
architectures? Think of stuff like ld.so.conf, where some plugins or
library path is coded in a configuration file.

-- 
 .''`.  Josselin Mouette
: :' :
`. `'
  `-


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1329230441.3297.378.camel@pi0307572

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

2012-02-14 Thread Jakub Wilk


* Raphael Hertzog hert...@debian.org, 2012-02-14, 14:17:
dpkg-buildpackage --binary-version ver --binary-changelog 'foo' could 
create debian/changelog.build with the given changelog version and 
changelog entry.


dpkg-parsechangelog could be taught to read debian/changelog.build 
before debian/changelog so that dpkg-parsechangelog continues to do the 
right thing (when called from debian/rules).


And dh_installchangelogs can be taught to install 
debian/changelog.build as 
/usr/share/doc/foo/changelog.Debian.build-$arch.


dpkg-buildpackage would clean up debian/changelog.build if it wasn't 
passed the proper option. dpkg-source would learn to not include it in 
generated source packages, too.


This looks like rather appealing to me. What do you think?


Yes, it does look appealing. But...

Are we sure than no existing package uses debian/changelog.build for 
their own purposes?


Are we sure that all existing packages (and helpers) that parse 
debian/changelog use dpkg-parsechangelog?


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120214144341.ga3...@jwilk.net

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

2012-02-14 Thread Ian Jackson

Guillem Jover writes (Re: Multiarch file overlap summary and proposal (was: 
Summary: dpkg shared / reference counted files and version match)):
 On Mon, 2012-02-13 at 22:43:04 -0800, Russ Allbery wrote:
  * The binNMU process is changed to add the binNMU changelog entry to an
arch-qualified file (changelog.Debian.arch, probably).  We need to
figure out what this means if the package being binNMU'd has a
/usr/share/doc/package symlink to another package, though; it's not
obvious what to do here.
 
 This requires IMO multitude of hacks when the simplest and obvious
 arch-qualified pkgname solves this cleanly, and allows debhelper to
 automatically deal with it. And for tools to just change where they
 always look for those files in the M-A:same case regardless of the
 package being binNMUed or not.

I agree that it would be nice to always arch-qualify the changelog
filename.  But that would involve a lot of changes to
changelog-reading tools which we perhaps don't want to do right now.

Note that even if we decide to always arch-qualify, we will still have
lots of old packages so all changelog-reading tools will need to look
in both places.

For most changelog-reading tools it won't be very troublesome if they
accidentally don't spot a binNMU entry.  So Russ's proposal is a good
step towards your proposal.  And if we decide we don't need to go all
the way then it's good enough for now.

 This still does not solve the other issues I listed, namely binNMUs
 have to be performed in lock-step, more complicated transitions /
 upgrades.

I don't think I see where this is coming from.  Are you talking about
variation in gzip output ?  Given the evidence we've seen here, in
practice I think that is not going to be a problem.  Certainly it
won't demand that binNMUs be performed in lock-step.

 So this is still pretty much unconvincing, and seems like clinging
 into the refcnt'ing “solution” while it makes things overall more
 complicated, will introduce inconsistency and incertainty to
 maintainers, needs way more global changes to keep it going, etc.

I think the refcounting approach is very worthwhile because it
eliminates unnecessary work (by human maintainers) in many simple
cases.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20282.28586.577528.890...@chiark.greenend.org.uk

Re: Multiarch file overlap summary and proposal

2012-02-14 Thread Sven Joachim

On 2012-02-14 15:28 +0100, Ian Jackson wrote:

 Guillem Jover writes:

 This still does not solve the other issues I listed, namely binNMUs
 have to be performed in lock-step, more complicated transitions /
 upgrades.

 I don't think I see where this is coming from.  Are you talking about
 variation in gzip output ?  Given the evidence we've seen here, in
 practice I think that is not going to be a problem.  Certainly it
 won't demand that binNMUs be performed in lock-step.

Guillem is referring to the need to to keep package versions in sync
across architectures, pretty much a necessity if you permit shared
files.

Cheers,
   Sven


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/871upxcvk4@turtle.gmx.de

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

Hi,

On Tue, 14 Feb 2012, Guillem Jover wrote:
  * All packages that want to be multiarch: same have to move all generated
documentation into a separate package unless the maintainer has very
carefully checked that the generated documentation will be byte-for-byte
identical even across minor updates of the documentation generation
tools and when run at different times.
 
 If packages have to be split anyway to cope with the other cases, then
 the number of new packages which might not be needed otherwise will be
 even smaller than the predicted amount, at which point it makes even
 less sense to support refcnt'ing.

Why are you so opposed to the refcnt'ing?

It's not such a big deal to maintain this feature in dpkg. And even if the
current implementation is not perfect, it can be improved later when dpkg
will store by itself checksums of provided files.

To me it looks like you don't like refcnt'ing and you're trying to find
some reasons to make it unacceptable.

 It also requires maintainers to carefully consider if the (doc, etc)
 toolchains will generate predictible ouput.

If the maintainer has to install files in non-standard path (because of
the need to arch-qualify it), it will also need maintainers to carefully
consider how to ensure that this move doesn't break anything.

It's not a white/black situation. You're trading one potential problem for
another. And the differing files are likely to be much more easy to spot
than other behaviour changes that might be implied by the move of some
files to arch qualified paths.

 Your proposal still requires papering over the other corner-cases.

Can you be explicit about which corner cases you're referring to ?

 This still does not solve the other issues I listed, namely binNMUs
 have to be performed in lock-step

Can you explain why? If the binnmu changelog is in a arch-specific file,
then we're free to bin-nmu packages separately.

dpkg must just ensure that all M-A: same packages have the same source
version (instead of the binary version as currently).

, more complicated transitions / upgrades.

We have no experience on this. It's a bit early to say whether those
constraints are going to be problematic or not.

 And introduces different solutions for different problems, while my
 proposal is generic for all cases.

There's nothing like a generic solution. You still have to decide whether
you move files to a -common package or if you arch qualify them and keep
them in the M-A: same package. And in both cases, you have to evaluate the
implications, in terms of package installation ordering in one case, in
terms of modifications to do to properly support the arch-qualified files
in the other one.

While it may sound like cleaner from a theoretical point of view, I'm
not convinced that it's better than the approach outlined by Russ.

Also you completely ignore the fact that what you're proposing is an
important change for multi-arch packages that have already been converted
both in Debian and in Ubuntu. You're pushing back the work to package
maintainers when there's not reason to not deal with this at the build
infrastructure level.

To reduce some of the downsides associated to compressed files in M-A:
same packages, we could/should investigate how to not compress files
in such packages instead of duplicating them needlessly.

 So this is still pretty much unconvincing, and seems like clinging
 into the refcnt'ing “solution” while it makes things overall more
 complicated, will introduce inconsistency and incertainty to
 maintainers, needs way more global changes to keep it going, etc.

This is not a fair characterization of the situation. IMO Global changes are
better than lots of maintainers having to do busy-work splitting their
packages.

You see inconsistency in Russ's proposal but you don't see
inconsistency/incertainty when you change the standard location of
changelog files.

And the more complicated, it might be true at the dpkg level, but I
don't believe that it's true from the maintainers points of view.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120214151318.ga14...@rivendell.home.ouaza.com

Re: Multiarch file overlap summary and proposal

Hi,

On Tue, 14 Feb 2012, Sven Joachim wrote:
  Guillem Jover writes:
 
  This still does not solve the other issues I listed, namely binNMUs
  have to be performed in lock-step, more complicated transitions /
  upgrades.
 
  I don't think I see where this is coming from.  Are you talking about
  variation in gzip output ?  Given the evidence we've seen here, in
  practice I think that is not going to be a problem.  Certainly it
  won't demand that binNMUs be performed in lock-step.
 
 Guillem is referring to the need to to keep package versions in sync
 across architectures, pretty much a necessity if you permit shared
 files.

It's a matter of defining in sync as having the same source version
instead of having the same binary version.

But such a change will require updates for APT too.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120214151637.gb14...@rivendell.home.ouaza.com

Re: Multiarch file overlap summary and proposal

2012-02-14 Thread Marvin Renich

* Russ Allbery r...@debian.org [120214 01:48]:
 If this is comprehensive, then I propose the following path forward, which
 is a mix of the various solutions that have been discussed:

I thought Goswin's suggestion in [1] of having dpkg use implicit
diversions has merit and deserves further scrutiny.  It essentially
implements refcnt-like behavior by using the existing diversion
mechanism.  I did not see any message in the thread that even
acknowledged that part of his message.  Did I miss something?

...Marvin

[1] http://lists.debian.org/debian-devel/2012/02/msg00511.html


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120214151437.ga14...@cleo.wdw

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

On Tue, 14 Feb 2012, Jakub Wilk wrote:
 Are we sure than no existing package uses debian/changelog.build for
 their own purposes?

No, but with debian/changelog.dpkg-build we should be safe.

 Are we sure that all existing packages (and helpers) that parse
 debian/changelog use dpkg-parsechangelog?

No, but I would consider anything else as a bug and we would notice
relatively quickly (we could even do a full rebuild to try to verify
pro-actively).

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120214152757.gc14...@rivendell.home.ouaza.com

Re: Multiarch file overlap summary and proposal

On Tue, 14 Feb 2012, Marvin Renich wrote:
 I thought Goswin's suggestion in [1] of having dpkg use implicit
 diversions has merit and deserves further scrutiny.

I don't. diversions support 2 packages, the diverted one and the
diverting one. Multi-Arch: same must support co-installation of
any number of packages.

So you can't reuse the existing logic without heavy modifications.

And changing the destination path at installation time is not a
good idea. What if the package really requires that specific version
of the file at the indicated path ? We will have configured the package
as if everything went fine when in fact it did not.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120214153350.gd14...@rivendell.home.ouaza.com

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

2012-02-14 Thread Guillem Jover

On Tue, 2012-02-14 at 14:28:58 +, Ian Jackson wrote:
 Guillem Jover writes (Re: Multiarch file overlap summary and proposal (was: 
 Summary: dpkg shared / reference counted files and version match)):
  On Mon, 2012-02-13 at 22:43:04 -0800, Russ Allbery wrote:
   * The binNMU process is changed to add the binNMU changelog entry to an
 arch-qualified file (changelog.Debian.arch, probably).  We need to
 figure out what this means if the package being binNMU'd has a
 /usr/share/doc/package symlink to another package, though; it's not
 obvious what to do here.
  
  This requires IMO multitude of hacks when the simplest and obvious
  arch-qualified pkgname solves this cleanly, and allows debhelper to
  automatically deal with it. And for tools to just change where they
  always look for those files in the M-A:same case regardless of the
  package being binNMUed or not.
 
 I agree that it would be nice to always arch-qualify the changelog
 filename.  But that would involve a lot of changes to
 changelog-reading tools which we perhaps don't want to do right now.

I've never proposed to arch-qualify the filename for the stuff under
/usr/share/doc/pkgname/, I've proposed to arch-qualify the pkgname in
the path (/usr/share/doc/pkgname:arch/), but only for M-A:same packages,
which are the only ones needing the disambiguation. This is how dpkg
handles pkgname output, or how it stores their data in the db too.

And it should be easy to ask a multiarch enabled dpkg-query for example
to normalize the pkgname output to be used on those paths, or otherwise
do it by hand:

  if M-A == same
pkgname:arch
  else
pkgname

 Note that even if we decide to always arch-qualify, we will still have
 lots of old packages so all changelog-reading tools will need to look
 in both places.

 For most changelog-reading tools it won't be very troublesome if they
 accidentally don't spot a binNMU entry.  So Russ's proposal is a good
 step towards your proposal.  And if we decide we don't need to go all
 the way then it's good enough for now.

How many tools are there that actually read the binary package changelog
file anyway? I only know of packages.d.o. Any other tool reading from
the installed path, cannot really rely on it being present at all
anyway, per policy.

And in addition, binNMU split changelogs are going to be there forever,
and as such their possible double locations. While the possible double
location for M-A:same packages using pkgname:arch qualified pathnames
would only be temporary and disappear once the packages have been rebuilt
with a new debhelper which automatically installs them in the correct
place.

  So this is still pretty much unconvincing, and seems like clinging
  into the refcnt'ing “solution” while it makes things overall more
  complicated, will introduce inconsistency and incertainty to
  maintainers, needs way more global changes to keep it going, etc.
 
 I think the refcounting approach is very worthwhile because it
 eliminates unnecessary work (by human maintainers) in many simple
 cases.

As I mentioned in Riku's reply, the amount of packages that would need
splitting that would otherwise not be needed should be even less than
before (which was predicted at around 700), also as I mentioned there
too, nothing prevents us from arch-qualifying paths (with Debian arch
or multiarch triplet depending on the case) if that's more convenient
or safer (as per your essential data example), and is what we've been
doing anyway for arch-indep data shipped in arch:any packages all along.
Given the amount of hacks or special casing piling up to make refcnt'ing
workable, when all that's really needed is a one time handling (or a
possible additional change for already converted packages, for things
that debhelper might not be able to handle) of moving qualifying paths
or splitting into new packages, it really does not seem worth it, no.

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120214164015.ga27...@gaara.hadrons.org

Re: Multiarch file overlap summary and proposal

2012-02-14 Thread Russ Allbery

Niels Thykier ni...@thykier.net writes:
 On 2012-02-14 07:43, Russ Allbery wrote:
 [...]
 
 * Lintian should recognize arch-qualified override files, and multiarch:
   same packages must arch-qualify their override files.  debhelper
   assistance is desired for this.
 
 [...]

 I have no problem with Lintian accepting arch-qualified override files,
 but I do not see the strict (i.e. must) requirement[1].  Lintian
 already allows you to do arch-specific overrides and 2.5.5 will even
 allow architecture wildcards as well[2].

Ah, yes, you're right, that's a good point.  If you use architecture
restrictions in the overrides, then you can install the same override file
on all architectures, so this doesn't need to be dealt with immediately
(if at all; we only have to do this if we want to support installing
different override files per architecture, which isn't strictly
necessary).

 [1] Exception being compressed override files, but of the 161 override
 files on my system not a single one of them are compressed.

I don't think we ever told anyone Lintian could support compressed
override files.  In fact, I didn't know it could.  :)

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/878vk58dq9@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

2012-02-14 Thread Guillem Jover

On Tue, 2012-02-14 at 14:28:58 +, Ian Jackson wrote:
 I think the refcounting approach is very worthwhile because it
 eliminates unnecessary work (by human maintainers) in many simple
 cases.

Aside from what I said on my other reply, I just wanted to note that
this seems to be a recurring point of tension in the project when it
comes to archive wide source package changes, where supposed short
term convenience (with its usually long term harmful effects) appears
to initially seduce people over what seems to be the cleaner although
slightly a bit more laborious solution.

Other recent-ish incarnations of this tension could be the build-arch
build-indep targets, or the build flag settings; where the former got
recently resolved so that the right thing to do is for *all* packages
needing to eventually support those targets, or for the latter which
got switched from the seemingly more convenient to the more laborious
but correct solution, that is, *all* packages need to set those build
flags by themselves.

This is a fundamental issue with how our source packages are handled,
and the freedom and power it gives to experiment and implement them
whatever way the maintainer wants, has the price that doing some
archive wide changes is sometimes more costly, than changing something
centrally and be done with it. But trying to workaround this by coming
up with stacks of hacked up solutions will not solve that fundamental
issue, and this kind of tension will keep coming up again and again,
as long as the foundation is not reworked. Either that, or the project
needs to accept that fact and learn to live with this kind of changes,
with patience.

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120215011510.ga15...@gaara.hadrons.org

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

2012-02-14 Thread Joey Hess

Guillem Jover wrote:
 Aside from what I said on my other reply, I just wanted to note that
 this seems to be a recurring point of tension in the project when it
 comes to archive wide source package changes, where supposed short
 term convenience (with its usually long term harmful effects) appears
 to initially seduce people over what seems to be the cleaner although
 slightly a bit more laborious solution.
 
 Other recent-ish incarnations of this tension could be the build-arch
 build-indep targets, or the build flag settings; where the former got
 recently resolved so that the right thing to do is for *all* packages
 needing to eventually support those targets, or for the latter which
 got switched from the seemingly more convenient to the more laborious
 but correct solution, that is, *all* packages need to set those build
 flags by themselves.
 
 This is a fundamental issue with how our source packages are handled,
 and the freedom and power it gives to experiment and implement them
 whatever way the maintainer wants, has the price that doing some
 archive wide changes is sometimes more costly, than changing something
 centrally and be done with it. But trying to workaround this by coming
 up with stacks of hacked up solutions will not solve that fundamental
 issue, and this kind of tension will keep coming up again and again,
 as long as the foundation is not reworked. Either that, or the project
 needs to accept that fact and learn to live with this kind of changes,
 with patience.

Very interesting mail. While I certianly agree with your examples, it's
worth remembering the counterexample of the /usr/doc transition which
took approximately 5 years to complete[1], and probably could have been
accomplished quickly and without pain with a simple hack to dpkg.

Anyway, my worry about the refcounting approach (or perhaps M-A: same in
general) is not the details of the implementation in dpkg, but the added
mental complexity of dpkg now being able to have multiple distinct
packages installed under the same name. I had a brief exposure to rpm,
which can install multiple versions of the same package, and that was
the main cause of much confusing behavior in rpm. While dpkg's invariant
that all co-installable package names be unique (and have unique files)
has certianly led to lots of ugly package names, it's kept the users'
and developers' mental models quite simple.

I worry that we have barely begun to scratch the surface of the added
complexity of losing this invariant.

-- 
see shy jo

[1] To the extent it was ever completed.. master.debian.org still has
a vestigial /usr/doc/


signature.asc
Description: Digital signature

Re: Multiarch file overlap summary and proposal

2012-02-14 Thread Russ Allbery

Joey Hess jo...@debian.org writes:

 Anyway, my worry about the refcounting approach (or perhaps M-A: same in
 general) is not the details of the implementation in dpkg, but the added
 mental complexity of dpkg now being able to have multiple distinct
 packages installed under the same name. I had a brief exposure to rpm,
 which can install multiple versions of the same package, and that was
 the main cause of much confusing behavior in rpm. While dpkg's invariant
 that all co-installable package names be unique (and have unique files)
 has certianly led to lots of ugly package names, it's kept the users'
 and developers' mental models quite simple.

 I worry that we have barely begun to scratch the surface of the added
 complexity of losing this invariant.

This does seem to be more M-A: same in general, to me, since whether we
have file overlaps or not we still have multiple packages with the same
name.  Which will force changes in everything that deals with packages,
like Puppet, to be able to specify packages with particular architectures.

I definitely agree on the complexity this adds.  But I don't think there's
an alternative to that complexity without using something like --sysroot
or mini-chroots, and I don't think those are satisfying solutions to the
set of problems we're trying to solve.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqdg4u5l@windlord.stanford.edu

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)