Bug#773256: dbus + triggers + apt issue (was: Bug#773256: pre-approval: unblock: dpkg/1.17.23)

2015-01-13 Thread Michael Vogt 
On Thu, Jan 08, 2015 at 02:47:05AM +0100, Guillem Jover wrote:
 [ @deity: please check out the dbus issue below. ]
[..]
 [ Re dbus issue ]
  Noted.  Do we (still?) have a (reliable) way of reproducing the dbus
  trigger issue?
 
 We didn't up to now (AFAIK), but there's now very valuable data from
 Karl Ljungkvist at #774124. I've been able to reproduce the issue with
 that status file and apt (over a clean debootstrap of jessie, and
 adding i386 as a foreign arch). 

Thanks for pointing us towards this bugreport, this contains some good
information. I followed up there with some more questions, it would be
really nice to be able to reproduce what error happend that caused
this broken state.

 And have been trying out several things:
 
   * It seems to affect both apt from jessie/sid and wheezy. :(
 apt gets into a state it cannot recover from by itself.

Indeed, apt sees the triggers-pending and triggers-awaited state
and tries to run dpkg --configure on all those packages. This
fails.

   * «dpkg --configure --pending» solves the issue.
   * Using dpkg --force-configure-any only fixes the issue partially, :(
 because apt does not notice that the package has been implicitly
 configured, and tries to configure the package again and dpkg exits
 with an error for that run, which apt does not like much either (as
 I feared). So I'd need to also change dpkg to not fail configuring
 an already configure package, or try to detect that specific case.

When you say implicitly configured what does that mean exactly? Is
the issue here that apt should ignore the triggers-awaited state
because those will be dealt with by dpkg entirely?

I also wonder if there are further implications for apt by the trigger
changes. The ordering code in apt builds on the premise that there is
no implict configuration, it builds the unpack/configure ordering
early and its static. Is this still a valid assumption?

[..] 
 Before considering either reverting, or trying to find a workaround
 for this in dpkg, I'd like to know if this is easily fixable in apt
 and the implications of this problem (i.e. can it affect similar
 situations regardless of the recent dpkg trigger changes?) and the
 implications of such a fix.

We have code in pkgApplyStatus that detects and fixes not-ok
packages. So far it considered packages with
triggers-{pending,awaited} as something to just dpkg --configure. We
could change the code to ignore those states and simply let dpkg
handle them (via Davids code that ensures 'dpkg --configure -a' gets
called). Does that sound sensible?

I'm still a bit nervous about the issue that lead to #774124. It would
be good to get more data here.

Cheers,
 Michael


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#773256: dbus + triggers + apt issue (was: Bug#773256: pre-approval: unblock: dpkg/1.17.23)

2015-01-13 Thread Guillem Jover 
Hi!

On Tue, 2015-01-13 at 15:08:01 +0100, Michael Vogt wrote:
 On Thu, Jan 08, 2015 at 02:47:05AM +0100, Guillem Jover wrote:
  [ Re dbus issue ]
   Noted.  Do we (still?) have a (reliable) way of reproducing the dbus
   trigger issue?
  
  We didn't up to now (AFAIK), but there's now very valuable data from
  Karl Ljungkvist at #774124. I've been able to reproduce the issue with
  that status file and apt (over a clean debootstrap of jessie, and
  adding i386 as a foreign arch). 
 
 Thanks for pointing us towards this bugreport, this contains some good
 information. I followed up there with some more questions, it would be
 really nice to be able to reproduce what error happend that caused
 this broken state.

The bug probably deserves to be reassigned to apt, I'll do that later.

Unfortunately it seems many if not all of these instances where also
affected by the apt bug eating dpkg's output. :(

I've some code locally that will log dpkg errors to the log file, but
didn't even try to get this into jessie due to the deep freeze. Will
be included with 1.18.x in any case.

  And have been trying out several things:
  
* It seems to affect both apt from jessie/sid and wheezy. :(
  apt gets into a state it cannot recover from by itself.
 
 Indeed, apt sees the triggers-pending and triggers-awaited state
 and tries to run dpkg --configure on all those packages. This
 fails.

Because I'm assuming apt is simply not taking into account the
unsatisfied dependencies for those package states?

* «dpkg --configure --pending» solves the issue.
* Using dpkg --force-configure-any only fixes the issue partially, :(
  because apt does not notice that the package has been implicitly
  configured, and tries to configure the package again and dpkg exits
  with an error for that run, which apt does not like much either (as
  I feared). So I'd need to also change dpkg to not fail configuring
  an already configure package, or try to detect that specific case.
 
 When you say implicitly configured what does that mean exactly? Is
 the issue here that apt should ignore the triggers-awaited state
 because those will be dealt with by dpkg entirely?

In normal conditions, when dpkg is called with an explicit list of
packages to configure, it will only configure those.

When using dpkg with --force-configure-any, if there's an unsatisfied
dependency that could be fixed by configuring another package, then
dpkg will also enqueue that package for configuration. Which means that
even though apt didn't request its configuration, it got configured
(implicitly) anyway.

Ideally apt would notice this through the status-fd updates, and not
try to configure that package explicitly afterwards, but it seems this
is not currently the case? And dpkg fails when apt requests to
configure the already configured package.

Otherwise enabling --force-configure-any by default in dpkg, would be
a sufficient workaround for this issue.

 I also wonder if there are further implications for apt by the trigger
 changes. The ordering code in apt builds on the premise that there is
 no implict configuration, it builds the unpack/configure ordering
 early and its static. Is this still a valid assumption?

As long as --force-configure-any is not used (which is the case
currently), then yes, there's no implicit configuration going on. I
was just checking if that would be a viable workaround for the dbus
issue. But due to the current apt behavior, that is not a viable
workaround by itself, it might require more changes in dpkg.

The only implication that I assume was not being taken into account
previously is that for dpkg to be able to process pending triggers,
the dependencies for that package need to be satisfied.

 [..] 
  Before considering either reverting, or trying to find a workaround
  for this in dpkg, I'd like to know if this is easily fixable in apt
  and the implications of this problem (i.e. can it affect similar
  situations regardless of the recent dpkg trigger changes?) and the
  implications of such a fix.
 
 We have code in pkgApplyStatus that detects and fixes not-ok
 packages. So far it considered packages with
 triggers-{pending,awaited} as something to just dpkg --configure. We
 could change the code to ignore those states and simply let dpkg
 handle them (via Davids code that ensures 'dpkg --configure -a' gets
 called). Does that sound sensible?

That should indeed in theory fix the issue (depending where apt runs
that command). The other option is to micromanage dpkg even more, by
checking for unsatisfied dependencies, and configuring those first. I'd
rather go with the first, as a way to start micromanaging dpkg less.

 I'm still a bit nervous about the issue that lead to #774124. It would
 be good to get more data here.

Indeed, that's why I wanted to get you guys to take a look at it
before proceeding with a decision or fixes/workarounds in dpkg. If it
ends up being a very localized problem of apt

Bug#773256: dbus + triggers + apt issue (was: Bug#773256: pre-approval: unblock: dpkg/1.17.23)

2015-01-07 Thread Guillem Jover 
[ @deity: please check out the dbus issue below. ]

Hi!

On Sat, 2014-12-27 at 18:45:38 +0100, Niels Thykier wrote:
 I hope you are feeling better now. :)

Yes, much better, thanks. :)

 TL;DR: Please upload dpkg/1.17.23 with the proposed fix for #771730 at
 your earliest convenience - by the looks of it, might fix one or two of
 our upgrade tests on jenkins.d.n.

Thanks for the unblock.

[ Re dbus issue ]
 Noted.  Do we (still?) have a (reliable) way of reproducing the dbus
 trigger issue?

We didn't up to now (AFAIK), but there's now very valuable data from
Karl Ljungkvist at #774124. I've been able to reproduce the issue with
that status file and apt (over a clean debootstrap of jessie, and
adding i386 as a foreign arch). And have been trying out several things:

  * It seems to affect both apt from jessie/sid and wheezy. :(
apt gets into a state it cannot recover from by itself.
  * «dpkg --configure --pending» solves the issue.
  * Using dpkg --force-configure-any only fixes the issue partially, :(
because apt does not notice that the package has been implicitly
configured, and tries to configure the package again and dpkg exits
with an error for that run, which apt does not like much either (as
I feared). So I'd need to also change dpkg to not fail configuring
an already configure package, or try to detect that specific case.

This is the (trimmed) «dpkg -C» output for the broken status file:

,---
The following packages have been unpacked but not yet configured.
They must be configured using dpkg --configure or the configure
menu option in dselect for them to work:
 bluezBluetooth tools and daemons
 bzip2high-quality block-sorting file compressor - utilities
 libdbus-1-3:amd64simple interprocess messaging system (library)
 libdbus-1-3:i386 simple interprocess messaging system (library)
 libdbus-1-dev:amd64  simple interprocess messaging system (development headers
 libperl5.20  shared Perl library
 perl Larry Wall's Practical Extraction and Report Language
 perl-modules Core Perl modules
 systemd-sysv system and service manager - SysV links
 tzdata-java  time zone and daylight-saving time data for use by java r

The following packages are awaiting processing of triggers that they
have activated in other packages.  This processing can be requested using
dselect or dpkg --configure --pending (or dpkg --triggers-only):
 systemd  system and service manager

The following packages have been triggered, but the trigger processing
has not yet been done.  Trigger processing can be requested using
dselect or dpkg --configure --pending (or dpkg --triggers-only):
 dbus simple interprocess messaging system (daemon and utilitie
`---

And here's the «apt-get install -f -s» output:

,---
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  libpam-systemd
The following packages will be upgraded:
  libpam-systemd
1 upgraded, 0 newly installed, 0 to remove and 238 not upgraded.
10 not fully installed or removed.
Conf systemd-sysv (215-8 Debian:testing [amd64]) [libpam-systemd:amd64 ]
Inst libpam-systemd [215-7] (215-8 Debian:testing [amd64])
Conf libdbus-1-3 (1.8.12-3 Debian:testing [amd64])
Conf libdbus-1-3:i386 (1.8.12-3  [i386])
Conf bzip2 (1.0.6-7+b2 Debian:testing [amd64])
Conf perl-modules (5.20.1-4 Debian:testing [all])
Conf perl (5.20.1-4 Debian:testing [amd64])
Conf bluez (5.23-2 Debian:testing [amd64])
Conf libdbus-1-dev (1.8.12-3 Debian:testing [amd64])
Conf libperl5.20 (5.20.1-4 Debian:testing [amd64])
Conf tzdata-java (2014j-1 Debian:testing [all])
Conf libpam-systemd (215-8 Debian:testing [amd64])
`---

Trying to run the first action gives the undesired unsatisfied trigger
dependency problem:

,---
$ dpkg --configure systemd-sysv
dpkg: dependency problems prevent processing triggers for dbus:
 dbus depends on libdbus-1-3 (= 1.7.6); however:
  Package libdbus-1-3:amd64 is not configured yet.

dpkg: error processing package dbus (--configure):
 dependency problems - leaving triggers unprocessed
[…]
dpkg: too many errors, stopping
Errors were encountered while processing:
 dbus
[…]
Processing was halted because there were too many errors.
`---

Before considering either reverting, or trying to find a workaround
for this in dpkg, I'd like to know if this is easily fixable in apt
and the implications of this problem (i.e. can it affect similar
situations regardless of the recent dpkg trigger changes?) and the
implications of such a fix.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-27 Thread Niels Thykier 
On 2014-12-23 04:36, Guillem Jover wrote:
 Hi!
 

Hi,

 On Sun, 2014-12-21 at 09:57:51 +0100, Niels Thykier wrote:
   It possibly still is since the version that introduced the trigger
 checks.  I hope we can have it resolved shortly.
 
 Yeah, I'm planning to upload tomorrow, sorry about the delay, was not
 feeling quite well the past couple of days.
 

I hope you are feeling better now. :)

TL;DR: Please upload dpkg/1.17.23 with the proposed fix for #771730 at
your earliest convenience - by the looks of it, might fix one or two of
our upgrade tests on jenkins.d.n.

 [...]

 Indeed I missed those.  For reference, pypy got fixed and gxine, icecc
 and mcollective will get auto-removed eventually.  My previous remark
 for gxine plus icecc applies equally to mcollective (and pypy, in case
 migration is stalled) as well.
   This leaves auctex as the only remaining blocker auctex.
 
 In the meantime mcollective also got fixed, which leaves auctex, gxine,
 icecc and wordpress. The latter has a fixed package sitting in NEW
 (but I'm not sure it will be accepted and allowed to migrate?).
 

Thanks for the update.  I will have a look at poking the remaining packages.

 I've added  versioned Breaks for the packages with fixed versions,
 and = for the ones with unfixed ones against the version in testing
 (as for auctex it differs against unstable).
 

Ok.

 [...]
 
 @deity/@dpkg: Right now, I am less concerned with whose fault it is
 and vastly more concerned with getting a functional upgrade path.
   If the correct fix is in APT and it can be provided in a reviewable
 diff in a reasonable time frame, I will happily take that.  Otherwise,
 the fix needs to be in dpkg (despite being wrong or a revert).  At
 this point in the freeze, we do not have the luxury of finding perfection.
 
 Sorry, my point was that I don't think we know exactly what lead
 to the dbus issue, which apt versions are affected, or if this is
 actually something that showed up due to the trigger changes, dbus
 maintscripts errors and if it was a latent issue that could as well
 show up with a system crash leaving dpkg in a broken state, etc.
 
 As I've mentioend before I don't mind at all adding a workaround in
 dpkg if that provides a smooth upgrade path, but I'd like to do that
 understanding what it's working around, and that it actually fixes
 the issue, and not just blindly.
 

Noted.  Do we (still?) have a (reliable) way of reproducing the dbus
trigger issue?

 The main issue for me is that besides this trigger issue, we also got
 the entire init system replacement on upgrades.  I fear this is
 uncharted territories right now because everybody is (mostly) stuck
 behind these trigger issues.
 
 While I think trigger cycles (be them real or bogusly detected as
 the ones in 1.17.22) should be RC, they are actually recoverable
 upgrade errors, as package involved in trigger cycles get reset into
 half-configured, and their pending triggers reset. Which means that
 a subsequent apt invocation should be able to continue from where it
 was left off. So I don't think this has gotten people stuck on upgrade
 problems (at least with dpkg = 1.17.22).
 

It is not about whether they are recoverable.  Most people do not
upgrade systems if they are aware of known unresolved upgrade issues.

The sooner we can say All of the dpkg and APT issues are now fixed; go
forth and test upgrade into Jessie, the sooner I get to the next level
of issues.

 The dbus problem seemed to be actually more severe, because although
 «dpkg --configure --pending» seemed to be able to recover from the
 situation, apt did not and got stuck there w/o being able to continue,
 at least from the logs and comments in the bug reports, but not many
 people have reported such issue so…
 

Noted.

 I also don't
 know what's the stance on requiring specific packaging tools to be
 upgraded first? Which might mean that if the issue is still there it
 could be fixed in apt proper.

 (@deity: You may want to have a look at this as well)

 As I recall, it is not a requirement - but I believe we can recommend it
 in the release-notes.  [...]
 
 If it's just a recommendation then I'm not sure that would be
 satisfactory as we cannot rely on it.

Indeed and even then, some people might not read the release notes until
after.

 In any case for apt-based (or
 cupt) frontends the important thing is to get the frontends upgraded
 first, because dpkg ends up being invoked many times, and the new
 version will be picked up once it gets upgraded. Also frontends seem
 to favor Essential packages so they will get upgraded very early and
 on its own dpkg run. But if something got to be added, then I guess
 just requiring to upgrade apt or cupt would be enough.
 
 Thanks,
 Guillem
 
 

If we add something, we might as well add dpkg as well. :)

[From a different mail than the one I am replying to]
 On Tue, 2014-12-23 at 04:36:07 +0100, Guillem Jover wrote:
 On Sun, 2014-12-21 at 09:57:51 +0100, Niels Thykier

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-23 Thread Guillem Jover 
Hi!

On Tue, 2014-12-23 at 02:52:01 +0100, Guillem Jover wrote:
 On Sun, 2014-12-21 at 21:38:31 +0100, Niels Thykier wrote:
  I do not recall (all of?) these trigger cycles being known.  @Guiliem,
  can you have a look at them and file bugs as necessary for these?
 
 These smell like instances of #771730 (more so when libc-bin is
 noawait), but I fired up a test upgrade with 1.17.23 to make sure.

Ok, after several GiBs of downloads and unpacks, the upgrade went fine
with dpkg 1.17.23 for education-thin-client-server. I'll leave one of
the other ones testing during the night, but I don't expect any
problems either.

On Tue, 2014-12-23 at 04:36:07 +0100, Guillem Jover wrote:
 On Sun, 2014-12-21 at 09:57:51 +0100, Niels Thykier wrote:
It possibly still is since the version that introduced the trigger
  checks.  I hope we can have it resolved shortly.
 
 Yeah, I'm planning to upload tomorrow, sorry about the delay, was not
 feeling quite well the past couple of days.

Actually, I just noticed the bug was not tagged confirmed, so given
this, the wordpress situation, and the questions you posed in the
previous email, I'll hold off the upload, which is tested and ready
for when I get a go.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-23 Thread Guillem Jover 
On Wed, 2014-12-24 at 02:26:24 +0100, Guillem Jover wrote:
 On Tue, 2014-12-23 at 02:52:01 +0100, Guillem Jover wrote:
  On Sun, 2014-12-21 at 21:38:31 +0100, Niels Thykier wrote:
   I do not recall (all of?) these trigger cycles being known.  @Guiliem,
   can you have a look at them and file bugs as necessary for these?
  
  These smell like instances of #771730 (more so when libc-bin is
  noawait), but I fired up a test upgrade with 1.17.23 to make sure.
 
 Ok, after several GiBs of downloads and unpacks, the upgrade went fine
 with dpkg 1.17.23 for education-thin-client-server. I'll leave one of
 the other ones testing during the night, but I don't expect any
 problems either.

… And the haskell one went well too.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-22 Thread Guillem Jover 
Hi!

On Sun, 2014-12-21 at 21:38:31 +0100, Niels Thykier wrote:
 Thanks.  We noticed the following results with the dpkg/1.17.22:
 
 
  19:49  h01ger 
  https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/job/chroot-installation_wheezy_install_haskell_upgrade_to_jessie/lastBuild/console
   not sure if i have seen this already:
  19:50  h01ger dpkg: cycle found while processing triggers:
  19:50  h01ger  chain of packages whose triggers are or may be responsible:
  19:50  h01ger   doc-base - libc-bin
  19:50  h01ger 
  https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/job/chroot-installation_wheezy_install_education-desktop-other_upgrade_to_jessie/lastBuild/console
   has
  19:50  h01ger   desktop-file-utils - libc-bin
  19:50  h01ger 
  https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/job/chroot-installation_wheezy_install_education-standalone_upgrade_to_jessie/lastBuild/console
   OTOH has
  19:50  h01ger   gnome-icon-theme - libc-bin
  19:51  h01ger education-workstation_upgrade_to_jessie is boringly 
  repeating education-desktop-other_upgrade_to_jessie ;)
  19:55  h01ger 
  https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/job/chroot-installation_wheezy_install_education-thin-client-server_upgrade_to_jessie/lastBuild/console
   has  something new:
  19:55  h01ger   gconf2 - desktop-file-utils
  19:55  h01ger nthykier: 
  https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/
   updated + done
 
 
 The last one with a bit more details:
 
  Setting up libdevmapper1.02.1:amd64 (2:1.02.90-2) ...
  dpkg: cycle found while processing triggers:
   chain of packages whose triggers are or may be responsible:
gconf2 - desktop-file-utils
   packages' pending triggers which are or may be unresolvable:
libc-bin: ldconfig
desktop-file-utils: /usr/share/applications
gconf2: /usr/share/gconf/schemas
  dpkg: error processing package libc-bin (--configure):
   triggers looping, abandoned
  Errors were encountered while processing:
   libc-bin
 

 I do not recall (all of?) these trigger cycles being known.  @Guiliem,
 can you have a look at them and file bugs as necessary for these?

These smell like instances of #771730 (more so when libc-bin is
noawait), but I fired up a test upgrade with 1.17.23 to make sure.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-22 Thread Guillem Jover 
Hi!

On Sun, 2014-12-21 at 09:57:51 +0100, Niels Thykier wrote:
   It possibly still is since the version that introduced the trigger
 checks.  I hope we can have it resolved shortly.

Yeah, I'm planning to upload tomorrow, sorry about the delay, was not
feeling quite well the past couple of days.

  It seems that only gxine and icecc are missing now.  If so, please go
  ahead with the .23 with versioned breaks for them as well.  Worst case,
  I will have them removed from testing - best case, they will be fixed.
I will take the political fall-out of this and notify the maintainers
  of the affected packages.  Let me know if I missed any packages.
  
  Unfortunately there's still auctex, gxine, icecc, mcollective, pypy
  and wordpress. And I'm not sure if the fix for all of them might be
  a strightforward switch to a -noawait directive.
 
 Indeed I missed those.  For reference, pypy got fixed and gxine, icecc
 and mcollective will get auto-removed eventually.  My previous remark
 for gxine plus icecc applies equally to mcollective (and pypy, in case
 migration is stalled) as well.
   This leaves auctex as the only remaining blocker auctex.

In the meantime mcollective also got fixed, which leaves auctex, gxine,
icecc and wordpress. The latter has a fixed package sitting in NEW
(but I'm not sure it will be accepted and allowed to migrate?).

I've added  versioned Breaks for the packages with fixed versions,
and = for the ones with unfixed ones against the version in testing
(as for auctex it differs against unstable).

  I've also not added the --force-configure-any default switch, because we
  don't really know what happened with apt and dbus there, and if apt from
  stable is affected or not. Given the recent dpkg, apt, and dbus changes
  I think I'd rather let this as is, and wait in case it shows up again,
  which should give us more information (due to the new apt not eating
  dpkg's output).
 
  Noted, though I sincerely hope it is fixed.  I /might/ be convinced to
  accept a .24 for this particular issue.
  
  Just to clarify, if there's still an issue, it's exclusively in apt.
  The problems with dbus were due to apt trying to configure stuff when
  libdbus was not yet properly configured (AFAIR). And there was similar
  issues in apt between 0.9.9 and 1.0.7 (see #767734), but maybe this is
  something else, although related, but only showing up when trigger
  states are involved, this might also be a problem in apt in stable or
  just with newer versions, that might have only emerged due to the
  recent changes in dpkg. It might be that this only happens when
  packages break on upgrade as it happened with some of the dbus
  versions that failed on postinst, etc. Dunno, really.

 @deity/@dpkg: Right now, I am less concerned with whose fault it is
 and vastly more concerned with getting a functional upgrade path.
   If the correct fix is in APT and it can be provided in a reviewable
 diff in a reasonable time frame, I will happily take that.  Otherwise,
 the fix needs to be in dpkg (despite being wrong or a revert).  At
 this point in the freeze, we do not have the luxury of finding perfection.

Sorry, my point was that I don't think we know exactly what lead
to the dbus issue, which apt versions are affected, or if this is
actually something that showed up due to the trigger changes, dbus
maintscripts errors and if it was a latent issue that could as well
show up with a system crash leaving dpkg in a broken state, etc.

As I've mentioend before I don't mind at all adding a workaround in
dpkg if that provides a smooth upgrade path, but I'd like to do that
understanding what it's working around, and that it actually fixes
the issue, and not just blindly.

 The main issue for me is that besides this trigger issue, we also got
 the entire init system replacement on upgrades.  I fear this is
 uncharted territories right now because everybody is (mostly) stuck
 behind these trigger issues.

While I think trigger cycles (be them real or bogusly detected as
the ones in 1.17.22) should be RC, they are actually recoverable
upgrade errors, as package involved in trigger cycles get reset into
half-configured, and their pending triggers reset. Which means that
a subsequent apt invocation should be able to continue from where it
was left off. So I don't think this has gotten people stuck on upgrade
problems (at least with dpkg = 1.17.22).

The dbus problem seemed to be actually more severe, because although
«dpkg --configure --pending» seemed to be able to recover from the
situation, apt did not and got stuck there w/o being able to continue,
at least from the logs and comments in the bug reports, but not many
people have reported such issue so…

  I also don't
  know what's the stance on requiring specific packaging tools to be
  upgraded first? Which might mean that if the issue is still there it
  could be fixed in apt proper.
 
 (@deity: You may want to have a look at this as well)
 
 As I recall, it

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-21 Thread Niels Thykier 
CC: @deity (look for a couple of @deitys below)

On 2014-12-19 20:35, Guillem Jover wrote:
 Hi!
 
 On Wed, 2014-12-17 at 22:18:12 +0100, Niels Thykier wrote:
 On 2014-12-16 06:22, Guillem Jover wrote:
 Package: release.debian.org
 Severity: normal
 User: release.debian@packages.debian.org
 Usertags: unblock
 Control: block -1 by 770627
 
 I'd prefer if 1.17.22 could be unblocked before uploading this, because
 that version is way better than the one currently in testing, and it is
 causing fewer upgrade issues. Otherwise I'll just merge both unblock
 requests.

 Apologies, but I am not entirely convinced here.  I would strongly
 prefer /not/ having trigger regressions right now.
 
 Sorry, it seems there was a communication breakdown somewhere, my
 fault.
 
 As I just mentioned on #771730, once tracked down I always thought
 this affected all dpkg versions doing trigger dependency checks
 (although other issues might have shadowed that specific problem),
 but it seems I thought I mentioned it somewhere but cannot find any
 reference now :(, and did not actually try to reproduce with older
 versions because it seemed a bit like a waste of time when the
 unblock didn't seem to be dependenent on that issue.
 
 So, yes, 1.17.22 is really better in any possible way to 1.17.21.
 But, obviously your call.
 

Okay, it was not clear to me that dpkg/1.17.21 was also affected by
#771730.  That changes it quite a bit as it is no longer a regression
between unstable and testing.
  It possibly still is since the version that introduced the trigger
checks.  I hope we can have it resolved shortly.

 In fact, I am honestly considering to request having the trigger change
 reverted if 1.17.23 does not solve the issues without introduce another
 regression.
 
 Ok. I'll do another pass over the code, and then try to improve the
 functional test suite to see if I catch something else before the
 upload.
 

Excellent.

   We are one and a half month into the freeze and we still do not have a
 clean upgrade path on a package level.  I am deeply concerned that we
 have been missing out on (e.g.) the systemd upgrade reports because of this.
 
 Sorry, I guess I should have tried to push a fix for the RC bug earlier
 w/o waiting for the translations deadline, but was not entirely clear
 on whether the disabled unblock was permanent or temporary until
 clarification on the dbus issue and enabling it back had not happened
 for unrelated reasons.
 
 Also given that #771730 was not a regression, it seemed prudent to let
 .22 migrate first.
 

Okay, I have decided to let dpkg/1.17.22 migrate.  I am still not
pleased with the dbus situation - regardless of whose fault it is.

@deity/@dpkg: On the RC bug fixes vs. translations.  Given that (one
of?) dpkg and APT is correctly breaking our upgrades, I am much more
interested in the RC bug fixes for these packages.
  I do not mind waiting up to 14 days on a translations only update
provided that it gets me *working* an upgrade path between Wheezy and
Jessie sooner.

 I've delayed a bit the request because there are still some packages
 with trigger cycles that have not been uploaded yet, I can start taking
 a look on delayed NMUs and wait for those or upload .23 right away and
 possibly prepare a .24 with those additional versioned Breaks, whichever
 you prefer.

 It seems that only gxine and icecc are missing now.  If so, please go
 ahead with the .23 with versioned breaks for them as well.  Worst case,
 I will have them removed from testing - best case, they will be fixed.
   I will take the political fall-out of this and notify the maintainers
 of the affected packages.  Let me know if I missed any packages.
 
 Unfortunately there's still auctex, gxine, icecc, mcollective, pypy
 and wordpress. And I'm not sure if the fix for all of them might be
 a strightforward switch to a -noawait directive.
 

Indeed I missed those.  For reference, pypy got fixed and gxine, icecc
and mcollective will get auto-removed eventually.  My previous remark
for gxine plus icecc applies equally to mcollective (and pypy, in case
migration is stalled) as well.
  This leaves auctex as the only remaining blocker auctex.

 I've also not added the --force-configure-any default switch, because we
 don't really know what happened with apt and dbus there, and if apt from
 stable is affected or not. Given the recent dpkg, apt, and dbus changes
 I think I'd rather let this as is, and wait in case it shows up again,
 which should give us more information (due to the new apt not eating
 dpkg's output).

 Noted, though I sincerely hope it is fixed.  I /might/ be convinced to
 accept a .24 for this particular issue.
 
 Just to clarify, if there's still an issue, it's exclusively in apt.
 The problems with dbus were due to apt trying to configure stuff when
 libdbus was not yet properly configured (AFAIR). And there was similar
 issues in apt between 0.9.9 and 1.0.7 (see #767734), but maybe this is
 something else, although

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-21 Thread Holger Levsen 
Hi,

On Sonntag, 21. Dezember 2014, Niels Thykier wrote:
 As I recall, it is not a requirement - but I believe we can recommend it
 in the release-notes.  Although, with the Breaks being added to dpkg for
 trigger issues, you would quite possibly pull in additional upgrades
 along with it.
 
 By upgrading specific packaging tools, I presume you mean something like:

I've now aded a second type of upgrade tests to 
https://jenkins.debian.net/view/chroot-installation/
so now there are eg wheezy2jessie upgrades and wheezy2jessie dpkgapt_first 
upgrade jobs, in which apt-get install -y dpkg apt is run before apt-get 
upgrade is tried as usual...

First results should be available in 1-2h, the relevant jobs have been 
triggered, but there is a queue of jobs waiting to be run :)

HTH! :)


cheers,
Holger


signature.asc
Description: This is a digitally signed message part.

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-21 Thread Niels Thykier 
On 2014-12-21 14:54, Holger Levsen wrote:
 Hi,
 
 On Sonntag, 21. Dezember 2014, Niels Thykier wrote:
 As I recall, it is not a requirement - but I believe we can recommend it
 in the release-notes.  Although, with the Breaks being added to dpkg for
 trigger issues, you would quite possibly pull in additional upgrades
 along with it.

 By upgrading specific packaging tools, I presume you mean something like:
 
 I've now aded a second type of upgrade tests to 
 https://jenkins.debian.net/view/chroot-installation/
 so now there are eg wheezy2jessie upgrades and wheezy2jessie dpkgapt_first 
 upgrade jobs, in which apt-get install -y dpkg apt is run before apt-get 
 upgrade is tried as usual...
 
 First results should be available in 1-2h, the relevant jobs have been 
 triggered, but there is a queue of jobs waiting to be run :)
 
 HTH! :)
 
 
 cheers,
   Holger
 


Thanks.  We noticed the following results with the dpkg/1.17.22:


 19:49  h01ger 
 https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/job/chroot-installation_wheezy_install_haskell_upgrade_to_jessie/lastBuild/console
  not sure if i have seen this already:
 19:50  h01ger dpkg: cycle found while processing triggers:
 19:50  h01ger  chain of packages whose triggers are or may be responsible:
 19:50  h01ger   doc-base - libc-bin
 19:50  h01ger 
 https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/job/chroot-installation_wheezy_install_education-desktop-other_upgrade_to_jessie/lastBuild/console
  has
 19:50  h01ger   desktop-file-utils - libc-bin
 19:50  h01ger 
 https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/job/chroot-installation_wheezy_install_education-standalone_upgrade_to_jessie/lastBuild/console
  OTOH has
 19:50  h01ger   gnome-icon-theme - libc-bin
 19:51  h01ger education-workstation_upgrade_to_jessie is boringly repeating 
 education-desktop-other_upgrade_to_jessie ;)
 19:55  h01ger 
 https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/job/chroot-installation_wheezy_install_education-thin-client-server_upgrade_to_jessie/lastBuild/console
  has  something new:
 19:55  h01ger   gconf2 - desktop-file-utils
 19:55  h01ger nthykier: 
 https://jenkins.debian.net/view/chroot-installation/view/upgrade%20wheezy2jessie/
  updated + done


The last one with a bit more details:

 Setting up libdevmapper1.02.1:amd64 (2:1.02.90-2) ...
 dpkg: cycle found while processing triggers:
  chain of packages whose triggers are or may be responsible:
   gconf2 - desktop-file-utils
  packages' pending triggers which are or may be unresolvable:
   libc-bin: ldconfig
   desktop-file-utils: /usr/share/applications
   gconf2: /usr/share/gconf/schemas
 dpkg: error processing package libc-bin (--configure):
  triggers looping, abandoned
 Errors were encountered while processing:
  libc-bin



I do not recall (all of?) these trigger cycles being known.  @Guiliem,
can you have a look at them and file bugs as necessary for these?

Thanks,
~Niels


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-19 Thread Guillem Jover 
Hi!

On Wed, 2014-12-17 at 22:18:12 +0100, Niels Thykier wrote:
 On 2014-12-16 06:22, Guillem Jover wrote:
  Package: release.debian.org
  Severity: normal
  User: release.debian@packages.debian.org
  Usertags: unblock
  Control: block -1 by 770627

  I'd prefer if 1.17.22 could be unblocked before uploading this, because
  that version is way better than the one currently in testing, and it is
  causing fewer upgrade issues. Otherwise I'll just merge both unblock
  requests.
 
 Apologies, but I am not entirely convinced here.  I would strongly
 prefer /not/ having trigger regressions right now.

Sorry, it seems there was a communication breakdown somewhere, my
fault.

As I just mentioned on #771730, once tracked down I always thought
this affected all dpkg versions doing trigger dependency checks
(although other issues might have shadowed that specific problem),
but it seems I thought I mentioned it somewhere but cannot find any
reference now :(, and did not actually try to reproduce with older
versions because it seemed a bit like a waste of time when the
unblock didn't seem to be dependenent on that issue.

So, yes, 1.17.22 is really better in any possible way to 1.17.21.
But, obviously your call.

 In fact, I am honestly considering to request having the trigger change
 reverted if 1.17.23 does not solve the issues without introduce another
 regression.

Ok. I'll do another pass over the code, and then try to improve the
functional test suite to see if I catch something else before the
upload.

   We are one and a half month into the freeze and we still do not have a
 clean upgrade path on a package level.  I am deeply concerned that we
 have been missing out on (e.g.) the systemd upgrade reports because of this.

Sorry, I guess I should have tried to push a fix for the RC bug earlier
w/o waiting for the translations deadline, but was not entirely clear
on whether the disabled unblock was permanent or temporary until
clarification on the dbus issue and enabling it back had not happened
for unrelated reasons.

Also given that #771730 was not a regression, it seemed prudent to let
.22 migrate first.

  I've delayed a bit the request because there are still some packages
  with trigger cycles that have not been uploaded yet, I can start taking
  a look on delayed NMUs and wait for those or upload .23 right away and
  possibly prepare a .24 with those additional versioned Breaks, whichever
  you prefer.
 
 It seems that only gxine and icecc are missing now.  If so, please go
 ahead with the .23 with versioned breaks for them as well.  Worst case,
 I will have them removed from testing - best case, they will be fixed.
   I will take the political fall-out of this and notify the maintainers
 of the affected packages.  Let me know if I missed any packages.

Unfortunately there's still auctex, gxine, icecc, mcollective, pypy
and wordpress. And I'm not sure if the fix for all of them might be
a strightforward switch to a -noawait directive.

  I've also not added the --force-configure-any default switch, because we
  don't really know what happened with apt and dbus there, and if apt from
  stable is affected or not. Given the recent dpkg, apt, and dbus changes
  I think I'd rather let this as is, and wait in case it shows up again,
  which should give us more information (due to the new apt not eating
  dpkg's output).
 
 Noted, though I sincerely hope it is fixed.  I /might/ be convinced to
 accept a .24 for this particular issue.

Just to clarify, if there's still an issue, it's exclusively in apt.
The problems with dbus were due to apt trying to configure stuff when
libdbus was not yet properly configured (AFAIR). And there was similar
issues in apt between 0.9.9 and 1.0.7 (see #767734), but maybe this is
something else, although related, but only showing up when trigger
states are involved, this might also be a problem in apt in stable or
just with newer versions, that might have only emerged due to the
recent changes in dpkg. It might be that this only happens when
packages break on upgrade as it happened with some of the dbus
versions that failed on postinst, etc. Dunno, really.

But in any case the --force-configure-any default change would be a
workaround in dpkg to cover for the dist-upgrade problem of apt not
upgrading itself as the first thing, and reexecuting. I also don't
know what's the stance on requiring specific packaging tools to be
upgraded first? Which might mean that if the issue is still there it
could be fixed in apt proper.

 Apologies for the less optimistic view on my end and thanks for your work.

Sure, fully understood, no problem.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-17 Thread Niels Thykier 
On 2014-12-16 06:22, Guillem Jover wrote:
 Package: release.debian.org
 Severity: normal
 User: release.debian@packages.debian.org
 Usertags: unblock
 Control: block -1 by 770627
 
 Hi!
 
 I'd like to upload dpkg 1.17.23 with the following changes. I've
 excluded and filtered translation changes. I'm including a patch series
 instead of a debdiff, because it should be clearer and easier to spot
 specific changes you might not want to approve?
 

Hi,

Thanks, the changes look reasonable.
It
 I'd prefer if 1.17.22 could be unblocked before uploading this, because
 that version is way better than the one currently in testing, and it is
 causing fewer upgrade issues. Otherwise I'll just merge both unblock
 requests.
 

Apologies, but I am not entirely convinced here.  I would strongly
prefer /not/ having trigger regressions right now.

In fact, I am honestly considering to request having the trigger change
reverted if 1.17.23 does not solve the issues without introduce another
regression.
  We are one and a half month into the freeze and we still do not have a
clean upgrade path on a package level.  I am deeply concerned that we
have been missing out on (e.g.) the systemd upgrade reports because of this.

 I've delayed a bit the request because there are still some packages
 with trigger cycles that have not been uploaded yet, I can start taking
 a look on delayed NMUs and wait for those or upload .23 right away and
 possibly prepare a .24 with those additional versioned Breaks, whichever
 you prefer.
 

It seems that only gxine and icecc are missing now.  If so, please go
ahead with the .23 with versioned breaks for them as well.  Worst case,
I will have them removed from testing - best case, they will be fixed.
  I will take the political fall-out of this and notify the maintainers
of the affected packages.  Let me know if I missed any packages.

 I've also not added the --force-configure-any default switch, because we
 don't really know what happened with apt and dbus there, and if apt from
 stable is affected or not. Given the recent dpkg, apt, and dbus changes
 I think I'd rather let this as is, and wait in case it shows up again,
 which should give us more information (due to the new apt not eating
 dpkg's output).
 

Noted, though I sincerely hope it is fixed.  I /might/ be convinced to
accept a .24 for this particular issue.

 Here's the full tentative changelog (with the translation changes
 already in master):
 
 ,---
 [...]
 `---
 
 Thanks,
 Guillem
 

Apologies for the less optimistic view on my end and thanks for your work.


~Niels


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#773256: pre-approval: unblock: dpkg/1.17.23

2014-12-15 Thread Guillem Jover 
Package: release.debian.org
Severity: normal
User: release.debian@packages.debian.org
Usertags: unblock
Control: block -1 by 770627

Hi!

I'd like to upload dpkg 1.17.23 with the following changes. I've
excluded and filtered translation changes. I'm including a patch series
instead of a debdiff, because it should be clearer and easier to spot
specific changes you might not want to approve?

I'd prefer if 1.17.22 could be unblocked before uploading this, because
that version is way better than the one currently in testing, and it is
causing fewer upgrade issues. Otherwise I'll just merge both unblock
requests.

I've delayed a bit the request because there are still some packages
with trigger cycles that have not been uploaded yet, I can start taking
a look on delayed NMUs and wait for those or upload .23 right away and
possibly prepare a .24 with those additional versioned Breaks, whichever
you prefer.

I've also not added the --force-configure-any default switch, because we
don't really know what happened with apt and dbus there, and if apt from
stable is affected or not. Given the recent dpkg, apt, and dbus changes
I think I'd rather let this as is, and wait in case it shows up again,
which should give us more information (due to the new apt not eating
dpkg's output).

Here's the full tentative changelog (with the translation changes
already in master):

,---
dpkg (1.17.23) UNRELEASED; urgency=low

  [ Guillem Jover ]
  * Use a matching group instead of ${^MATCH} in s/// in dselect build script.
  * Skip tar extractor tests if tar is not GNU tar = 1.27.
  * Reset the trigger cycle tracking on unsatisfied dependencies.
Closes: #771730
  * Fix out-of-bounds buffer read accesses when parsing field and trigger
names or checking package ownership of conffiles and directories.
Reported by Joshua Rogers megaman...@gmail.com.
  * Add versioned Breaks on packages creating trigger cycles. Namely
apt-cudf, ccache, cups, distcc, fusionforge-plugin-mediawiki, gap-core,
hoogle, libjs-protoaculous and xfonts-traditional.

  [ Updated programs translations ]
  * Basque (Iñaki Larrañaga Murgoitio). Closes: #771893
  * Catalan (Guillem Jover).
  * Czech (Miroslav Kure).
  * Esperanto (Felipe Castro).
  * French (Sébastien Poher).
  * Italian (Milo Casagrande).
  * Swedish (Peter Krefting).
  * Portuguese (Miguel Figueiredo).
  * Russian (Yuri Kozlov). Closes: #771691
  * Simplified Chinese (Zhou Mo). Closes: #771264
  * Spanish (Javier Fernández-Sanguino)
  * Thai (Theppitak Karoonboonyanan). Closes: #772965

  [ Updated scripts translations ]
  * Catalan (Guillem Jover).
  * Polish (Łukasz Dulny).
  * Russian (Yuri Kozlov). Closes: #772841

  [ Updated manpages translations ]
  * French (Sébastien Poher).
  * Italian (Beatrice Torracca). Closes: #771673

  [ Updated dselect translations ]
  * Catalan (Guillem Jover).
  * Czech (Miroslav Kure).
  * Norwegian Bokmål (Hans Fredrik Nordhaug).
  * Polish (Łukasz Dulny).
  * Portuguese (Miguel Figueiredo).
  * Russian (Yuri Kozlov). Closes: #771682
  * Spanish (Javier Fernández-Sanguino)
  * Vietnamese (Trần Ngọc Quân).

 -- Guillem Jover guil...@debian.org  Fri, 28 Nov 2014 02:41:17 +0100
`---

Thanks,
Guillem
From a5d6590b7a2a9f649f5aab01e3410bac62e12a76 Mon Sep 17 00:00:00 2001
From: Guillem Jover guil...@debian.org
Date: Fri, 5 Dec 2014 23:29:03 +0100
Subject: [PATCH 1/5] dselect: Use a matching group instead of ${^MATCH} in
 s///

It seems that this is not supported or does not work in perl 5.14.0,
although it should be since 5.10.0. Switch to a group matching to
allow using older perl version from stable.
---
 debian/changelog | 3 +++
 dselect/mkcurkeys.pl | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/debian/changelog b/debian/changelog
index 432d705..0ff1d9b 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,5 +1,8 @@
 dpkg (1.17.23) UNRELEASED; urgency=low
 
+  [ Guillem Jover ]
+  * Use a matching group instead of ${^MATCH} in s/// in dselect build script.
+
   [ Updated programs translations ]
   * Basque (Iñaki Larrañaga Murgoitio). Closes: #771893
   * Catalan (Guillem Jover).
diff --git a/dselect/mkcurkeys.pl b/dselect/mkcurkeys.pl
index d859376..342d491 100755
--- a/dselect/mkcurkeys.pl
+++ b/dselect/mkcurkeys.pl
@@ -140,6 +140,6 @@ sub capit {
 sub p {
 my ($k, $v) = @_;
 
-$v =~ s/[\\]/\\${^MATCH}/pg;
+$v =~ s/([\\])/\\$1/g;
 printf(  { %-15s \%-20s },\n, $k . ',', $v . '') or die $!;
 }
-- 
2.2.0.rc0.207.ga3a616c

From 88999a73fc4d77b3ab1072f3102cfbeaff94ab31 Mon Sep 17 00:00:00 2001
From: Guillem Jover guil...@debian.org
Date: Sat, 6 Dec 2014 00:11:14 +0100
Subject: [PATCH 2/5] libdpkg: Skip tar extractor tests if tar is not GNU tar
 = 1.27

This allows building on older systems.
---
 debian/changelog  |  1 +
 lib/dpkg/test/t-tar.t | 13 -
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/debian/changelog b/debian/changelog
index 0ff1d9b..da54d81 100644