Bug#850887: Decide proper solution for binutils' mips* bug

[James Cowgill]

Hi,

On 12/01/17 14:54, Lisandro Damián Nicanor Pérez Meyer wrote:
> I would like to point out that it would be preferable if, in case a patch is 
> preferable over going back to the last know version to work, either Matthias 
> or a mips porter points out which of the two proposed patches is preferable.
> 
> For the time being I'm testing the patch I submited to the bug, but I have no 
> preference over any of them (nor technical grounds to discuss).

Both patches posted in the upstream bug should work. The first one fixes
a bug in the MIPS back end so that local symbols are sorted before
global symbols. This is probably the safer (although larger) patch
because it only touches the MIPS back end to try and bring it into line
with other architectures. The second patch prevents the questionable
local symbols from every appearing (so no sorting is necessary). This
should also be correct, although it will visibly change the contents of
the dynamic symbol table on all arches so I am slightly more
apprehensive because of that.

Side note: the patch you uploaded is not totally correct because it
isn't applied when building cross binutils (__mips__ will not be defined
there).

Thanks,
James



signature.asc
Description: OpenPGP digital signature

Bug#850887: Decide proper solution for binutils' mips* bug

[Lisandro Damián Nicanor Pérez Meyer]

I would like to point out that it would be preferable if, in case a patch is 
preferable over going back to the last know version to work, either Matthias 
or a mips porter points out which of the two proposed patches is preferable.

For the time being I'm testing the patch I submited to the bug, but I have no 
preference over any of them (nor technical grounds to discuss).

Thanks, Lisandro.

-- 
Lisandro Damián Nicanor Pérez Meyer
http://perezmeyer.com.ar/
http://perezmeyer.blogspot.com/


signature.asc
Description: This is a digitally signed message part.

Bug#850887: [TIMELY for TC members] Interim Ballot Proposal: #850887 binutils mips

[Didier 'OdyX' Raboud]

Le mercredi, 11 janvier 2017, 22.38:33 h CET Sam Hartman a écrit :
> I heard back from doko today.  We can expect a reply tomorrow.  We also
> talked briefly about the issue.

Good. Thanks for this work.

> Realistically, i cannot imagine the TC coming to any final decision on
> something like this in under three weeks.  That timeline seems fairly
> aggressive actually.

Right. It implies that every involved party (Lisandro, the Release Team, 
Matthias, and the TC members) can provide a high bandwidth to that issue.

> However, I think the TC could act much more quickly in an interim
> capacity.

Yes.

> I personally believe that having packages building is a better interim
> state than the status quo.  There are risks to an interim measure.  We
> could have packages in the archive that build but fail to function
> correctly.

Ack.

> Depending on what we do long term, we could end up replacing
> packges currently in Stretch with packages we can no longer rebuild.

The worst case is needing to rebootstrap mips' stretch either from jessie, or 
in a cross-bootstrap situation, right ?

> I personally think that when I weigh those risks against my estimate of
> their probability, I think it makes sense to adopt an interim measure.

I agree.

> Roughly I propose to override the maintainer and permit an NMU to be
> made for this issue.

It would be much preferable if Matthias would accept that patch, or revert to 
the previous working version. But if it needs an NMU, so be it.

(Mid-term, I want to understand how it can make sense to change Debian's
 binutils' tracked branch (2.27→2.28) three days before the transition
 freeze.
)

> The decision stands until the maintainer fixes the bug or Stretch
> releases, or another resolution is passed (presumably with a more
> permanent decision).

Absolutely.

> Yes, that means that the maintainer could reintroduce the bug and revert
> the NMU immediately on the release of Stretch.

Absolutely. I wouldn't support a resolution enforcing that NMU in unstable 
forever. New release cycles are our reset button, really.

> I propose to be very agressive in calling for a vote on the following
> ballot.
> I plan to call for a vote in 24 hours if I get support from at least one
> TC member and no objections from within the TC or release team.

Let this mail be my support !

> Also, within that time, we should hear from doko.  His input may change
> my thinking even for an interim measure.

Yes, absolutely. There was only one mail from Matthias on the #844227, only to 
NAK the NMU, on an RC bug opened since November, his input is long overdue!

> 
> In #850887, the Debian Technical Committee was asked to choose a
> solution for #840227, a bug that prevents a significant number of
> packages from building on the mips architecture.  Given the upcoming
> Stretch freeze, this issue is urgent.
> 
> As an interim measure, using its powers under section 6.1.4 of the
> Debian Constitution, the Technical Committee overrules Matthias
> Klose's decision to revert the NMU of binutils fixing #840227.  The
> committee requests Lisandro Damián Nicanor Pérez Meyer to make a new
> NMU fixing #840227.
> 
> The committee requests the release team to support the interim nature of
> this solution and if a permanent solution is adopted before the release of
> Stretch, to consider including that solution in Stretch even if the freeze
> criteria would not normally permit such consideration.
> 
> In addition, the committee requests the stable release managers for Stretch
> to consider including the eventual upstream solution for this issue into a
> stretch update.
> 
> This interim decision stands until the release of Stretch, until it is
> replaced by resolution, or until the binutils maintainer fixes #840227 in
> some other manner.
> 
> 
> Choice 1:  Approve the Resolution (3:1 majority)
> Choice 2: Reject this Interim Measure
> Choice 3: Further Discussion
> 

I agree with the ballot including Ian's suggestion, and think we should start 
the vote as early as this week-end.

Cheers,
OdyX

signature.asc
Description: This is a digitally signed message part.

Bug#850967: marked as done (Clarify /usr/bin/foo should not be hardcoded even in upstream parts)

[Debian Bug Tracking System]

Your message dated Thu, 12 Jan 2017 15:08:05 +0100
with message-id <5566268.jkbec7e...@odyx.org>
and subject line Re: Bug#850967: Clarify /usr/bin/foo should not be hardcoded 
even in upstream parts [and 1 more messages]
has caused the Debian Bug report #850967,
regarding Clarify /usr/bin/foo should not be hardcoded even in upstream parts
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)


-- 
850967: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850967
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems
--- Begin Message ---
Package: tech-ctte
Control: block 850657 by -1

Policy 6.1 says
| Programs called from maintainer scripts should not normally have a
| path prepended to them.

Ie, programs that are on PATH should be found via the PATH rather than
by hardcoding /usr/bin/foo or whatever.  In general, I think we
normally, at least in software written specifically for Debian, apply
this not just to maintainer scripts but to all program execution.

However, this is not always uncontroversial with some upstreams.  This
causes a particular problem when the upstream is also heavily involved
with the Debian maintenance.


I would like the TC to:

 * Declare that Debian policy should be clarified to say that programs
   in Debian (not just maintainer scripts) should not hardcode the
   location of the binaries in /{usr/,}{bin,sbin} they execute.

 * Say that this applies even when the program needs to find pieces of
   the same (or closely related) package.

 * Say that where technically feasible, this should usually be done
   for other kinds of search paths (LD_LIBRARY_PATH, PYTHONPATH,
   PERLLIB, etc.), too.

 * Provide an informal opinion that this policy ought to apply to
   gpg-agent as requested in #850657.


I don't know if it is necessary to rehearse the arguments about this
issue for the TC, but I will try to do so briefly.


The argument in favour of finding programs (and libraries) on the
PATH:

This allows substitution and wrapping of programs (by adding a
directory to PATH containing a stunt version of the program, or by
adding a wrapper or replacement to /usr/local or ~/bin).  This can be
useful in a wide variety of situations.  It is a well-known debugging
technique.  It can be used by individual users, to modify the
behaviour of their system.  It can be profitably used by things like
test suites and audit tools.  It can be used by depending software to
work around bugs.

Most of these uses can be satisfied in some other way, but the other
ways have downsides.  For example, moving the actual binary aside is a
possibility (and supported by things like dpkg-divert); but it affects
the whole system, so requires administrator privilege; it is
unsuitable on a multiuser machine or for test suites; and if done
ad-hoc it is too easy to forget to put back and leave the system in a
weird state.

Explicit configuration or command line options to change which
subcommands are used are a good thing, but: they are usually fiddly
and certainly not always provided; this can often involve complicated
nesting (eg run `foo' with --bar-program pointing to a stunt `bar'
script which passes --wombat-program pointing to your stunt `wombat'
script); and lack of such an option at any level stymies this
approach.


I will quote the counterarguments from Daniel Kahn Gillmor, as they
are fairly typical of the responses from upstreams:

  In this bug report, you're asking a tightly-coupled suite of tools to
  invoke each other via the PATH, which offers another avenue of potential
  breakage; upstream regularly receives reports from people who *don't
  even know what version of gpg their system is running* because of
  multiple copies of the thing having been installed in various places on
  their system (either deliberately or incidentally, but in either case
  forgotten about by the local sysadmin).

  It's entirely reasonable for a tightly-coupled suite to want to invoke
  its own tools in this situation.  we have enough to debug in GnuPG as it
  is :/

Perhaps this is indeed a problem for some upstreams.  But I have two
responses to this which I think are each sufficient to rebut it:

Firstly, we are talking about this in the context of Debian.  In
Debian we have `reportbug', which the majority of people use to file
bug reports.

It would be simple for reportbug to report on these kind of anomalous
situations and mention them in the bug report.  It could, for example,
check to see if there is overlap between the package being reported
(and its dependencies), and /usr/local.  I don't know if it does
already, but if it doesn't 

Bug#850887: [TIMELY for TC members] Interim Ballot Proposal: #850887 binutils mips

[Sam Hartman]

> "Ian" == Ian Jackson  writes:


Ian> You should explicitly state whether you want this NMU to be
Ian> DELAYED.

Good point.
I think we don't want a delay.
Updated the ballot in git.

Bug#850887: [TIMELY for TC members] Interim Ballot Proposal: #850887 binutils mips

[Ian Jackson]

Sam Hartman writes ("Bug#850887: [TIMELY for TC members] Interim Ballot 
Proposal: #850887 binutils mips"):
> [stuff]

Thanks for pushing this issue, for your IMO correct approach to the
process, and for your clear and straightforward communication.

> 
> In #850887, the Debian Technical Committee was asked to choose a
> solution for #840227, a bug that prevents a significant number of
> packages from building on the mips architecture.  Given the upcoming
> Stretch freeze, this issue is urgent.
> 
> As an interim measure, using its powers under section 6.1.4 of the
> Debian Constitution, the Technical Committee overrules Matthias
> Klose's decision to revert the NMU of binutils fixing #840227.  The
> committee requests Lisandro Damián Nicanor Pérez Meyer to make a new
> NMU fixing #840227.

You should explicitly state whether you want this NMU to be DELAYED.

Ian.

-- 
Ian Jackson    These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts [and 1 more messages]

[Ian Jackson]

Josh Triplett writes ("Bug#850967: Clarify /usr/bin/foo should not be hardcoded 
even in upstream parts"):
> I find it surprising that this has ended up in front of the Technical
> Committee before it has, for instance, ended up on debian-policy or
> debian-devel for discussion.  While Policy would not make a change that
> would instantly declare a large number of packages de-jure buggy, those
> two lists nonetheless seem like the right place to have this discussion
> outside the context of any particular package.

Maybe I should have tried -devel.  -policy is no good because the
policy process cannot make controversial changes.  (As an aside, the
process also so little rewards the policy editors with autonomy that
it's not an attractive task, so the policy maintenance is slow due to
lack of effort.)

Stuart Prescott writes ("Bug#850967: Clarify /usr/bin/foo should not be 
hardcoded even in upstream parts"):
> Nikolaus highlights two failure modes that we see sufficiently often that 
> #debian even has boilerplate help text to deal with them:
>
> [python things]

Thanks for the data points.  This is troubling.  I know that Python
has historically had confusing approaches to its include path
management.

> By giving the local admin the ability to override tools using their
> PATH we give them great power and flexibility at the expense of
> robustness. We give the user a loaded gun, help them aim it at their
> foot and then stand back.

Perhaps this is indeed true for Python.


Anyway, I don't think this request of mine is going anywhere.  I'm
definitely still unconvinced about gpg-agent (#850657) and I'm
disapointed, but there is no point wasting any more of anyone's time
about this.

I suggest that the TC close #850967 NFA, and that the Debian gnupg
maintainers tag #850657 wontfix (and perhaps close it).

I won't promise to stop asking Debian maintainers of other packages to
make this change, when it trips me up.

Ian.

-- 
Ian Jackson    These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Josh Triplett]

On Wed, 11 Jan 2017 13:51:27 -0500 Daniel Kahn Gillmor  wrote:
> Please do not do this to gpg-agent unless upstream is fine with this
> change.  I have more important things i want to consider divergence from
> upstream about, and i don't think this particular scenario is a healthy
> use of debian policy.
> 
> I laud Ian's goals of making debugging easier, but the particular
> pattern he describes (wrapping stuff and putting the wrappers in $PATH)
> is not the only way to ease debugging, and is at least as vulnerable to
> the problems he associates with other techniques ("too easy to forget to
> put back", etc) as they are.

As another technical alternative, which I haven't seen mentioned
elsewhere in this thread or related bug reports: when I need to override
a packaged binary or file temporarily for debugging purposes, without
forgetting to restore it later, I tend to use "mount --bind
/my/replacement /usr/bin/foo".  For instance, for local testing while
developing dh-cargo, which required a newer version of Cargo than
packaged in Debian at the time, I built a local version of Cargo and did
"mount --bind ~/src/cargo/target/debug/cargo /usr/bin/cargo".  That
allowed me to easily test-build packages before the availability of a
Debian package of a sufficiently new Cargo.

This technique also has the advantage of vanishing on reboot, reverting
back to the system binary.

This seems appropriate for a local debugging technique with a temporary
lifetime.  Such local interposition hacks help determine or debug a
solution to some problem, which should ideally then lead to a fix to the
software itself, fixing the problem for everyone and allowing the
software to work out of the box for more people.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Josh Triplett]

On Wed, 11 Jan 2017 14:59:06 -0500 Sam Hartman  wrote:
> I'll note that the practice of hard-coding paths is fairly common.
> 
> 
> One common cause for this is programs that don't want to rely on PATH
> for calling exec.  Systemd is a particularly interesting example.
> ExecStart and related arguments in systemd units are required to include
> full paths.
> 
> I am very uncomfortable with the idea of setting policy here.  I find I
> tend to agree with Daniel's position a bit more than Ian's.
> In particular, I definitely think that for closely-related versions of
> software, making sure the same versions are used is helpful.
> I've hurt myself more by having parts of something built in /usr/local
> than I have not being able to override things for debugging.
> However, I
> think that both parties have valid points.
> So, I'd be much more comfortable if we wanted to help make people more
> aware of the tradeoffs than I would setting policy.

I've likewise run into far too many debugging and triage adventures
involving a program not running what it thought it was running, or for
interpreted languages, picking up a random local version of a module.
(At this point, when helping debug something on someone else's system, I
check early on for issues related to loading local copies of things
fairly early in the process.)  Today's clever hack becomes tomorrow's
technical debt.  And the more certainty a sysadmin has that they'd never
forget they had a locally installed override/hack, the deeper the
head-shaped indentations in the nearest desk or wall when it turns out
they do.

I don't want to argue that we should never invoke programs via $PATH, or
that we should never invoke programs via full pathnames.  I can see
cases for both.  Rather, I'd agree with Sam Hartman's comment above that
we should document both approaches and the tradeoffs between them.  But
in terms of ecosystem fragility, it seems far worse to have the software
in Debian behave differently in this particular regard than the software
upstream, making it even more challenging to debug.

I find it surprising that this has ended up in front of the Technical
Committee before it has, for instance, ended up on debian-policy or
debian-devel for discussion.  While Policy would not make a change that
would instantly declare a large number of packages de-jure buggy, those
two lists nonetheless seem like the right place to have this discussion
outside the context of any particular package.