Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Ansgar Burchardt]

Philip Hands writes:
> I stumbled across 'proot' while looking into the background for this,
> which seems to be able to provide the effect of a bind mount without
> needing root privilege, and would presumably deal with Ian's original
> problem quite nicely.

If you enable unprivileged user namespaces (the upstream kernel default;
disabled by a Debian patch if I remember correctly), you can just use
`unshare` and `mount --bind` on Linux:

  # echo 1 > /proc/sys/kernel/unprivileged_userns_clone

  $ unshare -r -m /bin/sh -c 'mount --bind /usr/bin/gpg /usr/bin/true; 
/usr/bin/true --version'
  gpg (GnuPG) 2.1.17
  [...]

Ansgar

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Josh Triplett]

On Fri, Jan 13, 2017 at 10:04:34AM -0500, Sam Hartman wrote:
> > "Josh" == Josh Triplett  writes:
> 
> Josh> As another technical alternative, which I haven't seen
> Josh> mentioned elsewhere in this thread or related bug reports:
> Josh> when I need to override a packaged binary or file temporarily
> Josh> for debugging purposes, without forgetting to restore it
> Josh> later, I tend to use "mount --bind /my/replacement
> Josh> /usr/bin/foo".  For instance, for local testing while
> Josh> developing dh-cargo, which required a newer version of Cargo
> Josh> than packaged in Debian at the time, I built a local version
> Josh> of Cargo and did "mount --bind ~/src/cargo/target/debug/cargo
> Josh> /usr/bin/cargo".  That allowed me to easily test-build
> Josh> packages before the availability of a Debian package of a
> Josh> sufficiently new Cargo.
> 
> O, cool, that's really need.
> 
> And as a throw-back to an alternate Plan9 history, you could presumably
> unshare your mount namespace and even do that for a subset of the
> processes on a system, getting almost all the benefits of PATH.

Yes.  Years ago, when Debian transitioned /bin/sh from bash to dash,
Marco d'Itri posted a sample workaround for any scripts assuming bash,
which involved unsharing the mount namespace, bind-mounting /bin/bash
over /bin/sh, and then exec'ing a program.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Philip Hands]

Sam Hartman  writes:

>> "Josh" == Josh Triplett  writes:
>
> Josh> As another technical alternative, which I haven't seen
> Josh> mentioned elsewhere in this thread or related bug reports:
> Josh> when I need to override a packaged binary or file temporarily
> Josh> for debugging purposes, without forgetting to restore it
> Josh> later, I tend to use "mount --bind /my/replacement
> Josh> /usr/bin/foo".  For instance, for local testing while
> Josh> developing dh-cargo, which required a newer version of Cargo
> Josh> than packaged in Debian at the time, I built a local version
> Josh> of Cargo and did "mount --bind ~/src/cargo/target/debug/cargo
> Josh> /usr/bin/cargo".  That allowed me to easily test-build
> Josh> packages before the availability of a Debian package of a
> Josh> sufficiently new Cargo.
>
> O, cool, that's really need.
>
> And as a throw-back to an alternate Plan9 history, you could presumably
> unshare your mount namespace and even do that for a subset of the
> processes on a system, getting almost all the benefits of PATH.

I stumbled across 'proot' while looking into the background for this,
which seems to be able to provide the effect of a bind mount without
needing root privilege, and would presumably deal with Ian's original
problem quite nicely.

It's currently broken in stretch though (#847292).

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Sam Hartman]

> "Josh" == Josh Triplett  writes:

Josh> As another technical alternative, which I haven't seen
Josh> mentioned elsewhere in this thread or related bug reports:
Josh> when I need to override a packaged binary or file temporarily
Josh> for debugging purposes, without forgetting to restore it
Josh> later, I tend to use "mount --bind /my/replacement
Josh> /usr/bin/foo".  For instance, for local testing while
Josh> developing dh-cargo, which required a newer version of Cargo
Josh> than packaged in Debian at the time, I built a local version
Josh> of Cargo and did "mount --bind ~/src/cargo/target/debug/cargo
Josh> /usr/bin/cargo".  That allowed me to easily test-build
Josh> packages before the availability of a Debian package of a
Josh> sufficiently new Cargo.

O, cool, that's really need.

And as a throw-back to an alternate Plan9 history, you could presumably
unshare your mount namespace and even do that for a subset of the
processes on a system, getting almost all the benefits of PATH.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts [and 1 more messages]

[Ian Jackson]

Josh Triplett writes ("Bug#850967: Clarify /usr/bin/foo should not be hardcoded 
even in upstream parts"):
> I find it surprising that this has ended up in front of the Technical
> Committee before it has, for instance, ended up on debian-policy or
> debian-devel for discussion.  While Policy would not make a change that
> would instantly declare a large number of packages de-jure buggy, those
> two lists nonetheless seem like the right place to have this discussion
> outside the context of any particular package.

Maybe I should have tried -devel.  -policy is no good because the
policy process cannot make controversial changes.  (As an aside, the
process also so little rewards the policy editors with autonomy that
it's not an attractive task, so the policy maintenance is slow due to
lack of effort.)

Stuart Prescott writes ("Bug#850967: Clarify /usr/bin/foo should not be 
hardcoded even in upstream parts"):
> Nikolaus highlights two failure modes that we see sufficiently often that 
> #debian even has boilerplate help text to deal with them:
>
> [python things]

Thanks for the data points.  This is troubling.  I know that Python
has historically had confusing approaches to its include path
management.

> By giving the local admin the ability to override tools using their
> PATH we give them great power and flexibility at the expense of
> robustness. We give the user a loaded gun, help them aim it at their
> foot and then stand back.

Perhaps this is indeed true for Python.


Anyway, I don't think this request of mine is going anywhere.  I'm
definitely still unconvinced about gpg-agent (#850657) and I'm
disapointed, but there is no point wasting any more of anyone's time
about this.

I suggest that the TC close #850967 NFA, and that the Debian gnupg
maintainers tag #850657 wontfix (and perhaps close it).

I won't promise to stop asking Debian maintainers of other packages to
make this change, when it trips me up.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Josh Triplett]

On Wed, 11 Jan 2017 14:59:06 -0500 Sam Hartman  wrote:
> I'll note that the practice of hard-coding paths is fairly common.
> 
> 
> One common cause for this is programs that don't want to rely on PATH
> for calling exec.  Systemd is a particularly interesting example.
> ExecStart and related arguments in systemd units are required to include
> full paths.
> 
> I am very uncomfortable with the idea of setting policy here.  I find I
> tend to agree with Daniel's position a bit more than Ian's.
> In particular, I definitely think that for closely-related versions of
> software, making sure the same versions are used is helpful.
> I've hurt myself more by having parts of something built in /usr/local
> than I have not being able to override things for debugging.
> However, I
> think that both parties have valid points.
> So, I'd be much more comfortable if we wanted to help make people more
> aware of the tradeoffs than I would setting policy.

I've likewise run into far too many debugging and triage adventures
involving a program not running what it thought it was running, or for
interpreted languages, picking up a random local version of a module.
(At this point, when helping debug something on someone else's system, I
check early on for issues related to loading local copies of things
fairly early in the process.)  Today's clever hack becomes tomorrow's
technical debt.  And the more certainty a sysadmin has that they'd never
forget they had a locally installed override/hack, the deeper the
head-shaped indentations in the nearest desk or wall when it turns out
they do.

I don't want to argue that we should never invoke programs via $PATH, or
that we should never invoke programs via full pathnames.  I can see
cases for both.  Rather, I'd agree with Sam Hartman's comment above that
we should document both approaches and the tradeoffs between them.  But
in terms of ecosystem fragility, it seems far worse to have the software
in Debian behave differently in this particular regard than the software
upstream, making it even more challenging to debug.

I find it surprising that this has ended up in front of the Technical
Committee before it has, for instance, ended up on debian-policy or
debian-devel for discussion.  While Policy would not make a change that
would instantly declare a large number of packages de-jure buggy, those
two lists nonetheless seem like the right place to have this discussion
outside the context of any particular package.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Josh Triplett]

On Wed, 11 Jan 2017 13:51:27 -0500 Daniel Kahn Gillmor  wrote:
> Please do not do this to gpg-agent unless upstream is fine with this
> change.  I have more important things i want to consider divergence from
> upstream about, and i don't think this particular scenario is a healthy
> use of debian policy.
> 
> I laud Ian's goals of making debugging easier, but the particular
> pattern he describes (wrapping stuff and putting the wrappers in $PATH)
> is not the only way to ease debugging, and is at least as vulnerable to
> the problems he associates with other techniques ("too easy to forget to
> put back", etc) as they are.

As another technical alternative, which I haven't seen mentioned
elsewhere in this thread or related bug reports: when I need to override
a packaged binary or file temporarily for debugging purposes, without
forgetting to restore it later, I tend to use "mount --bind
/my/replacement /usr/bin/foo".  For instance, for local testing while
developing dh-cargo, which required a newer version of Cargo than
packaged in Debian at the time, I built a local version of Cargo and did
"mount --bind ~/src/cargo/target/debug/cargo /usr/bin/cargo".  That
allowed me to easily test-build packages before the availability of a
Debian package of a sufficiently new Cargo.

This technique also has the advantage of vanishing on reboot, reverting
back to the system binary.

This seems appropriate for a local debugging technique with a temporary
lifetime.  Such local interposition hacks help determine or debug a
solution to some problem, which should ideally then lead to a fix to the
software itself, fixing the problem for everyone and allowing the
software to work out of the box for more people.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Stuart Prescott]

Nikolaus Rath wrote:

>>> Policy 6.1 says
>>> | Programs called from maintainer scripts should not normally have a
>>> | path prepended to them.
>>>
>>> Ie, programs that are on PATH should be found via the PATH rather than
>>> by hardcoding /usr/bin/foo or whatever.  In general, I think we
>>> normally, at least in software written specifically for Debian, apply
>>> this not just to maintainer scripts but to all program execution.
>>
>> I agree that it's a common practice for software written specifically
>> for Debian.  I'm not convinced it's a common practice elsewhere, and i'm
>> definitely not convinced that we should mandate divergence from upstream
>> for this purpose.
> 
> Just as a datapoint, in other communities there is actually a trend in
> the opposite direction. For example, for the Python ecosystem there is
> an increasing drive to establish a "system Python" that ignores Python
> modules in /usr/local and $PYTHONPATH, specifically to avoid potential
> interference of user installed modules with distribution Python scripts.
> 
> I have a lot of sympathie for this idea, and I think it would be good to
> keep this in mind when discussing potential clarifications of policy.

Nikolaus highlights two failure modes that we see sufficiently often that 
#debian even has boilerplate help text to deal with them:

* user installs their own version of python in /usr/local and then packaged 
python programs start failing (missing modules, incompatible interpreter 
etc), possibly even to the point of breaking package installs/updates such 
that the user can no longer use apt until they work out that is the problem 
and rectify it.

* user installs their own versions of python modules in /usr/local and then 
packaged python programs/modules start failing because the new versions are 
incompatible; this can also easily break maintainer scripts and so break 
package installs/upgrades.

By giving the local admin the ability to override tools using their PATH we 
give them great power and flexibility at the expense of robustness. We give 
the user a loaded gun, help them aim it at their foot and then stand back. 
Shouldn't we be trying to engineer robust solutions rather than footguns? Is 
helping people to accidentally break apt a good thing? Is opening the door 
to subvert gnupg¹ where we want to go? The principle should be one of 
isolation not accidental, unwanted and detrimental coupling.

cheers
Stuart


¹ wild (but fitting) hyperbole; then again, let me quietly add something as 
group:staff to /usr/local/bin...


-- 
Stuart Prescotthttp://www.nanonanonano.net/   stu...@nanonanonano.net
Debian Developer   http://www.debian.org/ stu...@debian.org
GPG fingerprint90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Nikolaus Rath]

>> Policy 6.1 says
>> | Programs called from maintainer scripts should not normally have a
>> | path prepended to them.
>>
>> Ie, programs that are on PATH should be found via the PATH rather than
>> by hardcoding /usr/bin/foo or whatever.  In general, I think we
>> normally, at least in software written specifically for Debian, apply
>> this not just to maintainer scripts but to all program execution.
>
> I agree that it's a common practice for software written specifically
> for Debian.  I'm not convinced it's a common practice elsewhere, and i'm
> definitely not convinced that we should mandate divergence from upstream
> for this purpose.

Just as a datapoint, in other communities there is actually a trend in
the opposite direction. For example, for the Python ecosystem there is
an increasing drive to establish a "system Python" that ignores Python
modules in /usr/local and $PYTHONPATH, specifically to avoid potential
interference of user installed modules with distribution Python scripts.

I have a lot of sympathie for this idea, and I think it would be good to
keep this in mind when discussing potential clarifications of policy.


Best,
Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts [and 2 more messages]

[Ian Jackson]

I see that I am Wrong On The Internet and my efforts not to distract
the TC have been futile.  Sorry.  Now I am falling subject to the same
problem but I really cannot let all of this go unchallenged.

Daniel Kahn Gillmor writes ("Re: Bug#850967: Clarify /usr/bin/foo should not be 
hardcoded even in upstream parts"):
> On Wed 2017-01-11 12:13:44 -0500, Ian Jackson 
> <ijack...@chiark.greenend.org.uk> wrote:
> > I think this argument is utterly wrong in principle.  It is contrary
> > to the whole point of Debian.
> 
> We clearly disagree here, though i think Ian might be overstating his
> case for rhetorical effect.

Certainly not.  Our primary priority should be to make it easy for
users to make their own choices.  The argument being made here is that
it should be made harder for users to make certain modifications
because that produces confusing bug reports.  Outrageous!

If there were a significant risk of *users* being troubled by their
own old locally-built stuff in /usr/local, then that would be a good
argument.  (For example, this might be true in the case of a program
which has an unstable command line API which means that it is not
compatible with other versions of its callers.  But of course such a
thing probably wouldn't be on PATH at all.)


Sam Hartman writes ("Re: Bug#850967: Clarify /usr/bin/foo should not be 
hardcoded even in upstream parts"):
> I'll note that the practice of hard-coding paths is fairly common.

It is a common bug.  There are many common bugs.  That does not stop
them being bugs.

Of course upstream projects often do this because they need their
software to be installable in nonstandard locations and still find all
of their pieces.  So such a thing is not necessarily a bug in an
upstream package.  But this is not necessary in Debian packages, and
then the restriction serves no purpose.

> One common cause for this is programs that don't want to rely on PATH
> for calling exec.  Systemd is a particularly interesting example.
> ExecStart and related arguments in systemd units are required to include
> full paths.

Do you really think that bringing utterly broken systemd behaviours
into this conversation would help ?  It's certainly raised my blood
pressure.


Didier 'OdyX' Raboud writes ("Re: Bug#850967: Clarify /usr/bin/foo should not 
be hardcoded even in upstream parts"):
> Dear Ian,
> >  * Say that this applies even when the program needs to find pieces of
> >the same (or closely related) package.
> 
> Reading this literally (which is what the Debian Policy is for),
> this would ask the Debian project to patch literally all its
> archive. For instance, that covers literally all shebang lines.

That shebang lines don't work this way is unfortunate but I'm not
suggesting it should be changed.

> I went and read #850657, and found myself agreeing with all points
> made by Daniel: if you want a gnupg that runs your custom gpg-agent
> for debugging purposes, building a patched src:gnupg is there for
> you; we provide source for a reason.

Have you ever talked some non-Debian person through the process of
building their own patched version of some program on their
Debian-derived system ?  This is not entirely straightforward, even if
they are an experienced software developer.  You will find yourself
making apologies (and also missing steps out, and making false
assumptions about the reasonableness of Debian source packages).

If you think this is so easy, I would welcome patches to this tutorial
manpage:
  
https://manpages.debian.org/cgi-bin/man.cgi?query=dgit-user=0=0=Debian+unstable+sid=html=en

Even though we can probably improve this (and we certainly should,
where we can), it's never going to be as easy as putting a wrapper
script on the path that invokes the subprogram with strace, or
whatever.

And it's complete overkill if what you wanted was to strace something
to see where it was going wrong (often you can't strace the parent for
some reason), or apply a ulimit, or pass a command line option that
the developers have for some reason decided shouldn't have a
corresponding config file option, or replace the program with a copy
of "true" or "false" to test the error handling, or whatever.

> All-in-all, I think that the decisions you would like the TC to
> issue would be against the project's consensus, and (which is more
> important for TC decisions) wouldn't be technically sound.

In many cases there is no way of ascertaining project consensus other
than a GR.  What you see in the mailing lists is the views of those
who are loud, stubborn and thick-skinned.  (FAOD a GR for this issue
would be ridiculous.)

But in this case, we can see fairly easily.  We have a large body of
software written specifically by Debian contributors to be part of
Debian.  We could see what that software does.

My experience is that in softw

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Didier 'OdyX' Raboud]

Dear Ian,

while I see where you come from, and can empathize with some of your points, I 
think that Daniel's following of upstream practices in using full paths is 
both fairly common, and a valid practice in Debian Packages.

Le mercredi, 11 janvier 2017, 17.13:44 h CET Ian Jackson a écrit :
> I would like the TC to:
> 
>  * Declare that Debian policy should be clarified to say that programs
>in Debian (not just maintainer scripts) should not hardcode the
>location of the binaries in /{usr/,}{bin,sbin} they execute.

Asking the TC to declare the above seems _very_ premature to me. In 
particular, in the frame of §6.3.6, I don't think enough efforts have been put 
in resolving this via consensus. Rather, I think it _is_ a resolved issue 
(over the course of years), but that you happen to disagree with the 
consensus.

>  * Say that this applies even when the program needs to find pieces of
>the same (or closely related) package.

Reading this literally (which is what the Debian Policy is for), this would 
ask the Debian project to patch literally all its archive. For instance, that 
covers literally all shebang lines.

>  * Provide an informal opinion that this policy ought to apply to
>gpg-agent as requested in #850657.

I went and read #850657, and found myself agreeing with all points made by 
Daniel: if you want a gnupg that runs your custom gpg-agent for debugging 
purposes, building a patched src:gnupg is there for you; we provide source for 
a reason.

All-in-all, I think that the decisions you would like the TC to issue would be 
against the project's consensus, and (which is more important for TC 
decisions) wouldn't be technically sound.

I'd be in favour of closing this bug above Further Discussion.

Cheers,
OdyX

P.S. No need to reply asking me to focus on a different issue…

signature.asc
Description: This is a digitally signed message part.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Ian Jackson]

Sam Hartman writes ("Re: Bug#850967: Clarify /usr/bin/foo should not be 
hardcoded even in upstream parts"):
> [some stuff]

Please concentrate on the MIPS binutils bug.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Tollef Fog Heen]

]] Sam Hartman 

> So, I'd be much more comfortable if we wanted to help make people more
> aware of the tradeoffs than I would setting policy.

Overall, I agree with Sam here, it's hard to come up with hard rules
about this, and as long as upstream and the Debian maintainer have
thought properly through the tradeoffs, I'd generally be inclined to
trust them.

(In the particular case of gpg and gpg-agent, I generally think
dpkg-diverting it should be fine for debugging, since few systems that
have private key material on them should be multiuser systems.)

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Sam Hartman]

I'll note that the practice of hard-coding paths is fairly common.


One common cause for this is programs that don't want to rely on PATH
for calling exec.  Systemd is a particularly interesting example.
ExecStart and related arguments in systemd units are required to include
full paths.

I am very uncomfortable with the idea of setting policy here.  I find I
tend to agree with Daniel's position a bit more than Ian's.
In particular, I definitely think that for closely-related versions of
software, making sure the same versions are used is helpful.
I've hurt myself more by having parts of something built in /usr/local
than I have not being able to override things for debugging.
However, I
think that both parties have valid points.
So, I'd be much more comfortable if we wanted to help make people more
aware of the tradeoffs than I would setting policy.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Ian Jackson]

Daniel Kahn Gillmor writes ("Re: Bug#850967: Clarify /usr/bin/foo should not be 
hardcoded even in upstream parts"):
> Ian, in the future please x-debbugs-cc me when you take our discussions
> to the tech-ctte :)

Sorry, I should have thought of that.

Thanks for your reply.  There are some things I might want to respond
in it but I don't want to distract the TC any further from #850887.
This issue isn't urgent, even though it is quite wide-ranging.
So, I won't press this now and instead I'll wait for a TC member to
pick it up.

Regards,
Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Daniel Kahn Gillmor]

Hi Ian (and hi tech-ctte!)--

On Wed 2017-01-11 12:13:44 -0500, Ian Jackson  
wrote:
> Package: tech-ctte
> Control: block 850657 by -1

Ian, in the future please x-debbugs-cc me when you take our discussions
to the tech-ctte :)

> Policy 6.1 says
> | Programs called from maintainer scripts should not normally have a
> | path prepended to them.
>
> Ie, programs that are on PATH should be found via the PATH rather than
> by hardcoding /usr/bin/foo or whatever.  In general, I think we
> normally, at least in software written specifically for Debian, apply
> this not just to maintainer scripts but to all program execution.

I agree that it's a common practice for software written specifically
for Debian.  I'm not convinced it's a common practice elsewhere, and i'm
definitely not convinced that we should mandate divergence from upstream
for this purpose.  Please see the examples i've already given in
#850657, or run

  find {/usr,}/{s,}bin -type f -print0 | xargs -0 strings | egrep 
'^(/usr)?/s?bin/[[:alnum:]]+$' | sort | uniq -c | sort -rn

on your own system to see that it's not actually particularly uncommon.

>  * Declare that Debian policy should be clarified to say that programs
>in Debian (not just maintainer scripts) should not hardcode the
>location of the binaries in /{usr/,}{bin,sbin} they execute.
>
>  * Say that this applies even when the program needs to find pieces of
>the same (or closely related) package.
>
>  * Say that where technically feasible, this should usually be done
>for other kinds of search paths (LD_LIBRARY_PATH, PYTHONPATH,
>PERLLIB, etc.), too.
>
>  * Provide an informal opinion that this policy ought to apply to
>gpg-agent as requested in #850657.

Please do not do this to gpg-agent unless upstream is fine with this
change.  I have more important things i want to consider divergence from
upstream about, and i don't think this particular scenario is a healthy
use of debian policy.

I laud Ian's goals of making debugging easier, but the particular
pattern he describes (wrapping stuff and putting the wrappers in $PATH)
is not the only way to ease debugging, and is at least as vulnerable to
the problems he associates with other techniques ("too easy to forget to
put back", etc) as they are.

> Firstly, we are talking about this in the context of Debian.  In
> Debian we have `reportbug', which the majority of people use to file
> bug reports.

the majority of debian users don't file bug reports at all, sadly.  they
complain about brokenness to their friends, they grouse about it on
social media, ask for help on stackoverflow, mention their troubles on
upstream mailing lists, or they just give up and move on.

diverging from upstream for this particular case means that on most of
those complaints, they might *also* hear "oh yeah, debian doesn't know
how to find the files from your specific installation because they
patched that out, so it's using whatever it found in $PATH".  Imagine
hearing this as someone who doesn't know what $PATH is, but just
followed a recipe on stackoverflow for some other problem that stuck
arbitrary crap in your $PATH four weeks ago and then you forgot about
the whole thing.  To such a user, they hear "debian broke this
package", which is not helpful to anyone.

Sophisticated users who do sophisticated debugging already have lots of
other options available to them.

> It would be simple for reportbug to report on these kind of anomalous
> situations and mention them in the bug report.  It could, for example,
> check to see if there is overlap between the package being reported
> (and its dependencies), and /usr/local.  I don't know if it does
> already, but if it doesn't and someone feels the information is
> valuable, then I'm sure the patches to reportbug would be welcome.

Indeed, patches welcome to automate this kind of inspection process as
part of reportbug.  if that infrastructure already existed and was
well-tested, i *might* be willing to consider this divergence from
upstream after talking it over with upstream and making sure they
understand why we aim to do it, and what additional ways we plan to help
them deal with any related bug reports.  As it stands, i don't think
that infrastructure exists, and i don't want to have that negotiation
with upstream.

> Secondly, I think the possibility that users may do things upstream
> consider undesirable, and even cause lossage, is precisely software
> freedom.

of course it is.  I would never tell users they can't build whatever
they want to build.  But i also want to support software in debian.
that means working well with upstream, managing expectations, and
minimizing the bugs that users encounter when they use the defaults.

> The thrust of the upstream argument here is that users must be
> prevented from messing with their software in certain ways because it
> makes those users' bug reports too hard to deal with.
>
> I think this argument is 

Bug#850967: Clarify /usr/bin/foo should not be hardcoded even in upstream parts

[Ian Jackson]

Package: tech-ctte
Control: block 850657 by -1

Policy 6.1 says
| Programs called from maintainer scripts should not normally have a
| path prepended to them.

Ie, programs that are on PATH should be found via the PATH rather than
by hardcoding /usr/bin/foo or whatever.  In general, I think we
normally, at least in software written specifically for Debian, apply
this not just to maintainer scripts but to all program execution.

However, this is not always uncontroversial with some upstreams.  This
causes a particular problem when the upstream is also heavily involved
with the Debian maintenance.


I would like the TC to:

 * Declare that Debian policy should be clarified to say that programs
   in Debian (not just maintainer scripts) should not hardcode the
   location of the binaries in /{usr/,}{bin,sbin} they execute.

 * Say that this applies even when the program needs to find pieces of
   the same (or closely related) package.

 * Say that where technically feasible, this should usually be done
   for other kinds of search paths (LD_LIBRARY_PATH, PYTHONPATH,
   PERLLIB, etc.), too.

 * Provide an informal opinion that this policy ought to apply to
   gpg-agent as requested in #850657.


I don't know if it is necessary to rehearse the arguments about this
issue for the TC, but I will try to do so briefly.


The argument in favour of finding programs (and libraries) on the
PATH:

This allows substitution and wrapping of programs (by adding a
directory to PATH containing a stunt version of the program, or by
adding a wrapper or replacement to /usr/local or ~/bin).  This can be
useful in a wide variety of situations.  It is a well-known debugging
technique.  It can be used by individual users, to modify the
behaviour of their system.  It can be profitably used by things like
test suites and audit tools.  It can be used by depending software to
work around bugs.

Most of these uses can be satisfied in some other way, but the other
ways have downsides.  For example, moving the actual binary aside is a
possibility (and supported by things like dpkg-divert); but it affects
the whole system, so requires administrator privilege; it is
unsuitable on a multiuser machine or for test suites; and if done
ad-hoc it is too easy to forget to put back and leave the system in a
weird state.

Explicit configuration or command line options to change which
subcommands are used are a good thing, but: they are usually fiddly
and certainly not always provided; this can often involve complicated
nesting (eg run `foo' with --bar-program pointing to a stunt `bar'
script which passes --wombat-program pointing to your stunt `wombat'
script); and lack of such an option at any level stymies this
approach.


I will quote the counterarguments from Daniel Kahn Gillmor, as they
are fairly typical of the responses from upstreams:

  In this bug report, you're asking a tightly-coupled suite of tools to
  invoke each other via the PATH, which offers another avenue of potential
  breakage; upstream regularly receives reports from people who *don't
  even know what version of gpg their system is running* because of
  multiple copies of the thing having been installed in various places on
  their system (either deliberately or incidentally, but in either case
  forgotten about by the local sysadmin).

  It's entirely reasonable for a tightly-coupled suite to want to invoke
  its own tools in this situation.  we have enough to debug in GnuPG as it
  is :/

Perhaps this is indeed a problem for some upstreams.  But I have two
responses to this which I think are each sufficient to rebut it:

Firstly, we are talking about this in the context of Debian.  In
Debian we have `reportbug', which the majority of people use to file
bug reports.

It would be simple for reportbug to report on these kind of anomalous
situations and mention them in the bug report.  It could, for example,
check to see if there is overlap between the package being reported
(and its dependencies), and /usr/local.  I don't know if it does
already, but if it doesn't and someone feels the information is
valuable, then I'm sure the patches to reportbug would be welcome.

Secondly, I think the possibility that users may do things upstream
consider undesirable, and even cause lossage, is precisely software
freedom.

The thrust of the upstream argument here is that users must be
prevented from messing with their software in certain ways because it
makes those users' bug reports too hard to deal with.

I think this argument is utterly wrong in principle.  It is contrary
to the whole point of Debian.


Thanks for your attention.

Ian.

PS: Please give the MIPS binutils bug priority.


-- 
Ian Jackson    These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.