Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Sean Whitton]

control: tag -1 +patch

Hello,

On Fri 24 Aug 2018 at 08:44PM -0700, Russ Allbery wrote:

> I'm looking for seconds for this patch to relax the current requirement
> back to a should.  After that, I think the next step would be to introduce
> automatic fixing of the #! line to debhelper, since that seems relatively
> uncontroversial, and then we can reconsider this later after that's had a
> chance to propagate through the archive.

I agree that this is how we should proceed.

> --- a/perl-policy.xml
> +++ b/perl-policy.xml
> @@ -533,7 +533,7 @@ $(MAKE) OPTIMIZE="-O2 -g -Wall"
>Script Magic
>
>
> -All packaged perl programs must start with
> +All packaged perl programs should start with
>  #!/usr/bin/perl and may append such flags as
>  are required.
>
> diff --git a/policy/ch-files.rst b/policy/ch-files.rst
> index f31a3b4..bc87573 100644
> --- a/policy/ch-files.rst
> +++ b/policy/ch-files.rst
> @@ -186,7 +186,7 @@ All command scripts, including the package maintainer 
> scripts inside the
>  package and used by ``dpkg``, should have a ``#!`` line naming the shell
>  to be used to interpret them.
>
> -In the case of Perl scripts this must be ``#!/usr/bin/perl``.
> +In the case of Perl scripts this should be ``#!/usr/bin/perl``.
>
>  When scripts are installed into a directory in the system PATH, the
>  script name should not include an extension such as ``.sh`` or ``.pl``

Seconded, and given the other seconds, committed to git.

>> My personal view is that the rule is the correct one though. Installing
>> a different perl for some application specific purpose is not uncommon -
>> some people choose to not use the system perl at all when they are
>> deploying a perl application - and they should be free to do that by
>> putting a different perl in the path. That doesn't mean that they
>> suddenly have to worry about parts of the packaged Debian system
>> breaking.  I certainly couldn't name every part of Debian that I rely on
>> that's written in perl!
>
> Yes, this is my feeling as well.  It's all well and good to say that if
> local administrators install their own Perl and things break, they get to
> keep both pieces, but this seems unnecessarily fragile.

Yes.

My first thought when seeing this thread was dpkg-divert, which hasn't
been mentioned yet.  If a local admin wants to override perl such that
all Debian-installed Perl command scripts use their custom perl,
dpkg-divert seems like a semantically correct way to do that.  And this
method is far less susceptible to the various examples of breakage that
have been brought up in this thread so far.

> There are various ways in which some other Perl could be added earlier
> in the PATH without the administrator having any intention whatsoever
> to supplant system Perl with it.  Consider, for instance, some local
> application written using bleed Perl installed in some non-standard
> path, some other program that with the best of intentions prepends the
> location of that application to the PATH, and some program that this
> application happens to run without even necessarily knowing it's
> written in Perl.  It seems better to ensure that this sort of pattern
> just works.

Yes.  Users should not have to think very hard about the language in
which stuff that Debian installs into /usr/bin is implemented.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Dominic Hargreaves]

On Fri, Aug 24, 2018 at 08:44:26PM -0700, Russ Allbery wrote:
> Dominic Hargreaves  writes:
> 
> > Clearly it should not be a must at this point given the deviation:
> > though it still looks to me like a must ever since it was added to the
> > perl policy, so if it is changed it should be changed in both places.
> 
> Hi all,
> 
> I'm looking for seconds for this patch to relax the current requirement
> back to a should.  After that, I think the next step would be to introduce
> automatic fixing of the #! line to debhelper, since that seems relatively
> uncontroversial, and then we can reconsider this later after that's had a
> chance to propagate through the archive.
> 
> --- a/perl-policy.xml
> +++ b/perl-policy.xml
> @@ -533,7 +533,7 @@ $(MAKE) OPTIMIZE="-O2 -g -Wall"
>Script Magic
>  
>
> -All packaged perl programs must start with
> +All packaged perl programs should start with
>  #!/usr/bin/perl and may append such flags as
>  are required.
>
> diff --git a/policy/ch-files.rst b/policy/ch-files.rst
> index f31a3b4..bc87573 100644
> --- a/policy/ch-files.rst
> +++ b/policy/ch-files.rst
> @@ -186,7 +186,7 @@ All command scripts, including the package maintainer 
> scripts inside the
>  package and used by ``dpkg``, should have a ``#!`` line naming the shell
>  to be used to interpret them.
>  
> -In the case of Perl scripts this must be ``#!/usr/bin/perl``.
> +In the case of Perl scripts this should be ``#!/usr/bin/perl``.
>  
>  When scripts are installed into a directory in the system PATH, the
>  script name should not include an extension such as ``.sh`` or ``.pl``

Seconded.

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Jonathan Nieder]

Hi Russ,

Russ Allbery wrote:

> I'm looking for seconds for this patch to relax the current requirement
> back to a should.
[...]
> --- a/perl-policy.xml
> +++ b/perl-policy.xml
> @@ -533,7 +533,7 @@ $(MAKE) OPTIMIZE="-O2 -g -Wall"
>Script Magic
>  
>
> -All packaged perl programs must start with
> +All packaged perl programs should start with
>  #!/usr/bin/perl and may append such flags as
>  are required.
>
> diff --git a/policy/ch-files.rst b/policy/ch-files.rst
> index f31a3b4..bc87573 100644
> --- a/policy/ch-files.rst
> +++ b/policy/ch-files.rst
> @@ -186,7 +186,7 @@ All command scripts, including the package maintainer 
> scripts inside the
>  package and used by ``dpkg``, should have a ``#!`` line naming the shell
>  to be used to interpret them.
>  
> -In the case of Perl scripts this must be ``#!/usr/bin/perl``.
> +In the case of Perl scripts this should be ``#!/usr/bin/perl``.
>  
>  When scripts are installed into a directory in the system PATH, the
>  script name should not include an extension such as ``.sh`` or ``.pl``

Seconded.

Thanks,
Jonathan

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Russ Allbery]

Dominic Hargreaves  writes:

> Clearly it should not be a must at this point given the deviation:
> though it still looks to me like a must ever since it was added to the
> perl policy, so if it is changed it should be changed in both places.

Hi all,

I'm looking for seconds for this patch to relax the current requirement
back to a should.  After that, I think the next step would be to introduce
automatic fixing of the #! line to debhelper, since that seems relatively
uncontroversial, and then we can reconsider this later after that's had a
chance to propagate through the archive.

--- a/perl-policy.xml
+++ b/perl-policy.xml
@@ -533,7 +533,7 @@ $(MAKE) OPTIMIZE="-O2 -g -Wall"
   Script Magic
 
   
-All packaged perl programs must start with
+All packaged perl programs should start with
 #!/usr/bin/perl and may append such flags as
 are required.
   
diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index f31a3b4..bc87573 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -186,7 +186,7 @@ All command scripts, including the package maintainer 
scripts inside the
 package and used by ``dpkg``, should have a ``#!`` line naming the shell
 to be used to interpret them.
 
-In the case of Perl scripts this must be ``#!/usr/bin/perl``.
+In the case of Perl scripts this should be ``#!/usr/bin/perl``.
 
 When scripts are installed into a directory in the system PATH, the
 script name should not include an extension such as ``.sh`` or ``.pl``

> My personal view is that the rule is the correct one though. Installing
> a different perl for some application specific purpose is not uncommon -
> some people choose to not use the system perl at all when they are
> deploying a perl application - and they should be free to do that by
> putting a different perl in the path. That doesn't mean that they
> suddenly have to worry about parts of the packaged Debian system
> breaking.  I certainly couldn't name every part of Debian that I rely on
> that's written in perl!

Yes, this is my feeling as well.  It's all well and good to say that if
local administrators install their own Perl and things break, they get to
keep both pieces, but this seems unnecessarily fragile.  There are various
ways in which some other Perl could be added earlier in the PATH without
the administrator having any intention whatsoever to supplant system Perl
with it.  Consider, for instance, some local application written using
bleed Perl installed in some non-standard path, some other program that
with the best of intentions prepends the location of that application to
the PATH, and some program that this application happens to run without
even necessarily knowing it's written in Perl.  It seems better to ensure
that this sort of pattern just works.

-- 
Russ Allbery (r...@debian.org)   

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Ian Jackson]

Norbert Preining writes ("Re: Bug#906901: debian-policy: Perl script shebang 
requirement is disturbing and inconsistent with rest of policy"):
> Hi Ian,
> > This confusing user experience only occurs if someone prepends a
> > different perl to $PATH.  Has anyone actually ever done this and got
> > useful results ?  Has anyone actually even wanted to do this ?
> 
> Not that I know off in a separate PATH, but I have seen perl in 
> /usr/local/bin quite some time.
> 
> >   Norbert, is your concern theoretical, or is this a thing you have
> >   actually wanted to do ?
> 
> I personally have no need for it, but I use servers where this is 
> done, mostly to get newer perl than what is shipped.

Right.

When this is done, does it matter whether the Debian scripts use
Debian's /usr/bin/perl, or the newer one provided in /usr/local/bin ?

I would guess (and hope!) not.

Ian.

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Norbert Preining]

Hi Ian,

> This confusing user experience only occurs if someone prepends a
> different perl to $PATH.  Has anyone actually ever done this and got
> useful results ?  Has anyone actually even wanted to do this ?

Not that I know off in a separate PATH, but I have seen perl in 
/usr/local/bin quite some time.

>   Norbert, is your concern theoretical, or is this a thing you have
>   actually wanted to do ?

I personally have no need for it, but I use servers where this is 
done, mostly to get newer perl than what is shipped.

> but it was buried in a longer mail and Norbert seems to have missed

Seems so, sorry.

> I certainly agree that this should be demoted to a SHOULD right away.

Agreed.

Best

Norbert

--
PREINING Norbert   http://www.preining.info
Accelia Inc. +JAIST +TeX Live +Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Ian Jackson]

Julian Andres Klode writes ("Bug#906901: debian-policy: Perl script shebang 
requirement is disturbing and inconsistent with rest of policy"):
> In fact, we're seeing upgrade failures due to people having
> different interpreter versions in /usr/local, and there was some
> discussion about dpkg specifying a safe PATH. There was no consent
> about that, but it seems likely that APT will start specifying a
> safe PATH=/usr/sbin:/usr/bin:/sbin:/bin eventually to prevent users
> from breaking their systems.

Please do not do this.

If you absolutely must do it, make it optional.

> I agree. Let's put it this way: As a user I don't want to know whether
> the program in $PATH is a Perl script or not and then have to deal with
> local perl installations. I installed a tool, not a script, it should
> not stop working due to some unrelated change.

IMO if you install your own Perl installation in /usr/local/bin and it
doesn't work right, then you should get to keep all the resulting
pieces.

If you install a Perl installation in /usr/local/bin and it _does_
work right then there is no problem.  AIUI perl XS has a reasonably
stable ABI ?  At least, old modules will work with new perl binaries ?

If I wanted to install my own Perl I would make sure to have
/usr/bin/perl point to it.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Ian Jackson]

Jonathan Nieder writes ("Bug#906901: debian-policy: Perl script shebang 
requirement is disturbing and inconsistent with rest of policy"):
> Dominic Hargreaves wrote:
> > On Tue, Aug 21, 2018 at 08:42:11PM -0700, Russ Allbery wrote:
...
> In this thread, there are three possible rules proposed:
> 
>  1. #!/usr/bin/env perl
>  2. #!/usr/bin/perl
>  3. Packager decides between 1 and 2, policy doesn't express a
> preference.
> 
> The passage you are quoting is about the confusing user experience
> that (3) provides:

This confusing user experience only occurs if someone prepends a
different perl to $PATH.  Has anyone actually ever done this and got
useful results ?  Has anyone actually even wanted to do this ?

I asked Norbert this on Wednesday:

  Norbert, is your concern theoretical, or is this a thing you have
  actually wanted to do ?

but it was buried in a longer mail and Norbert seems to have missed
it.

If no one has tried this, or is likely to, then it doesn't really
matter what we do because any of the options will have the same
effect.  In that case it's probably worth repeating upstream's
recommendation of /usr/bin/perl in our own policy as a SHOULD.

I certainly agree that this should be demoted to a SHOULD right away.
Maintainers should not be working, now, to change their scripts to
conform to a policy which we might change.

Ian.

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Julian Andres Klode]

On Wed, Aug 22, 2018 at 07:04:10PM -0700, Jonathan Nieder wrote:
> Hi,
> 
> Dominic Hargreaves wrote:
> > On Tue, Aug 21, 2018 at 08:42:11PM -0700, Russ Allbery wrote:
> 
> >> I do feel like allowing either based on the whim of the packager is just
> >> kind of bad.  It produces inconsistent behavior to no real benefit for
> >> anyone.  If you install a Perl earlier in your PATH, you get totally
> >> unpredictable behavior, and everyone will be unhappy half the time.
> >
> > My personal view is that the rule is the correct one though. Installing
> > a different perl for some application specific purpose is not uncommon -
> 
> In this thread, there are three possible rules proposed:
> 
>  1. #!/usr/bin/env perl
>  2. #!/usr/bin/perl
>  3. Packager decides between 1 and 2, policy doesn't express a
> preference.
> 
> The passage you are quoting is about the confusing user experience
> that (3) provides: when someone installs a different perl to $PATH,
> some Debian packages would use it and others wouldn't, with no
> organizing principle to predict how each new package will behave.
> 
> I believe you are arguing for (2) as a long-term goal: all Debian
> packages would then use /usr/bin/perl instead of the perl from $PATH,
> while the sysadmin's own custom scripts using '#!/usr/bin/env perl'
> would use perl from $PATH.  I agree with you that that's a good place
> to end up, similar to what we already do with python.
> 
> Ian and Russ also mentioned that there's a bit of an inconsistency in
> spirit here with policy 6.1:
> 
>   Programs called from maintainer scripts should not normally have a
>   path prepended to them. [...] These considerations really apply to
>   all shell scripts.
> 
> But as Ian mentioned, commands exist on a spectrum; most interpreters
> are at an extreme end where hard-coding the path in *packaged scripts*
> is generally the right policy, while commands like ls or git, say, are
> at the other extreme where hard-coding the path would not accomplish
> much useful.

In fact, we're seeing upgrade failures due to people having different
interpreter versions in /usr/local, and there was some discussion about
dpkg specifying a safe PATH. There was no consent about that, but it seems
likely that APT will start specifying a safe PATH=/usr/sbin:/usr/bin:/sbin:/bin
eventually to prevent users from breaking their systems.


> - for the sake of the long term, I agree that debhelper should rewrite
>   to the #!/usr/bin/perl form

I agree. Let's put it this way: As a user I don't want to know whether
the program in $PATH is a Perl script or not and then have to deal with
local perl installations. I installed a tool, not a script, it should
not stop working due to some unrelated change.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Jonathan Nieder]

Hi,

Dominic Hargreaves wrote:
> On Tue, Aug 21, 2018 at 08:42:11PM -0700, Russ Allbery wrote:

>> I do feel like allowing either based on the whim of the packager is just
>> kind of bad.  It produces inconsistent behavior to no real benefit for
>> anyone.  If you install a Perl earlier in your PATH, you get totally
>> unpredictable behavior, and everyone will be unhappy half the time.
>
> My personal view is that the rule is the correct one though. Installing
> a different perl for some application specific purpose is not uncommon -

In this thread, there are three possible rules proposed:

 1. #!/usr/bin/env perl
 2. #!/usr/bin/perl
 3. Packager decides between 1 and 2, policy doesn't express a
preference.

The passage you are quoting is about the confusing user experience
that (3) provides: when someone installs a different perl to $PATH,
some Debian packages would use it and others wouldn't, with no
organizing principle to predict how each new package will behave.

I believe you are arguing for (2) as a long-term goal: all Debian
packages would then use /usr/bin/perl instead of the perl from $PATH,
while the sysadmin's own custom scripts using '#!/usr/bin/env perl'
would use perl from $PATH.  I agree with you that that's a good place
to end up, similar to what we already do with python.

Ian and Russ also mentioned that there's a bit of an inconsistency in
spirit here with policy 6.1:

  Programs called from maintainer scripts should not normally have a
  path prepended to them. [...] These considerations really apply to
  all shell scripts.

But as Ian mentioned, commands exist on a spectrum; most interpreters
are at an extreme end where hard-coding the path in *packaged scripts*
is generally the right policy, while commands like ls or git, say, are
at the other extreme where hard-coding the path would not accomplish
much useful.

So while this distinction between the interpreter of a script and
commands invoked within a script may feel arbitrary, in practice it
seems to work out pretty well.  Best of all, it's predictable.

For some other languages (#!/usr/bin/env node?), matching convention
might lead us to a different conclusion, and that's okay.

tl/dr:

- in the short term, I agree that this should be lowered to a policy
  "should", not "must"

- for the sake of the long term, I agree that debhelper should rewrite
  to the #!/usr/bin/perl form

Thanks,
Jonathan

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Norbert Preining]

Hi,

> Addressing your inconsistency argument above: I can certainly see an
> argument that some types of perl scripts shipped in Debian might want
> to opt into being run by a different interpreter for special reasons,
> but I think they should be the exception rather than the rule. Having

I am fine with that, so what about returning to a "should" as policy
item and let it be with it?

Norbert

--
PREINING Norbert   http://www.preining.info
Accelia Inc. +JAIST +TeX Live +Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Dominic Hargreaves]

On Tue, Aug 21, 2018 at 08:42:11PM -0700, Russ Allbery wrote:
> Russ Allbery  writes:
> 
> > Did Lintian have some special case that was allowing /usr/bin/env perl
> > previously and then Lintian changed based on Policy?  That would be
> > unfortunate, since we thought we were changing to match Lintian
> 
> Sigh.  Yes, indeed.
> 
>   * checks/scripts.pm:
> + [CL] Policy 10.4 states that Perl scripts must use /usr/bin/perl
>   directly and not via /usr/bin/env, etc.  (Closes: #904414)
> 
> in Lintian 2.5.94.
> 
> Well, this is a mess.  Apparently a lot of people were ignoring that part
> of Policy, and now we've created a ton of buggy packages because I made a
> bad assumption about what Lintian was already checking for.

Oh, that's really unfortunate :(

> Perl folks, the short version is that Lintian wasn't actually checking for
> scripts that used /usr/bin/env perl, so our check when we closed #683495
> was bogus.  Lintian has now changed based on Policy, and it looks like
> there were around 2,000 scripts in Debian that were using the /usr/bin/env
> perl form.
> 
> Any feelings about where we should go from here?

Clearly it should not be a must at this point given the deviation:
though it still looks to me like a must ever since it was added to the
perl policy, so if it is changed it should be changed in both places.

> I do feel like allowing either based on the whim of the packager is just
> kind of bad.  It produces inconsistent behavior to no real benefit for
> anyone.  If you install a Perl earlier in your PATH, you get totally
> unpredictable behavior, and everyone will be unhappy half the time.

My personal view is that the rule is the correct one though. Installing
a different perl for some application specific purpose is not uncommon -
some people choose to not use the system perl at all when they are
deploying a perl application - and they should be free to do that by
putting a different perl in the path. That doesn't mean that they
suddenly have to worry about parts of the packaged Debian system breaking.
I certainly couldn't name every part of Debian that I rely on that's
written in perl!

Addressing your inconsistency argument above: I can certainly see an
argument that some types of perl scripts shipped in Debian might want
to opt into being run by a different interpreter for special reasons,
but I think they should be the exception rather than the rule. Having
a few special cases in Debian seems far better than having every single
perl script in Debian be at risk of breaking when /usr/local/bin/perl
appears.

Dominic.

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Stuart Prescott]

On Wednesday, 22 August 2018 13:26:30 AEST Ian Jackson wrote:
> In practice, replacing perl with a homebrewed actual perl is rare and
> in that case you want to put it in /usr/bin, because that's where all
> distributed perl scripts expect it to be.

(If you'd make it /usr/bin/perl anyway, isn't that really saying that there's 
no point in using env?)

> And IIRC with Python we have historically had the trouble that a
> Python compiled into /usr/local/bin has bizarre ideas about where to
> look for its modules, which means that this isn't a useful strategy
> anyway.

This is not a python-specific problem but since this assertion comes up here 
from time to time and you're keen to broaden this discussion, let's look at 
it.

For some interpreter /usr/local/bin/foo, there are two options for the module 
path:

* modules outside /usr/local are ignored; installing libbar-foo packages as 
dependencies no longer satisfies the functional requirement of providing 
module bar when running `foo`. One now gets to reimplement apt but with the 
added fun that the modules one installs aren't necessarily compatible with 
other foo-using distro packages that are installed. Good luck running 
anything.

* modules outside /usr/local are not ignored; distribution provided modules 
are not necessarily compatible with the interpreter for API or ABI reasons. 
When one runs code, it might be the distro provided module or a locally 
supplied module that is used. Some of them might even have been tested 
together and might even work. Good luck running anything.

Both options are pretty broken, sadly. (If this were easy, we wouldn't have 
significant disruption and lots of work to do every time new upstream releases 
of perl or python were uploaded, for instance.)

The problem might be more visible in python-land than perl-land but that is 
more to do with some combination of luck, popularity, development activity and 
perhaps admin experience. In the limit of a dead software ecosystem, one 
doesn't bother with newer interpreters/modules/libraries, and/or the chance of 
a backwards incompatible change is so tiny that one gets away with it.

Every place where /usr/local is earlier than /usr in a search path (PATH, 
LD_LIBRARY_PATH, @INC, sys.path etc) is a place where the local admin can 
accidentally break distribution-supplied tools while only doing things in the 
admin's /usr/local playground.

Flexibility sits on a scale between fragile and robust. The trick is working 
out how many footguns are appropriate to leave around our distribution.

cheers
Stuart

-- 
Stuart Prescotthttp://www.nanonanonano.net/   stu...@nanonanonano.net
Debian Developer   http://www.debian.org/ stu...@debian.org
GPG fingerprint90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Ian Jackson]

Russ Allbery writes ("Bug#906901: debian-policy: Perl script shebang 
requirement is disturbing and inconsistent with rest of policy"):
> Yeah, this is one of the points that I'm struggling with: I feel like our
> stance with Perl (and Python) and our stance with maintainer scripts is
> inconsistent.  We require maintainer scripts to support PATH
> interposition, but take the opposite stance for interpreters, and then
> that causes these sorts of problems.

There are significant differences between #! lines invoking
interpreters like perl and python, and maintainer scripts calling
random programs.

In practice, replacing perl with a homebrewed actual perl is rare and
in that case you want to put it in /usr/bin, because that's where all
distributed perl scripts expect it to be.

If one wants to do some kind of stunt, it is normally only a few
scripts that one wants to affect.  So it is easy enough to put
modified versions of (or wrappers for) those scripts on PATH.

Making $PATH contain a wrapper script for the whole perl intepreter is
a massively big hammer, and quite complicated to do right since one
has to reparse perl's arguments.

And IIRC with Python we have historically had the trouble that a
Python compiled into /usr/local/bin has bizarre ideas about where to
look for its modules, which means that this isn't a useful strategy
anyway.

To be honest: I'm a great fan of insisting on always using PATH.  I
have enountered numerous cases where this has allowed me to easily do
things which would otherwise be significantly harder.  But I have
never wanted to provide a stunt perl or a stunt python, or anything of
the kind.

Norbert, is your concern theoretical, or is this a thing you have
actually wanted to do ?

Next up: what about #!/bin/sh.

Having said all that, I think it would be right for policy to revert
the change immediately, and ask lintian to revert the change
immediately, before we have a big conversation about what to do about
this.

Reverting both changes will mean that maintainers of individual
packages are not being prompted to make changes we may later ask them
to reverse.  And it will remove the animus behind Norbert's setting of
this bug to `serious' - which I think is arguably justified.

Thanks,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Raphael Hertzog]

Hi,

On Tue, 21 Aug 2018, Russ Allbery wrote:
> Perl folks, the short version is that Lintian wasn't actually checking for
> scripts that used /usr/bin/env perl, so our check when we closed #683495
> was bogus.  Lintian has now changed based on Policy, and it looks like
> there were around 2,000 scripts in Debian that were using the /usr/bin/env
> perl form.
> 
> Any feelings about where we should go from here?

Yes, relax the requirement to a "should" again and downgrade the severity
of the lintian tag to a warning, but get #904409 implemented so that the
vast majority of packages maintainers have nothing to do to get it right.

> I do feel like allowing either based on the whim of the packager is just
> kind of bad.  It produces inconsistent behavior to no real benefit for
> anyone.  If you install a Perl earlier in your PATH, you get totally
> unpredictable behavior, and everyone will be unhappy half the time.

Ack, IMO the decision was correct but the tooling was not ready for this.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Norbert Preining]

Hi

> Apparently the Python packaging community in Debian already had this
> discussion and chose to standardize on /usr/bin paths for everything, and
> the debhelper add-ons for Python just rewrite the #! line accordingly.  So

Hmm, really? Looking throught the python scripts in TeX Live I find a
lot of different things:
/usr/share/texlive/texmf-dist/scripts] find . -name \*.py -exec head -n1 '{}' \;
#! /usr/bin/env python2.7
#!/usr/bin/env python
#!/usr/bin/env python
#!/usr/bin/env python2
#!/usr/bin/env python3
#!/usr/bin/env python
#!/usr/bin/env python2
#!/usr/bin/env python3
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#!/usr/bin/env python
#!/usr/bin/env python
#! /usr/bin/env python2
#!/usr/bin/env python
#!/usr/bin/env python
#!/usr/bin/env python
#!/usr/bin/env python
"""
#!/usr/bin/env python
#!/usr/bin/env python
#!/usr/bin/env python
#!/usr/bin/env python
#!/usr/bin/env python
"""Module to plot using Pgfplots.

Not all of them are linked to /usr/bin I guess, but at least some.

Searching for lua/texlua I get the same env texlua
$ find . -name \*.lua -exec head -n1 '{}' \; | grep ^#
#!/usr/bin/env texlua
#!/usr/bin/env texlua
#!/usr/bin/env texlua
#!/usr/bin/env texlua
#!/usr/bin/env texlua
#!/usr/bin/env texlua
#!/usr/bin/env texlua
#!/usr/bin/env texlua
#!/usr/bin/env texlua
#! /usr/bin/env texlua
#!/usr/bin/env texlua


> Note too that Policy *did* say that one "should" use /usr/bin/perl and the
> Perl Policy said that one "must" (and has for a while -- the DocBook

Yes, I am aware of that after (re)reading the discussion on debian-perl
that led to this change.

> conversion makes git annotate not as useful, but I didn't see any relevant
> changelog entry and suspect it may have been that way since it was
> incorporated into Policy).  So this isn't entirely out of the blue.

But it was never reported by lintian.

> less a thing than it used to be.  But surely this is still not too
> uncommon for stable users?  Maybe I'm just out of touch and everyone uses

Good question. I leave this to you. I only know one place that compiles
perl because it runs on an old CentOS ...

> debhelper to have it just automatically fix the #! lines, which would make
> quite a lot of that go away if they moved forward with that approach.
> That's apparently the approach Python took.

It would be a nice idea, but I don't see it actually working.

cd /usr/bin ; head -n 1 $(ls -l | grep texmf-dist | awk '{print$9}') 
gives me a list of files with /usr/bin/env style shebangs. Several
of them with python interpreters. Grepping for usr/bin/env I get:
1x  #!/usr/bin/env bash
59x #!/usr/bin/env perl
9x  #!/usr/bin/env python
1x  #!/usr/bin/env python2
1x  #!/usr/bin/env python2.7
19x #!/usr/bin/env texlua
1x  #!/usr/bin/env wish
(these are only the TeX Live scripts linked and with /usr/bin/env
shebang)

I don't see that this is acutally working in debhelper.

> Certainly we should avoid making every packager carry a bunch of diffs or

Yeah, I decided against diff and patch the files after they are
installed into the debian/$Pkg/ directories.

Best

Norbert

--
PREINING Norbert   http://www.preining.info
Accelia Inc. +JAIST +TeX Live +Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Russ Allbery]

Norbert Preining  writes:

> But we probably have the very same situation with any other interpreter.
> ANyone ever checked the shebangs of python/php/lua/whatever scripts?

Apparently the Python packaging community in Debian already had this
discussion and chose to standardize on /usr/bin paths for everything, and
the debhelper add-ons for Python just rewrite the #! line accordingly.  So
yeah, this is something other interpreters have looked at.  I'm not sure
what PHP and Lua have done, but Python at least decided it was worth
standardizing.

Note too that Policy *did* say that one "should" use /usr/bin/perl and the
Perl Policy said that one "must" (and has for a while -- the DocBook
conversion makes git annotate not as useful, but I didn't see any relevant
changelog entry and suspect it may have been that way since it was
incorporated into Policy).  So this isn't entirely out of the blue.
That's why we thought the change was uncontroversial.  I'm fairly sure
that 2,000 or so scripts is much less than the total number in Debian,
although I'm sure this was still an unpleasant surprise to a lot of people
since it doesn't look like we had any detection or enforcement of that.

I'm sorry for the surprise.  That's largely my fault.

> First of all, I don't see that this will happen on more then a very rare
> basis - and if it happens, the user/admin who did this should be aware
> that things might go not as smooth as expected.

Installing a separate local version of Perl?  Hm, I'm surprised that you
think that will only happen on a very rare basis, although I guess this is
less a thing than it used to be.  But surely this is still not too
uncommon for stable users?  Maybe I'm just out of touch and everyone uses
containers these days, or Perl is so stable that people don't tend to
install their own versions as much.

> And, if you decide on one of the two, again many packages will get
> lintian errors. You mentioned that the env perl had about 2000 scripts,
> I guess the /usr/bin/perl shebang will affect even more scripts.

I agree that this is surely a hassle, but there are things we could do to
make it easier.  For instance, it looks like there's a bug filed against
debhelper to have it just automatically fix the #! lines, which would make
quite a lot of that go away if they moved forward with that approach.
That's apparently the approach Python took.

Certainly we should avoid making every packager carry a bunch of diffs or
something to fix this if we decide to standardize and can make their lives
easier!  I totally agree with that.

-- 
Russ Allbery (r...@debian.org)   

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Norbert Preining]

Hi Russ,

> Unpredictable behavior, where every command written in Perl behaves
> randomly based on the whim of the packager?

Well, so it be?

> commands in maintainer scripts).  But picking one or the other essentially
> at random (from the perspective of the user) sounds awful.

But we probably have the very same situation with any other interpreter.
ANyone ever checked the shebangs of python/php/lua/whatever scripts?

First of all, I don't see that this will happen on more then a very rare
basis - and if it happens, the user/admin who did this should be aware
that things might go not as smooth as expected.

And, if you decide on one of the two, again many packages will get
lintian errors. You mentioned that the env perl had about 2000 scripts,
I guess the /usr/bin/perl shebang will affect even more scripts.

None of that sounds like a good alternative.

Best

Norbert

--
PREINING Norbert   http://www.preining.info
Accelia Inc. +JAIST +TeX Live +Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Russ Allbery]

Thank you for the background on Python!

Stuart Prescott  writes:

> Sidenote: Getting rid of env in shebangs is only part of the story to
> making packages robust against accidental breakage. Putting an
> incompatible interpreter into PATH (say, /usr/local/bin/{python,perl})
> ends up breaking maintainer scripts that call the interpreter without
> specifying the full path in the same way as a dysfunctional
> /usr/local/bin/rm would break maintainer scripts that call 'rm'. The
> differences are that people seldom compile their own coreutils and the
> interpreter is also reliant upon a compatible module tree and that is
> not necessarily available to the locally installed interpreter. (Python
> people are strongly encouraged not to use /usr/local/bin/ python for
> precisely this reason, using, say, virtualenv instead.)

Yeah, this is one of the points that I'm struggling with: I feel like our
stance with Perl (and Python) and our stance with maintainer scripts is
inconsistent.  We require maintainer scripts to support PATH
interposition, but take the opposite stance for interpreters, and then
that causes these sorts of problems.

Personally, I've never liked command interposition in maintainer scripts,
although I admit that it lets one do some things as a local sysadmin that
are otherwise quite irritating to do.  But apart from one's preferences on
both, it feels like we have a foundational inconsistency here.

-- 
Russ Allbery (r...@debian.org)   

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Russ Allbery]

Norbert Preining  writes:

>> If you install a Perl earlier in your PATH, you get totally
>> unpredictable behavior, and everyone will be unhappy half the time.

> Well, but this is what you have asked for when you installed a Perl
> earlier in your PATH. It is up to you.

Unpredictable behavior, where every command written in Perl behaves
randomly based on the whim of the packager?

I have a hard time imagining that's what anyone is really asking for.  It
just doesn't seem useful.  I can see the argument for consistently using
/usr/bin/perl (it means that any local installation of Perl doesn't break
Perl scripts from Debian packages, and they still find their dependencies
through other Debian packages as intended), and for consistently using
/usr/bin/env perl (the local sysadmin can interject some other version of
Perl, although is now on the hook for ensuring all necessary modules are
installed, following much the same logic as for not using full paths to
commands in maintainer scripts).  But picking one or the other essentially
at random (from the perspective of the user) sounds awful.

-- 
Russ Allbery (r...@debian.org)   

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Stuart Prescott]

On Tuesday, 21 August 2018 20:42:11 AEST Russ Allbery wrote:
> I do feel like allowing either based on the whim of the packager is just
> kind of bad.  It produces inconsistent behavior to no real benefit for
> anyone.  If you install a Perl earlier in your PATH, you get totally
> unpredictable behavior, and everyone will be unhappy half the time.

Perhaps a useful data point in looking at standard practice within Debian:

A parallel discussion was had around python packaging some time ago and these 
exact same arguments were felt to be persuasive: packaged utilities should be 
robust against accidental breakage where possible. (In the current python 
policy, it is only listed as "preferred", however.)

https://www.debian.org/doc/packaging-manuals/python-policy/programs.html

Avoiding "/usr/bin/env python" in packages is almost universal. The dh_python2 
and dh_python3 helpers do shebang rewriting by default and only one package 
elects to skip that. (There are 14 explicitly-python-using packages that don't 
use these helpers plus some packages with python scripts but no explicit 
python dependency, many of which have curiously overridden the lintian error 
complaining about that and are presumably content to ship broken scripts.)

https://codesearch.debian.net/search?q=--no-shebang-rewrite

https://lintian.debian.org/tags/python-depends-but-no-python-helper.html

https://lintian.debian.org/tags/python3-depends-but-no-python3-helper.html

https://lintian.debian.org/tags/python-script-but-no-python-dep.html

Sidenote: Getting rid of env in shebangs is only part of the story to making 
packages robust against accidental breakage. Putting an incompatible 
interpreter into PATH (say, /usr/local/bin/{python,perl}) ends up breaking 
maintainer scripts that call the interpreter without specifying the full path 
in the same way as a dysfunctional /usr/local/bin/rm would break maintainer 
scripts that call 'rm'. The differences are that people seldom compile their 
own coreutils and the interpreter is also reliant upon a compatible module 
tree and that is not necessarily available to the locally installed 
interpreter. (Python people are strongly encouraged not to use /usr/local/bin/
python for precisely this reason, using, say, virtualenv instead.)

cheers
Stuart

-- 
Stuart Prescotthttp://www.nanonanonano.net/   stu...@nanonanonano.net
Debian Developer   http://www.debian.org/ stu...@debian.org
GPG fingerprint90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Norbert Preining]

Hi Russ,

> there were around 2,000 scripts in Debian that were using the /usr/bin/env
> perl form.

That was a number I somehow expected. It is decades old practice to use
this shebang.

> Any feelings about where we should go from here?

Yes.

> I do feel like allowing either based on the whim of the packager is just
> kind of bad.  It produces inconsistent behavior to no real benefit for

I do feel that is is the correct way.

> anyone.  If you install a Perl earlier in your PATH, you get totally
> unpredictable behavior, and everyone will be unhappy half the time.

Well, but this is what you have asked for when you installed a Perl
earlier in your PATH. It is up to you.

I strongly support that the env perl form is acceptable.

Best

Norbert

--
PREINING Norbert   http://www.preining.info
Accelia Inc. +JAIST +TeX Live +Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Russ Allbery]

Russ Allbery  writes:

> Did Lintian have some special case that was allowing /usr/bin/env perl
> previously and then Lintian changed based on Policy?  That would be
> unfortunate, since we thought we were changing to match Lintian

Sigh.  Yes, indeed.

  * checks/scripts.pm:
+ [CL] Policy 10.4 states that Perl scripts must use /usr/bin/perl
  directly and not via /usr/bin/env, etc.  (Closes: #904414)

in Lintian 2.5.94.

Well, this is a mess.  Apparently a lot of people were ignoring that part
of Policy, and now we've created a ton of buggy packages because I made a
bad assumption about what Lintian was already checking for.

Perl folks, the short version is that Lintian wasn't actually checking for
scripts that used /usr/bin/env perl, so our check when we closed #683495
was bogus.  Lintian has now changed based on Policy, and it looks like
there were around 2,000 scripts in Debian that were using the /usr/bin/env
perl form.

Any feelings about where we should go from here?

I do feel like allowing either based on the whim of the packager is just
kind of bad.  It produces inconsistent behavior to no real benefit for
anyone.  If you install a Perl earlier in your PATH, you get totally
unpredictable behavior, and everyone will be unhappy half the time.

-- 
Russ Allbery (r...@debian.org)   

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Russ Allbery]

Russ Allbery  writes:
> Norbert Preining  writes:

>> If a user/system admin wants to replace Perl by prepending the path to
>> a self compiled Perl to the PATH, it is his right to do so, and Perl
>> scripts are expected to follow this decision. It is the obligation of
>> the one doing the change to ensure proper availability of modules and
>> support files.

> There were literally zero packages in Debian that did this that Lintian
> could find.  Did we miss something?

Oh, hm, now that same Lintian tag is reporting a bunch of packages that
were using /usr/bin/env, which makes me think we *did* miss something.
The intent of this change wasn't to declare a bunch of packages buggy, so
yeah, that's something we should look at.  (That said, this did already
violate a should in Policy, and I think it's questionable how much having
some random selection of Perl scripts honor the PATH when all the others
don't really would be to anyone.)

https://lintian.debian.org/tags/wrong-path-for-interpreter.html

Did Lintian have some special case that was allowing /usr/bin/env perl
previously and then Lintian changed based on Policy?  That would be
unfortunate, since we thought we were changing to match Lintian

-- 
Russ Allbery (ea...@eyrie.org)  

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Russ Allbery]

Control: severity -1 normal

Norbert Preining  writes:

> The recent addition to the Debian Policy to require a Perl shebang of
> /usr/bin/perl is inconsistent with the rest of script usage, and hinders
> the user/system administrator to freely override Perl.

This isn't recent.  All we did was change should to must in the main
policy, and it was already must in the Perl policy.

> If a user/system admin wants to replace Perl by prepending the path to a
> self compiled Perl to the PATH, it is his right to do so, and Perl
> scripts are expected to follow this decision. It is the obligation of
> the one doing the change to ensure proper availability of modules and
> support files.

There were literally zero packages in Debian that did this that Lintian
could find.  Did we miss something?

-- 
Russ Allbery (r...@debian.org)   

Bug#906901: debian-policy: Perl script shebang requirement is disturbing and inconsistent with rest of policy

[Norbert Preining]

Package: debian-policy
Version: 4.2.0.1
Severity: serious

The recent addition to the Debian Policy to require a Perl shebang of
/usr/bin/perl is inconsistent with the rest of script usage, and hinders
the user/system administrator to freely override Perl.

If a user/system admin wants to replace Perl by prepending the path to
a self compiled Perl to the PATH, it is his right to do so, and Perl
scripts are expected to follow this decision. It is the obligation of
the one doing the change to ensure proper availability of modules and
support files.

Package maintainers are free to use the more restrictive /usr/bin/perl
shebang when the consider it necessary, but this must not be a
requirement in the policy.

I hereby suggest removing this line
In the case of Perl scripts this must be "#!/usr/bin/perl".
from policy 10.4.

Sincerely yours

Norbert


-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.18.3 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages debian-policy depends on:
ii  libjs-sphinxdoc  1.7.6-1

debian-policy recommends no packages.

Versions of packages debian-policy suggests:
pn  doc-base  

-- no debconf information
Norbert

--
PREINING Norbert   http://www.preining.info
Accelia Inc. +JAIST +TeX Live +Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13