Bug#436466: dash: Please optimise single command given to -c to exec it

[Vincent Bernat]

 ❦ 27 May 2020 08:42 -07, Jonathan Nieder:

 Because of bug #642922 (archived, but has some interesting discussion),
 this patch for this bug ended up being backed out.
> [...]
>> I gather that the ocamlbuild bug has been fixed so can this be
>> reinstated please?
>
> Curious: the ocamlbuild bug appears to still be open upstream
> , but
> https://github.com/ocaml/ocamlbuild/commit/00e05f686e15b07ac4268fcd10cf3738a5c28467
> was applied with the proposed fix in ocamlbuild 0.9.0.
>
> I suspect this means that the ocamlbuild bug was indeed fixed, and
> that we should be able to reinstate the patch.
>
> Except...  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=642835;msg=55
> says the bug is present in ocamlbuild 0.10.1.  Ximin, do you remember
> where that version number came from?  Is it possible that the bug is
> *not* present in that version?

Once bullseye is released, would it be possible to reinstate the patch?
Or maybe in experimental. This way, it will be easier to check if the
problem is still present. Moreover, everyone should not carry the burden
of failures from ocamlbuild to correctly track processes. Another
workaround would be to use bash for ocamlbuild if the problem is still
here.

While the footprint of /bin/sh is small, it is not zero and since most
shells optimize this use case, many programs assume they can spawn
process through `/bin/sh -c` at no cost. In my case, I was curious why
processes spawned by i3 and its ecosystem were all leaving a /bin/sh
instance behind.
-- 
Many pages make a thick book, except for pocket Bibles which are on very
very thin paper.

Bug#436466: dash: Please optimise single command given to -c to exec it

[Jonathan Nieder]

Herbert Xu wrote:
>> Tim Connors wrote:

>>> Because of bug #642922 (archived, but has some interesting discussion),
>>> this patch for this bug ended up being backed out.
[...]
> I gather that the ocamlbuild bug has been fixed so can this be
> reinstated please?

Curious: the ocamlbuild bug appears to still be open upstream
, but
https://github.com/ocaml/ocamlbuild/commit/00e05f686e15b07ac4268fcd10cf3738a5c28467
was applied with the proposed fix in ocamlbuild 0.9.0.

I suspect this means that the ocamlbuild bug was indeed fixed, and
that we should be able to reinstate the patch.

Except...  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=642835;msg=55
says the bug is present in ocamlbuild 0.10.1.  Ximin, do you remember
where that version number came from?  Is it possible that the bug is
*not* present in that version?

Thanks,
Jonathan

Bug#436466: dash: Please optimise single command given to -c to exec it

[Herbert Xu]

On Sat, Nov 26, 2011 at 07:44:43PM -0600, Jonathan Nieder wrote:
> 
> Tim Connors wrote:
> 
> > Because of bug #642922 (archived, but has some interesting discussion),
> > this patch for this bug ended up being backed out.
> >
> > Hopefully this bug won't be forgotten by someone who understands the
> > problems in 642922, which seem to have been fixed elsewhere.
> 
> Thanks for the reminder, Tim.  Indeed, this won't be forgotten.

Looks it had been forgotten :)

I gather that the ocamlbuild bug has been fixed so can this be
reinstated please?

Thanks,
-- 
Email: Herbert Xu 
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Bug#436466: dash: Please optimise single command given to -c to exec it

[Jonathan Nieder]

tags 642835 = upstream
block 436466 by 642835
tags 436466 + fixed-upstream
fixed 436466 dash/0.5.7-1
found 436466 dash/0.5.7-2
quit

Tim Connors wrote:

 Because of bug #642922 (archived, but has some interesting discussion),
 this patch for this bug ended up being backed out.

 Hopefully this bug won't be forgotten by someone who understands the
 problems in 642922, which seem to have been fixed elsewhere.

Thanks for the reminder, Tim.  Indeed, this won't be forgotten.



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#436466: dash: Please optimise single command given to -c to exec it

[Jonathan Nieder]

Hi Gerrit,

 The only concern I have with this change is the handling of $PPID

 $ cat /tmp/t
 echo $PPID
 $ sh -c /tmp/t
 17254
 $ sh -c 'exec /tmp/t'
 13318
 $ 

 Would this be a regression and, if yes, an acceptable one?

It's a noticeable behavior change and so it should be documented in
the changelog, I agree.

I think the change would be acceptable.  Many other shells (e.g.,
ksh93, bash) follow the same rule so it doesn't hurt script
portability.  POSIX clearly allows it: when XCU 2.12 says

Utilites other than the special built-ins [...]
shall be invoked in a separate environment that consists of
the following.

it is very careful to not include the process ID as part of the
definition.  Natural enough.  Typically non-special builtins like cd
are run in the same process, even while they get their own set of
variables, cwd, redirections, and so on.

Regards,
Jonathan



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#436466: dash: Please optimise single command given to -c to exec it

[Gerrit Pape]

On Wed, Dec 08, 2010 at 12:10:33AM +0100, Jilles Tjoelker wrote:
 Regarding sh -c optimization, I am in favour of this. Uselessly waiting
 'sh -c' processes annoy me. I made the change for FreeBSD 8.0 sh, which
 is very similar to dash. The SVN changeset is r194128. The change
 appears to work for dash as well.

The only concern I have with this change is the handling of $PPID

$ cat /tmp/t
echo $PPID
$ sh -c /tmp/t
17254
$ sh -c 'exec /tmp/t'
13318
$ 

Would this be a regression and, if yes, an acceptable one?

Regards, Gerrit.



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#436466: dash: Please optimise single command given to -c to exec it

[Jonathan Nieder]

tags 436466 + patch
forwarded 436466 http://thread.gmane.org/gmane.comp.shells.dash/556
quit

Jilles Tjoelker wrote:

 Regarding sh -c optimization, I am in favour of this. Uselessly waiting
 'sh -c' processes annoy me. I made the change for FreeBSD 8.0 sh

Thanks.



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#436466: dash: Please optimise single command given to -c to exec it

[Jilles Tjoelker]

Drake Wilson wrote:
 - Point out that full tail execution optimization is not as
   localized as the anything without shell meta-stuff that some
   people think it is, and that partial tail execution optimization
   can lead to subtle problems if not done carefully.  Cf. a vaguely
   similar case where I was attempting to use an XSI shell builtin
   (which I was willing to rely on on the target system) as a single
   command in a series of commands in a makefile, and GNU Make
   decided that it didn't look enough like a shell thing and could be
   optimized into an exec, which naturally hosed the command.  Of
   course, the case of doing exec transformation _in_ the shell is
   not as bad as that.

I think the gmake problem is caused by a non-compliance to POSIX in the
operating system, possibly combined with an overzealous skipping of the
shell by gmake. It is rather different from the fork/exec problem with
sh -c.

POSIX requires that all standard utilities except special builtins be
available via the exec functions and the utilities that can invoke
user-specified other utilities (env, find, nice, nohup, time, xargs).

In SUSv4, the special builtins are . (dot), : (colon), break, continue,
eval, exec, exit, export, readonly, return, set, shift, times, trap and
unset; none of these are under the XSI option. An implementation is not
required to provide them as external programs (and there seems no point
in doing so). Gmake must know about these and not try to exec them
itself.

Other builtins that do not exist as external programs yet need to be
provided. An easy way is to link a shell script like the below to
/usr/bin/alias, /usr/bin/bg, /usr/bin/cd, /usr/bin/command, etc.:

#!/bin/sh
command -p ${0##*/} $@

The 'command -p' serves to avoid looping when the PATH is set to a value
like /usr/bin:%builtin which is an undocumented way to prefer
/usr/bin/FOO to a builtin command FOO. The 'command' utility itself is
not subject to this because dash implements the POSIX requirement that
various (usually builtin) utilities such as 'command' cannot be
overridden via PATH (but unlike the special builtins can be overridden
with a function).

Regarding sh -c optimization, I am in favour of this. Uselessly waiting
'sh -c' processes annoy me. I made the change for FreeBSD 8.0 sh, which
is very similar to dash. The SVN changeset is r194128. The change
appears to work for dash as well.

-- 
Jilles Tjoelker



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#436466: dash: Please optimise single command given to -c to exec it

[Drake Wilson]

Quoth Jonathan Nieder jrnie...@gmail.com, on 2010-11-26 01:10:47 -0600:
  Inserting, say, a
  debugging echo before the actual command is virtually guaranteed to
  not exec the final command directly.
 
 How do you like the ksh93 behavior?

I wasn't previously aware of the ksh93 behavior.  So they _do_ do that
in cases where the entire script is not a single simple command.  (My
previous experience was always otherwise, including with Bash, pdksh,
I think zsh though that appears to exhibit ksh93-esque behavior now so
I may have misremembered, and one or two other shells.)

ksh93 appears to also handle various other tail positions, including
subshells and  and ||.  That's more the sort of thing I could get
behind if I were made Dictator of Shells.  :-)

Obviously my information above was out of date.  I amend it to say in
many cases will not exec the final command directly.  (This means
that a program that wishes to be portable to multiple underlying
shells can rely even less on the process tree shape, so my previous
points mostly stand anyway.)

 Actually, let me take that back.  If you actually want a guarantee
 that future versions of dash will have or lack this feature,

Let me be a little clearer, at the risk of restating myself.  I don't
personally need a guarantee either way, especially since it's
something I'll have to deal with regardless of what dash does in the
future.  I mostly wanted to ensure that people reading the bug trail
were aware of some of the subtler ramifications of a decision to do
tail execution optimization in shells, and in particular, I wanted to:

  - Add a dissenting voice to some ideas from upthread that making the
last command a child of the shell is wrong and that relying on a
highly compact process tree shape that was not actually requested
is a sane thing to do in a POSIX or GNU/Linux environment, which
IMHO it is not.

  - Point out that full tail execution optimization is not as
localized as the anything without shell meta-stuff that some
people think it is, and that partial tail execution optimization
can lead to subtle problems if not done carefully.  Cf. a vaguely
similar case where I was attempting to use an XSI shell builtin
(which I was willing to rely on on the target system) as a single
command in a series of commands in a makefile, and GNU Make
decided that it didn't look enough like a shell thing and could be
optimized into an exec, which naturally hosed the command.  Of
course, the case of doing exec transformation _in_ the shell is
not as bad as that.

Thanks for the attention.

   --- Drake Wilson



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#436466: dash: Please optimise single command given to -c to exec it

[Jonathan Nieder]

Hi Drake,

Drake Wilson wrote:

 Depending on how it's implemented, this can also cause a sort of
 inconsistency fault line.
[...]
 Inserting, say, a
 debugging echo before the actual command is virtually guaranteed to
 not exec the final command directly.

How do you like the ksh93 behavior?

Actually, let me take that back.  If you actually want a guarantee
that future versions of dash will have or lack this feature, you
can work with upstream[1] or the Austin common standards revision
group[2].

Hints:
. Since ksh93 (like ksh88, iirc) and bash already support this
  optimization and, well, it's been good for them, POSIX is not likely
  to change to forbid it.

. Goals for dash include code size, speed, POSIX conformance,
  simplicity, compatibility.  Patches with clear justification are
  much better received than arguments without patches (though bug
  reports are always appreciated).

Needless to say, a change in behavior is not going to happen in time
for squeeze regardless.

Thanks for your interest,
Jonathan

[1] d...@vger.kernel.org
[2] http://www.opengroup.org/austin/



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#436466: dash: Please optimise single command given to -c to exec it

[Raphael Geissert]

If this is ever implemented, care should be taken not to break backwards 
compatibility. E.g.

$ bash -c 'readlink /proc/$$/exe'
/bin/readlink
$ bash -c ':; readlink /proc/$$/exe'
/bin/bash
$ dash -c 'readlink /proc/$$/exe'
/bin/dash

Cheers,
-- 
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#436466: dash: Please optimise single command given to -c to exec it

[Reuben Thomas]

Package: dash
Version: 0.5.4-1
Severity: wishlist

When you do bash -c foo, where foo is a simple command, it's execed.
It'd be nice if dash had this same optimisation (which should be
rather simpler than it is for bash).

It needn't even be done precisely, i.e. for all cases where it could
be used, as long as it covers the most common case:

sh -c foo

In general I believe that simply searching for the absence of a few
special characters `|(){}' is a good way of catching cases that can
be optimised, without of course catching all of them.

The other alternative is to study how bash performs the same
optimisation.

If this sounds interesting, I'd be happy to try preparing a patch.
-- System Information:
Debian Release: lenny/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: i386 (i686)

Kernel: Linux 2.6.21-2-686 (SMP w/1 CPU core)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dash depends on:
ii  libc6 2.6-2  GNU C Library: Shared libraries

dash recommends no packages.

-- debconf information:
* dash/sh: true


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]