subject:"Re\: bash exorcism experiment \('bug' 762923 \& 763012\)"

2014-10-07 15:03:05 +0200, Thorsten Glaser:
 On Sat, 4 Oct 2014, Russ Allbery wrote:
 
   If we were to decide that #309415 should be fixed in policy (and hence
   posh), then it should be done by requiring support for the obsolescent
 
 The problems with posh and dash are also the sheer number of bugs
 in corner cases, which the more actively developed shells fix.
 
 posh inherited all bugs from pdksh which I fixed in mksh, partially
 by rewriting the parser… so it had to be restarted from newer code.
 
 dash, well, just ugh.
 
 tglase@tglase:~ $ cat x
 a='space divded  argument
 here'
 IFS=\  ; set -- $a
 IFS= ; q=$* ; nq=$*
 printf '%s\n' $* $* $q $nq
 [ $q = $nq ]  echo =true || echo =false
 tglase@tglase:~ $ dash x
 spacedivdedargument
 here
 spacedivdedargument
 here
 spacedivdedargument
 here
 spacedivdedargument
 here
 =true

That's expected behaviour to me, and the intuitive continuation
of the Bourne shell.

In the Bourne shell, the * and @ variable were variables whose
content were the concatenation of the positional arguments with
  in between. And, as a special case $@ in list contexts
didn't undergo normal expansion, but instead expended to the
list of positional parameters as separate words.

$*, $@, $* were not special in any way. They just underwent
the same rules as other variables. Only $@ was.

That's why for instance:

bourne-sh -c 'set a b; IFS=; printf %s\n $*'
a b

dash is the same except that $* and $@ are the concatenation of
the positional parameters with the first character of IFS.

ksh introduced its bogus array feature and modified the meaning
of $* and $@ which are no longer normal variables (and in my
opinion make a lot less sense than the ash behaviour).

Now all shells have varying behaviours for corner cases of those
which means  that for portability, you should only use $* and
$@ quoted and only use $@ in list contexts.

The behaviour for a=$@ of ${a#$*} or ${a#$@} varies from shell
to shell.

-- 
Stephane


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141013204233.gb6...@chaz.gmail.com

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-13 12:21:33 +0200, Thorsten Glaser:
 On Mon, 13 Oct 2014, Dominik George wrote:
 
  foo='x[$(rm -rf /)]'
  echo $(( foo ))
  
  Guess when the array index is evaluated? Now mind that it could be
 
 This is fully and completely a user error. (User being the script.)
 
  user-provided.
 
 Never put “tainted” input into ksh arithmetics, period.
 (And always initialise your variables.)
 
 It could be documented better. Stéphane Chazelas said
 he may write it up in detail, which I have already promised
 will then be linked from the mksh manpage.
[...]

It's an error from a user not expecting arithmetic expressions
to be evaluated in such a silly way. Yet another design mistake
of Korn's.

No documentation will ever prevent users from doing

echo $(( ENV_VAR + 2 ))

That being a vector for arbitrary command execution is in breach
of the law of least astonishment.

I'd bet the first reaction of anyone finding it out would be
that the language is severely broken.

I and many others (and many others) have spent the last 20 years
telling people to quote their variable, that

echo $QUERYSTRING
or even
: ${QUERYSTRING:=foo}

is a DoS vector or worse
(QUERYSTRING=/*/*/*/../../../*/*/*/*/...)

, experimenting with teaching tools like the split+glob operator
(`echo $var` is applying the split+glob operator to the content
of $var)

to no avail. People still do:

echo $var

because it's the most intuitive thing to write. It's saying what
there should be in the tin. Many people don't understand or
don't believe you when you tell them you should actually use:

printf '%s\n' $var

So I do really wish that Debian's sh doesn't import any other
misfeature of the Korn shell.

If you need arrays, use $@ or use perl/python/ruby..., but
please don't break yet another shell with the Korn arrays or
arithmetics.

-- 
Stephane


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141013210234.gc6...@chaz.gmail.com

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-13 Thread Thorsten Glaser

Stephane Chazelas dixit:

[ a lot, with which I vehemently disagree ]

If you need arrays, use $@ or use perl/python/ruby..., but
please don't break yet another shell with the Korn arrays or
arithmetics.

The good part about mksh i̲s̲ that it’s a programming language,
a nice one to use, much more legible than Perl, besides having
“set -x” beats them all; lots of those other features like
associative multidimensional arrays are making their way into
mksh. I only hack o̲n̲ mksh because I want to program i̲n̲ mksh,
because, e.g. C sucks (UB). So I make it the best language
within its scope I can, and David Korn as well as those who
worked on pdsh/pdksh/oksh before laid good groundwork.

Newbies should be careful before using a̲n̲y̲ language. “or use
perl instead” doesn’t help if they forget -T either. Your
arguments are, thus, lacking hold.

bye,
//mirabilos
-- 
18:47⎜mirabilos:#!/bin/mksh well channels… you see, I see everything in the
same window anyway  18:48⎜xpt:#!/bin/mksh i know, you have some kind of
telnet with automatic pong 18:48⎜mirabilos:#!/bin/mksh haha, yes :D
18:49⎜mirabilos:#!/bin/mksh though that's more tinyirc – sirc is more comfy


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/pine.bsm.4.64l.1410132115280@herc.mirbsd.org

Re: bash exorcism experiment ('bug' 762923 763012)

2014-09-29 09:22:58 +1000, Russell Stuart:
 On Sun, 2014-09-28 at 16:47 +0200, Guillem Jover wrote:
   I've attempted to port the many shell scripts I've written over the
   years to dash.  The three irritants are:
   
 - pipefail,
  
  http://cfajohnson.com/shell/cus-faq-2.html#Q11.
 
 That's one of those scratch my eyes out solutions. A more readable
 solution is just to say the exit status of each command in a temporary
 file.  Given how infrequently the problem arises, it isn't a major
 issue.
[...]

I happen to have written that portion of the cus FAQ. There were
a few issues with that implementation. You can find an improved
version at http://unix.stackexchange.com/a/76171/22565

[...]
 No workaround for this one?  Pity.  This is what usually prevents
 conversion.

If you do find yourself needing arrays in shells, chances are
you're not doing it the right way or that shells are not what
you need.

You may want to consider perl/ruby/python/awk instead.
Imperative programming patterns don't really apply to shells.

In any case, IMO, ksh/bash arrays are not the solution, and
there's a good reason they were never standardized by POSIX.

POSIX shells have $@ which in most cases is more than enough.

-- 
Stephane


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141013211438.gd6...@chaz.gmail.com

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-02 10:06:50 -0400, shawn wilson:
[...]
 I hate the idea of dash. It's not more secure (see vmware cve for an
 example) and I think it was more of an accident than anything else this
 didn't hit dash too.
[...]

That CVE is not about a bug in dash. There are a few
misconceptions around that CVE.

See
https://bugs.launchpad.net/ubuntu/+source/dash/+bug/1215660/comments/4
for more details.

Whatever dash bugs may have are easily fixed. The good thing
with it is that it has *fewer* features, and especially fewer of
the ksh misfeatures (many of which copied by bash, some of them
fixed/improved in zsh), so it's more efficient and likely to be
safer than bash/mksh/zsh so is a much more obvious choice for
/bin/sh, that is  the system's command line interpreter (used in
system(), popen()) or interpreter for POSIX sh scripts.

By all means, use zsh or bash... as your interactive shell, but
please keep /bin/sh minimum, and don't bring as broken a feature
as the ksh arrays into the sh language.

Switching to dash for /bin/sh is one of the great things Debian
has done.

-- 
Stephane


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141013213741.ge6...@chaz.gmail.com

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-12 Thread Florian Weimer

* Russell Stuart:

   - array variables.

Array variables practically imply arithmetic evaluation, amd this is a
shell feature which is rather difficult to use correctly because
compatibility with other shell encourages both recursive evaluation
and access to the full shell language in a few corners.

If you need array variables, it's likely that the script has grown so
complex that switching to another language is a good idea.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87fvet3vgf@mid.deneb.enyo.de

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-12 Thread Marco d'Itri

On Oct 11, Theodore Ts'o ty...@mit.edu wrote:

 But if individual Debian developers were to fix their own packages, or
 suggest patches as non-RC bugs, there wouldn't be any real harm, and
 possibly some good (especially for those people who are very much into
 pedantry, and don't mind a slightly slower system --- but if a user
 wants to use /bin/posh, that's an individual user's choice :-)
We have no obligation to support every bad idea that people have.

-- 
ciao,
Marco


signature.asc
Description: Digital signature

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-12 Thread Holger Levsen

Hi Florian,

On Sonntag, 12. Oktober 2014, Florian Weimer wrote:
 Array variables practically imply arithmetic evaluation, amd this is a
 shell feature which is rather difficult to use correctly because
 compatibility with other shell encourages both recursive evaluation
 and access to the full shell language in a few corners.

and if I dont care about compatibilty?
 
 If you need array variables, it's likely that the script has grown so
 complex that switching to another language is a good idea.

oh, yeah. amen++


cheers,
Holger, just dealing with 25 lines of shell growing into 1000 lines
 and where I missed the point to switch to python or anything
 else sensible


signature.asc
Description: This is a digitally signed message part.

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-12 Thread Russell Stuart

On Sun, 2014-10-12 at 22:05 +0200, Florian Weimer wrote:
 Array variables practically imply arithmetic evaluation, amd this is a
 shell feature which is rather difficult to use correctly because
 compatibility with other shell encourages both recursive evaluation
 and access to the full shell language in a few corners.

I don't understand this.  I've had legions of bugs in shell scripts over
the years.  The number caused by arithmetic evaluation is tiny.  The
only trap I can recall is it being restricted to 32 bit signed
arithmetic, and that not being enough for times.

 If you need array variables, it's likely that the script has grown so
 complex that switching to another language is a good idea.

Not really.  One of, if not the primary function of the shell is to run
other programs.  One of the things you have to do when running programs
is construct and process argument lists.  An array variable is the only
sane way to represent an argument list in the shell scripting language.
The only other option is horrid hacks using `eval ...`.

Also, while I agree that shell script is a terrible language and nothing
over 100 lines or so should be written in it, real life doesn't always
pan out the way you plan.  It's probably true that any shell script I've
written started out under 100 lines, or at least I'm going to pretend I
thought they would would when I stared writing them.  But the things are
like dust bunnies in a cupboard.  They grow while you aren't paying
attention.

So now I have one 5000 line shell script, and a few around the 1000 line
mark.  No, this is not something I'm proud of, but I've got better
things to do in my life than rewrite 5000 line programs that have been
bug free for years.


signature.asc
Description: This is a digitally signed message part

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-12 Thread Dominik George

 Array variables practically imply arithmetic evaluation, amd this is
a
 shell feature which is rather difficult to use correctly because
 compatibility with other shell encourages both recursive evaluation
 and access to the full shell language in a few corners.

I think the idea here was not use correctly as in yielding the result you 
expect but as in guard against wreaking havoc:

foo='x[$(rm -rf /)]'
echo $(( foo ))

Guess when the array index is evaluated? Now mind that it could be 
user-provided.

-nik


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/8ce27f9c-7ab6-4a9f-9729-4fc417edb...@naturalnet.de

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-12 Thread Russell Stuart

On Mon, 2014-10-13 at 06:23 +0200, Dominik George wrote:
 foo='x[$(rm -rf /)]'
 echo $(( foo ))

 Guess when the array index is evaluated? Now mind that it could be 
 user-provided.

In dash it isn't executed which means on Debian at least it's most
harmless.  That's another bouquet for dash.  It's almost enough to make
you forgive the reason it doesn't parse: dash's buggy argument parser.


signature.asc
Description: This is a digitally signed message part

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-11 Thread Russ Allbery

Russell Stuart russell-deb...@stuart.id.au writes:

 Not really.  I'm about documentation reflecting reality.  Think of
 putting an electrical component whose documentation says its 200 degrees
 on a motherboard, only to find it fails at 190.  When you ask why, is
 well we design it for 200, but only test it to 180 a satisfying
 answer?

 You have convinced me that in this case it's going to have to be that
 way, so my prejudices notwithstanding.  I've rationalised the pain away
 by deciding it's no so bad as any competent programmer could see that is
 it only tested to 190 regardless of what the standards say.

Yeah, I do get that discomfort.  I would love for Policy to be more
accurate about what's actually happening in the archive.  I just don't
have much (any) time at the moment to try to push the wording in that
direction.

 It's attractive because makes Policy more relevant - but only because of
 that.  Now that I think about it, switching pbuilder to posh would be
 almost as good.  Any additional pain would not be worth the effort.

That would be interesting, although I think that would mostly pick up
build issues, which are somewhat different from the issues encountered
when running the packages on a system.  What we'd really like is something
like running autopkgtest tests with posh as a shell, but with much better
coverage than we currently have.

We're much better at testing our build processes than we are at testing
the constructed packages, at least currently.

 If Debian was going to switch to another shell, I'd vote for the one in
 busybox.  That's because on desktop machines it doesn't matter, but on
 embedded architectures it does - and they use busybox.  So switching to
 busybox would extend Debian's reach.

Wouldn't that pose a bunch of problems due to the huge number of built-ins
in busybox, most of which don't work the same way as the regular program?

 If the speed is comparable

 Here are two benchmarks.  I did others. These demonstrate the extremes:

 $ time dash -c 'i=0; while [ $i -lt 1000 ]; do echo -n; i=$(($i + 
 1)); done'
 real  0m16.695s
 user  0m16.684s
 sys   0m0.000s
 $ time posh -c 'i=0; while [ $i -lt 1000 ]; do echo -n; i=$(($i + 
 1)); done'
 real  0m41.899s
 user  0m41.872s
 sys   0m0.000s

Yeah, I seemed to remember posh being much slower than dash (although that
particular benchmark is a little artificial).

 It looks like moving to dash sped Debian up a little.

That was supported by boot timings, which are a pretty good simulation of
real-world load.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87lhom8q3t@hope.eyrie.org

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-11 Thread Theodore Ts'o

On Sat, Oct 11, 2014 at 10:37:26AM -0700, Russ Allbery wrote:
  You have convinced me that in this case it's going to have to be that
  way, so my prejudices notwithstanding.  I've rationalised the pain away
  by deciding it's no so bad as any competent programmer could see that is
  it only tested to 190 regardless of what the standards say.
 
 Yeah, I do get that discomfort.  I would love for Policy to be more
 accurate about what's actually happening in the archive.  I just don't
 have much (any) time at the moment to try to push the wording in that
 direction.

I assume that posh meets the strict definition of 10.4.  And so
without actually changing policy, someone _could_ try setting /bin/sh
to be /bin/posh, and then start filing RC bugs against packages that
have scripts that break.   Yes?

Given that the freeze is almost upon us, I could see how this might be
considered unfriendly, but if someone wanted to start filing bugs (at
some priority, perhaps RC, perhaps not) after Jessie ships, we could
in theory try to (slowly) move Debian to the point where enough
scripts in Debian worked under /bin/posh that it might be possible to
set it at a release goal, for some future release.   Yes?

Now, this might be considered not the best use of Debian Developers'
resources, and which is why it might be considered bad manners to do
mass bug filings, particularly mass RC bug filings at this stage of
the development/release cycle.

But if individual Debian developers were to fix their own packages, or
suggest patches as non-RC bugs, there wouldn't be any real harm, and
possibly some good (especially for those people who are very much into
pedantry, and don't mind a slightly slower system --- but if a user
wants to use /bin/posh, that's an individual user's choice :-)

Cheers,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141011215714.ga21...@thunk.org

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-07 Thread Russell Stuart

On Fri, 2014-10-03 at 09:20 -0700, Russ Allbery wrote:
 Russell Stuart russell-deb...@stuart.id.au writes:
  I looks to me like you are re-writing history.
 
 I'm not sure how you meant this, but to note, this sentence made me very
 sad, since it felt like you believe I'm being intentionally dishonest with
 you.  I'm really not, although I'm not sure how to convince you of that.

You convinced me I was wrong a few sentences later.

 I thought that's what you were getting at when talking about
 testing

Not really.  I'm about documentation reflecting reality.  Think of
putting an electrical component whose documentation says its 200 degrees
on a motherboard, only to find it fails at 190.  When you ask why, is
well we design it for 200, but only test it to 180 a satisfying
answer?

You have convinced me that in this case it's going to have to be that
way, so my prejudices notwithstanding.  I've rationalised the pain away
by deciding it's no so bad as any competent programmer could see that is
it only tested to 190 regardless of what the standards say.

 Oh!  I didn't realize or internalize that you were proposing switching the
 default shell to posh from dash.  Yes, that would certainly improve our
 compliance with Policy considerably.

It's attractive because makes Policy more relevant - but only because of
that.  Now that I think about it, switching pbuilder to posh would be
almost as good.  Any additional pain would not be worth the effort.

If Debian was going to switch to another shell, I'd vote for the one in
busybox.  That's because on desktop machines it doesn't matter, but on
embedded architectures it does - and they use busybox.  So switching to
busybox would extend Debian's reach.

 If the speed is comparable

Here are two benchmarks.  I did others. These demonstrate the extremes:

$ time dash -c 'i=0; while [ $i -lt 1000 ]; do echo -n; i=$(($i + 
1)); done'
real0m16.695s
user0m16.684s
sys 0m0.000s
$ time posh -c 'i=0; while [ $i -lt 1000 ]; do echo -n; i=$(($i + 
1)); done'
real0m41.899s
user0m41.872s
sys 0m0.000s
$ time busybox sh -c 'i=0; while [ $i -lt 1000 ]; do echo -n; 
i=$(($i + 1)); done'
real0m27.938s
user0m25.160s
sys 0m2.760s
$ time bash -c 'i=0; while [ $i -lt 1000 ]; do echo -n; i=$(($i + 
1)); done'
real1m7.971s
user1m7.928s
sys 0m0.000s

$ time dash -c 'x=aaa; t() { local x=$1; echo $x; }; 
while [ ${x%b} = ${x} ]; do y=; while :; do z=${x#b}; [ $z != $x ] || 
break; y=a$y x=$z; done; x=$(t ${y}b${x#a}); done'
real0m1.577s
user0m0.204s
sys 0m0.500s
$ time posh -c 'x=aaa; t() { local x=$1; echo $x; }; 
while [ ${x%b} = ${x} ]; do y=; while :; do z=${x#b}; [ $z != $x ] || 
break; y=a$y x=$z; done; x=$(t ${y}b${x#a}); done'
real0m2.232s
user0m0.316s
sys 0m0.536s
$ time busybox sh -c 'x=aaa; t() { local x=$1; echo $x; 
}; while [ ${x%b} = ${x} ]; do y=; while :; do z=${x#b}; [ $z != $x ] 
|| break; y=a$y x=$z; done; x=$(t ${y}b${x#a}); done'
real0m2.104s
user0m0.284s
sys 0m0.516s
$ time bash -c 'x=aaa; t() { local x=$1; echo $x; }; 
while [ ${x%b} = ${x} ]; do y=; while :; do z=${x#b}; [ $z != $x ] || 
break; y=a$y x=$z; done; x=$(t ${y}b${x#a}); done'
real0m4.849s
user0m0.892s
sys 0m0.740s
$

It looks like moving to dash sped Debian up a little.


signature.asc
Description: This is a digitally signed message part

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-07 Thread Thorsten Glaser

On Sat, 4 Oct 2014, Russ Allbery wrote:

  If we were to decide that #309415 should be fixed in policy (and hence
  posh), then it should be done by requiring support for the obsolescent

The problems with posh and dash are also the sheer number of bugs
in corner cases, which the more actively developed shells fix.

posh inherited all bugs from pdksh which I fixed in mksh, partially
by rewriting the parser… so it had to be restarted from newer code.

dash, well, just ugh.

tglase@tglase:~ $ cat x
a='space divded  argument
here'
IFS=\  ; set -- $a
IFS= ; q=$* ; nq=$*
printf '%s\n' $* $* $q $nq
[ $q = $nq ]  echo =true || echo =false
tglase@tglase:~ $ dash x
spacedivdedargument
here
spacedivdedargument
here
spacedivdedargument
here
spacedivdedargument
here
=true
tglase@tglase:~ $ ksh93 x
spacedivdedargument
here
space
divded
argument
here
spacedivdedargument
here
spacedivdedargument
here
=true

  It's already fixed:

  * ‘test’, if implemented as a shell built-in, must support ‘-a’ and ‘-o’
  as binary logical operators.

 Yeah, that's been there for a while.  They were too widely used, so
 although they're really confusing, we decided not to pick that fight.

Yeah, but Md is an arsehole anyway and requires printf to be
a /bin/sh builtin instead of just adding /usr/bin to $PATH,
especially now that the initrd mounts /usr already anyway,
and CTTE decided to rather offend me than Md because he is
maintainer of the more important packages, or those where
it’s hard to find someone else for.

bye,
//mirabilos
-- 
Yes, I hate users and I want them to suffer.
-- Marco d'Itri on gmane.linux.debian.devel.general


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/alpine.deb.2.11.1410071459300.23...@tglase.lan.tarent.de

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-07 Thread Neil McGovern

On Tue, Oct 07, 2014 at 03:03:05PM +0200, Thorsten Glaser wrote:
 Yeah, but Md is an arsehole anyway and requires printf to be
 a /bin/sh builtin instead of just adding /usr/bin to $PATH,
 especially now that the initrd mounts /usr already anyway,
 and CTTE decided to rather offend me than Md because he is
 maintainer of the more important packages, or those where
 it’s hard to find someone else for.
 

Thorsten,

Could you please keep your tone more civil? Personal attacks on fellow
project members and conspiracy theories does nothing to further your
technical arguments - in fact it makes me more likely to dismiss any
valid point you may have.

Neil
-- 


signature.asc
Description: Digital signature

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-07 Thread The Wanderer

On 10/07/2014 at 02:39 AM, Russell Stuart wrote:

 On Fri, 2014-10-03 at 09:20 -0700, Russ Allbery wrote:

 Oh!  I didn't realize or internalize that you were proposing
 switching the default shell to posh from dash.  Yes, that would
 certainly improve our compliance with Policy considerably.
 
 It's attractive because makes Policy more relevant - but only because
 of that.  Now that I think about it, switching pbuilder to posh would
 be almost as good.  Any additional pain would not be worth the
 effort.

Speaking as an uninvolved but interested observer, I though this
(switching the shell people use to build and test their packages, and
nothing else) was what you were proposing in the first place.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw


0x3428326B.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-07 Thread Thorsten Glaser

On Tue, 7 Oct 2014, Adam Borowski wrote:

 change your /bin/sh), 2. being (then) a violation of a must clause of
 the policy.

To be fair: my bug wasn’t about -a and -o, but about the printf builtin
which Policy is silent about. Some shells do have a builtin printf,
most don’t. printf(1) lives in /usr/bin, and Md’s init script set the
$PATH explicitly to /bin:/sbin yet still used printf(1), which, for a
POSIX sh script, is probably sensible anyway. He “just” barricaded all
three or four ways I could come up to fix this for the users.

bye,
//mirabilos
-- 
Yes, I hate users and I want them to suffer.
-- Marco d'Itri on gmane.linux.debian.devel.general


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/alpine.deb.2.11.1410072133250.9...@tglase.lan.tarent.de

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-05 Thread Bernd Zeimetz

On 09/28/2014 10:33 AM, Colin Watson wrote:
 On Sat, Sep 27, 2014 at 09:11:44PM -0500, Troy Benjegerdes wrote:
 Does update-menus really need bash? Why?
 
 pipefail is actually a fairly useful bashism.

Use mispipe from moreutils instead.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/54310554.2060...@bzed.de

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-04 Thread Colin Watson

On Fri, Oct 03, 2014 at 10:42:56AM +1000, Russell Stuart wrote:
 On Fri, 2014-10-03 at 00:04 +, brian m. carlson wrote:
  Unfortunately, some developers have outright refused to make their
  software using /bin/sh work with posh, even when provided with a patch
  (e.g. #309415), to the point that last time I tried to use posh
  as /bin/sh, the system wouldn't boot.
 
 If I understand what you are saying correctly this means policy 10.4 is
 mostly ignored.

I think you're making the mistake of inferring from a few notable
failures that the whole thing is broken.  It's not.  Policy 10.4 has
been incredibly useful in practice to short-circuit a whole load of what
would otherwise have been repetitive and time-wasting arguments about
what shells we ought to support, by having the argument once and then
being able to refer to its results in test code, bug reports, and so on.
Most people who don't care too much about the specifics will go along
with it and fix things up along with the various other things they do,
and it's been very useful in harnessing the efforts of a long tail of
developers that way.

 If we want Debian policy to reflect reality and make it easy for
 developers to test their scripts conform to policy, then it should say
 #!/bin/sh scripts must work with dash.

If we were to decide that #309415 should be fixed in policy (and hence
posh), then it should be done by requiring support for the obsolescent
XSI extensions test -a and test -o.  I might even have supported that
once except that the rules for parsing those are headache-inducing; yes,
I'm sure shell implementers can manage them, but they're a lot of
complexity to stuff into what people intuitively expect to be a simple
primitive operation.  As it is, I think recent versions of POSIX make an
excellent argument for why they're a terrible idea, and I've generally
found upstreams to be receptive to patches to avoid them.

-- 
Colin Watson   [cjwat...@debian.org]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141004084300.ga14...@riva.ucam.org

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-04 Thread Jakub Wilk


* Colin Watson cjwat...@debian.org, 2014-10-04, 09:43:
If we were to decide that #309415 should be fixed in policy (and hence 
posh), then it should be done by requiring support for the obsolescent 
XSI extensions test -a and test -o.


It's already fixed:

* ‘test’, if implemented as a shell built-in, must support ‘-a’ and ‘-o’ 
as binary logical operators.


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141004174941.ga3...@jwilk.net

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-04 Thread Russ Allbery

Jakub Wilk jw...@debian.org writes:
 * Colin Watson cjwat...@debian.org, 2014-10-04, 09:43:

 If we were to decide that #309415 should be fixed in policy (and hence
 posh), then it should be done by requiring support for the obsolescent
 XSI extensions test -a and test -o.

 It's already fixed:

 * ‘test’, if implemented as a shell built-in, must support ‘-a’ and ‘-o’
 as binary logical operators.

Yeah, that's been there for a while.  They were too widely used, so
although they're really confusing, we decided not to pick that fight.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87a95bzoht@hope.eyrie.org

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-04 Thread Colin Watson

On Sat, Oct 04, 2014 at 11:19:42AM -0700, Russ Allbery wrote:
 Jakub Wilk jw...@debian.org writes:
  * Colin Watson cjwat...@debian.org, 2014-10-04, 09:43:
  If we were to decide that #309415 should be fixed in policy (and hence
  posh), then it should be done by requiring support for the obsolescent
  XSI extensions test -a and test -o.
 
  It's already fixed:
 
  * ‘test’, if implemented as a shell built-in, must support ‘-a’ and ‘-o’
  as binary logical operators.
 
 Yeah, that's been there for a while.  They were too widely used, so
 although they're really confusing, we decided not to pick that fight.

Oh right, I should have checked. :-)  While I don't especially like it
as a technical decision, I can certainly respect the trade-off of
technical excellence vs. practical effort required.

Thanks,

-- 
Colin Watson   [cjwat...@debian.org]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141004205637.ga24...@riva.ucam.org

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-03 Thread Russell Stuart

On Thu, 2014-10-02 at 20:43 -0700, Russ Allbery wrote:
 A lot of people miss this about Policy 10.4.  People seem to think that
 Policy 10.4 is about requirements for shell scripts.  But it's just as
 much a standard for /bin/sh.

You wrote it, so I guess you get to say what it means.  But if you
hadn't said so just now, I'd be using the People's interpretation
rather than yours.  10.4 is after all titled Scripts, not a sh
language definition or some such.  Where it does define the shell
language, it does so in in this context: Scripts may assume
that /bin/sh implements   So to me it is addressing itself to
script writers, not sh language designers, and to extent it describe the
the shell language standard is does to purely to inform the script
writers what environment they can expect when they use #!/bin/sh.

 This was important when we were debating switching from bash to something
 else, and needed to be clear about what behavior the rest of the
 archive could expect /bin/sh to always satisfy.

I looks to me like you are re-writing history.  Right back in the
2.1.3.2 policy was pretty much what it was now.  #!/bin/sh scripts had
to be POSIX - albeit with less extensions than we allow today.  If
policy was so good at informing the debate about what #!/bin/sh
scripts did, I'd be surprised there was much in the way of a debate.
You could have switched to any shell that implemented the POSIX subset
with very little pain.

So this statement of yours: we ... needed to be clear about what
behavior the rest of the archive could expect /bin/sh to always satisfy
is puzzling, because there was pain.  Everyone knew what /bin/sh did -
it was defined in the bash man page.  Since bashism's worked just fine,
and evidently regardless of what policy said no one cared whether you
used bash'ism or not so they were used with gay abandon.  If as you
suggest Debian relied on policy for a clear description of how /bin/sh
scripts behaved, it was in for a rude shock.

It didn't of course.  Debian got the clear description it needed by
writing automated checkers like checkbashism, followed threats to
change /bin/sh away from /bin/bash, mass bug filings and finally in 2011
doing it.

I am not criticising any part of this process.  Standardising its shell
language was huge undertaking for Debian, and pull it off almost without
a hitch.  It's the sort of thing that makes me proud of the project.
What I am questioning is your assertion that policy that wasn't verified
let alone enforced was somehow key to it.

 I think people often don't realize what Policy is actually about, or what
 it can (and can't!) accomplish.  Policy is more a way for us to coordinate
 our work and agree on what we're actually talking about than an enforced
 set of rules that are followed.

Again, you've lost me.  Yes, policy that is followed and policed is very
useful.  It is very nice have man pages for almost everything.  For me
its essential I can rely on Debian's copyright policing.

But to use this example again, you are saying the agreeing that
#!/bin/sh scripts shall be POSIX shell scripts, and then largely
ignoring it for 10 years because it is unverifiable was helpful to the
project?  I don't see how it saved anyone any time.

 So yes, there's a lot of Policy that is ignored in practice.  You can take
 various attitudes towards that.  You can view that as meaning Policy is
 (at least partly) worthless because we're not enforcing it.  Or you can
 decide that Policy is more aspirational than descriptive.  Or you can
 focus on the change Policy has helped make happen.  I think all those
 viewpoints are accurate to a degree.

OK, but realise you are making life hard for some of us here.  Perhaps
you, as one of the policy author's know what bits are hard and fast
rules and which bits are purely aspirational.  I don't.  I guess if we
less knowledgeable folk finding ourselves disagreeing with some policy,
we can try assuming it's aspirational and ignore it.  Yes, it made me
cringe to write that.  But you are telling me it is the way Debian works
now.  And I get the impression you think this is a good thing.

 As bad as you think the compliance with Policy 10.4 is right now, I
 guarantee that the prospects of being able to use something else as 
 /bin/sh would be way worse if we did what you suggest.

Ah!  And here we is our fundamental point of difference.  It is beyond
me how you could think that could be so.  So much so that I'm doubting
my comprehension abilities.

I do have this right - the goal is to ensure #!/bin/sh scripts use a
standardised subset of the shell languages out there?  That way should a
user change a different /bin/sh, he can be reasonably sure it will work
if implements this well defined subset.  (And yes, I acknowledge the
subset is well defined in the current policy - well done.)

To achieve that end you are proposing all we do is ask developers nicely
to use that subset.  The alternative I am proposing is to link /bin/sh
to a

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-03 Thread Russ Allbery

Russell Stuart russell-deb...@stuart.id.au writes:
 On Thu, 2014-10-02 at 20:43 -0700, Russ Allbery wrote:

 A lot of people miss this about Policy 10.4.  People seem to think that
 Policy 10.4 is about requirements for shell scripts.  But it's just as
 much a standard for /bin/sh.

 You wrote it, so I guess you get to say what it means.  But if you
 hadn't said so just now, I'd be using the People's interpretation
 rather than yours.  10.4 is after all titled Scripts, not a sh
 language definition or some such.  Where it does define the shell
 language, it does so in in this context: Scripts may assume that
 /bin/sh implements   So to me it is addressing itself to script
 writers, not sh language designers, and to extent it describe the the
 shell language standard is does to purely to inform the script writers
 what environment they can expect when they use #!/bin/sh.

We had a lot of discussions about this on debian-policy at the time.  I
suppose it's possible that I'm misremembering, but I'm pretty sure I'm
not.  That's specifically what the phrasing scripts may assume is trying
to drive at: this is the functionality that you can assume that /bin/sh
has, which in turn creates a requirement for /bin/sh.

I'm sure the wording could be clearer.  Wording always can be.  :)

 This was important when we were debating switching from bash to
 something else, and needed to be clear about what behavior the rest of
 the archive could expect /bin/sh to always satisfy.

 I looks to me like you are re-writing history.

I'm not sure how you meant this, but to note, this sentence made me very
sad, since it felt like you believe I'm being intentionally dishonest with
you.  I'm really not, although I'm not sure how to convince you of that.

Maybe you actually meant to say misremembering (something that could be
accidental) rather than re-writing (which is usually taken to be
intentional and deliberate)?

Anyway, I think the key misunderstanding is here:

 So this statement of yours: we ... needed to be clear about what
 behavior the rest of the archive could expect /bin/sh to always satisfy
 is puzzling, because there was pain.  Everyone knew what /bin/sh did -
 it was defined in the bash man page.  Since bashism's worked just fine,
 and evidently regardless of what policy said no one cared whether you
 used bash'ism or not so they were used with gay abandon.  If as you
 suggest Debian relied on policy for a clear description of how /bin/sh
 scripts behaved, it was in for a rude shock.

That's not what I was getting at at all.  Rather, that portion of the
policy received a lot of work and attention, some rewordings, and a lot of
expansion during the time period when the project was working on replacing
/bin/sh with dash.  The specific list of exceptions was expanded based on
that work; they were the things that we chose to accept on top of a POSIX
shell because getting rid of them would have otherwise have been too much
effort.  That section of Policy was written hand-in-hand with writing
checkbashisms and with pushing the archive towards working with dash.

 It didn't of course.  Debian got the clear description it needed by
 writing automated checkers like checkbashism, followed threats to change
 /bin/sh away from /bin/bash, mass bug filings and finally in 2011 doing
 it.

Policy 10.4 is a key part of the specification for checkbashisms and the
justification for the mass bug filings.  These were not independent
efforts.

 I am not criticising any part of this process.  Standardising its shell
 language was huge undertaking for Debian, and pull it off almost without
 a hitch.  It's the sort of thing that makes me proud of the project.
 What I am questioning is your assertion that policy that wasn't verified
 let alone enforced was somehow key to it.

We did not ever go all the way to verifying that everything in the archive
would work with a shell that exactly conforms to the minimum Policy
requirements and no more.  The farthest we ever got was to get dash to
work.  I thought that's what you were getting at when talking about
testing, and I was agreeing with you that we didn't test against non-dash
shells.  What I don't agree with is the idea that Policy should therefore
just say that scripts have to support dash, rather than saying what it
says now.  (I thought you were arguing for that; maybe I'm wrong.)

 But to use this example again, you are saying the agreeing that
 #!/bin/sh scripts shall be POSIX shell scripts, and then largely
 ignoring it for 10 years because it is unverifiable was helpful to the
 project?  I don't see how it saved anyone any time.

I don't think it *was* particularly helpful prior to when we did the work
of switching to dash.  But during that effort, Policy was quite helpful in
communicating what script writers could rely upon, and in driving
specifications of other tools, like checkbashisms.  And providing sanction
to mass bug filings.

Now that dash works with the archive, this

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-02 Thread Thorsten Glaser

On Wed, 1 Oct 2014, Russell Stuart wrote:

 The only reason I ported things to dash is /bin/sh is now linked to it,
 which in view makes it the standard shell.  Every script starting with
 #!/bin/sh must work with.  If I can't get it working because of a

This is wrong. Every script starting with #!/bin/sh must work with a
POSIX shell that supports “local” and “echo -n” (Policy §10.4). There
are currently three implementations of that in Debian, maybe four
(posh). Do not port to dash. dash has bugs, and I’ve personally found
a dashism in Ubuntu’s checkroot.sh (by running hardy with mksh as
/bin/sh). Also, dash supports more nōn-standard things than, say,
posh (or heirloom-sh, which lacks “local” though).

bye,
//mirabilos
-- 
[16:04:33] bkix: veni vidi violini
[16:04:45] bkix: ich kam, sah und vergeigte...


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/alpine.deb.2.11.1410021146470.3...@tglase.lan.tarent.de

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-02 Thread shawn wilson

On Sep 30, 2014 7:59 PM, Russell Stuart russell-deb...@stuart.id.au
wrote:

 On Tue, 2014-09-30 at 13:08 +0200, Thorsten Glaser wrote:
  You really really should be looking at replacing any
  ash variant with mksh. It’s not that much bigger (at
  least if you add -DMKSH_SMALL to CPPFLAGS and build
  with klibc or dietlibc or so), but much saner.

 I am not a fan of any particular variant of *sh.  They are all horrible
 computer languages.  Nothing over a couple of lines should be written in
 them, as they are idiosyncratic, error prone and basic software
 engineering processes (like units tests) difficult.


I'm not a big fan of bash scripts either but that's because I don't know
bash as well as say perl. Bash /can/ look elegant and not need to use sed,
ask, grep, and cut every other line.

I hate the idea of dash. It's not more secure (see vmware cve for an
example) and I think it was more of an accident than anything else this
didn't hit dash too.

I use zsh as a user but keep everyone else bash by default as people are
more likely to test scripts with bash (in Linux) or sh (on anything else).
If redhst, suse, and oracle all moved their default system shell to dash,
I'd switch but for now, debian is an outlier here. And any other ash
variant would be tested less than even dash so there's no way in hell I'm
going there to run main system processes.

Re: bash exorcism experiment ('bug' 762923 763012)

shawn wilson ag4ve...@gmail.com writes:

 I hate the idea of dash. It's not more secure (see vmware cve for an
 example) and I think it was more of an accident than anything else this
 didn't hit dash too.

The fact that this specific problem didn't hit dash certainly isn't an
accident.  The exploited functionality simply doesn't exist in dash.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87fvf6xz9e@hope.eyrie.org

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-02 Thread shawn wilson

On Thu, Oct 2, 2014 at 11:33 AM, Russ Allbery r...@debian.org wrote:
 shawn wilson ag4ve...@gmail.com writes:

 I hate the idea of dash. It's not more secure (see vmware cve for an
 example) and I think it was more of an accident than anything else this
 didn't hit dash too.

 The fact that this specific problem didn't hit dash certainly isn't an
 accident.  The exploited functionality simply doesn't exist in dash.


I'm pretty sure dash never got a rewrite? So this just happened to be
a feature that got ripped out of dash. I'm not sure why it got
ripped out, but I'm pretty certain it wasn't because the devs saw a
security issue here (I should go looking to see if there's a public
repo and see if I can find where the feature was removed and why).

Now, if you're right and this was removed in dash because of a
security concern, that'd be more interesting than my theory that they
just got lucky.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/cah_obifoqpt1bjmdtphbfyapgksvpiltdhqcljhk1u3hm-f...@mail.gmail.com

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-02 Thread Simon McVittie

On 02/10/14 17:30, shawn wilson wrote:
 I'm pretty sure dash never got a rewrite? So this just happened to be
 a feature that got ripped out of dash.

You seem to be under the impression that dash is some sort of fork or
derivative of bash. It isn't; I don't think they even have a common
ancestor.

POSIX sh is a specification for how Unix-like shells should behave,
based on the language interpreted by the 1977 Bourne shell (sh). Debian
Policy requires /bin/sh to be a POSIX sh with at least a couple of
specified additional features (local is one of those features), and
optionally, other features beyond those. The default implementation was
originally bash, and was changed to dash in recent releases.

dash is an implementation of POSIX sh, derived from the Almquist shell
(ash) taken from NetBSD. As far as I know, ash was an independent
implementation (i.e. rewrite) of a POSIX sh. It has a small number of
non-POSIX features, including those required by Debian Policy.

bash is GNU's implementation (i.e. another rewrite) of the Bourne shell,
hence its name Bourne Again SHell. It has lots of non-POSIX features,
making it a considerably better interactive shell than dash, and more
capable for scripting. One of its non-POSIX features is the ability to
export functions, which is the feature being abused in this vulnerability.

 I'm not sure why it got ripped out

I don't think dash ever had this feature to begin with, so there was
nothing to rip out.

S


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/542d8629.8000...@debian.org

Re: bash exorcism experiment ('bug' 762923 763012)

shawn wilson ag4ve...@gmail.com writes:
 On Thu, Oct 2, 2014 at 11:33 AM, Russ Allbery r...@debian.org wrote:
 shawn wilson ag4ve...@gmail.com writes:

 I hate the idea of dash. It's not more secure (see vmware cve for an
 example) and I think it was more of an accident than anything else
 this didn't hit dash too.

 The fact that this specific problem didn't hit dash certainly isn't an
 accident.  The exploited functionality simply doesn't exist in dash.

 I'm pretty sure dash never got a rewrite? So this just happened to be
 a feature that got ripped out of dash.

The feature of exporting functions into the environment and importing them
from the environment has never been implemented in dash or ash so far as I
know.  I don't believe it's been implemented in anything except bash.
It's a bash-specific feature.

That's always been the point in favor of dash: it simply is smaller, has
fewer features, and tries to do much less.  That makes it a weaker
interactive shell for obvious reasons, but that's why it's faster and also
why it is less likely to have security vulnerabilities of this kind.  You
can't have implementation-based security vulnerabilities in code that
doesn't exist for features that aren't implemented.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/8761g2xunz@hope.eyrie.org

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-02 Thread shawn wilson

Ok then, I stand (doubly) corrected. Thanks

On Thu, Oct 2, 2014 at 1:06 PM, Simon McVittie s...@debian.org wrote:
On 02/10/14 17:30, shawn wilson wrote:
I'm pretty sure dash never got a rewrite? So this just happened to be
a feature that got ripped out of dash.

You seem to be under the impression that dash is some sort of fork or
derivative of bash. It isn't; I don't think they even have a common
ancestor.

POSIX sh is a specification for how Unix-like shells should behave,
based on the language interpreted by the 1977 Bourne shell (sh). Debian
Policy requires /bin/sh to be a POSIX sh with at least a couple of
specified additional features (local is one of those features), and
optionally, other features beyond those. The default implementation was
originally bash, and was changed to dash in recent releases.

dash is an implementation of POSIX sh, derived from the Almquist shell
(ash) taken from NetBSD. As far as I know, ash was an independent
implementation (i.e. rewrite) of a POSIX sh. It has a small number of
non-POSIX features, including those required by Debian Policy.

bash is GNU's implementation (i.e. another rewrite) of the Bourne shell,
hence its name Bourne Again SHell. It has lots of non-POSIX features,
making it a considerably better interactive shell than dash, and more
capable for scripting. One of its non-POSIX features is the ability to
export functions, which is the feature being abused in this vulnerability.

I'm not sure why it got ripped out

I don't think dash ever had this feature to begin with, so there was
nothing to rip out.

--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/542d8629.8000...@debian.org

--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
https://lists.debian.org/CAH_OBifeVy4YdYmi=vBpWgoH4co7WRWshynS24Y5Yt=dcym...@mail.gmail.com

Re: bash exorcism experiment ('bug' 762923 763012)

On Thu, 2014-10-02 at 11:48 +0200, Thorsten Glaser wrote:
 This is wrong. Every script starting with #!/bin/sh must work with a
 POSIX shell that supports “local” and “echo -n” (Policy §10.4).

Solid, working software is hard enough to produce.  A policy requiring
something you can't test for makes it near impossible.

IMO, if Debian has decided the in the default case /bin/sh == dash,
then the policy should say #!/bin/sh scripts must work with dash.  It
then becomes trivial for Developers to test their code conforms with
policy.  If we allow /bin/sh to be linked to other shells, policy should
say those shells must implement all the features /bin/dash implements so
that any script that works with dash must also work with them.

As is stands, the one thing you can guarantee we will get from our
policy saying #!/bin/sh scripts work with a shell that does not exist
and can't be tested against is scripts that have never been tested
against that policy.

If Debian really wants to implement the policy as described, then it
should do the work required to produce robust software that conforms
with it.  In this case that would mean producing a shell that behaves as
described, which we make /bin/sh by default.  Perhaps a flag to dash
stripping all of the features not described in SUSv3 features would
suffice.


signature.asc
Description: This is a digitally signed message part

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-02 Thread brian m. carlson

On Fri, Oct 03, 2014 at 09:39:29AM +1000, Russell Stuart wrote:
 IMO, if Debian has decided the in the default case /bin/sh == dash,
 then the policy should say #!/bin/sh scripts must work with dash.  It
 then becomes trivial for Developers to test their code conforms with
 policy.  If we allow /bin/sh to be linked to other shells, policy should
 say those shells must implement all the features /bin/dash implements so
 that any script that works with dash must also work with them.

This isn't possible, as there are some corner cases in which dash and
bash work differently, and those are both explicitly supported choices
for /bin/sh.  These areas are in extensions that are not part of POSIX
or the extended set of features required by Policy.

 As is stands, the one thing you can guarantee we will get from our
 policy saying #!/bin/sh scripts work with a shell that does not exist
 and can't be tested against is scripts that have never been tested
 against that policy.

The set of features that Debian's /bin/sh must support is fairly
limited.  It's not very difficult at all to implement a shell script
appropriately.  Every shell script I write these days will work in bash,
dash, and posh, whether I'm writing it to run on a Debian system or not.

 If Debian really wants to implement the policy as described, then it
 should do the work required to produce robust software that conforms
 with it.  In this case that would mean producing a shell that behaves as
 described, which we make /bin/sh by default.  Perhaps a flag to dash
 stripping all of the features not described in SUSv3 features would
 suffice.

The shell you're describing is posh.  It implements exactly those
features, and nothing more.  Unfortunately, some developers have
outright refused to make their software using /bin/sh work with posh,
even when provided with a patch (e.g. #309415), to the point that last
time I tried to use posh as /bin/sh, the system wouldn't boot.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187


signature.asc
Description: Digital signature

Re: bash exorcism experiment ('bug' 762923 763012)

On Fri, 2014-10-03 at 00:04 +, brian m. carlson wrote:
 The shell you're describing is posh.  It implements exactly those
 features, and nothing more.

You've got me to look at posh.  Thanks for that.

So we do have a shell that developers can use to test their scripts
match Debian policy.

 Unfortunately, some developers have outright refused to make their
 software using /bin/sh work with posh, even when provided with a patch
 (e.g. #309415), to the point that last time I tried to use posh
 as /bin/sh, the system wouldn't boot.

If I understand what you are saying correctly this means policy 10.4 is
mostly ignored.  Worse, if you try to configure a system that conforms
closely to that policy it doesn't boot.  This is the sort of thing that
that makes open source programmers look like rank amateurs.

If we want Debian policy to reflect reality and make it easy for
developers to test their scripts conform to policy, then it should say
#!/bin/sh scripts must work with dash.

As an aside, I'm not sure why the preference for posh over dash, given:

$ size $(which posh)
   textdata bss dec hex filename
 12326048682920  131048   1ffe8 /bin/posh
$ size $(which dash)
   textdata bss dec hex filename
 1083765192   11240  124808   1e788 /bin/dash



signature.asc
Description: This is a digitally signed message part

Re: bash exorcism experiment ('bug' 762923 763012)

Russell Stuart r...@debian.org writes:

 IMO, if Debian has decided the in the default case /bin/sh == dash,
 then the policy should say #!/bin/sh scripts must work with dash.  It
 then becomes trivial for Developers to test their code conforms with
 policy.

Up until dash changes, and then you have absolutely no idea what to do
with that sort of policy.  There's a reason why no standards document I've
ever seen says something like this.  The ISO C standard isn't going to say
that anything that compiles with gcc is valid code.

Policy already says that testing the script with dash is a good indicator
that it may be okay, and at least is free of bashisms, which is about as
strong of a statement that you can make.

 If we allow /bin/sh to be linked to other shells, policy should say
 those shells must implement all the features /bin/dash implements so
 that any script that works with dash must also work with them.

Ugh, no.  That's a possibly unbounded future set.

Instead, we started with POSIX and then added the absolute minimum of
features on top of what POSIX required that we decided we really could not
live without.  I think that's a reasonable approach, and it means that
multiple shell authors can make their shells follow the Policy
requirements and work properly with scripts in Debian.  And script authors
in Debian can check the POSIX documentation plus the list of features in
Policy and have a pretty solid idea of what is allowed.

And we *do* have checkbashisms as a way to test.  It's not perfect, but
it's not nothing.  You can also test with posh.

 If Debian really wants to implement the policy as described, then it
 should do the work required to produce robust software that conforms
 with it.  In this case that would mean producing a shell that behaves as
 described, which we make /bin/sh by default.

Being pedantic is not the only, or even the main, goal of our /bin/sh.
Performance, for example, is also quite important.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/878ukyufnf@hope.eyrie.org

Re: bash exorcism experiment ('bug' 762923 763012)

2014-10-02 Thread Henrique de Moraes Holschuh

On Fri, 03 Oct 2014, Russell Stuart wrote:
 On Fri, 2014-10-03 at 00:04 +, brian m. carlson wrote:
  The shell you're describing is posh.  It implements exactly those
  features, and nothing more.
 
 You've got me to look at posh.  Thanks for that.
 
 So we do have a shell that developers can use to test their scripts
 match Debian policy.

posh is useful to test if a script restricts itself to POSIX features.

Debian policy mandates that /bin/sh implement a _superset_ of POSIX, which
is out of scope for posh.

-- 
  One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie. -- The Silicon Valley Tarot
  Henrique Holschuh


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141003015055.ga4...@khazad-dum.debian.net

Re: bash exorcism experiment ('bug' 762923 763012)

Henrique de Moraes Holschuh h...@debian.org writes:
 On Fri, 03 Oct 2014, Russell Stuart wrote:

 You've got me to look at posh.  Thanks for that.

 So we do have a shell that developers can use to test their scripts
 match Debian policy.

 posh is useful to test if a script restricts itself to POSIX features.

 Debian policy mandates that /bin/sh implement a _superset_ of POSIX,
 which is out of scope for posh.

The p in posh stands for Policy, not POSIX.  See the package
description.

I think it would be really cool to do some sort of continuous integration
testing on a system with posh as /bin/sh.  We don't have a particularly
great story around integration testing right now, but even with what we've
got it might still be interesting.  Of course, that relies on maintainers
caring, which is often the hard part.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87y4sxuc3m@hope.eyrie.org

Re: bash exorcism experiment ('bug' 762923 763012)

On Thu, 2014-10-02 at 22:50 -0300, Henrique de Moraes Holschuh wrote:
 Debian policy mandates that /bin/sh implement a _superset_ of POSIX, which
 is out of scope for posh.

Regardless, posh implements all the additional features mandated by
10.4:

  echo -n, if implemented as a shell built-in, must not generate a
  newline.

  $ posh -c echo -n x
  x$

  test, if implemented as a shell built-in, must support -a and -o as
  binary logical operators.

  $ posh -c test -z '' -a -z ''  echo x; test -z '' -a -n ''  echo y;
  x
  $

  $ posh -c test -n '' -o -n ''  echo x; test -n '' -o -z ''  echo y;
  y
  $

  local to create a scoped variable must be supported, ...

  $ posh -c 'a=a; x() { local a b c=C; a=A; echo $a$c; }; x; echo $a$c'
  AC
  a

   The XSI extension to kill allowing kill -signal, where signal is
   either the name of a signal or one of the numeric signals ...

  $ posh -c 'kill -KILL $$'
  Killed
  $   

   The XSI extension to trap allowing numeric signals must be
   supported. In addition to the signal numbers listed in the
   extension, which are the same as for kill above, 13 (SIGPIPE) must
   be allowed.

$ posh -c 'trap echo sigterm TERM; kill -TERM $$'
sigterm
$



signature.asc
Description: This is a digitally signed message part

Re: bash exorcism experiment ('bug' 762923 763012)

On Thu, 2014-10-02 at 18:05 -0700, Russ Allbery wrote:
 Up until dash changes, and then you have absolutely no idea what to do
 with that sort of policy.  There's a reason why no standards document I've
 ever seen says something like this.  The ISO C standard isn't going to say
 that anything that compiles with gcc is valid code.

Correct.  But we aren't we aren't ISO C, or POSIX and we aren't in the
standards business.  We are in the business of producing working
reliable systems.  Yes we do use standards to do that, we even write a
few of our own.  But unlike ISO and POSIX they are for our internal
consumption.  The thing we create and release to others is the archives,
and it would be a long stretch to call them a standard.

Onto your then you have absolutely no idea what to do with that sort of
policy comment.  

Why have policy 10.4 at all?  Presumably to make producing a reliable,
robust Debian easier.  But 10.4 doesn't proscribe a standard way of
doing that or checking it has been done.  Unsurprisingly adherence is at
best sporadic.

So now to the answer to your then you have absolutely no idea what to
do with that sort of policy comment.  This is what you do.  You, as a
developer create unit tests the scripts using /bin/posh (or whatever
shell implements the policy), and if you do your job well you deliver a
system that works reliably under a prescribed set of conditions (ie posh
is installed as /bin/sh). You have a reasonable chance of this remaining
so because the hopefully the unit tests are run every time the package
is build.  The standard may not be well enough defined for your tastes,
but in my world repeatable and reliable take a far higher precedence.

I can also tell you how what to do with that sort of policy applies
the current policy.  What is done is we have the occasional spat about
bash'ism and discussion on shell syntax.  Having bashism's (or not) is
NOT the same as working.  Yet, that's about the best we can do.  Thus
the policy is not robustly and objectively enforced (and worse, can not
be).  brian's observation should come as no surprise: when you configure
Debian with /bin/sh conforming strictly to 10.4, Debian doesn't boot.
Surely you aren't going to say this is an example of policy working
well?

AFAICT defining posh as the shell all #!/bin/sh scripts must work with
is better in every way, and where ever possible all Debian policy should
be like that.  Which is to say it should be obvious what you have to do
to comply, it should be automatically verifiable you have complied, and
if you comply it should help in ensuring Debian is robust and reliable.
The current policy 10.4 fails the first two tests.  Since could be
re-worded so it doesn't, there is no doubt in my mind it could be
better.

This isn't to say we should delete the statements on what we expect of
the shell.  They are excellent good documentation.  I guess in summing
up, my main point is good documentation doesn't necessarily make good
policy.


signature.asc
Description: This is a digitally signed message part

Re: bash exorcism experiment ('bug' 762923 763012)