Re: pipes and sub-shells

[Thorsten Glaser]

Jean Delvare dixit:

>On mer., 2017-03-22 at 10:12 +0100, Jean Delvare wrote:
>> One of our customers recently asked us about the different behavior of
>> constructs involving pipes in ksh scripts. I explained to them that
>> mksh spawns a sub-shell for the right hand side of each pipe, and
>> therefore whatever happens on that side of the pipe has no effect on
>> the main shell.
>
>I have a hard time convincing the customer that the behavior is as
>expected. They quote the manual page:
>
>"A function can be made to finish immediately using the return
>command;"
>
>Which indeed isn't what's happening.

Of course it isn’t because the return command is not called from
within a function but from within a subshell which, while contained
in a function, is not “in a function”.

This is no different to any non-last part of a pipeline in such
shells — calling return in anything BEFORE a pipe would also not
“work” as requested.

>The read loop FAQ (about pipes spawning subshells) mentions the
>inability to propagate variable changes back to the original shell, but
>says nothing about the altered effect of "return" or "exit".

It has not come up as a problem so far.

>May I suggest that the manual page gets updated in the following ways:

Yes, thank you.

>I can prepare and send a patch doing (some of) these changes if you
>agree with them.
>
>Maybe it would also make sense to add a specific paragraph or section
>somewhere about subshells. In various places, the manual page explains

Feel free to send such a patch.

>that a given construct will (or will not) spawn a subshell, but there
>is no central place explaining what a subshell is nor the limitations
>that it creates. You probably know more than I do on the topic. I know
>about the variable propagation (or more generally the separate
>environments), return and exit, but maybe there is more?

A subshell involves fork(), it’s as easy as that.

Re: pipes and sub-shells

[Dr. Werner Fink]

On Wed, Mar 22, 2017 at 07:39:21PM +, Thorsten Glaser wrote:
> 
> Dr. Werner Fink dixit:
> 
> >Just to be mentioned, I can mksh change in such a way that without job 
> >control
> >I can do
> >
> >   ~/rpmbuild/BUILD/mksh> ./mksh -c 'echo xxx | read yyy; echo $yyy'
> >   xxx
> >
> >that is it does work similar to the lastpipe shell option of the bash
> >I had initiated now 7 years back. The trick is to avouid forking the
> >last element in a pipe chain.
> 
> I’ve thought about this… years ago, too. But seeing as POSIX allows
> either behaviour, I decided that this (running all sides of pipelines
> in subshells) is a language feature now, for simplicity of user scripts
> (as they can just use co-processes when they don’t want it).
> 
> So, thanks but no. (I’ve got scripts that actually rely on that now.)

OK .. nevertheless, only for the records, the change might look
similar to that in the attachment I guess.

Werner

-- 
  "Having a smoking section in a restaurant is like having
  a peeing section in a swimming pool." -- Edward Burr
--- mksh/exec.c
+++ mksh/exec.c	2017-03-27 10:33:11.0 +
@@ -156,6 +156,8 @@ execute(struct op * volatile t,
 			}
 		}
 
+	flags &= ~XPIPELS;
+
 	switch (t->type) {
 	case TCOM:
 		rv = comexec(t, tp, (const char **)ap, flags, xerrok);
@@ -192,8 +194,23 @@ execute(struct op * volatile t,
 		restfd(1, e->savefd[1]);
 		/* no need to re-restore this */
 		e->savefd[1] = 0;
-		/* Let exchild() close 0 in parent, after fork, before wait */
-		i = exchild(t, flags | XPCLOSE | XPIPEST, xerrok, 0);
+
+		tp = NULL;
+		if (Flag(FLASTPIPE) && !Flag(FPIPEFAIL) && t->args) {
+			up = eval(t->args, t->u.evalflags | DOBLANK | DOGLOB | DOTILDE);
+			ap = (const char **)up;
+			if (ap[0])
+tp = findcom(ap[0], FC_BI|FC_FUNC);
+		}
+
+		if (tp && tp->type == CSHELL) {
+			flags &= ~XFORK;
+			flags |= XPIPELS;
+			i = execute(t, flags | XPIPELS | XERROK, xerrok);
+		} else
+			/* Let exchild() close 0 in parent, after fork, before wait */
+			i = exchild(t, flags | XPCLOSE | XPIPEST, xerrok, 0);
+
 		if (!(flags) && !(flags))
 			rv = i;
 		break;
@@ -453,6 +470,12 @@ execute(struct op * volatile t,
 
 	/* restores IO */
 	quitenv(NULL);
+
+#ifndef MKSH_UNEMPLOYED
+	if (flags & XPIPELS)
+		j_restore(flags);
+#endif
+
 	if ((flags))
 		/* exit child */
 		unwind(LEXIT);
--- mksh/jobs.c
+++ mksh/jobs.c	2017-03-27 10:34:26.0 +
@@ -641,6 +641,23 @@ exchild(struct op *t, int flags,
 	return (rv);
 }
 
+#ifndef MKSH_UNEMPLOYED
+void
+j_restore(int flags)
+{
+	if (!Flag(FMONITOR) || (flags))
+		return;
+	/* job control set up */
+
+	if (!ttypgrp_ok)
+		return;
+
+	if (kshpgrp == kshpid && kshpgrp != tcgetpgrp(tty_fd))
+		if (tcsetpgrp(tty_fd, kshpid) < 0)
+			ttypgrp_ok = false;
+}
+#endif
+
 /* start the last job: only used for $(command) jobs */
 void
 startlast(void)
--- mksh/main.c
+++ mksh/main.c	2017-03-27 10:45:27.0 +
@@ -250,6 +250,15 @@ main_init(int argc, const char *argv[],
 	 */
 	Flag(FBRACEEXPAND) = 1;
 
+#if 1
+	/*
+	 * Turn on lastpipe by default ??
+	 * If set the shell runs the last command of a pipeline  not
+	 * executed in the background in the current shell environment.
+	 */
+	Flag(FLASTPIPE) = 1;
+#endif
+
 	/*
 	 * Turn on "set -x" inheritance by default.
 	 */
--- mksh/mksh.1
+++ mksh/mksh.1	2017-03-27 10:58:39.000 +
@@ -4291,6 +4291,9 @@ Do not reset
 .Fl o Ic xtrace
 upon entering functions.
 This is enabled by default.
+.It Fl o Ic lastpipe
+If set the shell runs the last command of a pipeline not executed in the
+background in the current shell environment.
 .It Fl o Ic nohup
 Do not kill running jobs with a
 .Dv SIGHUP
--- mksh/sh.h
+++ mksh/sh.h	2017-03-27 10:47:08.0 +
@@ -1653,6 +1653,7 @@ struct op {
 #define TTIME		20	/* time pipeline */
 #define TEXEC		21	/* fork/exec eval'd TCOM */
 #define TCOPROC		22	/* coprocess |& */
+#define TLPIPE		23	/* last in pipe chain  */
 
 /*
  * prefix codes for words in command tree
@@ -1714,6 +1715,7 @@ struct ioword {
 #define XCOPROC BIT(9)		/* starting a co-process */
 #define XTIME	BIT(10)		/* timing TCOM command */
 #define XPIPEST	BIT(11)		/* want PIPESTATUS */
+#define XPIPELS	BIT(12)		/* Last element in pipe chain */
 
 /*
  * flags to control expansion of words (assumed by t->evalflags to fit
@@ -2142,6 +2144,7 @@ void j_init(void);
 void j_exit(void);
 #ifndef MKSH_UNEMPLOYED
 void j_change(void);
+void j_restore(int);
 #endif
 int exchild(struct op *, int, volatile int *, int);
 void startlast(void);
--- mksh/sh_flags.opt
+++ mksh/sh_flags.opt	2017-03-27 10:40:06.0 +
@@ -79,6 +79,10 @@ FN("interactive", FTALKING, OF_CMDLINE
 >k|
 FN("keyword", FKEYWORD, OF_ANY
 
+/* ./.	if set the shell runs the last command of a pipeline not executed in the background in the current shell environment */
+>|
+FN("lastpipe", FLASTPIPE, OF_ANY
+
 /* -l	login shell */
 >l|!SHFLAGS_NOT_CMD
 FN("login", FLOGIN, OF_CMDLINE


signature.asc
Description: PGP 

Re: POSIX character classes (was Re: pipes and sub-shells)

[Martijn Dekker]

Op 23-03-17 om 22:02 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> * BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
>> [expressions] are not supported in glob patterns.
> 
> I really really REALLY hate that this will make mksh really big.
> We’re talking about 36K .rodata even without titlecase conversion
> and BMP-only (16-bit Unicode) here.

I sympathise.

Even fnmatch(3) is not compliant on all systems; the BSDs don't seem to
have caught up yet. :(  I don't suppose using that is an option in any
case because of mksh's extended globbing functionality.

Is adding 36k really that much in 2017? On my system, the current
development binary of mksh is 283k after stripping when built with -O2,
235k with -Os. Adding 36k would make it 316k/271k, still quite small.

If that's too much, I guess you should continue to not support them. The
reason modernish detects BUG_NOCHCLASS is not to make some sort of
statement, but to enable programs using the library to easily check for
the presence of the issue and implement alternative methods (such as
falling back to external commands, or just matching ASCII only without
character classes).

> Can I get by making them match ASCII only even in UTF-8 mode?

IMHO, that would defeat their primary purpose, namely locale-dependent
class matching, so no, not really. :)

If Greeks or Russians (or Germans, for that matter) can't count on
[:upper:] matching an upper case letter in their alphabets, then I'd say
for them it would be better to have no support than broken support.

> Strictly speaking, POSIX requires only support for the C locale,
[...]

Yes, but on systems supporting other locales (e.g. UTF-8), it would not
be conforming for character classes to match ASCII only. You either
support UTF-8 or you don't.

- M.

POSIX character classes (was Re: pipes and sub-shells)

[Thorsten Glaser]

Martijn Dekker dixit:

>* BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
>[expressions] are not supported in glob patterns.

I really really REALLY hate that this will make mksh really big.
We’re talking about 36K .rodata even without titlecase conversion
and BMP-only (16-bit Unicode) here.

Can I get by making them match ASCII only even in UTF-8 mode?

Strictly speaking, POSIX requires only support for the C locale,
and our UTF-8 mode is only close to POSIX anyway, and currently
(though this will change, there have been good points made for
locale tracking) enabled using a mksh-specific set flag.

If I implemented that, we could then say that “lksh -o posix”
is, in the C locale, fully POSIX conformant. (Perhaps — but
certainly a goal to work for, even despite the uselessness
of standards.)

bye,
//mirabilos
-- 
(gnutls can also be used, but if you are compiling lynx for your own use,
there is no reason to consider using that package)
-- Thomas E. Dickey on the Lynx mailing list, about OpenSSL

Re: pipes and sub-shells

[Martijn Dekker]

Op 23-03-17 om 10:49 schreef Jean Delvare:
> Apparently it requires a more recent version of mksh than we are
> shipping:
> 
> $ echo $KSH_VERSION
> @(#)MIRBSD KSH R50 2014/06/29 openSUSE

That version is quite ancient, so you should consider upgrading it to
the latest. FYI, modernish 
currently detects the following bugs on it that are relevant for
cross-shell programming. All except BUG_LNNOALIAS, BUG_LNNOEVAL and
BUG_NOCHCLASS have been fixed in the current release. The former two are
fixed in current cvs. The latter is a design decision from Thorsten that
is nonetheless a bug in POSIX terms.

* BUG_CMDPV: 'command -pv' does not find builtins.

* BUG_CMDSPEXIT: preceding a special builtin with 'command' does not
stop it from exiting the shell if the builtin encounters an error.

* BUG_CMDVRESV: 'command -v' does not find reserved words such as "if".

* BUG_LNNOALIAS: $LINENO is always expanded to 0 when used within an alias.

* BUG_LNNOEVAL: $LINENO is always expanded to 0 when used in 'eval'.

* BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
[expressions] are not supported in glob patterns.

* BUG_PP_01: POSIX says that empty "$@" generates zero fields but empty
'' or "" or "$emptyvariable" generates one empty field. This means
concatenating "$@" with one or more other, separately quoted, empty
strings (like "$@""$emptyvariable") should still produce one empty
field. With this bug, this erroneously produces zero fields.

* BUG_PP_02: Like BUG_PP_01, but with unquoted $@ and only with
"$emptyvariable"$@, not $@"$emptyvariable".

* BUG_PP_03: Assigning the positional parameters to a variable using
either var=$* or var="$*" or both doesn't work as expected, using either
default, empty, unset or custom settings of $IFS.

* BUG_PP_04: Like BUG_PP_03, but for a default assignment within a
parameter substitution, i.e. ${var=$*} or ${var="$*"}.

* BUG_SELECTRPL: In a 'select' loop, input that is not a menu item is
not stored in the REPLY variable as it should be.

* BUG_TESTERR0: test/[ exits successfully (exit status 0) if an invalid
argument is given to an operator.

Hope this helps,

- M.

Re: pipes and sub-shells

[Jean Delvare]

Hi Martijn,

On mer., 2017-03-22 at 10:26 +0100, Martijn Dekker wrote:
> Op 22-03-17 om 10:12 schreef Jean Delvare:
> > 
> > Concretely, the customer's code looks like this:
> > 
> > command | while read line
> > do
> > if 
> > then
> > exit
> > fi
> > process $line
> > done
> [...]
> > 
> > I was wondering if there is any other trick you can suggest that would work 
> > in mksh?
> 
> A here-string with a command substitution:
> 
> while read line
> do
>   if 
>   then
>   exit
>   fi
>   process $line
> done <<<$(command)

Apparently it requires a more recent version of mksh than we are
shipping:

$ echo $KSH_VERSION
@(#)MIRBSD KSH R50 2014/06/29 openSUSE
$ ./test_return.sh
./test_return.sh[10]: syntax error: '(' unexpected

But indeed works find with the latest upstream version, thanks for the
pointer.

Thanks,
-- 
Jean Delvare
SUSE L3 Support

Re: pipes and sub-shells

[Thorsten Glaser]

Martijn Dekker dixit:

>Op 22-03-17 om 10:12 schreef Jean Delvare:
>> Concretely, the customer's code looks like this:
>>
>> command | while read line
>> do
>>  if 
>>  then
>>  exit
>>  fi
>>  process $line
>> done
>[...]
>> I was wondering if there is any other trick you can suggest that would work 
>> in mksh?
>
>A here-string with a command substitution:

That works but if not very performant.

This is actually a FAQ ;-) Use co-processes!

command |& while read -p line; do …


Dr. Werner Fink dixit:

>Just to be mentioned, I can mksh change in such a way that without job control
>I can do
>
>   ~/rpmbuild/BUILD/mksh> ./mksh -c 'echo xxx | read yyy; echo $yyy'
>   xxx
>
>that is it does work similar to the lastpipe shell option of the bash
>I had initiated now 7 years back. The trick is to avouid forking the
>last element in a pipe chain.

I’ve thought about this… years ago, too. But seeing as POSIX allows
either behaviour, I decided that this (running all sides of pipelines
in subshells) is a language feature now, for simplicity of user scripts
(as they can just use co-processes when they don’t want it).

So, thanks but no. (I’ve got scripts that actually rely on that now.)

bye,
//mirabilos
-- 
Solange man keine schmutzigen Tricks macht, und ich meine *wirklich*
schmutzige Tricks, wie bei einer doppelt verketteten Liste beide
Pointer XORen und in nur einem Word speichern, funktioniert Boehm ganz
hervorragend.   -- Andreas Bogk über boehm-gc in d.a.s.r

Re: pipes and sub-shells

[Martijn Dekker]

Op 22-03-17 om 10:12 schreef Jean Delvare:
> Concretely, the customer's code looks like this:
> 
> command | while read line
> do
>   if 
>   then
>   exit
>   fi
>   process $line
> done
[...]
> I was wondering if there is any other trick you can suggest that would work 
> in mksh?

A here-string with a command substitution:

while read line
do
if 
then
exit
fi
process $line
done <<<$(command)

Hope this helps,

- M.