Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
Scraping through this - thanks for the lessons aka explanations. On 18/03/2021 16:08, Chet Ramey wrote: On 3/18/21 5:53 AM, Michael Felt wrote: Yes, something to test. Thx. The ojdk scenario is: /usr/bin/printf > >(tee -a stdout.log) 2> >(tee -a stderr.log). So, yes, in this case it is working because printf is the parent - (which I never seemed to find actually calling open() of the file. It seems to be using the fd opened by the child - in a magical way). It's the redirection. The shell does the open, since the filename resulting from process substitution is the target of a redirection operator. This is a common idiom -- so common, in fact, that people have interpreted it to mean that the entire `> >(xxx)' is a single operator. However, the shell expands redirections in the child process it forks to exec printf, so that child shell is what does the process substitution. That might be the problem here. The command itself doesn't do anything, though. `tee' just sits there waiting for data to write to log files. It has no purpose. I'm not sure what the intent is. If you wrapped that command into a script, it's unlikely that either `tee' would exit (why would they?) before `printf' exits and the script completes. In bash-5.0, there would be nothing to remove the FIFOs. If I understand correctly, the commands are generated by gmake as it processes targets. This is defined to provide `diff' with two arguments. Let's call them /var/tmp/sh-np12345 and /var/tmp/sh-np67890 So diff runs, sees two arguments, opens both files, and does its thing. Diff has to see two filenames when it runs, otherwise it's an error. But what I thoght I was seeing is that diff is the PARENT calling substitute_process() that create(s) a child process that reads/writes to a fifo file. Yes and no. Process substitution is a word expansion that results in a filename. The stuff between the parens defines a command that writes to or reads from a pipe expressed as a filename (/dev/fd/NN or a FIFO) that is the result of the word expansion. In this case, the process substitution is the target of a redirection, so the shell performs that word expansion before it execs diff. a) the child process never returns - it `exits` via, iirc, sh_exit(result) and the end of the routine It executes the specified command and exits. Got it: must remember - initially it is bash busy with word expansion (a new bash child for each 'process substitution' b) the parent gets the filename (pathname) - but I never see it actually opening it - only (when using bash -x) seeing the name in the -x command expansion. It doesn't have to. The filename itself is the expansion: it's an object you can use to communicate with an asynchronous process. This is how you can have programs that expect a filename use program-generated output, for instance, without using a temp file. Yes - for me at least it is much easier to fathom as input - that ends and behaves/looks/feels like EOF. In this case, it opens the FIFO because it is the target of a redirection. Nods: just as it would if it was the output from a program than had just run 'moments' before. Now, let's say your change is there. The shell still runs diff /var/tmp/sh-np12345 /var/tmp/sh-np67890 but, depending on how processes get scheduled, the shell forked to run the process substitutions has already unlinked those FIFOs. Diff will error out. The user will be unhappy. I will get bug reports. You have introduced a race condition. You may not get hit by it, but you cannot guarantee that no one will. No I cannot - and for now it is a `hack` to solve a bigger issue. With 3500 calls in a single build I hope the race occurs - and I'll finally see where the PARENT actually uses the name returned. You mean in terms of using the filename as an argument to a shell builtin? Otherwise you'll have to trace into other child process execution. /usr/bin/printf is not a built-n (afaik) If I understand correctly - from printf perspective we have /usr/bin/printf "Some formatted message" > /tmp/sh-np.123456 2> /tmp/sh-np.9876543 & And, if for ease of discussion we say program1 is PID-123456 and program is PID-987654 - these programs have no way of knowing their stdin is named /tmp/sh-np-something? BING: as the dutch (used to) say - the quarter drops - the other programs (e.g., tee) have no fifo knowledge - they are who/what they are. What maybe needed for this situation - is rather than directly execev() the program - yet another fork (for the execve() - and wait for that program to hit it's EOF on input and then the sleeping 'word expansion child' cleans up the fifo file it created for the communication path. * Am I getting closer? :) The shell can't unlink the FIFO until it can guarantee that the proce
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 17/03/2021 23:12, Chet Ramey wrote: On 3/17/21 3:29 PM, Michael Felt wrote: I tried as many combinations of commands as I could - and it seems that the regular behavior of dup2 on the opened fifo is enough to maintain communication. It's not, since FIFOs exist in the file system and have to be available to open(2) when the other process (consumer, producer) wants to use them. I didn't expect it to be perfect (see thx etc below). Better: I needed help in telling when it would most like fail! :) Going into a system test (ie. a normal AdoptOpemJDK build process) that has nearly 3500 commands, each with two process_substitution commands. Consider the following scenario. You want to perform a regression test of two versions of a program, each of which produces textual output. You run diff <(program-version-1 args) <(program-version-2 args) Yes, something to test. Thx. The ojdk scenario is: /usr/bin/printf > >(tee -a stdout.log) 2> >(tee -a stderr.log). So, yes, in this case it is working because printf is the parent - (which I never seemed to find actually calling open() of the file. It seems to be using the fd opened by the child - in a magical way). This is defined to provide `diff' with two arguments. Let's call them /var/tmp/sh-np12345 and /var/tmp/sh-np67890 So diff runs, sees two arguments, opens both files, and does its thing. Diff has to see two filenames when it runs, otherwise it's an error. But what I thoght I was seeing is that diff is the PARENT calling substitute_process() that create(s) a child process that reads/writes to a fifo file. a) the child process never returns - it `exits` via, iirc, sh_exit(result) and the end of the routine b) the parent gets the filename (pathname) - but I never see it actually opening it - only (when using bash -x) seeing the name in the -x command expansion. Now, let's say your change is there. The shell still runs diff /var/tmp/sh-np12345 /var/tmp/sh-np67890 but, depending on how processes get scheduled, the shell forked to run the process substitutions has already unlinked those FIFOs. Diff will error out. The user will be unhappy. I will get bug reports. You have introduced a race condition. You may not get hit by it, but you cannot guarantee that no one will. No I cannot - and for now it is a `hack` to solve a bigger issue. With 3500 calls in a single build I hope the race occurs - and I'll finally see where the PARENT actually uses the name returned. The shell can't unlink the FIFO until it can guarantee that the processes that need to open it have opened it, and it can't guarantee that in the general case. It has to wait until the process completes, at least, and even that might not be correct. Again, my issue was with >(command) substitution - where the `files` get written to by the parent - rather than reading them. p.s. it is not my call to ask why they do not use regular redirection or pipes. Feels much simpler - but some people cannot miss the opportunity to use something blinky and shiney. p.p.s. - If you have `words of wisdom` re: why this approach is much better than `standard` redirection - I am all ears! *** Thanks again for the time to reply *** That's why the last-ditch approach is to remove all remaining FIFOs when the shell exits. btw: other than the one open in the middle of process_substitution() I did not see anywhere where another process even tries to open the file. They are not necessarily shell processes, but what if they were? Since a FIFO is an object in the file system, you can just open(2) it. That's ostensibly the advantage of FIFOs. what I also noticed is that the process, (iirc) that opens the file - never 'returns' - it ends via sh_exit() and the end of the routine. Of course. It's a process that is forked to run the command specified in the process substitution. What else does it need to do? OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
I tried as many combinations of commands as I could - and it seems that the regular behavior of dup2 on the opened fifo is enough to maintain communication. Going into a system test (ie. a normal AdoptOpemJDK build process) that has nearly 3500 commands, each with two process_substitution commands. Maybe that fails - and then I'll have yet another test. btw: other than the one open in the middle of process_substitution() I did not see anywhere where another process even tries to open the file. what I also noticed is that the process, (iirc) that opens the file - never 'returns' - it ends via sh_exit() and the end of the routine. Next time - I'll save all of my debug changes. Got a bit too rigorous when I cleaned up. On 17/03/2021 19:03, Chet Ramey wrote: On 3/17/21 11:52 AM, Michael Felt wrote: OK - this process on github has not gone exactly as I intended - merged with master - while I wanted to update, ie., merge with branch 5.0.18. So, the link may not be accurate. This is not correct. Process substitution is a word expansion that results in a pathname. You can't just remove the pathname after the child opens it. How will other processes that want to communicate with the process substitution use it? OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
OK - this process on github has not gone exactly as I intended - merged with master - while I wanted to update, ie., merge with branch 5.0.18. So, the link may not be accurate. The change is simple: diff --git a/subst.c b/subst.c index 843c9d39..3792e45c 100644 --- a/subst.c +++ b/subst.c @@ -5926,6 +5926,8 @@ process_substitute (string, open_for_read_in_child) #if !defined (HAVE_DEV_FD) /* Open the named pipe in the child. */ fd = open (pathname, open_for_read_in_child ? O_RDONLY : O_WRONLY); + /* now that the file is open (or not) * unlink it to keep garbage down */ + unlink(pathname); if (fd < 0) { /* Two separate strings for ease of translation. */ On 17/03/2021 16:17, Michael Felt wrote: On 11/03/2021 18:11, Chet Ramey wrote: On 3/11/21 11:28 AM, Michael Felt wrote: Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' I believe this is fixed in bash-5.1. I added some debug statements to try and catch what is not happening. It seems that the fifo_list[i].proc value is never being set to (pid_t)-1 so any call to `unlink_fifo()` or `unlink_fifo_list()` does not unlink the special file created. Probably because the process substitution does not exit before the shell does. I spent several days debugging - and, basically, they never get cleared because the fifo_struct never gets the (pid_t) -1 value assigned. Although the `reap` function does get called - there is never anything to do. The routine that does assign the (pid_t) -1 value is `wait`*something - and this is only called via an interrupt (aka signal) - as far as I could see. in the end I came up with a very simple - basically historical solution - for working with tempoary files that do not need to survive the process - unlink() the file immediately after open()> As I need to document for AdoptOpenJDK I created a mirror of savannah (git) and created a PR: https://github.com/aixtools/bash/pull/2 I expect much more testing is warrented - as to potential side-effects with the fifo struct (that is no longer accurate as the file may (read should) already be unlinked. Hope this helps, Michael OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 11/03/2021 18:11, Chet Ramey wrote: On 3/11/21 11:28 AM, Michael Felt wrote: Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' I believe this is fixed in bash-5.1. I added some debug statements to try and catch what is not happening. It seems that the fifo_list[i].proc value is never being set to (pid_t)-1 so any call to `unlink_fifo()` or `unlink_fifo_list()` does not unlink the special file created. Probably because the process substitution does not exit before the shell does. I spent several days debugging - and, basically, they never get cleared because the fifo_struct never gets the (pid_t) -1 value assigned. Although the `reap` function does get called - there is never anything to do. The routine that does assign the (pid_t) -1 value is `wait`*something - and this is only called via an interrupt (aka signal) - as far as I could see. in the end I came up with a very simple - basically historical solution - for working with tempoary files that do not need to survive the process - unlink() the file immediately after open()> As I need to document for AdoptOpenJDK I created a mirror of savannah (git) and created a PR: https://github.com/aixtools/bash/pull/2 I expect much more testing is warrented - as to potential side-effects with the fifo struct (that is no longer accurate as the file may (read should) already be unlinked. Hope this helps, Michael OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 16/03/2021 16:21, Chet Ramey wrote: On 3/16/21 11:07 AM, Michael Felt wrote: On 16/03/2021 14:38, Chet Ramey wrote: On 3/16/21 8:04 AM, Michael Felt wrote: Decided to give bash-5.1 a try. I doubt it is major, but I get as far as: "../../../src/bash-5.1.0/lib/sh/tmpfile.c", line 289.11: 1506-068 (W) Operation between types "char*" and "int" is not allowed. ld: 0711-317 ERROR: Undefined symbol: .mkdtemp Then how does configure find it? It's a POSIX function, and that file includes the appropriate headers. Haven't looked at configure yet - but do not find it in the usual places: root@x065:[/data/prj/gnu/bash/bash-5.0.18]grep mkdtemp /usr/include/*.h root@x065:[/data/prj/gnu/bash/bash-5.0.18]grep mkdtemp /usr/include/sys/*.h Also, not found on AIX 6.1 (TL9), but did find on AIX 7.1 TL4. Sure, but configure checks for it, and bash only uses mkdtemp if configure finds it. Why does configure find it? Not sure. I'll have time again Thursday. Will be back. OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 16/03/2021 14:38, Chet Ramey wrote: On 3/16/21 8:04 AM, Michael Felt wrote: Decided to give bash-5.1 a try. I doubt it is major, but I get as far as: "../../../src/bash-5.1.0/lib/sh/tmpfile.c", line 289.11: 1506-068 (W) Operation between types "char*" and "int" is not allowed. ld: 0711-317 ERROR: Undefined symbol: .mkdtemp Then how does configure find it? It's a POSIX function, and that file includes the appropriate headers. Haven't looked at configure yet - but do not find it in the usual places: root@x065:[/data/prj/gnu/bash/bash-5.0.18]grep mkdtemp /usr/include/*.h root@x065:[/data/prj/gnu/bash/bash-5.0.18]grep mkdtemp /usr/include/sys/*.h Also, not found on AIX 6.1 (TL9), but did find on AIX 7.1 TL4. Hope this helps, Michael OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
On 11/03/2021 22:27, Chet Ramey wrote: On 3/11/21 3:55 PM, Michael Felt (aixtools) wrote: Sent from my iPhone On 11 Mar 2021, at 18:15, Chet Ramey wrote: On 3/11/21 11:28 AM, Michael Felt wrote: Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' I believe this is fixed in bash-5.1. Would it be difficult to give me a hint for 5.0. I could test further now i have a command that generates the issue. I can't reproduce it, but you can look at unlink_all_fifos() in bash-5.1. It's defined in subst.c and called in shell.c. FYI: Been digging in bash-5.0.18 - learning... Decided to give bash-5.1 a try. I doubt it is major, but I get as far as: "../../../src/bash-5.1.0/lib/sh/tmpfile.c", line 289.11: 1506-068 (W) Operation between types "char*" and "int" is not allowed. ld: 0711-317 ERROR: Undefined symbol: .mkdtemp ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information. make: 1254-004 The error code from the last command is 8. Using AIX 5.3 TL12 and xlc/xlC-v11, and a largely stripped system Of other OSS). I'll also give it a go on a more public server. Michael Probably because the process substitution does not exit before the shell does. I was hoping that is what the wait routines were for. Also noticed that the second fifo never gets a pid. Bash doesn't wait for asynchronous processes before it exits unless you use `wait'. OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: so-called pipe files (sh-np-*) do not get deleted when processes close.
Sent from my iPhone > On 11 Mar 2021, at 18:15, Chet Ramey wrote: > > On 3/11/21 11:28 AM, Michael Felt wrote: >> Hi, >> Issue: AdoptOpenJDK build process makes bash calls in a particular way. An >> abbreviated (shorter pathnames) example is: >> ``` >> bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image >> test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >> >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a >> /home/aixtools/build.log >&2) >> Building targets 'product-images legacy-jre-image test-image' in >> configuration 'aix-ppc64-normal-server-release' > > I believe this is fixed in bash-5.1. Would it be difficult to give me a hint for 5.0. I could test further now i have a command that generates the issue. > > >> I added some debug statements to try and catch what is not happening. It >> seems that the fifo_list[i].proc value is never being set to (pid_t)-1 so >> any call to `unlink_fifo()` or `unlink_fifo_list()` does not unlink the >> special file created. > > Probably because the process substitution does not exit before the shell does. I was hoping that is what the wait routines were for. Also noticed that the second fifo never gets a pid. > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer >``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
so-called pipe files (sh-np-*) do not get deleted when processes close.
Hi, Issue: AdoptOpenJDK build process makes bash calls in a particular way. An abbreviated (shorter pathnames) example is: ``` bash-5.0$ /usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' ``` back to ksh: ``` $ cat ~/build.log Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release' ``` I added some debug statements to try and catch what is not happening. It seems that the fifo_list[i].proc value is never being set to (pid_t)-1 so any call to `unlink_fifo()` or `unlink_fifo_list()` does not unlink the special file created. I have forced some debug info: a) unset USE_MKTEMP b) added the PID of the process creating the special file c) additional debug info using fprintf(FILE, ...) fprintf(unlink_lst,"%d:%s:%d:%16s:%2d:%12d:%s\n", getpid(), __FILE__, __LINE__, "unlink_fifo_list", i, fifo_list[i].proc, fifo_list[i].file); That gives me the following information from the command above: 25231596:../src/bash-5.0.18/execute_cmd.c:752:execute_command_internal:/usr/bin/printf "Building targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-normal-server-release'\n" > >(/usr/bin/tee -a /home/aixtools/build.log) 2> >(/usr/bin/tee -a /home/aixtools/build.log >&2) 21233868:../src/bash-5.0.18/subst.c:5372: add_fifo_list: 0: 0:/tmp//sh-np-21233868-1115804781 27918500:../src/bash-5.0.18/execute_cmd.c:752:execute_command_internal:/usr/bin/tee -a /home/aixtools/build.log 21233868:../src/bash-5.0.18/subst.c:5372: add_fifo_list: 1: 0:/tmp//sh-np-21233868-3761770506 27918500:../src/bash-5.0.18/subst.c:5506:reap_procsubs-calls:23 27918500:../src/bash-5.0.18/subst.c:5510: unlink_if==-1: 0: 0:/tmp//sh-np-21233868-1115804781 26869806:../src/bash-5.0.18/execute_cmd.c:752:execute_command_internal:/usr/bin/tee -a /home/aixtools/build.log 1>&2 26869806:../src/bash-5.0.18/subst.c:5506:reap_procsubs-calls:23 26869806:../src/bash-5.0.18/subst.c:5510: unlink_if==-1: 0: 27918500:/tmp//sh-np-21233868-1115804781 26869806:../src/bash-5.0.18/subst.c:5510: unlink_if==-1: 1: 0:/tmp//sh-np-21233868-3761770506 And this is remaining: $ ls -l /tmp/sh-np* | grep 21233868 prw--- 1 aixtools staff 0 Mar 11 08:07 /tmp/sh-np-21233868-1115804781 prw--- 1 aixtools staff 0 Mar 11 08:07 /tmp/sh-np-21233868-3761770506 Getting back to AdoptOpenJDK - a build process has roughly 3750 of these commands - leaving 7500 files behind in /tmp. On a busy day this can lead to over 100k empty files in /tmp. Thanks for any assistance. Regards, Michael OpenPGP_0x722BFDB61F396FC2.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: issue with vredir6.sub and AIX - bash (sub-shell) is crashing durig manual replication of test
OK. Thanks for the reply - and apologies for slow response. I was expecting a summary of the test results. I thought the other projects you lead ended with test summaries. If you care to help me find where the wrong error number is being returned I'll try and come up with a PR to fix this test for AIX. If not, we can consider this thread 'closed'. Sincerely, Michael On 16/10/2020 15:16, Chet Ramey wrote: > On 10/16/20 6:31 AM, Michael Felt wrote: > >> OK. While - perhaps the root cause is differences in error-codes, or >> something like that - and not to be overly concerned about (as I am not >> with the UTF-8 'errors') I am concerned that 'make test' terminates at >> this point - rather than continuing with the rest of the test suite. > There is no "rest of the test suite." That is the last test script. Bash > runs all the tests to completion. >
Re: issue with vredir6.sub and AIX - bash (sub-shell) is crashing durig manual replication of test
FYI: modified the script to: ``` root@x065:[/data/prj/gnu/bash/bash-5.0.18]cat -n *sub 1 unset v 2 exec {v}= 10 )); then echo ok 1; else echo bad 1; fi 5 6 exec {v}<&- 7 8 ulimit -n 6 9 10 exec On 15/10/2020 16:21, Chet Ramey wrote: >> On 10/15/20 3:03 AM, Michael Felt wrote: >>> Hi. >>> >>> I don't actually use bash myself - so something that would be apparent >>> to a bash user is invisible to me. >>> >>> As part of the packaging of bash-5.0.18 (i.e., 5.0 at patch level 18) I >>> ran the test suite. >>> >>> a) is there a flag I can pass so that it ignores the UTF-8 tests? I do >>> not want to not build UTF-8 support - for anyone who does load those >>> language filesets, but I would prefer to not see that test show up as >>> failed - rather as skipped. >> There isn't. There isn't any way to make any part of the test suite >> optional. I'd argue that UTF-8 support is the rule, rather than the >> exception, anyway. > Yes - AIX - without UTF-8 is the exception for bash. I was not concerned > about the report. > I know that AIX is 'exceptional' :) pun intended. >>> b) I have - apparently - been a LONG time since I last ran `make test` >>> for bash, as I get the same error on bash-4.4 patch level 23, and on >>> bash-5.0 patch level 11 and bash-5.0 patch level 18. >>> >>> run-vredir >>> 14,16c14,16 >>> < ./vredir.tests: line 25: $v: A file descriptor does not refer to an >>> open file. >>> < ./vredir.tests: line 26: $v: A file descriptor does not refer to an >>> open file. >>> < ./vredir.tests: line 27: $v: A file descriptor does not refer to an >>> open file. >>> --- >>>> ./vredir.tests: line 25: $v: Bad file descriptor >>>> ./vredir.tests: line 26: $v: Bad file descriptor >>>> ./vredir.tests: line 27: $v: Bad file descriptor >>> 90,91c90,91 >>> < ./vredir6.sub: redirection error: cannot duplicate fd: The process >>> file table is full. >>> < ./vredir6.sub: line 13: /dev/null: The process file table is full. >>> --- >>>> ./vredir6.sub: redirection error: cannot duplicate fd: Invalid argument >>>> ./vredir6.sub: line 13: /dev/null: Invalid argument >> These are simply different error messages for the same underlying error >> (AIX is famous for these) or system calls returning different values of >> errno for the same underlying failure (the last example above is EMFILE >> vs. EINVAL, for instance). Some of the tests warn about this: >> >> warning: the text of a system error message may vary between systems and >> warning: produce diff output. > OK. While - perhaps the root cause is differences in error-codes, or > something like that - and not to be overly concerned about (as I am not > with the UTF-8 'errors') I am concerned that 'make test' terminates at > this point - rather than continuing with the rest of the test suite. >>> $ exec >> $ exit >> You changed the shell's input -- from where it is reading commands -- to >> /dev/null, the next read returned EOF, and the shell exited. It printed >> `exit' so you know what's going on, just as if you had typed ^D at an >> interactive shell prompt. This doesn't happen when you're executing a >> script because the shell is reading commands from the script file. > I should have known that - but shell scripting is not my favorite > pasttime :) >>> root@x065:[/data/prj/gnu/bash/bash-5.0.18] >>> >>> ``` >>> >>> As you can see by the return of the original PS1 - the sub-shell >>> (./bash) 'crashed' -- I did not type 'exit' - that is a result of the >>> 'exec > That's not a crash. >> > >
Re: issue with vredir6.sub and AIX - bash (sub-shell) is crashing durig manual replication of test
On 15/10/2020 16:21, Chet Ramey wrote: > On 10/15/20 3:03 AM, Michael Felt wrote: >> Hi. >> >> I don't actually use bash myself - so something that would be apparent >> to a bash user is invisible to me. >> >> As part of the packaging of bash-5.0.18 (i.e., 5.0 at patch level 18) I >> ran the test suite. >> >> a) is there a flag I can pass so that it ignores the UTF-8 tests? I do >> not want to not build UTF-8 support - for anyone who does load those >> language filesets, but I would prefer to not see that test show up as >> failed - rather as skipped. > There isn't. There isn't any way to make any part of the test suite > optional. I'd argue that UTF-8 support is the rule, rather than the > exception, anyway. Yes - AIX - without UTF-8 is the exception for bash. I was not concerned about the report. I know that AIX is 'exceptional' :) pun intended. > >> b) I have - apparently - been a LONG time since I last ran `make test` >> for bash, as I get the same error on bash-4.4 patch level 23, and on >> bash-5.0 patch level 11 and bash-5.0 patch level 18. >> >> run-vredir >> 14,16c14,16 >> < ./vredir.tests: line 25: $v: A file descriptor does not refer to an >> open file. >> < ./vredir.tests: line 26: $v: A file descriptor does not refer to an >> open file. >> < ./vredir.tests: line 27: $v: A file descriptor does not refer to an >> open file. >> --- >>> ./vredir.tests: line 25: $v: Bad file descriptor >>> ./vredir.tests: line 26: $v: Bad file descriptor >>> ./vredir.tests: line 27: $v: Bad file descriptor >> 90,91c90,91 >> < ./vredir6.sub: redirection error: cannot duplicate fd: The process >> file table is full. >> < ./vredir6.sub: line 13: /dev/null: The process file table is full. >> --- >>> ./vredir6.sub: redirection error: cannot duplicate fd: Invalid argument >>> ./vredir6.sub: line 13: /dev/null: Invalid argument > These are simply different error messages for the same underlying error > (AIX is famous for these) or system calls returning different values of > errno for the same underlying failure (the last example above is EMFILE > vs. EINVAL, for instance). Some of the tests warn about this: > > warning: the text of a system error message may vary between systems and > warning: produce diff output. OK. While - perhaps the root cause is differences in error-codes, or something like that - and not to be overly concerned about (as I am not with the UTF-8 'errors') I am concerned that 'make test' terminates at this point - rather than continuing with the rest of the test suite. > >> $ exec > $ exit > You changed the shell's input -- from where it is reading commands -- to > /dev/null, the next read returned EOF, and the shell exited. It printed > `exit' so you know what's going on, just as if you had typed ^D at an > interactive shell prompt. This doesn't happen when you're executing a > script because the shell is reading commands from the script file. I should have known that - but shell scripting is not my favorite pasttime :) > >> root@x065:[/data/prj/gnu/bash/bash-5.0.18] >> >> ``` >> >> As you can see by the return of the original PS1 - the sub-shell >> (./bash) 'crashed' -- I did not type 'exit' - that is a result of the >> 'exec That's not a crash. >
Re: issue with vredir6.sub and AIX - bash (sub-shell) is crashing durig manual replication of test
On 15/10/2020 16:11, k...@plushkava.net wrote: > On 15/10/2020 08:03, Michael Felt wrote: > >> $ exec > $ exit >> root@x065:[/data/prj/gnu/bash/bash-5.0.18] >> >> ``` >> >> As you can see by the return of the original PS1 - the sub-shell >> (./bash) 'crashed' -- I did not type 'exit' - that is a result of the >> 'exec Upon running "exec attached to your terminal emulator. Consequently, bash tries to read > from /dev/null, only to encounter EOF. As there is no further input to > be processed, it exits. > > You can prevent this from happening by running a script, or by using > the -c option to convey the code. Doing so will launch a > non-interactive instance of bash that does not read commands from the > standard input. > > Incidentally, it is not a subshell but an ordinary subprocess. Thanks - I'll try and give that a try to help understand what is going on in the test. As I'll mention in my reply to Chet - the concern is because 'make test' stops, rather than just report an error.
Re: issue with vredir6.sub and AIX - bash (sub-shell) is crashing durig manual replication of test
Just repeated this on a system with bash that I did not package - and I get the same results: BASH_VERSINFO=([0]="4" [1]="4" [2]="0" [3]="1" [4]="release" [5]="powerpc-ibm-aix6.1.0.0") BASH_VERSION='4.4.0(1)-release' ... ``` _=/usr/bin/bash aixtools@gcc119:[/home/aixtools]PS1='$ ' $ unset v $ exec {v} Hi. > > I don't actually use bash myself - so something that would be apparent > to a bash user is invisible to me. > > As part of the packaging of bash-5.0.18 (i.e., 5.0 at patch level 18) I > ran the test suite. > > a) is there a flag I can pass so that it ignores the UTF-8 tests? I do > not want to not build UTF-8 support - for anyone who does load those > language filesets, but I would prefer to not see that test show up as > failed - rather as skipped. > > b) I have - apparently - been a LONG time since I last ran `make test` > for bash, as I get the same error on bash-4.4 patch level 23, and on > bash-5.0 patch level 11 and bash-5.0 patch level 18. > > run-vredir > 14,16c14,16 > < ./vredir.tests: line 25: $v: A file descriptor does not refer to an > open file. > < ./vredir.tests: line 26: $v: A file descriptor does not refer to an > open file. > < ./vredir.tests: line 27: $v: A file descriptor does not refer to an > open file. > --- >> ./vredir.tests: line 25: $v: Bad file descriptor >> ./vredir.tests: line 26: $v: Bad file descriptor >> ./vredir.tests: line 27: $v: Bad file descriptor > 90,91c90,91 > < ./vredir6.sub: redirection error: cannot duplicate fd: The process > file table is full. > < ./vredir6.sub: line 13: /dev/null: The process file table is full. > --- >> ./vredir6.sub: redirection error: cannot duplicate fd: Invalid argument >> ./vredir6.sub: line 13: /dev/null: Invalid argument > root@x065:[/data/prj/gnu/bash/bash-4.4.23] > > To try and understand what is going on - I ran the following commands > from the CLI. I hope you can give me some suggestions on what to look at. > > ``` > > root@x065:[/data/prj/gnu/bash/bash-5.0.18]./bash > root@x065:[/data/prj/gnu/bash/bash-5.0.18]PS1='$ ' > $ unset v > $ exec {v} $ echo $v > 10 > $ exec {v}<&- > $ ulimit -n > 2 > $ ulimit -n 6 > $ ulimit -n > 6 > $ exec $ exit > root@x065:[/data/prj/gnu/bash/bash-5.0.18] > > ``` > > As you can see by the return of the original PS1 - the sub-shell > (./bash) 'crashed' -- I did not type 'exit' - that is a result of the > 'exec > Hope this helps, > > Michael (aka aixtools). > > > >
issue with vredir6.sub and AIX - bash (sub-shell) is crashing durig manual replication of test
Hi. I don't actually use bash myself - so something that would be apparent to a bash user is invisible to me. As part of the packaging of bash-5.0.18 (i.e., 5.0 at patch level 18) I ran the test suite. a) is there a flag I can pass so that it ignores the UTF-8 tests? I do not want to not build UTF-8 support - for anyone who does load those language filesets, but I would prefer to not see that test show up as failed - rather as skipped. b) I have - apparently - been a LONG time since I last ran `make test` for bash, as I get the same error on bash-4.4 patch level 23, and on bash-5.0 patch level 11 and bash-5.0 patch level 18. run-vredir 14,16c14,16 < ./vredir.tests: line 25: $v: A file descriptor does not refer to an open file. < ./vredir.tests: line 26: $v: A file descriptor does not refer to an open file. < ./vredir.tests: line 27: $v: A file descriptor does not refer to an open file. --- > ./vredir.tests: line 25: $v: Bad file descriptor > ./vredir.tests: line 26: $v: Bad file descriptor > ./vredir.tests: line 27: $v: Bad file descriptor 90,91c90,91 < ./vredir6.sub: redirection error: cannot duplicate fd: The process file table is full. < ./vredir6.sub: line 13: /dev/null: The process file table is full. --- > ./vredir6.sub: redirection error: cannot duplicate fd: Invalid argument > ./vredir6.sub: line 13: /dev/null: Invalid argument root@x065:[/data/prj/gnu/bash/bash-4.4.23] To try and understand what is going on - I ran the following commands from the CLI. I hope you can give me some suggestions on what to look at. ``` root@x065:[/data/prj/gnu/bash/bash-5.0.18]./bash root@x065:[/data/prj/gnu/bash/bash-5.0.18]PS1='$ ' $ unset v $ exec {v}
Re: issues in bash found while creating command result display
Sent from my iPhone > On 17 Apr 2019, at 01:37, Paul Wise wrote: > >> On Tue, 2019-04-16 at 14:57 -0400, Chet Ramey wrote: >> >> Why take so much effort to (imperfectly) figure out and display >> things you already know? > > Correctness. If what the user knows You mean think they know, better is believe. > doesn't match what the program > knows And, if not a bug, which someone who knows will “know”, better, recognize or suspect as a bug. Otherwise the program does not “know” but better (re)acts as instructed. In short, you can never, for 100%, enhance the UI to an extent a program can inform the user they’re trying to do something “unexpected” or explain why something is not the recommended approach. My 2 cents! 😃 > then they might think that the program is buggy or that there is > something malicious on their system causing errors in order to make > them doubt their own mind. > > -- > bye, > pabs > > https://bonedaddy.net/pabs3/
Re: make distclean and bash-4.4 - FYI
> On 11/1/2018 12:43 PM, Michael Felt wrote: >>> I am mainly surprised by "process file table is full" - is there >>> something specific I can do to look at this more closely? >> No, it's expected. That script tests the behavior when the process's open >> file limit (`ulimit -n') is set to something very small. > Using defaults: > root@x065:[/]ulimit -n > 2000 > >> It's interesting >> that AIX (you're running AIX, right?) chooses to return -1/EMFILE even >> though the process doesn't have the maximum number of file descriptors open. > Well, it is AIX 5.3 (TL7). Maybe there are more choices for errno today, than > there was in 2007. I'll create a clean source to patch/build oot, and > compare AIX 5.3, 61. and 7.1 (no ready access to 7.2, sorry). > > In case you are interested.. AIX 6.1 and 7.1 stop at the same point, but get there MUCH faster. I started the AIX 5.3 make test even before I started the builds on AIX 6.1 and 7.1, AIX 5.3 is still showing: warning: if the text of an error message concerning `redir1.*' not being warning: found or messages concerning bad file descriptors produce diff warning: output, please do not consider it a test failure while AIX 6.1 and AIX 7.1 have already "make error" status. FYI (who really cares about AIX 5.3 these days :p )
Re: make distclean and bash-4.4 - FYI
> On 10/31/2018 9:31 PM, Chet Ramey wrote: >> On 10/31/18 3:13 PM, Michael Felt wrote: >> >> Running "make test", and I amy have forgotten something I did in the past. >> a) running tests as root (initially) >> b) ends with: >> run-vredir >> 14,16c14,16 >> < ./vredir.tests: line 25: $v: A file descriptor does not refer to an >> open file. >> < ./vredir.tests: line 26: $v: A file descriptor does not refer to an >> open file. >> < ./vredir.tests: line 27: $v: A file descriptor does not refer to an >> open file. >> --- >>> ./vredir.tests: line 25: $v: Bad file descriptor >>> ./vredir.tests: line 26: $v: Bad file descriptor >>> ./vredir.tests: line 27: $v: Bad file descriptor > These are just different messages for the same value of errno. > >> 90,91c90,91 >> < ./vredir6.sub: redirection error: cannot duplicate fd: The process >> file table is full. >> < ./vredir6.sub: line 13: /dev/null: The process file table is full. >> --- >>> ./vredir6.sub: redirection error: cannot duplicate fd: Invalid argument >>> ./vredir6.sub: line 13: /dev/null: Invalid argument > Same. > >> I am mainly surprised by "process file table is full" - is there >> something specific I can do to look at this more closely? > No, it's expected. That script tests the behavior when the process's open > file limit (`ulimit -n') is set to something very small. Using defaults: root@x065:[/]ulimit -n 2000 > It's interesting > that AIX (you're running AIX, right?) chooses to return -1/EMFILE even > though the process doesn't have the maximum number of file descriptors open. Well, it is AIX 5.3 (TL7). Maybe there are more choices for errno today, than there was in 2007. I'll create a clean source to patch/build oot, and compare AIX 5.3, 61. and 7.1 (no ready access to 7.2, sorry).
Re: make distclean and bash-4.4 - FYI
> On 10/31/2018 9:24 PM, Chet Ramey wrote: >> On 10/31/18 3:03 PM, Michael Felt wrote: >> Hi, >> >> I finally got around to patching and packaging bash-4.4.23 and had to >> run a "make distclean" because I had copied the old version and without >> the make distclean make kept looking for "bashversion" in the old path >> .../bash-4.4.19/bashversion > The build directory is written to the Makefile so the build artifacts > (like bashversion) get removed from the right place. This is how you > support multiple build directories from a single source directory, for > instance. > >> This is just FYI - I'll install yacc, which is what make is complaining >> about now - but I also wonder if I could have avoided this by trying to >> build oot (out of tree). Is this supported, or even recommended? > You should install bison, not yacc, otherwise you'll run into problems > resulting from yacc's inability to support recursive parsers. And yes, > a build directory separate from the source directory is the recommended > configuration. If I "must", I shall build and install yacc. However, my hope is that: a) I won't need either after I create a source-directory, and build oot b) I prefer to work, as much as possible, from platform supplied tools. Thanks for the extra info. > >> Thanks for a great product! > Thanks for your kind words. > > Chet >
Re: make distclean and bash-4.4 - FYI
On 10/31/2018 9:03 PM, Michael Felt wrote: > Hi, > > I finally got around to patching and packaging bash-4.4.23 and had to > run a "make distclean" because I had copied the old version and without > the make distclean make kept looking for "bashversion" in the old path > .../bash-4.4.19/bashversion > > This is just FYI - I'll install yacc, make worked well after yacc was installed > which is what make is complaining > about now - but I also wonder if I could have avoided this by trying to > build oot (out of tree). Is this supported, or even recommended? > > Thanks for a great product! > > Michael > Running "make test", and I amy have forgotten something I did in the past. a) running tests as root (initially) b) ends with: run-vredir 14,16c14,16 < ./vredir.tests: line 25: $v: A file descriptor does not refer to an open file. < ./vredir.tests: line 26: $v: A file descriptor does not refer to an open file. < ./vredir.tests: line 27: $v: A file descriptor does not refer to an open file. --- > ./vredir.tests: line 25: $v: Bad file descriptor > ./vredir.tests: line 26: $v: Bad file descriptor > ./vredir.tests: line 27: $v: Bad file descriptor 90,91c90,91 < ./vredir6.sub: redirection error: cannot duplicate fd: The process file table is full. < ./vredir6.sub: line 13: /dev/null: The process file table is full. --- > ./vredir6.sub: redirection error: cannot duplicate fd: Invalid argument > ./vredir6.sub: line 13: /dev/null: Invalid argument I am mainly surprised by "process file table is full" - is there something specific I can do to look at this more closely? Michael
make distclean and bash-4.4 - FYI
Hi, I finally got around to patching and packaging bash-4.4.23 and had to run a "make distclean" because I had copied the old version and without the make distclean make kept looking for "bashversion" in the old path .../bash-4.4.19/bashversion This is just FYI - I'll install yacc, which is what make is complaining about now - but I also wonder if I could have avoided this by trying to build oot (out of tree). Is this supported, or even recommended? Thanks for a great product! Michael
Re: make distclean breaks "later "configure && makes"", i.e., removes a file in distro that a build needs (cannot process parse.y)
I'll start all over again - using the following structure: cd dist; wget distro..tar.gz cd ../src; gzip -dc ../dist/distro.tar.gz | tar xf - mkdir ../distro; cd ../distro ../src/distro/configure --arguments; make; make distclean ../src/distro/configure --arguments; make As I have not been build OOT, maybe this fixes it - I forget why I am not building OOT. Maybe because I was not being handy when having to update a source tree with a patch. IF above works - I'll say "my mistake", if not - might still be my mistake, but not one I can figure out. On 2/14/2018 4:32 PM, Chet Ramey wrote: There is code in the Makefile that attempts to detect when the build and source directories are not the same, and removes y.tab.[ch] from the build directory when they are not. The problem is that when you run `./configure', the source directory gets set to `.' and the build directory gets set to the absolute pathname. Those don't compare as equal, and the files get removed. I could solve that problem with a short `same_dir' script, but maybe there's an easier way.
make distclean breaks "later "configure && makes"", i.e., removes a file in distro that a build needs (cannot process parse.y)
I can down the distribution, e.g. ftp://ftp.gnu.org/gnu/bash/bash-4.4.18.tar.gz, unpack it, goto base directory and run configure. Thought I had reported this earlier, but guess not! If after a successful build, I run "make distclean" - "./configure && make" no longer works. e.g.: root@x065:[/data/prj/gnu/bash/bash-4.4.19]make distclean rm -f shell.o eval.o y.tab.o general.o make_cmd.o print_cmd.o dispose_cmd.o execute_cmd.o variables.o copy_cmd.o error.o expr.o flags.o jobs.o subst.o hashcmd.o hashlib.o mailcheck.o trap.o input.o unwind_prot.o pathexp.o sig.o test.o version.o alias.o array.o arrayfunc.o assoc.o braces.o bracecomp.o bashhist.o bashline.o list.o stringlib.o locale.o findcmd.o redir.o pcomplete.o pcomplib.o syntax.o xmalloc.o bash bashbug rm -f .build .made version.h ... rm -f config.h config.cache config.status config.log stamp-h po/POTFILES config.status.lineno tags TAGS rm -f signames.h recho zecho printenv tests/recho tests/zecho tests/printenv xcase tests/xcase mksignames lsignames.h mksyntax syntax.c bashversion bashversion.o buildversion.o mksignames.o signames.o buildsignames.o Makefile Makefile builtins/Makefile doc/Makefile lib/readline/Makefile lib/glob/Makefile lib/sh/Makefile lib/tilde/Makefile lib/malloc/Makefile lib/termcap/Makefile examples/loadables/Makefile examples/loadables/Makefile.inc examples/loadables/perl/Makefile support/Makefile lib/intl/Makefile po/Makefile po/Makefile.in pathnames.h root@x065:[/data/prj/gnu/bash/bash-4.4.19]CPPFLAGS="-I/opt/include" CFLAGS="-I/opt/aixtools/include -O2 -qmaxmem=-1 -qarch=pwr5"\ ./configure\ --prefix=/opt/aixtools \ --sysconfdir=/var/bash/etc\ --sharedstatedir=/var/bash/com\ --localstatedir=/var/bash\ --mandir=/usr/share/man\ --infodir=/opt/share/info/bash \ > .buildaix/configure.out configure: WARNING: bison not available; needed to process parse.y + /usr/bin/make > .buildaix/make.out yacc: not found make: 1254-004 The error code from the last command is 1. Stop. /usr/bin/make returned an error
Re: Bash-4.4-beta available for FTP
AIX is "no more entitled" to it, but they are using it - and "it" is stopping make from succeeding. + > .buildaix/make.out ../src/bash-4.4/parse.y: warning: 1 shift/reduce conflict [-Wconflicts-sr] "../src/bash-4.4/execute_cmd.c", line 4655.16: 1506-068 (W) Operation between types "struct array*" and "volatile struct array*" is not allowed. "../src/bash-4.4/execute_cmd.c", line 4657.16: 1506-068 (W) Operation between types "struct array*" and "volatile struct array*" is not allowed. "/usr/include/mbstr.h", line 47.22: 1506-334 (S) Identifier mbchar_t has already been defined on line 175 of "../src/bash-4.4/include/shmbchar.h". make: 1254-004 The error code from the last command is 1. This is from a build done from the beta release from 24 hours ago. Michael On Mon, Oct 26, 2015 at 4:40 PM, Chet Ramey wrote: > On 10/23/15 1:50 PM, Michael Felt wrote: > > I do not mind installing yacc :) > > You would need to install bison anyway. > > > How about the redefine of mbchar_t ? > > See my previous reply; AIX is no more entitled to this symbol than anyone > else. Bash doesn't actually use this symbol internally, since it only uses > some of the functions from the gnulib implementation, so I imagine I could > just excise it from the header file if I had to. > > Chet > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, ITS, CWRUc...@case.edu > http://cnswww.cns.cwru.edu/~chet/ >
Re: Bash-4.4-beta available for FTP
I do not mind installing yacc :) How about the redefine of mbchar_t ? On Fri, Oct 23, 2015 at 2:45 PM, Chet Ramey wrote: > On 10/22/15 3:47 PM, aixtools wrote: >> On 2015-10-15 16:23, Chet Ramey wrote: >>> The first beta release of bash-4.4 is now available with the URL >>> >>> ftp://ftp.cwru.edu/pub/bash/bash-4.4-beta.tar.gz >>> >>> >> First attempt at build on AIX >> >> configure warns that bison is not available - good. >> >> configure finishes - good >> >> make fails immediately because yacc is not installed - not good > > The next release will have this fixed; the modtimes on y.tab.[ch] were > wrong. > > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Issue with patching from bash-4.3-patches
I am using patch 2.7.0 - and I get the warnings about dangerous files (what is a dangerous file I ask myself) - but patching proceeds without any problems. Michael On Thu, Oct 2, 2014 at 11:09 PM, Chet Ramey wrote: > On 9/30/14, 2:18 PM, Matthew Gessner wrote: > > I apologize, but I somehow sent this from gmail prematurely. > > I am having issues with patch working properly with the patches in > > bash-4.3-patches. > > Admittedly, I don't use patch very often, but having asked on freenode > > ##bash and others telling me they are not having issues, they suggested I > > post this here. > > > > I've seen similar issues using patch 2.7.1 and patch 2.6. > > Sorry, I'm using patch 2.5. There's probably an option to later patch > versions to force patching in the presence of `dangerous' filenames. > > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, ITS, CWRUc...@case.edu > http://cnswww.cns.cwru.edu/~chet/ > >