Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Chet Ramey]

On 11/18/24 12:55 AM, Martin D Kealey wrote:



On Sun, 17 Nov 2024, 03:32 Chet Ramey, > wrote:


On 11/16/24 3:36 AM, Martin D Kealey wrote:
I don't think a disclaimer saying "this manual is not what it does not
claim to be" is particularly useful.


That's valid, but how else do we get people to read the correct document 
for their skill level, when their self-assessment will invariably be wrong? 
How can we reasonably expect users to assess their own skill level and 
choose a 'tutorial' when their instincts are telling them "yep I understand 
almost all of this stuff"?


Maybe I have a higher opinion of users than this? This is pretty scathing.

The Shell language /looks/ simple, while actually being one of the most 
complex languages in common use, and its multitude of subtle fishhooks 
effectively turns half the populace into Dunning-Kruger sufferers. 
Moreover, we are complicit in this: every new feature added to Bash makes 
it more useful, more attractive to new users.


OK, you have a problem with the language and its definition. Let's not
project that on everyone.


The morally right thing to do would be to shoo them away, not attract them.
In the meantime we owe them a duty of care.


Wow, you really don't think much of new users.

I've used Bash for about 30 years, and even I have to wonder whether the 
Shell manual should start with this warning instead:


"/If you're writing a new script from scratch, *STOP NOW*, and go and 
choose a different language - ANY language. No matter what you choose, it 
will be a better choice than the Shell language.


Maybe spend some time away from this. I don't think adding that helps
anyone -- me, users of any experience level, people who want to help --
anyone. It's just alienating.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Oğuz]

On Monday, November 18, 2024, Martin D Kealey 
wrote:
>
> That's valid, but how else do we get people to read the correct document
> for their skill level,


How is it our responsibility?


> I've used Bash for about 30 years, and even I have to wonder whether the
> Shell manual should start with this warning instead:
>
> "
> *If you're writing a new script from scratch, STOP NOW, and go and choose a
> different language - ANY language. No matter what you choose, it will be a
> better choice than the Shell language.*
>

Maybe it's time "you" switched shells and stop bothering "us".


-- 
Oğuz

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Martin D Kealey]

On Sun, 17 Nov 2024, 03:32 Chet Ramey,  wrote:

> On 11/16/24 3:36 AM, Martin D Kealey wrote:
> I don't think a disclaimer saying "this manual is not what it does not
> claim to be" is particularly useful.
>

That's valid, but how else do we get people to read the correct document
for their skill level, when their self-assessment will invariably be wrong?
How can we reasonably expect users to assess their own skill level and
choose a 'tutorial' when their instincts are telling them "yep I understand
almost all of this stuff"?

The Shell language *looks* simple, while actually being one of the most
complex languages in common use, and its multitude of subtle fishhooks
effectively turns half the populace into Dunning-Kruger sufferers.
Moreover, we are complicit in this: every new feature added to Bash makes
it more useful, more attractive to new users.

The morally right thing to do would be to shoo them away, not attract them.
In the meantime we owe them a duty of care.

I've used Bash for about 30 years, and even I have to wonder whether the
Shell manual should start with this warning instead:

"
*If you're writing a new script from scratch, STOP NOW, and go and choose a
different language - ANY language. No matter what you choose, it will be a
better choice than the Shell language.*
*The Shell language makes bad coding easy, and good coding hard, so almost
everything you write will turn out to have bugs that may lurk for years
before biting you - or your successor.** It lacks sane error handling. It
lacks sane variable scopes. It lacks data structures. It lacks modules. It
looks like other languages, but doesn't behave like them.*
*And in case you're wondering, no, Bash can't be changed to make a saner
language, because it has to meet the POSIX standard, as well as 40 years of
backwards compatibility.*
*Don't say we didn't warn you.*

*(But don't simply choose a different implementation of the Shell; that
would be worse.)*"

-Martin

>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Chet Ramey]

On 11/16/24 3:36 AM, Martin D Kealey wrote:


(Insert this between the "Copyright" and "Description" sections)


I don't think a disclaimer saying "this manual is not what it does not
claim to be" is particularly useful.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Martin D Kealey]

On Sat, 16 Nov 2024 at 15:01, Robert Elz  wrote:

> Date:Sat, 16 Nov 2024 14:05:17 +1000
> From:Martin D Kealey 
> Message-ID:   d1_ebrgbusb2u1sxhyloqq...@mail.gmail.com>
>
>   | I know, but "explicitly" is not the same as "highlighted", let alone
> "well
>   | enough".
>
> Please, nothing at all like you're suggesting.   Man pages are not, and
> not intended to be, tutorials.
>

In the general case I agree; man pages should be reference manuals, not
tutorials.

But in this case we have a language that simultaneously *looks* familiar,
but *actually* has an unfamiliar grammar, radically different semantics,
and a large number of intricate behind-the-scenes interactions. As a
result, some significant fraction of users are going to overestimate how
well they understand it, and prematurely skip from the "tutorial" phase to
the "reference manual" phase.

If nothing else, the manual needs to point out, in unequivocal ways that
cannot be missed, that they really *don't* understand it as well as they
think they do, and that they need to go study a tutorial before doing
anything else (and especially before submitting a bug report). And that
needs to be right up front, where they cannot miss it.

I would hope that this idea isn't controversial, but if you feel my idea
misses something important then please let me know. If you agree with the
idea, then I would welcome suggestions for improving the wording below,
which I offer as a tentative implementation of the idea above.

(Insert this between the "Copyright" and "Description" sections)


*“PrefaceDear reader, if this is your introduction to the Shell programming
language, please be aware that although it may look like other languages
you are familiar with, such similarity is deceptive. The shell language is
different in subtle and non-obvious ways from all (*1) other languages, and
making assumptions based on what you know from other languages will lead
you into traps. Please read the "Shell Grammar" section, allowing **time to
digest it **slowly and thoroughly, and then read at least one of the
recommended tutorials listed in appendix B. If you are unsure whether you
fully understand the Shell, please take the test in appendix A.”*


(Insert this after the "Bugs" section)





*“Appendix A: a self-test to check understanding of basic principles.If you
have trouble answering these questions then you should read one of the
tutorials in Appendix B before consulting this manual. It will also be
helpful to read the "Shell Grammar" section. Please write down your answer
to each question, and then run its code (*2) to see if its behaviour
matches your answer. There is no pass or fail mark, but any mismatch
indicates that you would likely benefit from revision.*
*1. What will be displayed when you run this: foo=bar ; echo zot | read foo
; echo "$foo"*

*2. What will be displayed when you run this: export foo=bar ; foo=zot ;
env | grep ^foo=*
*3.  ...” *(additional questions welcome)


*“Appendix B: Recommended TutorialsBefore consulting this manual, we
recommend reading: https://mywiki.wooledge.org/BashGuide
 *http://guide.bash.academy/* etc...
We recommend avoiding the so-called Advanced Bash-Scripting Guide on *
tldp.org 
*...”*(additional reading suggestions welcome)

-Martin

(*1: yes I know it's an offshoot of the UNIX v7 shell, and other similar
languages were designed around the same time, but they're all dead now, and
certainly won't be encountered by a neophyte reading this manual page in
2024.)

(*2: this may require some finessing of nroff to ensure that it does NOT
substitute curly quotes or otherwise mess with the shell syntax, so that
users can simply copy & paste the example code to test it)

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Oğuz]

On Saturday, November 16, 2024, Martin D Kealey 
wrote:

> (Insert this between the "Copyright" and "Description" sections)
>
>
> *“PrefaceDear reader, if this is your introduction to the Shell programming
> language, please be aware that although it may look like other languages
> you are familiar with, such similarity is deceptive. The shell language is
> different in subtle and non-obvious ways from all (*1) other languages, and
> making assumptions based on what you know from other languages will lead
> you into traps. Please read the "Shell Grammar" section, allowing **time to
> digest it **slowly and thoroughly, and then read at least one of the
> recommended tutorials listed in appendix B. If you are unsure whether you
> fully understand the Shell, please take the test in appendix A.”*
>
>
> (Insert this after the "Bugs" section)
>
>
>
>
>
> *“Appendix A: a self-test to check understanding of basic principles.If you
> have trouble answering these questions then you should read one of the
> tutorials in Appendix B before consulting this manual. It will also be
> helpful to read the "Shell Grammar" section. Please write down your answer
> to each question, and then run its code (*2) to see if its behaviour
> matches your answer. There is no pass or fail mark, but any mismatch
> indicates that you would likely benefit from revision.*
> *1. What will be displayed when you run this: foo=bar ; echo zot | read foo
> ; echo "$foo"*
>
> *2. What will be displayed when you run this: export foo=bar ; foo=zot ;
> env | grep ^foo=*
> *3.  ...” *(additional questions welcome)
>
>
> *“Appendix B: Recommended TutorialsBefore consulting this manual, we
> recommend reading: https://mywiki.wooledge.org/BashGuide
>  *http://guide.bash.academy/*
> etc...
> We recommend avoiding the so-called Advanced Bash-Scripting Guide on *
> tldp.org 
> *...”*(additional reading suggestions welcome)
>
> -Martin
>
> (*1: yes I know it's an offshoot of the UNIX v7 shell, and other similar
> languages were designed around the same time, but they're all dead now, and
> certainly won't be encountered by a neophyte reading this manual page in
> 2024.)
>
> (*2: this may require some finessing of nroff to ensure that it does NOT
> substitute curly quotes or otherwise mess with the shell syntax, so that
> users can simply copy & paste the example code to test it)
>

This belongs in separate, unofficial document; not the reference manual,
which is fine as is.


-- 
Oğuz

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Robert Elz]

Date:Sat, 16 Nov 2024 14:05:17 +1000
From:Martin D Kealey 
Message-ID:  


  | I know, but "explicitly" is not the same as "highlighted", let alone "well
  | enough".

Please, nothing at all like you're suggesting.   Man pages are not, and
not intended to be, tutorials.

People who need tutorials should go get exactly that - there are books on
shell programming, and I expect, at least one that is precisely on bash.

If that kind of thing isn't good enough, talk to their authors.   Refer
obvious newbies to those, not to the man page.

kre

ps: that is perhaps except for the separate man pages for built-ins.
I have been considering doing something like that - document the generic
properties of the built-ins, make passing reference to some shell specific
variations, and refer the user to their specific shell documentation for
shell specific details.   Then the shell man page needs a reference to the
man page for the built-in, followed by any variations that particular shell
makes to how it works.

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Martin D Kealey]

On Fri, 15 Nov 2024 at 00:20, Chet Ramey  wrote:

> On 11/13/24 11:40 PM, Martin D Kealey wrote:
>
> > The fact that pipeline components are implicitly run in subshells is
> > arguably not highlighted well enough [...]
>


> The man page states that explicitly.
>

I know, but "explicitly" is not the same as "highlighted", let alone "well
enough".

Novice shell users tend to have a weak-at-best grasp of process separation,
and an abysmal-to-non-existent grasp of subshells, so the documentation for
Bash needs to drive those home at every opportunity.

I know that the FSF believes that "info" is the *one true documentation
format*, but that's pretty unrealistic in most environments. If people want
documentation from the command line, they type "man whatever", because it
works. (Or they ask Facebook or Bing or Google or Reddit or StackExchange,
and get ... wherever.)

The manual page runs to 7000+ lines, all in one document, so most people
*don't* read it from start to finish; rather they search for keywords.

If a warning or explanation isn't shown in response to such a search, it
might as well not exist.

There are no hyperlinks in man pages. Most $PAGER programs have a search
function, but it's a challenge to convince a human under the age of 40 to
use it. So I conclude that *writing* "see also" is not enough; any warning
text itself has to be repeated, immediately adjacent to each description
where it's germane.

And yes that would probably blow up the current 7000 lines to 1+ lines.
Perhaps the Bash man page needs to be broken into many separate pages,
including one-per-builtin.

-Martin

PS: I would envisage maybe as many as a hundred repetitions of something
like this: *“The built-in Foo command changes the Bar setting only in the
current process, which may or may not be your script's main process. Please
read "man bash-subshell" for an explanation of when new processes are
created and how this affects **each of the **built-in commands”* or this: *“The
Zot construct creates a subshell. This will limit the effects of built-in
commands within it, preventing them from changing settings in the main
process. Please read "man bash-subshell" for an explanation of when new
processes are created and how this affects **each of the **built-in
commands”.*

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

aidd , thxx .. ++

On Thu, Nov 14, 2024, 8:45 PM Chet Ramey  wrote:

> On 11/14/24 2:28 PM, #!microsuxx wrote:
> > maybe set -a 'd help ?
> > if it can backwards export the vars
>
> No. A child process can't affect its parent's environment without inventing
> some kind of IPC.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Chet Ramey]

On 11/14/24 2:28 PM, #!microsuxx wrote:

maybe set -a 'd help ?
if it can backwards export the vars


No. A child process can't affect its parent's environment without inventing
some kind of IPC.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

maybe set -a 'd help ?
if it can backwards export the vars

On Thu, Nov 14, 2024, 3:01 PM Chet Ramey  wrote:

> On 11/13/24 4:48 PM, Yuri wrote:
>
> > The first 'source' command didn't set variables set or exported in the
> > child.sh script, even though 'source' is executed in the current
> script's
> > context,
>
> The `source' command isn't "executed in the current script's context." It
> is executed in a subshell, which can't affect its parent's variables.
>
> The bash man page says:
>
> "Each command in a multi-command pipeline, where pipes are  created,  is
>   executed  in a subshell, which is a separate process."
>
> POSIX says:
>
> "Changes made to the subshell environment shall not affect the shell
>   environment. Command substitution, commands that are grouped with
>   parentheses, and asynchronous AND-OR lists shall be executed in a
> subshell
>   environment. Additionally, each command of a multi-command pipeline is in
>   a subshell environment;"
>
> Some shells, including bash in some circumstances, will execute the last
> pipeline element in the current shell, but no shell executes the first
> element in the current shell context.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Chet Ramey]

On 11/13/24 11:40 PM, Martin D Kealey wrote:


The fact that pipeline components are implicitly run in subshells is
arguably not highlighted well enough, but it IS stated that all components
of a pipeline run in parallel, so it is logically deducible that this must
involve separate processes.


The man page states that explicitly.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Chet Ramey]

On 11/13/24 5:12 PM, Yuri wrote:

On 11/13/24 14:02, Greg Wooledge wrote:
The commands within a pipeline are executed in subshells (child 
processes), so all variable changes are discarded when the subshell exits.



This sounds like an implementation detail that should be invisible that 
affects how the 'source' feature works.


You might consider it an implementation detail, but it's how shells work.


This makes it a very confusing from the user's perspective.


I think people are pretty much used to it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Chet Ramey]

On 11/13/24 4:48 PM, Yuri wrote:

The first 'source' command didn't set variables set or exported in the 
child.sh script, even though 'source' is executed in the current script's 
context, 


The `source' command isn't "executed in the current script's context." It
is executed in a subshell, which can't affect its parent's variables.

The bash man page says:

"Each command in a multi-command pipeline, where pipes are  created,  is
 executed  in a subshell, which is a separate process."

POSIX says:

"Changes made to the subshell environment shall not affect the shell
 environment. Command substitution, commands that are grouped with
 parentheses, and asynchronous AND-OR lists shall be executed in a subshell
 environment. Additionally, each command of a multi-command pipeline is in
 a subshell environment;"

Some shells, including bash in some circumstances, will execute the last
pipeline element in the current shell, but no shell executes the first
element in the current shell context.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Yuri]

Hi Martin,


On 11/13/24 20:40, Martin D Kealey wrote:

If you need to separate the output of `set -x` from your script's other
output, consider setting BASH_XTRACEFD:

exec 3>> /path/to/xtrace-logfile.txt
BASH_XTRACEFD=3
set -x
source your_file
set +x
exec 3>&-



I didn't know about BASH_XTRACEFD.

In case I'd really need to help users to diagnose their scripts down to 
the line number I'd use it.



Thank you for this very practical and relevant advise.


Yuri

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Martin D Kealey]

The disappearance of the variables that you export within your sourced file
is not a feature of the source command. That will happen to ANY command
that changes the shell's internal state, when run in a subshell.

The fact that pipeline components are implicitly run in subshells is
arguably not highlighted well enough, but it IS stated that all components
of a pipeline run in parallel, so it is logically deducible that this must
involve separate processes.

If you need to separate the output of `set -x` from your script's other
output, consider setting BASH_XTRACEFD:

exec 3>> /path/to/xtrace-logfile.txt
BASH_XTRACEFD=3
set -x
source your_file
set +x
exec 3>&-

or in recent versions of Bash:

exec {BASH_XTRACEFD}>> /path/to/xtrace-logfile.txt
set -x
source your_file
set +x
exec {BASH_XTRACEFD}>&-

(This lets Bash choose an available filedescriptor, rather than hard-coding
"3")

-Martin

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Robert Elz]

Date:Thu, 14 Nov 2024 00:30:32 +0100
From:"#!microsuxx" 
Message-ID:  



  | i just dunno the exec cmd to bring back the fd 1 and 2 after usage back to
  | tty , or restore from saved

Assuming that this is even slightly relevant to the question here (whether
any of this is really necessary), that you accomplish by a sequence like:

exec 7<&0 8>&1 9>&2
exec 0<... 1>... 2>...   # (whatever needs to be done)

# commands to use the modified
# stdin, stdout, stderr, (or some subset)
# go here

exec <&7 >&8 2>&9  7<&- 8>&- 9>&-   # to restore things to as they 
were.

You can use anything you like instead of 7, 8 & 9, they just need to be
unused file descriptors that the shell you're using will allow you to
access - all shells allow fd's 0..9 to be used by scripts (but 0, 1 & 2 are
already in use), some allow lots more than that.  You don't need to save
or restore anything you don't need to modify, obviously (ie: don't bother
saving stdin and then not assigning it elsewhere, and if you don't save
it don't attempt to restore it later).

kre

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

another code is

if u wanna not use . or eval or bash cmd
( . is source )

shopt -s expand_aliases
alias script=" $( < user.bash ) "
script

On Thu, Nov 14, 2024, 1:10 AM #!microsuxx  wrote:

> also u need to $log
> may i ask what criteria ur scripts are about
>
> a $log can be
>
> log=/tmp/my.log.$USER.$SRANDOM
>
> On Thu, Nov 14, 2024, 1:04 AM #!microsuxx  wrote:
>
>> maybe ur email formats it wrong
>> those are two lines , by greg
>> not one
>>
>> anyway if u append more than one cmd to logfile
>> more than one source cmd
>> u need to use >>logfile instead of >logfile
>>
>> On Thu, Nov 14, 2024, 1:00 AM #!microsuxx  wrote:
>>
>>>
>>>
>>> On Thu, Nov 14, 2024, 12:57 AM Yuri  wrote:
>>>
 On 11/13/24 15:44, Greg Wooledge wrote:
 > If the "user script" runs quickly enough, then: source userscript
 > >logfile 2>&1 cat logfile


 This would fail to save the user script's output in case it would
 execute "exit 1"

>>>
>>> the >logfile makes it overwrite
>>> just from the two quoted lines
>>>


 Yuri

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Greg Wooledge]

On Wed, Nov 13, 2024 at 15:56:57 -0800, Yuri wrote:
> On 11/13/24 15:44, Greg Wooledge wrote:
> > If the "user script" runs quickly enough, then: source userscript
> > >logfile 2>&1 cat logfile
> 
> 
> This would fail to save the user script's output in case it would execute
> "exit 1"

Then don't do that.

These questions with unclear mission statements and moving goal posts
are extremely frustrating.

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

i agree
its hard to , say , comm clearly together

i suggest for next times more real life true examples
op's msgs werent so much detailed , more bla-neric :))

greets all

On Thu, Nov 14, 2024, 1:35 AM Robert Elz  wrote:

> Date:Wed, 13 Nov 2024 15:18:58 -0800
> From:Yuri 
> Message-ID:  
>
>   | I need to (1) source the user script, (2) save this script's output to
> a
>   | dedicated file, (3) keep the same output in stdout, and (4) keep
>   | environment variables that the user script sets for later commands.
>   | It doesn't look like any of the above 3 lines do all these 4
> requirements.
>
> Sometimes you can't always have what you need - there probably are
> methods here, but before solving this problem someone should find out
> whether it really needs to be solved.
>
> That is, far too often people determine what they think they need to do
> to solve some problem,  discover that what they think they need to do is
> hard, so ask for help doing that hard thing - often with mixed results.
>
> If there was an explanation of what the issue really is that led you to
> believe that you need those 4 things, perhaps someone might be able to
> show you that you really don't - that this is not the correct approach
> to solving the real problem, and there is a much better way.
>
> But as long as you continue chasing down this rabbit hole, that will
> never happen.
>
> kre
>
>
>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Robert Elz]

Date:Wed, 13 Nov 2024 15:18:58 -0800
From:Yuri 
Message-ID:  

  | I need to (1) source the user script, (2) save this script's output to a 
  | dedicated file, (3) keep the same output in stdout, and (4) keep 
  | environment variables that the user script sets for later commands.
  | It doesn't look like any of the above 3 lines do all these 4 requirements.

Sometimes you can't always have what you need - there probably are
methods here, but before solving this problem someone should find out
whether it really needs to be solved.

That is, far too often people determine what they think they need to do
to solve some problem,  discover that what they think they need to do is
hard, so ask for help doing that hard thing - often with mixed results.

If there was an explanation of what the issue really is that led you to
believe that you need those 4 things, perhaps someone might be able to
show you that you really don't - that this is not the correct approach
to solving the real problem, and there is a much better way.

But as long as you continue chasing down this rabbit hole, that will
never happen.

kre

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

also u need to $log
may i ask what criteria ur scripts are about

a $log can be

log=/tmp/my.log.$USER.$SRANDOM

On Thu, Nov 14, 2024, 1:04 AM #!microsuxx  wrote:

> maybe ur email formats it wrong
> those are two lines , by greg
> not one
>
> anyway if u append more than one cmd to logfile
> more than one source cmd
> u need to use >>logfile instead of >logfile
>
> On Thu, Nov 14, 2024, 1:00 AM #!microsuxx  wrote:
>
>>
>>
>> On Thu, Nov 14, 2024, 12:57 AM Yuri  wrote:
>>
>>> On 11/13/24 15:44, Greg Wooledge wrote:
>>> > If the "user script" runs quickly enough, then: source userscript
>>> > >logfile 2>&1 cat logfile
>>>
>>>
>>> This would fail to save the user script's output in case it would
>>> execute "exit 1"
>>>
>>
>> the >logfile makes it overwrite
>> just from the two quoted lines
>>
>>>
>>>
>>> Yuri
>>>
>>>
>>>
>>>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

maybe ur email formats it wrong
those are two lines , by greg
not one

anyway if u append more than one cmd to logfile
more than one source cmd
u need to use >>logfile instead of >logfile

On Thu, Nov 14, 2024, 1:00 AM #!microsuxx  wrote:

>
>
> On Thu, Nov 14, 2024, 12:57 AM Yuri  wrote:
>
>> On 11/13/24 15:44, Greg Wooledge wrote:
>> > If the "user script" runs quickly enough, then: source userscript
>> > >logfile 2>&1 cat logfile
>>
>>
>> This would fail to save the user script's output in case it would
>> execute "exit 1"
>>
>
> the >logfile makes it overwrite
> just from the two quoted lines
>
>>
>>
>> Yuri
>>
>>
>>
>>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

On Thu, Nov 14, 2024, 12:57 AM Yuri  wrote:

> On 11/13/24 15:44, Greg Wooledge wrote:
> > If the "user script" runs quickly enough, then: source userscript
> > >logfile 2>&1 cat logfile
>
>
> This would fail to save the user script's output in case it would
> execute "exit 1"
>

the >logfile makes it overwrite
just from the two quoted lines

>
>
> Yuri
>
>
>
>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Greg Wooledge]

On Wed, Nov 13, 2024 at 15:18:58 -0800, Yuri wrote:
> I need to (1) source the user script, (2) save this script's output to a
> dedicated file, (3) keep the same output in stdout, and (4) keep environment
> variables that the user script sets for later commands.

If the "user script" runs quickly enough, then:

source userscript >logfile 2>&1
cat logfile

If the user script take a long time to run, and you need the output
in the terminal in real time, then there isn't any simple answer I'm
aware of.  You're going to be looking at variants of:

exec 3> >(tee logfile)
source userscript >&3 2>&3
exec 3>&-

There will be visible "cracks" in all of these, if you look hard enough
for them.  But maybe they'll be good enough for your purposes.

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

try ..

exec 3> >( tee -a log ) # or 3>>
. user.bash >&3
cmds ; cmds ; vars ; cmds

On Thu, Nov 14, 2024, 12:56 AM #!microsuxx  wrote:

> u need to , save out of . script1 to log1 but what was your second
> sentense alike of this first one ?
> u run source over log1 to produce log2 ?
>
> i thought ur looking for errors and for this set -x and stderr ? no need
> for both ?
>
> i need some details clarified ..
>
> On Thu, Nov 14, 2024, 12:48 AM Yuri  wrote:
>
>> Hi !microsuxx,
>>
>>
>> But I need to save the output of the user script into a dedicated log
>> file.
>> This script should run, should save its output into a dedicated log, and
>> then many other commands should use these environment variables.
>>
>> Their logs can't be combined into one.
>>
>>
>> Yuri
>>
>>
>>
>> On 11/13/24 15:36, #!microsuxx wrote:
>> > what u need to do with the vars include all code in tee or try exec >
>> > >( tee -a log ) 2>&1 set -x . my.bash ... what i firstly meant :
>> > #!/bin/bash tee -a log < <( all code here set -x ; . code ; other code
>> > that needs the vars set maybe needs a 2> smth ) try ( set -x . mybash
>> > vars_code ) |& tee -a log
>>
>>
>>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Yuri]

On 11/13/24 15:44, Greg Wooledge wrote:
If the "user script" runs quickly enough, then: source userscript 
>logfile 2>&1 cat logfile



This would fail to save the user script's output in case it would 
execute "exit 1"





Yuri

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

u need to , save out of . script1 to log1 but what was your second sentense
alike of this first one ?
u run source over log1 to produce log2 ?

i thought ur looking for errors and for this set -x and stderr ? no need
for both ?

i need some details clarified ..

On Thu, Nov 14, 2024, 12:48 AM Yuri  wrote:

> Hi !microsuxx,
>
>
> But I need to save the output of the user script into a dedicated log file.
> This script should run, should save its output into a dedicated log, and
> then many other commands should use these environment variables.
>
> Their logs can't be combined into one.
>
>
> Yuri
>
>
>
> On 11/13/24 15:36, #!microsuxx wrote:
> > what u need to do with the vars include all code in tee or try exec >
> > >( tee -a log ) 2>&1 set -x . my.bash ... what i firstly meant :
> > #!/bin/bash tee -a log < <( all code here set -x ; . code ; other code
> > that needs the vars set maybe needs a 2> smth ) try ( set -x . mybash
> > vars_code ) |& tee -a log
>
>
>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Yuri]

Hi !microsuxx,


But I need to save the output of the user script into a dedicated log file.
This script should run, should save its output into a dedicated log, and 
then many other commands should use these environment variables.


Their logs can't be combined into one.


Yuri



On 11/13/24 15:36, #!microsuxx wrote:
what u need to do with the vars include all code in tee or try exec > 
>( tee -a log ) 2>&1 set -x . my.bash ... what i firstly meant : 
#!/bin/bash tee -a log < <( all code here set -x ; . code ; other code 
that needs the vars set maybe needs a 2> smth ) try ( set -x . mybash 
vars_code ) |& tee -a log 

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

if u dont need stderr even simpler

On Thu, Nov 14, 2024, 12:36 AM #!microsuxx  wrote:

> what u need to do with the vars
> include all code in tee
> or try
>
> exec > >( tee -a log ) 2>&1
> set -x
> . my.bash
> ...
>
> what i firstly meant :
>
> #!/bin/bash
>  tee -a log < <(
> all code here
> set -x ; . code ; other code that needs the vars set
> maybe needs a 2> smth
>  )
>
> try
>
>  (
> set -x
> . mybash
> vars_code
>  ) |& tee -a log
>
>
> On Thu, Nov 14, 2024, 12:19 AM Yuri  wrote:
>
>> On 11/13/24 14:48, #!microsuxx wrote:
>> > 1st source cmd , no extra cmds 2. { . foo ; code ; code ; } | tee 3rd
>> > tee < <( . foo ; cmds )
>>
>>
>> I need to (1) source the user script, (2) save this script's output to a
>> dedicated file, (3) keep the same output in stdout, and (4) keep
>> environment variables that the user script sets for later commands.
>> It doesn't look like any of the above 3 lines do all these 4 requirements.
>>
>>
>> Yuri
>>
>>
>>
>>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

what u need to do with the vars
include all code in tee
or try

exec > >( tee -a log ) 2>&1
set -x
. my.bash
...

what i firstly meant :

#!/bin/bash
 tee -a log < <(
all code here
set -x ; . code ; other code that needs the vars set
maybe needs a 2> smth
 )

try

 (
set -x
. mybash
vars_code
 ) |& tee -a log


On Thu, Nov 14, 2024, 12:19 AM Yuri  wrote:

> On 11/13/24 14:48, #!microsuxx wrote:
> > 1st source cmd , no extra cmds 2. { . foo ; code ; code ; } | tee 3rd
> > tee < <( . foo ; cmds )
>
>
> I need to (1) source the user script, (2) save this script's output to a
> dedicated file, (3) keep the same output in stdout, and (4) keep
> environment variables that the user script sets for later commands.
> It doesn't look like any of the above 3 lines do all these 4 requirements.
>
>
> Yuri
>
>
>
>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

another solution
if u parse output u can intercept and parse without piping

 exec 2> >( stderr_parser_cmd ) 1> >( stdout parser cmd )

or both in one
or or

another one :

exec >>log 2>&1
set -x
. cmd

i just dunno the exec cmd to bring back the fd 1 and 2 after usage back to
tty , or restore from saved

u can also

eval " &( < a.bash ) "

and anotherone is using $LINENO or some similiar value , on error

On Thu, Nov 14, 2024, 12:09 AM #!microsuxx  wrote:

> source is not tty dependand , so try ..
>
>  tee -a log < <(
> exec 2>&1
> set -x
> . user.bash
>  )
>
> On Wed, Nov 13, 2024, 11:52 PM Yuri  wrote:
>
>> On 11/13/24 14:45, #!microsuxx wrote:
>> > depending on actual purpose instead bs demo code , there are serval
>> > approaches to code running code
>>
>>
>> The original code in my project runs 'source x.sh > log' where x.sh is
>> some user-provided script.
>>
>> I wanted to trace the code using 'set -x' in order to report errors
>> locations in scripts to the user.
>> However, the trace also goes to log instead of stdout.
>> When I changed that to 'source x.sh | tee log' - environment variables
>> that the user script x.sh also sets disappeared due to the problem in
>> SUBJECT.
>>
>>
>> This problem makes it very inconvenient to implement features around
>> scripts using 'source'.
>>
>>
>>
>> Yuri
>>
>>
>>
>>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Yuri]

On 11/13/24 14:48, #!microsuxx wrote:
1st source cmd , no extra cmds 2. { . foo ; code ; code ; } | tee 3rd 
tee < <( . foo ; cmds ) 



I need to (1) source the user script, (2) save this script's output to a 
dedicated file, (3) keep the same output in stdout, and (4) keep 
environment variables that the user script sets for later commands.

It doesn't look like any of the above 3 lines do all these 4 requirements.


Yuri

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

source is not tty dependand , so try ..

 tee -a log < <(
exec 2>&1
set -x
. user.bash
 )

On Wed, Nov 13, 2024, 11:52 PM Yuri  wrote:

> On 11/13/24 14:45, #!microsuxx wrote:
> > depending on actual purpose instead bs demo code , there are serval
> > approaches to code running code
>
>
> The original code in my project runs 'source x.sh > log' where x.sh is
> some user-provided script.
>
> I wanted to trace the code using 'set -x' in order to report errors
> locations in scripts to the user.
> However, the trace also goes to log instead of stdout.
> When I changed that to 'source x.sh | tee log' - environment variables
> that the user script x.sh also sets disappeared due to the problem in
> SUBJECT.
>
>
> This problem makes it very inconvenient to implement features around
> scripts using 'source'.
>
>
>
> Yuri
>
>
>
>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

eg in ur puroose u dont need source
and in ur code when u understood | othercmd is in another subshell , u see
it doesnt well
if u pipe to tee , .. well the scope of needed

1st source cmd , no extra cmds
2. { . foo ; code ; code ; } | tee
3rd tee < <( . foo ; cmds )

On Wed, Nov 13, 2024, 11:45 PM #!microsuxx  wrote:

> depending on actual purpose instead bs demo code , there are serval
> approaches to code running code
>
> On Wed, Nov 13, 2024, 11:13 PM Yuri  wrote:
>
>> On 11/13/24 14:02, Greg Wooledge wrote:
>> > The commands within a pipeline are executed in subshells (child
>> > processes), so all variable changes are discarded when the subshell
>> exits.
>>
>>
>> This sounds like an implementation detail that should be invisible that
>> affects how the 'source' feature works.
>> This makes it a very confusing from the user's perspective.
>>
>>
>>
>>
>> Yuri
>>
>>
>>
>>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Yuri]

On 11/13/24 14:45, #!microsuxx wrote:
depending on actual purpose instead bs demo code , there are serval 
approaches to code running code



The original code in my project runs 'source x.sh > log' where x.sh is 
some user-provided script.


I wanted to trace the code using 'set -x' in order to report errors 
locations in scripts to the user.

However, the trace also goes to log instead of stdout.
When I changed that to 'source x.sh | tee log' - environment variables 
that the user script x.sh also sets disappeared due to the problem in 
SUBJECT.



This problem makes it very inconvenient to implement features around 
scripts using 'source'.




Yuri

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[#!microsuxx]

depending on actual purpose instead bs demo code , there are serval
approaches to code running code

On Wed, Nov 13, 2024, 11:13 PM Yuri  wrote:

> On 11/13/24 14:02, Greg Wooledge wrote:
> > The commands within a pipeline are executed in subshells (child
> > processes), so all variable changes are discarded when the subshell
> exits.
>
>
> This sounds like an implementation detail that should be invisible that
> affects how the 'source' feature works.
> This makes it a very confusing from the user's perspective.
>
>
>
>
> Yuri
>
>
>
>

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Yuri]

On 11/13/24 14:02, Greg Wooledge wrote:
The commands within a pipeline are executed in subshells (child 
processes), so all variable changes are discarded when the subshell exits.



This sounds like an implementation detail that should be invisible that 
affects how the 'source' feature works.

This makes it a very confusing from the user's perspective.




Yuri

Re: The 'source x' command doesn't keep variables set by x when source output is piped into other command

[Greg Wooledge]

On Wed, Nov 13, 2024 at 13:48:18 -0800, Yuri wrote:
> # source is piped into tee
> source child.sh | tee /dev/null
> echo "XVAR=$XVAR"
> echo "YVAR=$YVAR"
> 
> # source is un-piped
> source child.sh
> echo "XVAR=$XVAR"
> echo "YVAR=$YVAR"

> The first 'source' command didn't set variables set or exported in the
> child.sh script, even though 'source' is executed in the current script's
> context, and it should behave as if these commands were executed in-place.

The commands within a pipeline are executed in subshells (child processes),
so all variable changes are discarded when the subshell exits.

See:
https://mywiki.wooledge.org/BashFAQ/024
https://mywiki.wooledge.org/BashFAQ/106