Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-24 Thread Koichi Murase
2024年5月25日(土) 8:45 Lawrence Velázquez :
> On Fri, May 24, 2024, at 9:04 AM, Koichi Murase wrote:
> > 2024年5月24日(金) 19:18 Martin D Kealey :
> >> On Tue, 21 May 2024 at 23:16, Koichi Murase  wrote:
> >>> However, I personally do not think the FPATH mechanism is useful
> >>> because a file can only contain one function per file. Significantly
> >>> non-trivial functions are usually implemented by a set of helper
> >>> functions or sub-functions.
> >>
> >> Defining extra (private) functions in a file loaded from FPATH does no 
> >> harm,
> >> as long as its name doesn't conflict.
> >
> > [...] and registers it through `autoload -U func', it would result in a big 
> > function
> > containing function definitions:
> >
> >   func() { internal1() { ...; }; internal2() { ...; }; func() { ...; }; }
> >
> > This is not an expected one.
>
> Moving the extra "func" call into the definition itself takes that
> off the caller's hands [...]
>
> % cat /tmp/func
> internal1() { echo internal1 }
> internal2() { echo internal2 }
> func() { internal1; internal2 }
> func

I thought we couldn't include multiple public functions and create
symbolic links that way.

However, I now tried $funcstack, and it seems to be working:

  internal1() { echo i1; }
  internal2() { echo i2; }
  public1() { printf p1; (($#)) && printf '<%s>' "$@"; echo; internal1; }
  public2() { printf p2; (($#)) && printf '<%s>' "$@"; echo;
internal1; internal2; }
  $funcstack[1] "$@"

> while maintaining the ability to use "func"
> as a standalone script (the motivation for the zsh style).

I see, I was wondering about the motivation.

--
Koichi



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-24 Thread Lawrence Velázquez
On Fri, May 24, 2024, at 9:04 AM, Koichi Murase wrote:
> 2024年5月24日(金) 19:18 Martin D Kealey :
>> On Tue, 21 May 2024 at 23:16, Koichi Murase  wrote:
>>> However, I personally do not think the FPATH mechanism is useful
>>> because a file can only contain one function per file. Significantly
>>> non-trivial functions are usually implemented by a set of helper
>>> functions or sub-functions.
>>
>> Defining extra (private) functions in a file loaded from FPATH does no harm,
>> as long as its name doesn't conflict.
>
> I was assuming the Zsh's implementation of $FPATH.  I thought the Zsh
> implementation of $FPATH doesn't allow it, but I now learned that ksh's
> implementation of $FPATH is different.  I'm not a user of Zsh, so maybe I miss
> something, but Zsh's default autoload assumes that each file contains the
> *body* of the function (instead of the function declaration).  So if one puts 
> a
> set of functions in a file `func':
>
>   internal1() { ...; }; internal2() { ...; }; func() { ...; }
>
> and registers it through `autoload -U func', it would result in a big function
> containing function definitions:
>
>   func() { internal1() { ...; }; internal2() { ...; }; func() { ...; }; }
>
> This is not an expected one.

On first run "func" would define all "inner" functions -- including
replacing itself -- and subsequently behave just as if it had been
autoloaded ksh-style.  So it's possible to define helper functions
in zsh-style autoloaded functions, but it goes against the grain a bit.

% cat /tmp/func
internal1() { echo internal1 }
internal2() { echo internal2 }
func() { internal1; internal2 }
% (autoload -Uk /tmp/func; func)
internal1
internal2
% (autoload -Uz /tmp/func; func; func)
internal1
internal2

Moving the extra "func" call into the definition itself takes that
off the caller's hands while maintaining the ability to use "func"
as a standalone script (the motivation for the zsh style).

% cat /tmp/func
internal1() { echo internal1 }
internal2() { echo internal2 }
func() { internal1; internal2 }
func
% autoload -U /tmp/func
% func
internal1
internal2

-- 
vq



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-24 Thread Koichi Murase
2024年5月24日(金) 19:18 Martin D Kealey :
> On Tue, 21 May 2024 at 23:16, Koichi Murase  wrote:
>> 2024年5月21日(火) 14:56 Phi Debian :
>> > 'May be' bash could investigate the ksh93/zsh $FPATH autoload, but don't
>> > know if that would be good enough for the initial purpose.
>>
>> There are already shell-function implementations at
>> /examples/functions/autoload* in the Bash source. They reference FPATH
>> to load functions, though one needs to call `autoload' for each
>> function in advance (by e.g. `autoload "$fpath_element"/*' ).
>
> My solution to this was to call 'autoload --all', which would gather all the
> filenames in FPATH (*1) and create autoload stubs for them.

Thank you for your comments.

These were just a comment on the implementations /example/functions/autoload*,
but yeah, I haven't noticed that `autoload.v4' actually supports the equivalent
option `autoload -f'.  If you would define your own `autoload', of course, you
can define the useful feature.

I think there is still a minor (and maybe negligible?) issue when a new path is
added to FPATH after performing `autoload --all'.  One would need to
additionally call `autoload "$fpath_added_element"/*' (or one could run
`autoload --all' again, but this would scan all the directories that were
already processed).

> Alternatively one could define a commandnotfound function to defer this until
> actually needed.

I thought about it before, but `command_not_found_handle' is executed in a
subshell, so the loaded functions will not be reflected in the original shell
process.

> (*1 I actually used a different variable name, since I wasn't providing
> exactly the same semantics as ksh, but that's relatively cosmetic)

I intended to talk about the specific semantics of ksh93/zsh's FPATH, but if we
extend the discussion to different variable names and different semantics, I
actually have my own `autoload' [1] referring the variable
`bleopt_import_path', where the function names are specified to `autoload' with
an explicit module name defining them.  An example use is found in
e.g. Ref. [2], although the module is specified by an absolute path in this
case.

[1] 
https://github.com/akinomyoga/ble.sh/blob/b72d78a97e6867bbf43de6a645dd6e387525443f/src/util.sh#L5071
[2] 
https://github.com/akinomyoga/ble.sh/blob/b72d78a97e6867bbf43de6a645dd6e387525443f/lib/core-syntax-def.sh#L62-L79

>> However, I personally do not think the FPATH mechanism is useful
>> because a file can only contain one function per file. Significantly
>> non-trivial functions are usually implemented by a set of helper
>> functions or sub-functions.
>
> Defining extra (private) functions in a file loaded from FPATH does no harm,
> as long as its name doesn't conflict.

I was assuming the Zsh's implementation of $FPATH.  I thought the Zsh
implementation of $FPATH doesn't allow it, but I now learned that ksh's
implementation of $FPATH is different.  I'm not a user of Zsh, so maybe I miss
something, but Zsh's default autoload assumes that each file contains the
*body* of the function (instead of the function declaration).  So if one puts a
set of functions in a file `func':

  internal1() { ...; }; internal2() { ...; }; func() { ...; }

and registers it through `autoload -U func', it would result in a big function
containing function definitions:

  func() { internal1() { ...; }; internal2() { ...; }; func() { ...; }; }

This is not an expected one.  However, TIL one can put function definitions in
ksh's style, and ksh's style can be used also in Zsh with `autoload -Uk func'.

>> Also, in libraries, we usually have a set
>> of functions that are closely related to one another and share the
>> implementations. I don't think it is practical to split those
>> functions into dozens or hundreds of files.
>
> I would hesitate to call what I've done "a manager", but my approach has been
> to allow a file to "declare" all the public functions it defines, and then
> simply have symlinks (or even hard links) to a single underlying file.

OK, this should work as ksh's FPATH implementation allows including function
definitions.

In that sense, this is just my personal preference, but I prefer to prepare a
`header' script file (such as [2]) that contains `autoload' declarations and is
intended to be sourced on the initialization stage instead of creating as many
symbolic links as the public functions.

--
Koichi



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-24 Thread Martin D Kealey
On Tue, 21 May 2024 at 23:16, Koichi Murase  wrote:

> 2024年5月21日(火) 14:56 Phi Debian :
> > 'May be' bash could investigate the ksh93/zsh $FPATH autoload, but don't
> > know if that would be good enough for the initial purpose.
>
> There are already shell-function implementations at
> /examples/functions/autoload* in the Bash source. They reference FPATH
> to load functions, though one needs to call `autoload' for each
> function in advance (by e.g. `autoload "$fpath_element"/*' ).
>

My solution to this was to call 'autoload --all', which would gather all
the filenames in FPATH (*1) and create autoload stubs for them.

Alternatively one could define a commandnotfound function to defer this
until actually needed.

(*1 I actually used a different variable name, since I wasn't providing
exactly the same semantics as ksh, but that's relatively cosmetic)

However, I personally do not think the FPATH mechanism is useful
> because a file can only contain one function per file. Significantly
> non-trivial functions are usually implemented by a set of helper
> functions or sub-functions.


Defining extra (private) functions in a file loaded from FPATH does no
harm, as long as its name doesn't conflict.


> Also, in libraries, we usually have a set
> of functions that are closely related to one another and share the
> implementations. I don't think it is practical to split those
> functions into dozens or hundreds of files.


I would hesitate to call what I've done "a manager", but my approach has
been to allow a file to "declare" all the public functions it defines, and
then simply have symlinks (or even hard links) to a single underlying file.

I copied Perl and named my command "require" because it's a run-time check,
not a parse-time one.

Each file should contain a "provides" statement for each (public) function
it defines; except it's optional for a function name that matches the only
possible "require" name. Using this mechanism it's also possible to require
a module rather than an individual function.

The autoloader stubs I mentioned above is just:
  func() { require --from=/path/to/library/func.bash func && func "$@" ; }

(In order to get the correct search precedence, these would need to be
generated by scanning FPATH in reverse order, so that later ones will be
replaced by earlier ones; amongst other things, I define my own FPATH-like
var to have the reverse precedence order.)

Originally I just made autoloader stubs like
  func() { source /path/to/library/func.bash && func "$@" ; }
however it turned out that using "require" instead of "source" simplified
"provides" since it could then rely on local variables from "require" being
available.

It would also be slow to
> read many different files, which requires access to random positions
> on the disk.
>

My approach allows multiple functions per file, and only loads each file
once, no matter how many names it has.

-Martin


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-21 Thread Martin D Kealey
On Tue, 21 May 2024 at 03:44, Chet Ramey  wrote:

> On 5/17/24 1:15 PM, Robert Elz wrote:
>
> >| If `nosort' means no sorting, there is no imposed ordering, and
> ascending
> >| and descending are meaningless.
> >
> > Sure, but directory order, and reverse directory order aren't (and that's
> > just a difference between the order in which you create the list as each
> > new dirent is read from the directory - does it go at the head or tail).
>
> That's changing from one random order to another.
>

In the *general* case yes, the order should be treated as random.
For example, readdir on a Linux ext4 fs returns directory entries in
pseudo-random order; this is necessary to allow successive readdir calls to
traverse a large directory, without skipping or repeating an existing
entries, and without locking (so allowing additions, removals, and
renamings to continue). (Connected to this, file positions returned by
lseek(dir_fd, 0, SEEK_CUR) are pseudorandom numbers identifying the next
directory entry to be fetched, not an indication of bytes read so far.)

But it's not *always* random.
For example, readdir on Linux's /proc/$pid/fd returns filedescriptors in
ascending numerical order; likewise readdir on /proc returns process IDs in
ascending numerical order (preceded by a bunch of other stuff, in an order
that's well-defined in the kernel though perhaps not obvious to most users).
Reversing those could be useful to some people.

-Martin


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-21 Thread Chet Ramey

On 5/21/24 1:55 AM, Phi Debian wrote:

The idea started before the current discussion as a way to avoid
overloading PATH in the source builtin:

https://lists.gnu.org/archive/html/help-bash/2024-04/msg00022.html



Some thought 'source' was a good candidate for this, except that source is
a read/parse/eval feature while the intention was more a read/parse for the
benefit of the 'importer'


You can't import function definitions without the `eval' step.



so construct like
hobbit:~$ PATH=. source bar

is perfectly ledgit if we are honest and admit bar is an import only and
execute no code, simply bring functions.

So PATH=$BASH_SOURCE_DIR . greatlib

is the way to go, still assuming libs are functions/vars only and no code
execution.

As soon as the 'library' concept assume that eval is also possible, along
with read/parse, then kaboom we need PATH back so the
PATH=$PATH:/path/to/lib .


There are workarounds for this, but they would require these package
managers to change.


Real life package actually require the eval, because it is the way to
configure a 'library' depending the user context.


Same.



'May be' bash could investigate the ksh93/zsh $FPATH autoload, but don't
know if that would be good enough for the initial purpose.


Apparently not, since there's been an example autoload function in the tree
for 30 years.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-21 Thread Phi Debian
On Tue, May 21, 2024 at 1:17 PM Koichi Murase 
wrote:


> There are already shell-function implementations at
> /examples/functions/autoload* in the Bash source. They reference FPATH
> to load functions, though one needs to call `autoload' for each
> function in advance (by e.g. `autoload "$fpath_element"/*' ).
>

> However, I personally do not think the FPATH mechanism is useful
> because a file can only contain one function per file. Significantly
> non-trivial functions are usually implemented by a set of helper
> functions or sub-functions. Also, in libraries, we usually have a set
> of functions that are closely related to one another and share the
> implementations. I don't think it is practical to split those
> functions into dozens or hundreds of files. It would also be slow to
> read many different files, which requires access to random positions
> on the disk.
>

Jeez, I forgot about this examples/functions/autoload.

First of all thanx for pointing it, it demonstrate that things can be
implemented as bash code (not C hack in bash) at least in the experimental
phase.

Second, there is no such limitation about 1 source 1 function mapping,
since it is sourced any functions defines in the files become available.
This open two path.

1 package setup function is autoloaded bringing all the other function that
are now callable

Say file foo defines foo() bar() then autoload foo() brings both foo() and
bar(), if foo is the setup thing for file foo then the doc sez autoload foo
to bring up bar() along.

Another path is to have a symlink on each 'exported' function on the file
foo, so a ln -s foo bar would make bar autoloadable. Note I used 'exported'
here meaning the foo file may define a lot of function, only the exposed
API one need a symlink.

The real divergence from this autoload, and the ksh93 one is that ksh93
will try an autoload on foo using FPATH on the 'command-not-found'
situation, something bash don't handle in the user context (it does it in a
subshell for unknown reasons).
So basically in ksh93 script, beside setting FPATH, you simply call
functions directly, on command-not-found then try FPATH to find your
function and if found proceed with the load, define func, run setup code,
and run the func with parameters...

This is neat, but the fact we don't have that on bash is no big deal, we
can live without it, again no need to break things that works.


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-21 Thread Chet Ramey

On 5/21/24 12:49 AM, konsolebox wrote:


`Properly' is clearly in the eye of the beholder.


I'm sorry if I have to sound assertive on this one but I thoroughly
thought about these things years ago and made a framework that works
on a multitude of shells, allows both simple and pattern-based
targets, and can be compiled (merged into a single file) with simple
usage.  Not to mention I actually wrote a multi-script application I
know what people who write one want. 


Thanks, this proves my point. You clearly have your idea about what makes
such a feature valuable.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-21 Thread Phi Debian
On Tue, May 21, 2024 at 1:15 PM Greg Wooledge  wrote:

> On Tue, May 21, 2024 at 10:12:55AM +, Matheus Afonso Martins Moreira
> wrote:
> > > the schizophrenic nature of the feature
> >
> > First the feature was "irritating" ... Now it's "schizophrenic" ?
> > I must be mentally ill for trying to contribute this?
> >
> > Yeah, I'm done.
>
> I don't think "schizophrenic" was used as an insult.  Rather, it looks
> like an attempt to describe the fact that everyone in the thread has a
> different concept and/or experience regarding how 'source' and '.'
> should work.
>
>
Well, what I really meant was that there are 2 directions one with
read/parse the other read/parse/eval and not being able to make a mind,
like the Buridan ass
https://en.wikipedia.org/wiki/Buridan%27s_ass

'import' read/parse (along with a PATH var) make sense and don't break
interfere with source

A 'source' read/parse/eval is there with its semantic and should not go
away.

An 'import' with source look alike and tweaking its flags and semantic is
border line, besides, it and can be implemented in shell only with no bash
modification, there is no perf consideration here, we are not on the perf
path, unless one come up with a source in the inner loop :-)

No offense, just pointing that these two path sounds not mixable, an
implementation of your $BASH_SOURCE_PATH can be all done in shell in one of
your libs, (say your bootstrap lib) whence this one is loaded, it can load
all the others.


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-21 Thread alex xmb sw ratchev
On Tue, May 21, 2024, 13:18 Koichi Murase  wrote:

> 2024年5月21日(火) 14:56 Phi Debian :
> > Apparently konsolebox wrote a package manager, and survived the source
> 'as
> > is', yet he still advocate for 'enhancement'. I do have a package manager
> > too, and survived source 'as is' and don't require any enhancement.
>
> I'm not sure if it is the case for konsolebox, but I think it is valid
> to suggest including the features that turned out to be useful in
> maintaining a package manager as built-in ones. Those suggested
> features are meant to replicate the useful part of the features of the
> package manager, so it is not surprising that the suggested features
> are not needed for the package manager itself.
>
> That said, I think it would be difficult to reach an agreement on the
> detailed behavior of the suggested features to make them consistent
> with many existing package managers.
>
> > 'May be' bash could investigate the ksh93/zsh $FPATH autoload, but don't
> > know if that would be good enough for the initial purpose.
>
> There are already shell-function implementations at
> /examples/functions/autoload* in the Bash source. They reference FPATH
> to load functions, though one needs to call `autoload' for each
> function in advance (by e.g. `autoload "$fpath_element"/*' ).
>
> However, I personally do not think the FPATH mechanism is useful
> because a file can only contain one function per file. Significantly
> non-trivial functions are usually implemented by a set of helper
> functions or sub-functions. Also, in libraries, we usually have a set
> of functions that are closely related to one another and share the
> implementations. I don't think it is practical to split those
> functions into dozens or hundreds of files. It would also be slow to
> read many different files, which requires access to random positions
> on the disk.
>

for the overall thread : sorry , i cant read that much text of ya' all
but i have a question and related information to share

question : is this one-thing-per-file sourcing ?

i made long ago a project that has different dirs , contain different type
of files ( function , alias , .. )
each one per file , and the name of the file , in the dir , is then its
identifier in the code
say proj/func/bla will result in bla() with the files comtent

.. is it such u discuss here ..
if so .. i have big experience and wanna talk more about such
just .. ..

greets ..

--
> Koichi
>
>


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-21 Thread Koichi Murase
2024年5月21日(火) 14:56 Phi Debian :
> Apparently konsolebox wrote a package manager, and survived the source 'as
> is', yet he still advocate for 'enhancement'. I do have a package manager
> too, and survived source 'as is' and don't require any enhancement.

I'm not sure if it is the case for konsolebox, but I think it is valid
to suggest including the features that turned out to be useful in
maintaining a package manager as built-in ones. Those suggested
features are meant to replicate the useful part of the features of the
package manager, so it is not surprising that the suggested features
are not needed for the package manager itself.

That said, I think it would be difficult to reach an agreement on the
detailed behavior of the suggested features to make them consistent
with many existing package managers.

> 'May be' bash could investigate the ksh93/zsh $FPATH autoload, but don't
> know if that would be good enough for the initial purpose.

There are already shell-function implementations at
/examples/functions/autoload* in the Bash source. They reference FPATH
to load functions, though one needs to call `autoload' for each
function in advance (by e.g. `autoload "$fpath_element"/*' ).

However, I personally do not think the FPATH mechanism is useful
because a file can only contain one function per file. Significantly
non-trivial functions are usually implemented by a set of helper
functions or sub-functions. Also, in libraries, we usually have a set
of functions that are closely related to one another and share the
implementations. I don't think it is practical to split those
functions into dozens or hundreds of files. It would also be slow to
read many different files, which requires access to random positions
on the disk.

--
Koichi



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-21 Thread Greg Wooledge
On Tue, May 21, 2024 at 10:12:55AM +, Matheus Afonso Martins Moreira wrote:
> > the schizophrenic nature of the feature
> 
> First the feature was "irritating" ... Now it's "schizophrenic" ?
> I must be mentally ill for trying to contribute this?
> 
> Yeah, I'm done.

I don't think "schizophrenic" was used as an insult.  Rather, it looks
like an attempt to describe the fact that everyone in the thread has a
different concept and/or experience regarding how 'source' and '.'
should work.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-21 Thread Matheus Afonso Martins Moreira
> the schizophrenic nature of the feature

First the feature was "irritating" ... Now it's "schizophrenic" ?
I must be mentally ill for trying to contribute this?

Yeah, I'm done.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Phi Debian
On Mon, May 20, 2024 at 7:54 PM Greg Wooledge  wrote:

> On Mon, May 20, 2024 at 07:43:10PM +0200, Andreas Kähäri wrote:
> > On Mon, May 20, 2024 at 05:31:05PM +, Matheus Afonso Martins Moreira
> wrote:
> >
> >   PATH=${BASH_SEARCH_PATH-$PATH} . file
> >
> > without the need to add any options to . or to source.  But maybe that
> > too pedestrian?
>
> Are we going in circles yet?  This would clobber the value of PATH for
> the duration of sourcing "file", which would potentially cause commands
> in "file" to break.
>
> hobbit:~$ cat bar
> echo hi | cat
> hobbit:~$ PATH=. source bar
> bash: cat: command not found
>

Yes we are probably circling because of the schizophrenic nature of the
feature.

At the beginning, I think not sure I remember well, the whole idea was to
get bash 'libraries' described as bringing more functions written by other
into a runnable (-x) script. This was perceived (I guess) as an 'include'
but since it was function may be as a 'link' or both combined, and as such
required a special treatment ala -I -L compiler like using INC_PATH,
LIB_PATH kind of thing.

Some thought 'source' was a good candidate for this, except that source is
a read/parse/eval feature while the intention was more a read/parse for the
benefit of the 'importer'

so construct like
hobbit:~$ PATH=. source bar

is perfectly ledgit if we are honest and admit bar is an import only and
execute no code, simply bring functions.

So PATH=$BASH_SOURCE_DIR . greatlib

is the way to go, still assuming libs are functions/vars only and no code
execution.

As soon as the 'library' concept assume that eval is also possible, along
with read/parse, then kaboom we need PATH back so the
PATH=$PATH:/path/to/lib .

Real life package actually require the eval, because it is the way to
configure a 'library' depending the user context.

So back to square one, what we need is source as it is and should not be
broken.

Apparently konsolebox wrote a package manager, and survived the source 'as
is', yet he still advocate for 'enhancement'. I do have a package manager
too, and survived source 'as is' and don't require any enhancement.

'May be' bash could investigate the ksh93/zsh $FPATH autoload, but don't
know if that would be good enough for the initial purpose.

Don't know if that FPATH thing was investigated before, if so, why it was
rejected. If good enough and could be adopted, it would be some convergence.


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread konsolebox
On Tue, May 21, 2024 at 4:10 AM Chet Ramey  wrote:
>
> On 5/20/24 3:39 AM, konsolebox wrote:
>
> > It's rare for a topic about adding an import/include function to get
> > so much attention like this I think it's time Bash takes the
> > opportunity to implement it and hopefully properly.
>
> `Properly' is clearly in the eye of the beholder.

I'm sorry if I have to sound assertive on this one but I thoroughly
thought about these things years ago and made a framework that works
on a multitude of shells, allows both simple and pattern-based
targets, and can be compiled (merged into a single file) with simple
usage.  Not to mention I actually wrote a multi-script application I
know what people who write one want.  I found the simplest formula
that isn't limiting and is sound as proven by other shells besides
bash that also work with it.  The framework's design was heavily
forged.

Of course we're only talking about improving Bash here but that just
means I have a solid reference and I know where to simplify things and
where to adapt.  This is why I was able to make the prototype script
quickly the moment I decided to write it instead of a C
implementation.  Deciding what helper function to use was just a
struggle.  Error handling also makes the code verbose it's better
presented in a script first before it's approved.

So what I'm saying is it's not just in everyone's own perception.
It's also in the experience.


-- 
konsolebox



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Koichi Murase
2024年5月17日(金) 4:36 Greg Wooledge :
> On Thu, May 16, 2024 at 11:31:55AM -0400, Chet Ramey wrote:
> > On 5/15/24 11:31 AM, Koichi Murase wrote:
> > > Maybe it was not clear to use `source name' and `source -i name' to
> > > describe the idea. I meant I assumed the search domain being
> > >
> > > * BASH_SOURCE_PATH + PATH + PWD without the option
> >
> > It seems to me that this isn't useful.

I tried to convince myself about the behavior that the existence of
BASH_SOURCE_PATH disables PATH and PWD. However, after rethinking, I
still think there is a compatibility issue with it. I still think the
search domain of the source builtin being

* BASH_SOURCE_PATH + PATH + PWD without the option
* BASH_SOURCE_PATH with the option

> > The whole reason to have and use
> > BASH_SOURCE_PATH is not to use PATH; a fallback to PATH if something
> > isn't found in BASH_SOURCE_PATH doesn't allow that.

For this point, I'm convinced that only providing "BASH_SOURCE_PATH +
PATH + PWD" wouldn't satisfy the whole reason for adding
BASH_SOURCE_PATH. Maybe we need the option.

> > And if you're using
> > BASH_SOURCE_PATH, you have to set it, and if you want `.' in there,
> > add it.
>
> Yes, I'm inclined to agree with this.  If you want it to fall back to
> $PATH, you can append the contents of $PATH to BASH_SOURCE_PATH as well.
> You get the most control if BASH_SOURCE_PATH is used exclusively when
> it's set.

The problem is that this changes the global behavior of the `source'
builtin, and setting BASH_SOURCE_PATH breaks other libraries that
assume PATH.

On Thu, May 16, 2024 at 11:31:55AM -0400, Chet Ramey wrote:
> (from the user's perspective;
> I'm not sure what I think about a loaded `library' changing the global
> user settings by setting it),

If the loaded "library" is a single-file library, there is no reason
for the library to set BASH_SOURCE_PATH. However, the original
motivation for adding BASH_SOURCE_PATH is to provide a mechanism to
specify the directory of a library that consists of many files, where
only the files necessary for the current purpose are loaded. Maybe I
should call it such a library as "a set of libraries" or "a library
framework". Such a library framework for Bash wants to add its own
path to BASH_SOURCE_PATH.

However, another library framework can similarly set PATH to achieve a
similar effect (despite the problem of the name conflicts in the mixed
namespace with the executable files). Such a library framework can be
an old library framework designed for Bash < 5.3. Or it can be a
library framework designed for POSIX sh.

If setting BASH_SOURCE_PATH disables PATH and PWD (on which a library
framework for POSIX sh would rely) in the search for the conventional
use of the `source' builtin, it means that we cannot use the two types
of the library framework in a session or a program: i.e., library
frameworks utilizing BASH_SOURCE_PATH vs library frameworks for POSIX
sh. To allow the existence of both types of library frameworks,

a. The search can fall back to PATH and PWD when the path is not found
in BASH_SOURCE_PATH. When the library framework for Bash wants to
restrict the search, the library can use the option as `source -i'
.etc (the letter `i' can be a different one). The POSIX-sh library
framework can continue to use the plain `source filename' without the
option.

b. Another option would be to require everyone to include `PATH'
and `.' in BASH_SOURCE_PATH. However, if we require that, "the whole
reason to add BASH_SOURCE_PATH" is again lost. In addition, a later
change of PATH wouldn't be reflected in BASH_SOURCE_PATH that way.

c. Even another option is to require people not to globally set
BASH_SOURCE_PATH and only use it through `BASH_SOURCE_PATH=
source filename'. However, I anticipate that if we add this feature,
library frameworks or distribution bashrcs would start to set
BASH_SOURCE_PATH, which would break library frameworks assuming PATH.

d. Is there any better solution under the design that BASH_SOURCE_PATH
disables PATH and PWD even without the option?

I now changed my mind so that the `source' builtin should have a
separate way to restrict the search in BASH_SOURCE_PATH. While
`source' without the option continues to consider also PATH and PWD,
`source' with the option can disable PATH and PWD.

If BASH_SOURCE_PATH is required to disable PATH and PWD in the
conventional uses of the `source' builtin (in the form `source
filename' without the option), maybe I'm opposed to adding
BASH_SOURCE_PATH at all.

--
Koichi



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/20/24 4:17 PM, Matheus Afonso Martins Moreira wrote:

And yet people are so opposed to the option that they want ridiculous
restrictions like `you can never alias source="source -i"'.


Yeah. I already feel like I'm insane for even suggesting it.


People feel strongly, it seems. More strongly than I, at least, guessed.




Because other people don't see the issue the same way you do,
and have their own way of doing things?


I suppose. I tried to argue for my viewpoint but convinced no one.


I had no idea that my prompting discussion would result in this much.

I suppose we've all learned that there are a dozen, all slightly
different, bash library/package managers.


For the record I think `source -p $VAR` is an even better solution
than what I proposed. I can send a patch if necessary.


It has some of the same semantic issues. Does using it disable searching
$PWD? Should it fall back to $PATH? What do you do with a null value? (Yes,
no, same as you do today.) It just avoids introducing a new variable.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Matheus Afonso Martins Moreira
> And yet people are so opposed to the option that they want ridiculous
> restrictions like `you can never alias source="source -i"'.

Yeah. I already feel like I'm insane for even suggesting it.

> Because other people don't see the issue the same way you do,
> and have their own way of doing things?

I suppose. I tried to argue for my viewpoint but convinced no one.
Maybe I just suck at communicating. Either way, there's no point
in me trying to justify this any longer. In the end it's up to you.

For the record I think `source -p $VAR` is an even better solution
than what I proposed. I can send a patch if necessary.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/20/24 3:39 AM, konsolebox wrote:


It's rare for a topic about adding an import/include function to get
so much attention like this I think it's time Bash takes the
opportunity to implement it and hopefully properly.


`Properly' is clearly in the eye of the beholder.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/19/24 6:31 AM, Matheus Afonso Martins Moreira wrote:


There is absolutely no doubt that this is useful.
Not long after I sent the patches, a list of about
a dozen projects implementing module loading
and management in bash was posted.
Clearly this is something people want.


I think the point of that list was to demonstrate that there already exist
solutions to this request, that they have existed in some form for many
years, and your solution would have to accommodate them, or at least learn
from them, in some way.


It cannot be done some other way.
Not without running into limitations.


Obviously these limitations have not proved fatal this far. That doesn't
mean they're irrelevant, only that they're not insurmountable.



That one is simple ... when it ('.' in sh, "source" in csh) was added
there were no functions, while read & eval might have been enough for
  simple one line scripts, they wouldn't work for more complex ones, to
  invent your own '.' command that way you'd need to do it in a moderately
  complex function, which didn't exist at the time.


At the very least you would have had to do something like

data="`cat filename`"
eval "$data"

every time you wanted to source a file.



Well all those components exist now.
Why not use them instead of source?


Because source/. exists, is required, and is not going away. Why not use
it?




I agree that modifying PATH isn't a great solution


Discussing this matter always seems to end like this.
I can't figure out if people actually agree with me or not.
Apparently I'm right but the feature _still_ shouldn't be
added because of reasons I can't seem to understand.


Because the current behavior is understood, and other people don't see a
great enough advantage to use something else.


generally only improves the status quo,


I don't think everyone agrees with this. Certainly it's disputed even in
the discussion here.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/19/24 4:58 AM, Matheus Afonso Martins Moreira wrote:


Bash has the command builtin which can be used to
ensure a real utility is executed instead of an alias or
function. It also works for builtins such as source.
I could also propose a new flag for it that restricts
its operation to builtins only, eliminating the PATH
search and ensuring that builtins cannot be replaced
by a program on the file system either.


This exists as the `builtin' builtin, and has for many years.


I'm not sure what the maintainer would think about it
but I could also propose a shopt that prohibits the
aliasing of builtins and other reserved words.


I would decline, as I have in the past. If you want to prohibit aliasing
reserved words and special builtins, posix mode will do that for you. If
you want to prevent builtin names from being alias expanded, use the
`builtin' builtin.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/18/24 7:00 PM, Matheus Afonso Martins Moreira wrote:

The issue is that what you want, and what I want,
and what konsolebox wants (and everyone else)
are all slighty different.


The option solution which I proposed is not really in opposition
to anything that anyone else wants. It will not break anything,
it supports the same inputs and use cases as before, including
the absolute path case you cited.


I don't think Phi agrees with that.



Pure bash solutions have two choices and both of them break stuff.
Either they clobber PATH, or they resolve the file's path on their own
and pass an absolute path to the source builtin. If you clobber PATH,
sourced scripts can't easily find commands anymore.


No. If you want to run commands out of one of these `library' files, there
are a couple of ways to do it. Either the library can change, or you can
run `PATH=$BASH_SOURCE_PATH:$PATH .' (or similar).


That completely
changes the way that scripts work and therefore the way they must
be written.


Not at all.


If you resolve the path, you're just reinventing the source
builtin. It also depends on source actually accepting the absolute path
which renders it incompatible with restricted shells.


Only if you want to run it after the startup files are read and the
restrictions are enforced. In a restricted environment -- not so common
anymore -- I can't see a use for this after the startup files are read.


The option solution completely avoids all that.


And yet people are so opposed to the option that they want ridiculous
restrictions like `you can never alias source="source -i"'.


I don't really understand why there seems to be so much
opposition to this.


Because other people don't see the issue the same way you do, and have
their own way of doing things?

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/18/24 11:17 AM, Phi Debian wrote:


an enforcement that no
alias source='source -i' could ever be possible.


This seems ridiculous. There should never be a prohibition against a user
doing something that is not dangerous.


All this is because one writing a 'main' script (#!/bin/bash), sourcing a
package of mine through my package management system, expecting the current
source behavior, and later add the loading of a 'libraries' from a friend
(terminal color jazz) that in turn start to mess around with alias
source='source -i' and BASH_SOURCE_PATH cold start to break my way of
finding my packages.


If you're worried about this, unalias source after loading this friend's
`library'. You're already changing your environment to include it.



An alternative to this would be
BASH_SOURCE_PATH="" don't do nothing with 'source', and allow the
'libraries' designer to do
typeset -r BASH_SOURCE_PATH=""


No. If you want to make sure it's unset, force it to remain unset, and
prevent anyone from setting it,

unset BASH_SOURCE_PATH
readonly BASH_SOURCE_PATH

will do the job.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/18/24 8:57 PM, Robert Elz wrote:

 Date:Sat, 18 May 2024 23:00:41 +
 From:Matheus Afonso Martins Moreira 
 Message-ID:  



   | If you resolve the path, you're just reinventing the source builtin.

Nonsense, the real work of the source ('.' really) utility is to
read the file, parse it, and execute it.   None of the proposed
solutions here do that.   What file names work for this purpose is
an irrelevant consideration.

   | It also depends on source actually accepting the absolute path
   | which renders it incompatible with restricted shells.

That's the only point here that has any merit.   And a better solution
to that would be to remove that restriction, which has no purpose that
I can imagine.


It's present because it's always been part of restricted shells. The
historical Bourne shell and Korn shell lines both have it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/17/24 11:40 AM, Robert Elz wrote:


Now of course, everyone is going to say "too difficult to do that all
the time", and they'd be right.  So the one extra feature would be that
if used without the "file" arg (which is normally mandatory), as in
. -p path-like-value
it would set the default search path for later uses of the '.' command
(and if anyone feels that being able to read back what has been set, an
additional option could make that info appear on stdout, and source
nothing).


There is nothing different between that and setting another variable,
one that's user-visible like BASH_SOURCE_PATH, except that it's more
obscure and fragile.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/20/24 1:43 PM, Andreas Kähäri wrote:


Or even

PATH=${BASH_SEARCH_PATH-$PATH} . file

without the need to add any options to . or to source.  


Yeah, that's where we started.

https://lists.gnu.org/archive/html/bug-bash/2024-05/msg00038.html


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Andreas Kähäri
On Mon, May 20, 2024 at 01:54:20PM -0400, Greg Wooledge wrote:
> On Mon, May 20, 2024 at 07:43:10PM +0200, Andreas Kähäri wrote:
> > On Mon, May 20, 2024 at 05:31:05PM +, Matheus Afonso Martins Moreira 
> > wrote:
> > > >> Why not add a -p option to '.' to specify the path to search.
> > > >> That is
> > > >>. -p "${BASH_SEARCH_PATH-${PATH}}" file
> > > >> would work if someone decided to use the
> > > >> BASH_SOURCE_PATH var name to store the path to use
> > > >> (defaulting to $PATH if that one isn't set).
> > > 
> > > > Believe it or not, I had thought of that as well.
> > > > It sidesteps the whole BASH_SOURCE_PATH
> > > > variable brouhaha altogether.
> > > 
> > > I think this is a really good solution. I hadn't thought of it.
> > > Users can even make an alias to set a default for themselves.
> > > 
> > >   -- Matheus
> > 
> > Or even
> > 
> > PATH=${BASH_SEARCH_PATH-$PATH} . file
> > 
> > without the need to add any options to . or to source.  But maybe that
> > too pedestrian?
> 
> Are we going in circles yet?  This would clobber the value of PATH for
> the duration of sourcing "file", which would potentially cause commands
> in "file" to break.
> 
> hobbit:~$ cat bar
> echo hi | cat
> hobbit:~$ PATH=. source bar
> bash: cat: command not found

So it is, at least if $BASH_SEARCH_PATH does not include $PATH.

I'll just go back to the sidelines and be quiet.

-- 
Andreas (Kusalananda) Kähäri
Uppsala, Sweden

.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Greg Wooledge
On Mon, May 20, 2024 at 07:43:10PM +0200, Andreas Kähäri wrote:
> On Mon, May 20, 2024 at 05:31:05PM +, Matheus Afonso Martins Moreira 
> wrote:
> > >> Why not add a -p option to '.' to specify the path to search.
> > >> That is
> > >>. -p "${BASH_SEARCH_PATH-${PATH}}" file
> > >> would work if someone decided to use the
> > >> BASH_SOURCE_PATH var name to store the path to use
> > >> (defaulting to $PATH if that one isn't set).
> > 
> > > Believe it or not, I had thought of that as well.
> > > It sidesteps the whole BASH_SOURCE_PATH
> > > variable brouhaha altogether.
> > 
> > I think this is a really good solution. I hadn't thought of it.
> > Users can even make an alias to set a default for themselves.
> > 
> >   -- Matheus
> 
> Or even
> 
>   PATH=${BASH_SEARCH_PATH-$PATH} . file
> 
> without the need to add any options to . or to source.  But maybe that
> too pedestrian?

Are we going in circles yet?  This would clobber the value of PATH for
the duration of sourcing "file", which would potentially cause commands
in "file" to break.

hobbit:~$ cat bar
echo hi | cat
hobbit:~$ PATH=. source bar
bash: cat: command not found



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Matheus Afonso Martins Moreira
> Or even
>
> PATH=${BASH_SEARCH_PATH-$PATH} . file
>
> without the need to add any options to . or to source.

If you do that, you'll interfere with command search
in the sourced file. They will not be able to find the
programs that would normally be in the PATH if the
variable is overridden prior to sourcing the script.

> But maybe that too pedestrian?

I don't know about "pedestrian"
but it's certainly not equivalent
to the proposed solution.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Andreas Kähäri
On Mon, May 20, 2024 at 05:31:05PM +, Matheus Afonso Martins Moreira wrote:
> >> Why not add a -p option to '.' to specify the path to search.
> >> That is
> >>. -p "${BASH_SEARCH_PATH-${PATH}}" file
> >> would work if someone decided to use the
> >> BASH_SOURCE_PATH var name to store the path to use
> >> (defaulting to $PATH if that one isn't set).
> 
> > Believe it or not, I had thought of that as well.
> > It sidesteps the whole BASH_SOURCE_PATH
> > variable brouhaha altogether.
> 
> I think this is a really good solution. I hadn't thought of it.
> Users can even make an alias to set a default for themselves.
> 
>   -- Matheus

Or even

PATH=${BASH_SEARCH_PATH-$PATH} . file

without the need to add any options to . or to source.  But maybe that
too pedestrian?

-- 
Andreas (Kusalananda) Kähäri
Uppsala, Sweden

.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Matheus Afonso Martins Moreira
>> Why not add a -p option to '.' to specify the path to search.
>> That is
>>. -p "${BASH_SEARCH_PATH-${PATH}}" file
>> would work if someone decided to use the
>> BASH_SOURCE_PATH var name to store the path to use
>> (defaulting to $PATH if that one isn't set).

> Believe it or not, I had thought of that as well.
> It sidesteps the whole BASH_SOURCE_PATH
> variable brouhaha altogether.

I think this is a really good solution. I hadn't thought of it.
Users can even make an alias to set a default for themselves.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/17/24 9:47 PM, Martin D Kealey wrote:



On Fri, 17 May 2024 at 04:18, Chet Ramey > wrote:


On 5/16/24 11:54 AM, G. Branden Robinson wrote:
 > At 2024-05-16T11:36:50-0400, Chet Ramey wrote:
 >> On 5/15/24 6:27 PM, Robert Elz wrote:
 >>> and any attempt to use a relative path (and you
 >>> can exclude ./anything or ../anything from that if you prefer - ie:
 >>
 >> Those are not relative paths.
 >
 > !
 >
 > POSIX 1003.1-202x/D4, §3.311 defines "relative pathname" thus:
 >
 > "A pathname not beginning with a  character."
 >
 > Can you clarify?  Does Bash have its own definition of this term?

In this specific case, I suppose. In default mode, `source' doesn't use
$PATH for ./x and ../x, but does for other relative pathnames.


I assumed that "default mode" means "not posix mode", but if so that 
doesn't hold up:


No, you're right. It forces a PATH search, but the path search itself
won't search for pathnames containing a slash. (Which is arguably a
mistake, but one we made 30+ years ago.)

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/17/24 1:15 PM, Robert Elz wrote:


   | If `nosort' means no sorting, there is no imposed ordering, and ascending
   | and descending are meaningless.

Sure, but directory order, and reverse directory order aren't (and that's
just a difference between the order in which you create the list as each
new dirent is read from the directory - does it go at the head or tail).


That's changing from one random order to another.


But in any case, it apparently isn't documented, if +nosort and -nosort
are to mean the same thing, then it should say so.  


OK, that's easy enough.



   | I'll consider it. What name would you propose for it?

Now that's the difficult question, I hate inventing names for
things - perhaps "number" or "numeric" or "numname" or ...


I like `numeric'.



All that matters is that it fall back to be "names" if the filenames are
not numeric (or at least if they don't start with digits, so atoi()
would just return 0), fully numeric names are the only ones that matter
to me.


Sure, but what do you prefer to do when you have results with a mix of
numeric values and string values? Pre-scan the results before sorting?
Let the comparison function handle it?



   | Because it's a complication that may or may not be worth implementing
   | without some critical mass of people asking for it. Let's see how this
   | goes first.

The doc also doesn't indicate what happens when the sort method chosen
produces equal answers - that can't happen for names, since the same name
can't occur twice in the same directory, but for everything else that's
in the list it can.   Does it then sort by name, by directory order, or
just by whatever order the sort algorithm (qsort?) happens to leave them
in when it is done?


Yes, it's unspecified.


Actually, one more option that might be worth adding (as a sort type)
might be "random" - giving different results every time the same pattern
is expanded. 


I don't see this as worth the implementation cost.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/17/24 11:40 AM, Robert Elz wrote:

 Date:Fri, 17 May 2024 10:09:06 -0400
 From:Chet Ramey 
 Message-ID:  

   | If you don't want `.' to search $PATH, turn off the `sourcepath' shell
   | option.

That part I assumed was already taken care of (that is, would be) just
by use of the BASH_SOURCE_PATH variable.  If it is set, PATH isn't used
(unless BASH_SOURCE_PATH=$PATH of course).


The extended semantics I had in mind would simply extend not searching PATH
(there is no other PATH variable to search right now) to not searching PATH
or BASH_SOURCE_PATH.



   | and we could make that part of the semantics of BASH_SOURCE_PATH.

That exactly was the suggestion.


OK, if it ever gets added.




Why not add a -p option to '.' to specify the path to search.  That is

. -p "${BASH_SEARCH_PATH-${PATH}}" file

would work if someone decided to use the BASH_SOURCE_PATH var name to
store the path to use (defaulting to $PATH if that one isn't set).


Believe it or not, I had thought of that as well. It sidesteps the whole
BASH_SOURCE_PATH variable brouhaha altogether.



Now of course, everyone is going to say "too difficult to do that all
the time", and they'd be right.  So the one extra feature would be that
if used without the "file" arg (which is normally mandatory), as in
. -p path-like-value
it would set the default search path for later uses of the '.' command
(and if anyone feels that being able to read back what has been set, an
additional option could make that info appear on stdout, and source
nothing).


Interesting, but obscure and potentially problematic if the filename is
stored in a variable that might be empty, e.g.,

source -p path $FILENAME

(yes, you know people are going to do that.) I would prefer to require

source -p $BASH_SOURCE_PATH filename

or whatever the user wants to call it, for clarity.


It is kind of attractive to make all of this just be an addition to the '.'
command, with no other externally visible intrusions elsewhere - that's
something I could probably support (still with the " . -p '' " semantic
of suppressing all "searches" of everything, full paths only, naturally).



I've just about given up on BASH_SOURCE_PATH anyway.


The problem with doing that is that it means making sure that the "same
semantics" are accurately copied, even in a shell where there are other
differences, and then maintained to match changes that bash makes to how
the variable works.   Not something I'm willing to undertake, and then
leave for my successors to deal with.   Nor I suspect would most others.


If someone else wanted to implement one of the BASH_ variables, I'd be
willing to talk about supporting a different name (as a synonym, in bash's
case) if we wanted to potentially standardize it. Some variables, like
SRANDOM, are already like that, and available for other shells to
implement.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Matheus Afonso Martins Moreira
> Bash would not set this to any value, since it doesn't install anything
> that would use it. To set it is to apply policy.

What I meant to say is that I as the user
can set it globally on my bash initialization
script so that I can always make use of the
new feature. If the act of setting it changes
source's behavior, I am prevented from
doing that since it could break scripts.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Chet Ramey

On 5/17/24 11:32 AM, Matheus Afonso Martins Moreira wrote:


You don't have to export variables. I would recommend not exporting
such a variable to begin with unless you're sure of its effects.


It could be made safe so that it's possible to export it.
Then it can always be set to some reasonable value.


Bash would not set this to any value, since it doesn't install anything
that would use it. To set it is to apply policy.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Matheus Afonso Martins Moreira
> It's rare for a topic about adding an import/include function to get
> so much attention like this I think it's time Bash takes the
> opportunity to implement it and hopefully properly.

Yes!! Conpletely agree with you!



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread Matheus Afonso Martins Moreira
> Are you referring to BASH_SOUCE_PATH being a source-only extension to PATH?

I mean that I won't be able to commit a standard loadable script
location to my dotfiles repository if setting BASH_SOURCE_PATH
changes the behavior of source. It could break too many things.

> Yes it allows the location of library scripts to be specified
> separately but it's not really useful.

It's a quite useful as a simple way for users to reuse code.
It's not meant to be a full module manager but that doesn't
mean it's not useful.

> You also would prefer `source` to not touch anything in PATH if it's
> meant to look for libraries.  Mixing it up is just programmatically
> unsound no matter how rare binaries in PATH might cause issues.
> Encapsulation of the feature is the right way to do it.

I agree. That's what the -i option does.
It keeps BASH_SOURCE_PATH completely encapsulated.

> I hate that I have to rely on an external binary just to load
> a script relative to the executable.  Example:
>
> source "$(realpath -m "${BASH_SOURCE}/../../lib/script.bash")" || exit
>
> If the `source -i` feature won't load a script using the calling
> script as reference it's pointless and I'd rather see it not
> implemented.

Hey for what it's worth I think that's a really good idea.
I'd really like additional features like that for source.
For example, I could add a -r option which prevents
searching and assumes the script is relative to the
current one. I'd also like to ensure that each script
gets loaded exactly once no matter how many times
it's been sourced.

I can't even seem to convince folks
of the value of source -i though.
So I haven't implemented more
functionality than that.

> BASH_SOUCE_PATH on its own also adds little value.

The variable only solution is not ideal for me
but I'd still get a lot of use out of it.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread konsolebox
On Mon, May 20, 2024 at 4:43 AM Robert Elz  wrote:
>   | A builtin is far more convenient, isn't it?
>   | Much more ergonomic.
>
> Only when it is really needed.

If there's a standard way of writing multi-script applications (and
reusing libraries) in bash, people will ask less about it in SO or
#bash and just have to read the manual.  If they still have to ask, at
least the answers will become much simpler and more straightforward.

Hi-voted questions like this with workaroundish
external-binary-dependent answers wouldn't exist:
https://stackoverflow.com/questions/59895/how-do-i-get-the-directory-where-a-bash-script-is-located-from-within-the-script

That's the highest voted bash-related question in SO by the way.

Not to mention people who like to write multi-script applications like
the way they do in C, Ruby or other languages will be happy if they
are able to do it in Bash too.

I wrote my include solution years ago and people still write their own
solutions from scratch.  So much time wasted for an obvious feature
almost every language has.

It's rare for a topic about adding an import/include function to get
so much attention like this I think it's time Bash takes the
opportunity to implement it and hopefully properly.


--
konsolebox



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-20 Thread konsolebox
On Sun, May 19, 2024 at 7:01 AM Matheus Afonso Martins Moreira
 wrote:
>
> > The issue is that what you want, and what I want,
> > and what konsolebox wants (and everyone else)
> > are all slightly different.
>
> The option solution which I proposed is not really in opposition
> to anything that anyone else wants. It will not break anything,
> it supports the same inputs and use cases as before, including
> the absolute path case you cited.

Just in case people might misinterpret it, I'll make it clear that I'm
not opposed to adding the `source -i` feature in builtin form.

I just don't like the way it's proposed to be implemented here.  I
described how `source -i` should work using a script with some
explanations but it's only a prototype.

I could implement it in C if it definitely gets accepted or before it
does if I get the urge to.

> On the other hand, the variable solution which everyone else
> seems to favor _does_ get in the way of what I want to do.

Are you referring to BASH_SOUCE_PATH being a source-only extension to PATH?

Yes it allows the location of library scripts to be specified
separately but it's not really useful.

You also would prefer `source` to not touch anything in PATH if it's
meant to look for libraries.  Mixing it up is just programmatically
unsound no matter how rare binaries in PATH might cause issues.
Encapsulation of the feature is the right way to do it.

> I don't really understand why there seems to be so much
> opposition to this. Ability to do something in pure bash is
> not an argument against this feature, especially since it
> overcomes limitations present in the pure bash solutions.
> Those are _real_ limitations, not some "personal hangups"
> as others called it. The pure bash solutions simply _cannot_
> both of these things, they must sacrifice one of them.

Indeed.  I hate that I have to rely on an external binary just to load
a script relative to the executable.  Example:

source "$(realpath -m "${BASH_SOURCE}/../../lib/script.bash")" || exit

If the `source -i` feature won't load a script using the calling
script as reference it's pointless and I'd rather see it not
implemented.

BASH_SOUCE_PATH on its own also adds little value.


-- 
konsolebox



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-19 Thread Phi Debian
On Sun, May 19, 2024 at 10:58 AM Matheus Afonso Martins Moreira <
math...@matheusmoreira.com> wrote:

>
> This is not a problem that's introduced by this patch.
> People can already do that today. Anyone could write
> `alias source='rm -rf --no-preserve-root /*'` right now,
> nothing stops them. So I don't see why this would be
> a reason against inclusion of this feature.
>

I was not talking about breakage of a script, anybody knows you can stick
an rm -rf anywhere in your script (BTW protection/containers,... mitigate
the damages). I was talking of two incompatible package manager, where one
can break the other, yours can break mine, mine can't break yours, I was
asking for a way to secure the old (current) way for mine.

So far I saw typeset -r BASH_SOURCE_PATH='' in my init path, but not even
sure it will be enough, beside I don't control my users who may miss the
init update but still can mix'n'match the package managers.

Here, we are not talking adding one more feature into bash, we are talking
about one feature predate (intentionally or not) another one, it is normal
that some head's up occurs.


> I understand your concern though. Perhaps we can find
> ways to work around it or fully address it.
>
Not too sure about that.


>
> > Another things that bugs me, why not implementing your invention in a
> > dynamically loaded builtin, advertise it on github, and let the one who
> > really needs this great thing be their choice download it, and use it.
>
> I don't really want to maintain such a thing. I'm struggling to find time
> to work on my personal projects as it is. We are lucky to have someone
> who maintains bash and who is considering this feature for addition.
>

Strange argument, you want to forward the hot potatoes to someone else
regarding maintenance.

First of all you claim the feature is tiny, almost bug free in its
inception, secondly it is vastly needed.

If those assertion are true, then your maintenance workload is almost nil
for your feature you can always access on top of bash with dynload.

If there is a vast need for it, and if unfortunately a bug shows up, then
in this vast population, there will be a good soul providing a pull request
for it, your load would be minimal.

That's the magic of open software :-)

So I think a dynloadable builtin of yours would be a perfect use case of
dynload, and a good way to challenge your invention, i.e measure the vast
population, bug rate, etc... I think it is a good 'preview' way, before
eventually move from dyn load to static load one day. Dynloadable builtins
exist, use it.



> I don't want to fork bash. I want to be able to download the latest version
> with my package manager and get access to this.


It is perfectly doable, clone, configure, build, stick you patch and
provide it to you (or your users) not a big deal of effort.
You can have that today, all that hidden in... bingo a library :-)



> If I had to fork bash to
> get this, I would simply give up and continue working on my own
> programming language instead.
>

I think this is a great valuable idea!

PS : On the citations side, I remember one, a software is bug free when you
trash it :-)


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-19 Thread G. Branden Robinson
At 2024-05-20T03:42:04+0700, Robert Elz wrote:
> My memory is hopeless, and I'm sure someone on the list can supply the
> correct attribution, but one of the big name CS people once said
> something along the lines of "perfection isn't when there's nothing
> left to add, but when there's nothing left to remove".

That would be Antoine de Saint-Exupéry, author of _La Petit Prince_,
which you may have read as a child, or read to children at some point.

Wikipedia says he was a "writer, poet, and aviator".

A reminder, were one needed, that intellectual silos are costly!  :)

(I only had this factoid ready at hand because Peter Schaffter, author
of the mom macro package for groff, uses it in his .signature file.)

Regards,
Branden


signature.asc
Description: PGP signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-19 Thread Robert Elz
Date:Sun, 19 May 2024 10:31:30 +
From:Matheus Afonso Martins Moreira 
Message-ID:  
<5LZgqrwhk0KEXRJqJIfPJzZvaJA8FVWbs_hnJHSPrPnzAzkjo7oA926voEMtU_3FGuZdylouhgpNXciyEJZPFdsScZ4pK0i4GLqv7ttR0nI=@matheusmoreira.com>

  | > the real work of the source ('.' really) utility
  | > is to read the file
  |
  | That's the work of read.
  |
  | > parse it, and execute it.
  |
  | That's the work of eval.

It could be.

  | The point is they probably could.
  | So why not do it?

Because there's no need, '.' exists, and is never going away,
so rather than doing lots of work, just use it.   If functions
had been implemented initially, and '.' not, then perhaps it
might be done that way, and we wouldn't need a '.' command, and
any suggestion to add one could be defeated by an argument
like that.   But that's all in the past, and can no longer be changed.

  | A builtin is far more convenient, isn't it?
  | Much more ergonomic.

Only when it is really needed.

  | Not at all. If one solution is capable of
  | accepting more inputs than another
  | without sacrificing any functionality,
  | then it's the better solution.

That's not true at all.   My memory is hopeless, and I'm sure someone
on the list can supply the correct attribution, but one of the big name
CS people once said something along the lines of "perfection isn't
when there's nothing left to add, but when there's nothing left to
remove".And that's right - but it works only during initial design,
once something has been made available, people use it, and it is never
possible after that to remove it.   (Unless you're python, and don't care
about breaking people's code.)   But it does apply when adding
anything new (or considering that) - new stuff should be added only
when absolutely essential, not just because it is convenient for someone
(even many someones).

  | > And a better solution to that would be to remove that restriction,
  | Is removing things from bash and breaking users a valid choice here?

No, of course not, but removing that restriction wouldn't break
anything I can see.

  | > which has no purpose that I can imagine.
  |
  | I didn't even know this restricted shell thing existed
  | until about a week ago. The fact remains that it's in
  | bash and that someone out there is probably using it.

Of course.   Oh, I see, you think I was suggesting removing the
restricted shell.   No, I wouldn't suggest that, no matter how
stupid the whole idea is.   All I suggested removing (for the
purposes relevant here anyway) was the restriction that the '.'
command cannot read files with a '/' in their names.   That
restriction seems pointless.  Removing the restriction cannot
break any existing rbash setups (they can't be using '.' with
a name containing a '/') unless it created a vulnerability,
which I don't see as happening here.

  | If the builtin works with the restricted shells
  | while the pure shell function does not,
  | the builtin is the better solution.

"If" yes, perhaps (though restricted shells work now without your
proposed changes) - which is why I suggested removing the restriction,
then the pure shell function would work with restricted shells too,
and no bash modifications would be needed (except to remove what seems
to be a pointless restriction.)

  | Restricted shells do not allow command names with slashes.

That's fine, but they don't prohibit args to (most) commands
containing slashes, and in any case, using shell rules to provide
access control is silly, we have file and directory permissions,
and sometimes ACLs as well, to do that.

  | > Just remove that restriction and this point goes away
  | That would be a breaking change.

Breaking what?

  | Not long after I sent the patches, a list of about
  | a dozen projects implementing module loading
  | and management in bash was posted.
  | Clearly this is something people want.

I don't see the connection.   All you're proposing is a different
way to convert a relative pathname in a '.' command into a
full one (in effect - that is, to locate it precisely in the
filesystem).   Since that can be done with a function, which
can then operate according to whatever rules the writer of that
function wants (things relative to the script location, relative
to $PWD relative to $PWD when the script started running, relative
to $HOME, relative to some other fixed point in the filesystem, or ...)
and applying whatever other name transformations it desires.
Much more flexible, and better suited to varying desires.

  | Well all those components exist now.
  | Why not use them instead of source?

Because that would be stupid - one uses the best tools that
are available, but every new job does not require the invention
of a new tool just to handle that job.  If you start doing that
you end up with so many tools, that no-one knows what exists
any more.

  | If I just started deleting things I couldn't care less about,
  | there wouldn't be much of bash left 

Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-19 Thread Matheus Afonso Martins Moreira
> the real work of the source ('.' really) utility
> is to read the file

That's the work of read.

> parse it, and execute it.

That's the work of eval.

> None of the proposed solutions here do that.

Of course not. I read the proposals.
The point is they probably could.
So why not do it?

A builtin is far more convenient, isn't it?
Much more ergonomic.

> What file names work for this purpose
> is an irrelevant consideration.

Not at all. If one solution is capable of
accepting more inputs than another
without sacrificing any functionality,
then it's the better solution.

> And a better solution to that would be to remove that restriction,

Is removing things from bash and breaking users a valid choice here?

> which has no purpose that I can imagine.

I didn't even know this restricted shell thing existed
until about a week ago. The fact remains that it's in
bash and that someone out there is probably using it.
Therefore integrating new features with it is desirable.

If the builtin works with the restricted shells
while the pure shell function does not,
the builtin is the better solution.

> The commands run by the source builtin would be restricted

That seems to be the whole point of restricted shells.

> and whatever they are, could simply be typed as input to the same effect
> so stopping a rbash user from accessing stuff in some common (full path
>  named) file seems pointless to me.

Restricted shells do not allow command names with slashes.

> Just remove that restriction and this point goes away

That would be a breaking change.

> Unless it proves useful

There is absolutely no doubt that this is useful.
Not long after I sent the patches, a list of about
a dozen projects implementing module loading
and management in bash was posted.
Clearly this is something people want.

> and unable to be done some other way

It cannot be done some other way.
Not without running into limitations.

The limitations might not matter to you,
but they still exist and still should to be
properly worked around and integrated.

Frankly I couldn't care less about those
limitations either but I still put in effort
needed to understand them and make
the patches work with them instead of
just dismissing them out of hand as
irrelevant.

> That one is simple ... when it ('.' in sh, "source" in csh) was added
> there were no functions, while read & eval might have been enough for
>  simple one line scripts, they wouldn't work for more complex ones, to
>  invent your own '.' command that way you'd need to do it in a moderately
>  complex function, which didn't exist at the time.

Well all those components exist now.
Why not use them instead of source?

> I wouldn't, as I don't care about restricted shells (never have, not
> even when I was running a system used by many hundreds of (*evil*)
> students).And that's the only real restriction, which should be
> removed anyway.

If I just started deleting things I couldn't care less about,
there wouldn't be much of bash left by the end of it.

I'm not even against it. In fact I've actually done it before.
The HTML templating language which currently powers
my website is a fork of an existing one and I literally
deleted everything I didn't care about. Literally.
Works great for me.

The point is bash would stop being bash if I did that.
It wouldn't be bash anymore, it'd be some new shell language.
I'm trying to improve bash, not replace it.

> I agree that modifying PATH isn't a great solution

Discussing this matter always seems to end like this.
I can't figure out if people actually agree with me or not.
Apparently I'm right but the feature _still_ shouldn't be
added because of reasons I can't seem to understand.

> though for any use I'd have of '.' that wouldn't bother me either

Well it absolutely bothers _me_ and that's why I suggested this
feature and developed and sent these patches in the first place.
If it doesn't bother you, doesn't break anything, doesn't really get
in anyone's way and generally only improves the status quo,
what exactly is the problem? I sincerely don't understand
what the problem seems to be and it's really frustrating.

> both because I've never used a '.' command that actually searches PATH,
> and because I don't put anything which would require a PATH search
> in files I use via '.'

Just because you don't generally do that
doesn't mean that others aren't doing it l
or that they don't want to do it.

I want to do exactly that.

> '.' is used when the current shell environment needs to be modified,
> so is useful for defining functions, and assigning values to variables

That's not under question. This is to allow better organization of the files.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-19 Thread Matheus Afonso Martins Moreira
> the library (the top one may be) decide that source -i
> is the general way to go a decide a despotic alias
> source='source -i' this could be a general setup of
> this package manager.

This is not a problem that's introduced by this patch.
People can already do that today. Anyone could write
`alias source='rm -rf --no-preserve-root /*'` right now,
nothing stops them. So I don't see why this would be
a reason against inclusion of this feature.

I understand your concern though. Perhaps we can find
ways to work around it or fully address it.

Bash has the command builtin which can be used to
ensure a real utility is executed instead of an alias or
function. It also works for builtins such as source.
I could also propose a new flag for it that restricts
its operation to builtins only, eliminating the PATH
search and ensuring that builtins cannot be replaced
by a program on the file system either.

I'm not sure what the maintainer would think about it
but I could also propose a shopt that prohibits the
aliasing of builtins and other reserved words.

These should completely solve that problem
for new scripts but not for the code you have
already deployed and can't change or update.
I'm not sure what to do about them.
However, the fact is there's any number of things
that other people can do to break them already.

> Another things that bugs me, why not implementing your invention in a
> dynamically loaded builtin, advertise it on github, and let the one who
> really needs this great thing be their choice download it, and use it.

I don't really want to maintain such a thing. I'm struggling to find time
to work on my personal projects as it is. We are lucky to have someone
who maintains bash and who is considering this feature for addition.

> I personally have my own bash and own ksh for our internal needs, we
> maintain them, and we have our own set of internally crafted builtins, I
> would not dare push them to shell maintainers, fearing breaking all who
> depend on bash/ksh/dash whatever for their OS startup, init script,
> configure script etc...

I don't want to fork bash. I want to be able to download the latest version
with my package manager and get access to this. If I had to fork bash to
get this, I would simply give up and continue working on my own
programming language instead.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-19 Thread Phi Debian
On Sun, May 19, 2024 at 1:17 AM Matheus Afonso Martins Moreira <
math...@matheusmoreira.com> wrote:

>
> Those should continue to work without any changes.
> They should even be compatible with each other:
> there should be no problems with sourcing -i a script
> which then sources another script normally or uses
> one of the module management solutions.
>
>   -- Matheus
>

Not true, the hack to break the whole thing is

old source.sh relying on the current {source,.,$PATH} behavior
The source.sh is alive (maintained)

One come along a say look I got this neat library to colorize stuff, it
depend on a couple of other library (meaning they do source -i inside) the
library (the top one may be) decide that source -i is the general way to go
a decide a despotic alias source='source -i' this could be a general setup
of this package manager.

Back to good'ol source.sh now any 'source' it does are biased.

I agry that true to for other shell options/variable,
'libraries/packages/imports' can break the whole thing too though (IFS is a
good one) but so far we managed :-)

So at the end, the poor one that rely on actual {source,.,$PATH}would have
to warn in its coding convention, never ever source a .sh that rely on
BASH_SOURCE_PATH,
I know a grep would suffice to enforce it, but there is no reasons to force
that.

Another things that bugs me, why not implementing your invention in a
dynamically loaded builtin, advertise it on github, and let the one who
really needs this great thing be their choice download it, and use it.

I personally have my own bash and own ksh for our internal needs, we
maintain them, and we have our own set of internally crafted builtins, I
would not dare push them to shell maintainers, fearing breaking all who
depend on bash/ksh/dash whatever for their OS startup, init script,
configure script etc...


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Robert Elz
Date:Sat, 18 May 2024 23:00:41 +
From:Matheus Afonso Martins Moreira 
Message-ID:  



  | If you resolve the path, you're just reinventing the source builtin.

Nonsense, the real work of the source ('.' really) utility is to
read the file, parse it, and execute it.   None of the proposed
solutions here do that.   What file names work for this purpose is
an irrelevant consideration.

  | It also depends on source actually accepting the absolute path
  | which renders it incompatible with restricted shells.

That's the only point here that has any merit.   And a better solution
to that would be to remove that restriction, which has no purpose that
I can imagine.   The commands run by the source builtin would be restricted,
and whatever they are, could simply be typed as input to the same effect,
so stopping a rbash user from accessing stuff in some common (full path
named) file seems pointless to me.   Just remove that restriction and
this point goes away, and there's no reason left to modify bash in one
particular way (rather than some other way that someone else would prefer).

[Aside: the restriction on rbash using "exec" to replace the shell makes
 no sense either - obviously whatever is exec'd would need to be a command
 the user could run without exec, and assuming that condition is maintained,
 then a user can just do "cmd" and stay in that until they're done, which
 is effectively identical to "exec cmd" except the former keeps the rbash
 process hanging around for no useful purpose.

 I also find it hard to believe that anyone can usefully use any shell
 which doesn't allow stdout to be redirected - what files can be used,
 that can be restricted, but simply banning redirection?   Weird.

 Then again, my general opinion of restricted shells is that they were a
 joke when first invented (long long ago when lots of users shared a
 computer, which is a very rare setup these days) and while the joke has
 become stale, that's still what they are.]

  | I don't really understand why there seems to be so much
  | opposition to this.

It is extra stuff that once added needs to be maintained forever.
Unless it proves useful, and unable to be done some other way,
adding it is typically the wrong thing to do.

  | So why add the source builtin to begin with?

That one is simple ... when it ('.' in sh, "source" in csh) was added
there were no functions, while read & eval might have been enough for
simple one line scripts, they wouldn't work for more complex ones, to
invent your own '.' command that way you'd need to do it in a moderately
complex function, which didn't exist at the time.   And like anything
else, once added, it is there forever.

  | > if I needed something like that, I could write my own, similar ones.
  | And you would run into the limitations described above.

I wouldn't, as I don't care about restricted shells (never have, not
even when I was running a system used by many hundreds of (*evil*)
students).And that's the only real restriction, which should be
removed anyway.

I agree that modifying PATH isn't a great solution - though for any
use I'd have of '.' that wouldn't bother me either (both because I've
never used a '.' command that actually searches PATH, and because I
don't put anything which would require a PATH search in files I use
via '.' - such scripts are simply run as scripts.  '.' is used when
the current shell environment needs to be modified, so is useful for
defining functions, and assigning values to variables (and perhaps
the odd option setting) - almost never anything more than that (at
least for me - so I wouldn't run into the limitations described.)

kre




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Matheus Afonso Martins Moreira
> distros, sysadmins, and users can already break your scripts
> in plenty of ways -- changing existing shell variables, setting
> shopts your scripts don't account for, etc.  

Existing footguns doesn't justify addition of new ones.
We have the ability to prevent a lot of breakage here.
Shouldn't we take that opportunity?

> By this logic, ALL future shell variables ought to be gated similarly.

Maybe. I haven't really thought that far
but it would probably be a good thing.

> it seems a bit heavy-handed

Why? Most if not all of the other builtins have options.

> Note that "source -i" precludes a sort of mirror situation.
> Someone who sourced scripts via PATH but wanted to
> switch to BASH_SOURCE_PATH would not be able to
> simply move the relevant directories from one variable
> to the other but would have to modify all sourcing scripts
>  -- including third-party ones -- to use "source -i". 

Yes, that's intentional. This ensures that users who wrote code
expecting PATH semantics actually get those semantics.
Then the new variable can be set in the environment
without impacting those users and their scripts.

> If they declined to do this, then they'd have to keep using
> PATH for at least some scripts, possibly indefinitely.

Those should continue to work without any changes.
They should even be compatible with each other:
there should be no problems with sourcing -i a script
which then sources another script normally or uses
one of the module management solutions.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Matheus Afonso Martins Moreira
> The issue is that what you want, and what I want,
> and what konsolebox wants (and everyone else)
> are all slighty different.

The option solution which I proposed is not really in opposition
to anything that anyone else wants. It will not break anything,
it supports the same inputs and use cases as before, including
the absolute path case you cited.

On the other hand, the variable solution which everyone else
seems to favor _does_ get in the way of what I want to do. 

> And as konsolebox's functions show, all this can be implemented,
> reasonably easily, outside of the bash source code

No. By discussing this I ended up discovering additional reasons
why it should actually be in bash as a builtin feature.

Pure bash solutions have two choices and both of them break stuff.
Either they clobber PATH, or they resolve the file's path on their own
and pass an absolute path to the source builtin. If you clobber PATH,
sourced scripts can't easily find commands anymore. That completely
changes the way that scripts work and therefore the way they must
be written. If you resolve the path, you're just reinventing the source
builtin. It also depends on source actually accepting the absolute path
which renders it incompatible with restricted shells.

The option solution completely avoids all that. It leaves PATH alone,
and as a builtin it can integrate with restricted shells. I even made
BASH_SOURCE_PATH be read only in that context, just like PATH.
And it accomplishes all that in a compatible way that's fully opt in,
anyone who does not want it can simply not use it. Existing code
is not impacted because existing code simply does not pass -i
as an option to the source builtin.

I don't really understand why there seems to be so much
opposition to this. Ability to do something in pure bash is
not an argument against this feature, especially since it
overcomes limitations present in the pure bash solutions.
Those are _real_ limitations, not some "personal hangups"
as others called it. The pure bash solutions simply _cannot_
both of these things, they must sacrifice one of them.

Even if that wasn't the case, it wouldn't be an argument.
By this logic, the source builtin itself should not have been
added either. I mean it's just read and eval, and bash has
both of those primitives built into it already. There's a
comment which even explains source in these terms:
it reads in the file into a string and just evaluates it.
So why add the source builtin to begin with?

> if I needed something like that, I could write my own, similar ones.

And you would run into the limitations described above.

> I think all this just shows that there's no real need to add any
> code to bash at all for this

I don't know about "need" but there are certainly advantages.

> use full path names instead

Incompatible with restricted shells.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Lawrence Velázquez
On Sat, May 18, 2024, at 11:17 AM, Phi Debian wrote:
> I am xing fingers the consensus, if any of theses takes off,
> is to have [...] protection against alias source='source -i'

This seems out of scope, regardless of what happens with "-i".


> I really depend on actual 'source' behavior, and can't update my packages
> (distributed since long). I don't plan to use BASH_SOURCE_PATH, so if one,
> distro, sysadmin, and ultimatly the user fiddle with BASH_SOURCE_PATH in
> the ENV (*rc files), changing the good'ol 'source' behavior, it would be a
> serious regression to me.

A reasonable concern, but distros, sysadmins, and users can already
break your scripts in plenty of ways -- changing existing shell
variables, setting shopts your scripts don't account for, etc.


> So to me BASH_SOURCE_PATH enablement must be guarded by a never used yet
> option to 'source or dot' (here -i apparently) and an enforcement that no
> alias source='source -i' could ever be possible.

By this logic, ALL future shell variables ought to be gated similarly.
No doubt some would embrace this, but it seems a bit heavy-handed.


> All this is because one writing a 'main' script (#!/bin/bash), sourcing a
> package of mine through my package management system, expecting the current
> source behavior, and later add the loading of a 'libraries' from a friend
> (terminal color jazz) that in turn start to mess around with alias
> source='source -i' and BASH_SOURCE_PATH cold start to break my way of
> finding my packages.

There's only so much you or bash itself can do to prevent users
from breaking their own setups.

Note that "source -i" precludes a sort of mirror situation.  Someone
who sourced scripts via PATH but wanted to switch to BASH_SOURCE_PATH
would not be able to simply move the relevant directories from one
variable to the other but would have to modify all sourcing scripts
-- including third-party ones -- to use "source -i".  If they
declined to do this, then they'd have to keep using PATH for at
least some scripts, possibly indefinitely.


-- 
vq



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Robert Elz
Date:Sat, 18 May 2024 08:39:57 -0300
From:Matheus Afonso Martins Moreira 
Message-ID:  


  | As the user, it should be my prerogative to set the
  | variable on my environment and/or rc so that I can
  | organize my sourceable scripts however I want

Of course.   The issue is that what you want, and what I want, and
what konsolebox wants (and everyone else) are all slighty different.

And as konsolebox's functions show, all this can be implemented,
reasonably easily, outside of the bash source code, in code that
implements whatever you want.   Those functions aren't what I'd
want (for several reasons, not least because they are very bash
dependent) but if I needed something like that, I could write my
own, similar ones.

I think all this just shows that there's no real need to add any
code to bash at all for this, it is OK as it is now (and everyone
should just continue ignoring the PATH search when using '.' (or
"source") and use full path names instead - whether generated by
a function, or an install script, or simply written that way).

If Chet wants to alter the way this works in bash, he can do it
however seems best to him for what the majority of users are likely
to prefer, or just whatever it is he prefers - I am fairly sure it
wouldn't break the simple ". /path/to/file" usage, which is all we
really need to work to be able to do anything we personally like.

kre




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Lawrence Velázquez
On Sat, May 18, 2024, at 11:41 AM, Matheus Afonso Martins Moreira wrote:
>> I am xing fingers the consensus, if any of theses takes off,
>> is to have the -i implemented (not omitted)
>
> Me too. Do you know anyone else who could be impacted?
> Consensus might move in that direction if more voices make
> themselves heard on this list.

Consensus does not depend on how many people openly agree with you,
except insofar as additional discussion sways those who disagree.

-- 
vq



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Matheus Afonso Martins Moreira
> I am xing fingers the consensus, if any of theses takes off,
> is to have the -i implemented (not omitted)

Me too. Do you know anyone else who could be impacted?
Consensus might move in that direction if more voices make
themselves heard on this list.

> if one,  distro, sysadmin, and ultimatly the user
> fiddle with BASH_SOURCE_PATH in the ENV (*rc files),
> changing the good'ol 'source' behavior,
> it would be a serious regression to me.

That's exactly what I mean.
  
> So to me BASH_SOURCE_PATH enablement must be guarded by a never used yet
> option to 'source or dot' (here -i apparently)

Agree.

I initially chose -l which stands for library
but that terminology proved problematic.
So I chose -i which stands for isolated,
the option "isolates" source to only the
directories defined in the new variable.
Maybe there's a better word/character.
Naming things is hard.

> and an enforcement that no alias
> source='source -i' could ever be possible.

I don't really have any objections to that
but I'd say there are probably too many
ways for people to get around such a filter.
It would probably be ineffective.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Phi Debian
On Sat, May 18, 2024 at 3:23 PM Matheus Afonso Martins Moreira <
math...@matheusmoreira.com> wrote:

>
> > That would cause shell scripts to see it and exhibit a change in
> behavior.
>
> Only if the -i option is not implemented.
> If it is, there will be zero change in behavior
> whether BASH_SOURCE_PATH is set or not.
> Only if the script explicitly passes -i to source
> will the behavior change.
>

I am xing fingers the consensus, if any of theses takes off, is to have the
-i implemented (not omitted) along with a protection against alias
source='source -i'

I really depend on actual 'source' behavior, and can't update my packages
(distributed since long). I don't plan to use BASH_SOURCE_PATH, so if one,
distro, sysadmin, and ultimatly the user fiddle with BASH_SOURCE_PATH in
the ENV (*rc files), changing the good'ol 'source' behavior, it would be a
serious regression to me.

So to me BASH_SOURCE_PATH enablement must be guarded by a never used yet
option to 'source or dot' (here -i apparently) and an enforcement that no
alias source='source -i' could ever be possible.

All this is because one writing a 'main' script (#!/bin/bash), sourcing a
package of mine through my package management system, expecting the current
source behavior, and later add the loading of a 'libraries' from a friend
(terminal color jazz) that in turn start to mess around with alias
source='source -i' and BASH_SOURCE_PATH cold start to break my way of
finding my packages.

An alternative to this would be
BASH_SOURCE_PATH="" don't do nothing with 'source', and allow the
'libraries' designer to do
typeset -r BASH_SOURCE_PATH=""
in the early step of the main script, so nobody will ever be able to do
anything clever with BASH_SOURCE_PATH, or could add import from
incompatible packages system. But even this idea is a pb to me because it
would require a change an my side that is not really doable.

Well in short don't break the old source semantic.


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Matheus Afonso Martins Moreira
> You've made contradictory statements here.

I don't see how.

> First you said you wanted to put it into your *environment*.

Yes.

> That would cause shell scripts to see it and exhibit a change in behavior.

Only if the -i option is not implemented.
If it is, there will be zero change in behavior
whether BASH_SOURCE_PATH is set or not.
Only if the script explicitly passes -i to source
will the behavior change.

> Next you said you would like to set it in your .bashrc file.
> That's totally different.

How is it different? If I export a variable in .bashrc,
and bash reads and evaluates that file at startup,
then that variable gets exported to the shell and
all processes spawned by it.

> Setting it in .bashrc *without* putting it into the
> environment (no export) is "safe".  It will only affect
> your interactive shell, and not any scripts that you run.

I know. I'm just saying it doesn't have to be that way.

I'm saying that in addition to it being safe if unset or
unexported, it could _also_ be made safe if exported.
The -i option provides that safety by requiring that
source code be changed in order to enable it.
Existing code will never be impacted by the feature
whether the variable is set or not, simply because
they do not pass -i to the source builtin.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Greg Wooledge
On Sat, May 18, 2024 at 08:39:57AM -0300, Matheus Afonso Martins Moreira wrote:
> > Setting the variable at all is opting in to new behavior, and you do
> > that at your own risk, after reading the documentation and deciding
> > that this is what you want.
> 
> As the user, it should be my prerogative to set the
> variable on my environment and/or rc so that I can
> organize my sourceable scripts however I want and
> have the scripts which enabled the isolated sourcing
> just work no matter where I chose to install them.
> 
> As the user, I should be able to run shell scripts
> regardless of whether they support this or not.
> I should be to set this variable on my .bashrc
> and enjoy the nice infrastructure it provides
> without worrying about whether or not
> some script takes it into account.

You've made contradictory statements here.

First you said you wanted to put it into your *environment*.  That would
cause shell scripts to see it and exhibit a change in behavior.

Next you said you would like to set it in your .bashrc file.  That's
totally different.  Setting it in .bashrc *without* putting it into the
environment (no export) is "safe".  It will only affect your interactive
shell, and not any scripts that you run.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-18 Thread Matheus Afonso Martins Moreira
> Setting the variable at all is opting in to new behavior, and you do
> that at your own risk, after reading the documentation and deciding
> that this is what you want.

As the user, it should be my prerogative to set the
variable on my environment and/or rc so that I can
organize my sourceable scripts however I want and
have the scripts which enabled the isolated sourcing
just work no matter where I chose to install them.

As the user, I should be able to run shell scripts
regardless of whether they support this or not.
I should be to set this variable on my .bashrc
and enjoy the nice infrastructure it provides
without worrying about whether or not
some script takes it into account.
I shouldn't be forced to choose between
legacy shell scripts which don't support this
and new shell scripts which do. There's no
technical reason why I shouldn't be able
to run both in the same bash, or even
have one type of script source the other.

Environment variables are recursively passed
down the process tree. If you choose the
variable approach, my choice to "opt in"
will get applied recursively as well.
I'll be making that choice for every
script that I run and every script sourced
by those scripts. That's actually a breaking
change that fails to be compatible.

All that could be achieved without incompatibility
by simply adding the -i option to source. Then my
choice remains mine alone when I pass -i to source
while interacting with bash via the terminal.
Then the script writer's choice becomes theirs alone
when they write their scripts. It becomes a local choice,
not a recursive and inheritable one.

The recursive and inheritable nature of environment variables
are useful for programs _after_ they have chosen to use them.
If the presence of the variable becomes a choice unto itself,
that choice will end up being inadvertently forced on others.

> I do not foresee people setting BASH_SOURCE_PATH
> in their basic interactive shell environments and login
> sessions.

That should definitely be foreseen because that's _exactly_
what I plan to do if this feature gets merged.

I will set it on my .bashrc just like this:

export BASH_SOURCE_PATH="${HOME}"/.local/share/bash/source

Then I will commit that to my dotfiles repository.

> Why would they do that?
> What purpose would it serve?

I will set the variable in order to create a fixed location
for loading libraries. Then I will break up my growing
collection of shell scripts into reusable functions,
put them into separate files and then put those files
in the BASH_SOURCE_PATH. Then I can easily source
those libraries of functions from my bash programs.

The purpose is to be able to more easily write
more complex bash scripts. This will allow me
to write scripts by following a process like:

source -i terminal
source -i options

# use option parsing functions to get inputs
# compute some useful output
# color output with terminal functions

> But you wouldn't want it to be set in your other windows.

I absolutely want it set on every bash process,
including interactive ones.

I always want the ability to source the nice functions
that I put effort into writing and debugging and organizing.
I will even add shorthand aliases for them so that they will
be easy to type on the terminal.

> What I'm imagining here is that the variable will be used almost
exclusively by scripts, to set the location of their resource files,
which they source.

I have nothing against that use case but that's not at all
what I'm imagining. For me this is very much a feature
for the _users_ to help them manage their shell script
collections. It should be in no way exclusive to the scripts.

> They're not useful outside of the script which sources them
> so only that script needs to set BASH_SOURCE_PATH

False. The variable will be extremely and directly useful to me
as a bash user. That's why I proposed and implemented it.

I didn't even intend for the scripts to override it.
The whole idea is scripts would give source some symbol
that represents what they need and source would take care
of loading it so that the functions are available. They need not
care where or even what the scripts are, they can just say:

source -i my-functions

And it gets done.

I have no objections to scripts overriding it at all.
However I don't think that's an argument against
users being able to control their environment just
like they have control over PATH.

  -- Matheus


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Martin D Kealey
On Fri, 17 May 2024 at 04:18, Chet Ramey  wrote:

> On 5/16/24 11:54 AM, G. Branden Robinson wrote:
> > At 2024-05-16T11:36:50-0400, Chet Ramey wrote:
> >> On 5/15/24 6:27 PM, Robert Elz wrote:
> >>> and any attempt to use a relative path (and you
> >>> can exclude ./anything or ../anything from that if you prefer - ie:
> >>
> >> Those are not relative paths.
> >
> > !
> >
> > POSIX 1003.1-202x/D4, §3.311 defines "relative pathname" thus:
> >
> > "A pathname not beginning with a  character."
> >
> > Can you clarify?  Does Bash have its own definition of this term?
>
> In this specific case, I suppose. In default mode, `source' doesn't use
> $PATH for ./x and ../x, but does for other relative pathnames.
>

I assumed that "default mode" means "not posix mode", but if so that
doesn't hold up:

$ mkdir tmp/a
$ cat >tmp/a/b
echo in B
$ ( PATH=$PWD/tmp/a source b )
in B
$ ( PATH=$PWD/tmp source a/b )
bash-latest: a/b: No such file or directory
$ echo $BASH_VERSION
5.3.0(2)-alpha
$ ( p=$( realpath "$0" ) ; echo "git commit ${p##*/bash/}" )
git commit aadb6ffb93359891760c58008539f549f06c5140/bin/bash
$ shopt -o posix
posix   off

-Martin


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Robert Elz
Date:Fri, 17 May 2024 10:20:57 -0400
From:Chet Ramey 
Message-ID:  

  | If you set the variable to something that needs it, you have to accept
  | the stat overhead -- otherwise, how could it do what you've requested?

Of course, I understand that, I'm just not sure about the average user
who might start complaining about how slow the shell has become when
expanding glob patterns.   (There's also going to be some memory overhead,
as once you have stat'd the file once, you'd want to remember the value
of interest along with the name, or you'd have way too many stat() calls
if that was repeated for every comparison done by the sort algorithm).

  | If `nosort' means no sorting, there is no imposed ordering, and ascending
  | and descending are meaningless.

Sure, but directory order, and reverse directory order aren't (and that's
just a difference between the order in which you create the list as each
new dirent is read from the directory - does it go at the head or tail).

But in any case, it apparently isn't documented, if +nosort and -nosort
are to mean the same thing, then it should say so.   Another possibility
(apart from reverse directory order, which I agree doesn't seem useful)
would be for "-nosort" to either be an error, or just be ignored, leading
to "names").

  | I'll consider it. What name would you propose for it?

Now that's the difficult question, I hate inventing names for
things - perhaps "number" or "numeric" or "numname" or ...

(but for something like this, what string to use makes no difference
at all really, they could be s0 s1 s2 s3 ... and it would work
equally well - just slightly harder to remember which one to use).

All that matters is that it fall back to be "names" if the filenames are
not numeric (or at least if they don't start with digits, so atoi()
would just return 0), fully numeric names are the only ones that matter
to me.

  | Because it's a complication that may or may not be worth implementing
  | without some critical mass of people asking for it. Let's see how this
  | goes first.

The doc also doesn't indicate what happens when the sort method chosen
produces equal answers - that can't happen for names, since the same name
can't occur twice in the same directory, but for everything else that's
in the list it can.   Does it then sort by name, by directory order, or
just by whatever order the sort algorithm (qsort?) happens to leave them
in when it is done?

If you want common implementations of this, so it could eventually be
standardised, then all these kinds of details need to be made very precise
(even though your average man page reader probably wouldn't care.)

Actually, one more option that might be worth adding (as a sort type)
might be "random" - giving different results every time the same pattern
is expanded.   That might be useful to help shake out bugs in some code.
(and no, I have no idea what -random would mean.)   You'd just run a PRNG
to supply the sort key, where for the others you extract data from stat().

kre




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Greg Wooledge
On Fri, May 17, 2024 at 03:32:23PM +, Matheus Afonso Martins Moreira wrote:
> > You don't have to export variables. I would recommend not exporting
> > such a variable to begin with unless you're sure of its effects.
> 
> It could be made safe so that it's possible to export it.
> Then it can always be set to some reasonable value.

"unset" is a very reasonable default value.  If BASH_SOURCE_PATH is
unset, then you get the historic bash behavior (or the POSIX behavior
if your shell is in POSIX mode).

Setting the variable at all is opting in to new behavior, and you do
that at your own risk, after reading the documentation and deciding
that this is what you want.

I do not foresee people setting BASH_SOURCE_PATH in their basic
interactive shell environments and login sessions.  Why would they
do that?  What purpose would it serve?

I could maybe see it being used in some sort of bash analogue of Python's
virtual environments.  Maybe you're building some bash project with
multiple resource files that get sourced from a subdirectory, and while
working in this project, you find it helpful to set BASH_SOURCE_PATH.
But you wouldn't want it to be set in your other windows.

What I'm imagining here is that the variable will be used almost
exclusively by scripts, to set the location of their resource files,
which they source.  These files may be "library code", or configuration
variables, or whatever.  They're not useful outside of the script which
sources them, so only that script needs to set BASH_SOURCE_PATH to find
them.

Effectively, it allows scripts to change from this:

#!/bin/bash
RSRCDIR=/opt/me/share
source "$RSRCDIR/foo"
source "$RSRCDIR/bar"

to this:

#!/bin/bash
BASH_SOURCE_PATH=/opt/me/share
source foo
source bar

Anything beyond that is up to you.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Robert Elz
Date:Fri, 17 May 2024 10:09:06 -0400
From:Chet Ramey 
Message-ID:  

  | If you don't want `.' to search $PATH, turn off the `sourcepath' shell
  | option.

That part I assumed was already taken care of (that is, would be) just
by use of the BASH_SOURCE_PATH variable.  If it is set, PATH isn't used
(unless BASH_SOURCE_PATH=$PATH of course).

  | the skipping $PWD has never been
  | historical behavior and would be genuinely new,

Yes.   For executables (the normal PATH search) it makes no sense,
nor does it for CDPATH, because of the way that both are typically
used, but for '.' (aka the old csh name "source") use without a full
path is very rare, and probably an accident (or one waiting to happen).
Being able to prevent it would be useful, I think.

  | and we could make that part of the semantics of BASH_SOURCE_PATH.

That exactly was the suggestion.

  | Sure.
  | https://lists.gnu.org/archive/html/bug-bash/2024-05/msg00262.html

Yes, I saw that after I sent the previous reply - when messages get
addressed to me, they end up in my inbox, which I read & process
much more quickly than I ever get to mailing list mail, which goes
to its appropriate list mailbox instead.   This one is the same!

  | If you implement the functionality later, maybe we can talk about a
  | different name that's acceptable to more implementors. But for now, it's
  | bash-specific if it gets implemented at all.

Actually, I have a different idea, which might avoid the var name issue,
and be easier to get implemented elsewhere.   It is kind of like the
earlier suggestion to use -i to mean something different, but not that.

Why not add a -p option to '.' to specify the path to search.  That is

. -p "${BASH_SEARCH_PATH-${PATH}}" file

would work if someone decided to use the BASH_SOURCE_PATH var name to
store the path to use (defaulting to $PATH if that one isn't set).

Now of course, everyone is going to say "too difficult to do that all
the time", and they'd be right.  So the one extra feature would be that
if used without the "file" arg (which is normally mandatory), as in
. -p path-like-value
it would set the default search path for later uses of the '.' command
(and if anyone feels that being able to read back what has been set, an
additional option could make that info appear on stdout, and source
nothing).

It is kind of attractive to make all of this just be an addition to the '.'
command, with no other externally visible intrusions elsewhere - that's
something I could probably support (still with the " . -p '' " semantic
of suppressing all "searches" of everything, full paths only, naturally).

The default value of this hidden internal path would be $PATH (using
PATH's value at the time of the '.' command in question).

  | Sure, other shells shouldn't implement variable names that start with BASH_
  | if they have different semantics than bash gives them, or if bash doesn't
  | implement them, but there's nothing wrong with other shells adding BASH_
  | variables that have the same semantics. Other than parochialism, of course.

The problem with doing that is that it means making sure that the "same
semantics" are accurately copied, even in a shell where there are other
differences, and then maintained to match changes that bash makes to how
the variable works.   Not something I'm willing to undertake, and then
leave for my successors to deal with.   Nor I suspect would most others.

  | Sure, and then if bash were to implement SOURCEPATH, or something similar,
  | then there would be objections similar to
  |
  | "it's barely tolerable to add more action-at-a-distance by introducing a
  | new variable"
  |
  | but with the addition that it intrudes on the user's namespace.

Yes, I know ... and it was thinking on that after reading this message
originally that led to the idea above.

And just for fun, we could even propose adding the very first ever (since
Bourne created them) special parameter, likely "$." instead of a '.' command
option, to make the value of the '.' command's path known).  That one isn't
likely to succeed (as a proposal), but I've always wanted for someone to
make a new special param, just because it's never been done -- I temporarily
had a "$+" which was the inverse of "$-" -- the option flags that are
disabled ... but that ended up not really having enough reason to exist,
especially as there's no agreement as to what counts as belonging in $-
anyway, just the options that "set" accepts, or also those that apply only
on the command line (like -c).

  | >| I have high hopes for `GLOBSORT', though.
  | > What does it do?
  |
  | Sorts the results of pathname expansion in different contexts using
  | different criteria. It's a bash-5.3 feature.

Yes, Lawrence provided me with the info (I could have gone and found the
5.3 alpha man page source somewhere and read that, if I'd been sure it
would be there, but...).You can see my reaction to that on the 

Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Matheus Afonso Martins Moreira
> As with everything.

It doesn't have to be that way. It could be made so that it's harmless,
so that it has no effect at all unless explicitly enabled by an option.
Then there's no need to carefully manage it. Users and packagers
can set it wherever they want without fear of breaking things.

> You don't have to export variables. I would recommend not exporting
> such a variable to begin with unless you're sure of its effects.

It could be made safe so that it's possible to export it.
Then it can always be set to some reasonable value.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Chet Ramey

On 5/16/24 10:23 PM, Matheus Afonso Martins Moreira wrote:

If you don't set BASH_SOURCE_PATH (from the user's perspective;
I'm not sure what I think about a loaded `library' changing the global
user settings by setting it), you get the historical behavior.


It's also important to consider the opposite situation:
distributions and/or users setting BASH_SOURCE_PATH globally
which then results in existing scripts _not_ getting the historical
behavior which matches what their authors expected at the time.


I'm more confident in users being able to manage their own shell
environment.


In order to preserve that behavior, users would have to very carefully
manage the scope of their BASH_SOURCE_PATH variable.


As with everything.



Even if users resorted to that, the environment variables
are passed down to new processes,


You don't have to export variables. I would recommend not exporting
such a variable to begin with unless you're sure of its effects.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Chet Ramey

On 5/17/24 2:42 AM, Robert Elz wrote:

 Date:Thu, 16 May 2024 23:54:12 -0400
 From:=?UTF-8?Q?Lawrence_Vel=C3=A1zquez?= 
 Message-ID:  <0b3fa8d4-7a3f-4bd6-8e8d-f7d1746be...@app.fastmail.com>

   | This is the description from the bash.1 that's currently in my local
   | repository:

Thanks.

Possibly useful I guess - though I'm not really sure about a possible
need to stat() every file from a glob result because that got set to
one of the options that needs it. 


If you set the variable to something that needs it, you have to accept
the stat overhead -- otherwise, how could it do what you've requested?


"nosort" seems likely to be the most
useful of the available options (that description didn't say what "-nosort"
means though - backwards from the order in the directory?)


If `nosort' means no sorting, there is no imposed ordering, and ascending
and descending are meaningless.



One option that might be useful (and cheap) which isn't there would be
to do a numeric sort (falling back to simple strcmp on equality) on the
file names - I use MH (nmh) for my mail, and needing to use ? ?? ???  ?
to get the files in the "correct" order (ie: not 1 10 100 1000 1 10001 ...)
would be useful.


I'll consider it. What name would you propose for it?


And if you're going to do all of that, why not have a list of sort
specs, so one can do "blocks,-size,mtime,..." to specify what happens
to files with the same number of blocks (and in this case, size, mod time).


Because it's a complication that may or may not be worth implementing
without some critical mass of people asking for it. Let's see how this
goes first.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Chet Ramey

On 5/16/24 11:36 PM, Robert Elz wrote:

 Date:Thu, 16 May 2024 11:36:50 -0400
 From:Chet Ramey 
 Message-ID:  <613852f3-c9ef-43d8-821b-37be3d9e9...@case.edu>

   | You can already do this; there's no reason to add these semantics to a
   | variable setting.

How?   What I'm looking for is a way to catch accidental ". foo" type
things (by causing them to fail, regardless of whether "./foo" (or some
other random "foo") exists and contains valid sh code).


If you don't want `.' to search $PATH, turn off the `sourcepath' shell
option. If you want to temporarily change it, toggle the option. That
addresses the path searching issue; the skipping $PWD has never been
historical behavior and would be genuinely new, and we could make that
part of the semantics of BASH_SOURCE_PATH.



   | > can exclude ./anything or ../anything from that if you prefer - ie:
   | Those are not relative paths.

Of course they are, that "." (or "..") is relative to something, it
isn't referring to a fixed location in the file system tree (like "/foo"
would be).

You might choose to categorise them differently, that's fine, but
there's no semantic difference between "./foo" and "dir/foo" - both
look for some directory ("." or "dir") within some other directory,
which usually depends upon other factors (such as the current PWD).


Sure.

https://lists.gnu.org/archive/html/bug-bash/2024-05/msg00262.html



   | > exclude anything which would skip the PATH-like search anyway) would
   | > simply fail (not just return itself and be attempted relative to ".")
   |
   | Is this really valuable?

I think so, and given that any other reasonable interpretation I can
think of has alternative (just as simple) mechanisms to accomplish,
it seems to me like it would be reasonable.   If you want BASH_SOURCE_PATH=.
then that's exactly that easy to accomplish.   You can make the default
when BASH_SOURCE_PATH is unset be whatever you like (like using PATH,
or doing something entirely different), but having it set to "" seems
like it ought mean "no path search".


That's what it means now (or would mean now), since interpreting "" as
equivalent to "." is standard for PATH-like variables. But that's not
what you want. It's easy to say that if you use BASH_SOURCE_PATH, `source'
doesn't fall back to $PWD if something is not found in the path. In that
case, it's up to the user to put "." (or equivalent) into BASH_SOURCE_PATH,
or not.



   | Maybe. We'll have to see. People will object no matter what the name is.

It isn't a matter of objecting, I have no objection to BASH_SEARCH_PATH,
but I'm never going to implement that var (just as I wouldn't implement
BASH_VERSION or any of the other BASH_XXX vars that exist). 


If you implement the functionality later, maybe we can talk about a
different name that's acceptable to more implementors. But for now, it's
bash-specific if it gets implemented at all.



and other shells than bash, never).


Sure, other shells shouldn't implement variable names that start with BASH_
if they have different semantics than bash gives them, or if bash doesn't
implement them, but there's nothing wrong with other shells adding BASH_
variables that have the same semantics. Other than parochialism, of course.


It's hard to get consensus on something when it isn't possible to even
agree on a name all can use


Sure, and then if bash were to implement SOURCEPATH, or something similar,
then there would be objections similar to

"it's barely tolerable to add more action-at-a-distance by introducing a
new variable"

but with the addition that it intrudes on the user's namespace.

But there's nothing set in stone, since nothing's been implemented yet.



   | I have high hopes for `GLOBSORT', though.

That is at least a plausible name - never heard of it, and my bash man
page (still at 5.2, I have a 5.3-a1 binary, but haven't installed its
man page anywhere) doesn't mention it.   What does it do?


Sorts the results of pathname expansion in different contexts using
different criteria. It's a bash-5.3 feature.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Greg Wooledge
On Fri, May 17, 2024 at 01:42:43PM +0700, Robert Elz wrote:
> [GLOBSORT]
> Possibly useful I guess - though I'm not really sure about a possible
> need to stat() every file from a glob result because that got set to
> one of the options that needs it.  "nosort" seems likely to be the most
> useful of the available options

My immediate first thought was the "mtime" option.  No more need to
use "ls --magic-options-here -t" and eval the output stream, to get
the 10 oldest/newest log files?  Yes please!

Other than that one niche case, I don't see an immediate use for it,
but it's a potentially powerful tool to have in the box.  I'm sure
someone will find clever uses for it.

> (that description didn't say what "-nosort"
> means though - backwards from the order in the directory?)

Heh.  I would imagine -nosort is the same as nosort, but hell, Chet
might surprise me on this one.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-17 Thread Robert Elz
Date:Thu, 16 May 2024 23:54:12 -0400
From:=?UTF-8?Q?Lawrence_Vel=C3=A1zquez?= 
Message-ID:  <0b3fa8d4-7a3f-4bd6-8e8d-f7d1746be...@app.fastmail.com>

  | This is the description from the bash.1 that's currently in my local
  | repository:

Thanks.

Possibly useful I guess - though I'm not really sure about a possible
need to stat() every file from a glob result because that got set to
one of the options that needs it.  "nosort" seems likely to be the most
useful of the available options (that description didn't say what "-nosort"
means though - backwards from the order in the directory?)

One option that might be useful (and cheap) which isn't there would be
to do a numeric sort (falling back to simple strcmp on equality) on the
file names - I use MH (nmh) for my mail, and needing to use ? ?? ???  ?
to get the files in the "correct" order (ie: not 1 10 100 1000 1 10001 ...)
would be useful.

And if you're going to do all of that, why not have a list of sort
specs, so one can do "blocks,-size,mtime,..." to specify what happens
to files with the same number of blocks (and in this case, size, mod time).

kre




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread Lawrence Velázquez
On Thu, May 16, 2024, at 11:36 PM, Robert Elz wrote:
> Date:Thu, 16 May 2024 11:36:50 -0400
> From:Chet Ramey 
> Message-ID:  <613852f3-c9ef-43d8-821b-37be3d9e9...@case.edu>
>
>   | I have high hopes for `GLOBSORT', though.
>
> That is at least a plausible name - never heard of it, and my bash man
> page (still at 5.2, I have a 5.3-a1 binary, but haven't installed its
> man page anywhere) doesn't mention it.   What does it do?

This is the description from the bash.1 that's currently in my local
repository:

GLOBSORT

Control how the results of pathname expansion are sorted.
The value of this variable specifies the sort criteria and
sort order for the results of pathname expansion.  If this
variable is unset or set to the null string, pathname
expansion uses the historical behavior of sorting by name.
If set, a valid value begins with an optional `+', which
is ignored, or `-', which reverses the sort order from
ascending to descending, followed by a sort specifier.  The
valid sort specifiers are `name', `size', `mtime', `atime',
`ctime', and `blocks', which sort the files on name, file
size, modification time, access time, inode change time,
and number of blocks, respectively.  For example, a value
of `-mtime' sorts the results in descending order by
modification time (newest first).  A sort specifier of
`nosort' disables sorting completely; the results are
returned in the order they are read from the file system.
If the sort specifier is missing, it defaults to `name',
so a value of `+' is equivalent to the null string, and a
value of `-' sorts by name in descending order.  Any invalid
value restores the historical sorting behavior.

-- 
vq



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread Robert Elz
Date:Thu, 16 May 2024 11:36:50 -0400
From:Chet Ramey 
Message-ID:  <613852f3-c9ef-43d8-821b-37be3d9e9...@case.edu>

  | You can already do this; there's no reason to add these semantics to a
  | variable setting.

How?   What I'm looking for is a way to catch accidental ". foo" type
things (by causing them to fail, regardless of whether "./foo" (or some
other random "foo") exists and contains valid sh code).

  | > can exclude ./anything or ../anything from that if you prefer - ie:
  | Those are not relative paths.

Of course they are, that "." (or "..") is relative to something, it
isn't referring to a fixed location in the file system tree (like "/foo"
would be).

You might choose to categorise them differently, that's fine, but
there's no semantic difference between "./foo" and "dir/foo" - both
look for some directory ("." or "dir") within some other directory,
which usually depends upon other factors (such as the current PWD).

  | > exclude anything which would skip the PATH-like search anyway) would
  | > simply fail (not just return itself and be attempted relative to ".")
  |
  | Is this really valuable?

I think so, and given that any other reasonable interpretation I can
think of has alternative (just as simple) mechanisms to accomplish,
it seems to me like it would be reasonable.   If you want BASH_SOURCE_PATH=.
then that's exactly that easy to accomplish.   You can make the default
when BASH_SOURCE_PATH is unset be whatever you like (like using PATH,
or doing something entirely different), but having it set to "" seems
like it ought mean "no path search".

  | Maybe. We'll have to see. People will object no matter what the name is.

It isn't a matter of objecting, I have no objection to BASH_SEARCH_PATH,
but I'm never going to implement that var (just as I wouldn't implement
BASH_VERSION or any of the other BASH_XXX vars that exist).  If some
script wants to set BASH_xxx for any random xxx, that's fine, but it
will never be anything but be a variable (and because of the possibility
that the var picked might one day conflict with one you add, I'd suggest
that people not generally do that, and other shells than bash, never).

It's hard to get consensus on something when it isn't possible to even
agree on a name all can use - if I'm using a different name for a similar
function, I have no reason to implement the same sematic as you do, and
all we end up with is yet another divergent system, which would be where
the posix people come along and invent POSIX_SOURCE_PATH (which no-one
implements) with some random sub/super-set of the way the various
implementations work.   That's not something to work towards, IMO.

  | I have high hopes for `GLOBSORT', though.

That is at least a plausible name - never heard of it, and my bash man
page (still at 5.2, I have a 5.3-a1 binary, but haven't installed its
man page anywhere) doesn't mention it.   What does it do?

kre




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread Greg Wooledge
On Fri, May 17, 2024 at 02:23:10AM +, Matheus Afonso Martins Moreira wrote:
> It's also important to consider the opposite situation:
> distributions and/or users setting BASH_SOURCE_PATH globally
> which then results in existing scripts _not_ getting the historical
> behavior which matches what their authors expected at the time.

They'll quickly learn not to do that.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread Matheus Afonso Martins Moreira
> If you don't set BASH_SOURCE_PATH (from the user's perspective;
> I'm not sure what I think about a loaded `library' changing the global
> user settings by setting it), you get the historical behavior.

It's also important to consider the opposite situation:
distributions and/or users setting BASH_SOURCE_PATH globally
which then results in existing scripts _not_ getting the historical
behavior which matches what their authors expected at the time.

In order to preserve that behavior, users would have to very carefully
manage the scope of their BASH_SOURCE_PATH variable.

Maybe they'll resort to wrapping the source builtin invocation
in a function with a local BASH_SOURCE_PATH. It'd be just like
the pure bash solution for safely overriding PATH for source
that many suggested on this list. I think users should never
have any need for such a function, source should be enough.

In the worst case users would resort to assigning it
before every source command to avoid the risk of
breaking existing scripts. It'd turn into one of the
many shell features that shellcheck tells users to
be careful about, and the fix would be to add the
variable assignment to every source command.
In that case, it'd work just like the -i option since
it would have to be included in every invocation,
but it'd be more verbose and less ergonomic.

Even if users resorted to that, the environment variables
are passed down to new processes, new subshells and
new in-shell execution contexts. All bash scripts sourced
while BASH_SOURCE_PATH was defined will have different
behavior, so users of BASH_SOURCE_PATH won't be able to
safely source existing scripts which themselves use source.
It's as if invoking source -i caused the -i option to be passed
to the source builtin of every subshell in the process tree
instead of just that one invocation.

It ends up recreating the same problem we're trying to fix.
We're adding a new variable to avoid the need to touch PATH
but at the same time we're forcing users to avoid touching
BASH_SOURCE_PATH. It also creates two mutually exclusive
categories of scripts: those written for PATH and those written
for BASH_SOURCE_PATH. They can't call into each other.

The advantage of the -i option is that users can set
BASH_SOURCE_PATH in their rc files or environment
and nothing happens by default. Only new code which
explicitly passes the -i option to the source builtin get
to use this new behavior. It also remains compatible
with existing scripts since the option is not passed down
the process tree like the environment variables. Only that
invocation gets the new behavior, other code still has to
opt into it by passing the -i option. It also has the advantage
of being much shorter to type which makes it very ergonomic
for interactive shell users.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread Greg Wooledge
On Thu, May 16, 2024 at 11:31:55AM -0400, Chet Ramey wrote:
> On 5/15/24 11:31 AM, Koichi Murase wrote:
> > Maybe it was not clear to use `source name' and `source -i name' to
> > describe the idea. I meant I assumed the search domain being
> > 
> > * BASH_SOURCE_PATH + PATH + PWD without the option
> 
> It seems to me that this isn't useful. The whole reason to have and use
> BASH_SOURCE_PATH is not to use PATH; a fallback to PATH if something
> isn't found in BASH_SOURCE_PATH doesn't allow that. And if you're using
> BASH_SOURCE_PATH, you have to set it, and if you want `.' in there,
> add it.

Yes, I'm inclined to agree with this.  If you want it to fall back to
$PATH, you can append the contents of $PATH to BASH_SOURCE_PATH as well.
You get the most control if BASH_SOURCE_PATH is used exclusively when
it's set.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread Chet Ramey

On 5/16/24 11:54 AM, G. Branden Robinson wrote:

At 2024-05-16T11:36:50-0400, Chet Ramey wrote:

On 5/15/24 6:27 PM, Robert Elz wrote:

and any attempt to use a relative path (and you
can exclude ./anything or ../anything from that if you prefer - ie:


Those are not relative paths.


!

POSIX 1003.1-202x/D4, §3.311 defines "relative pathname" thus:

"A pathname not beginning with a  character."

Can you clarify?  Does Bash have its own definition of this term?


In this specific case, I suppose. In default mode, `source' doesn't use
$PATH for ./x and ../x, but does for other relative pathnames.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread G. Branden Robinson
At 2024-05-16T11:36:50-0400, Chet Ramey wrote:
> On 5/15/24 6:27 PM, Robert Elz wrote:
> > and any attempt to use a relative path (and you
> > can exclude ./anything or ../anything from that if you prefer - ie:
> 
> Those are not relative paths.

!

POSIX 1003.1-202x/D4, §3.311 defines "relative pathname" thus:

"A pathname not beginning with a  character."

Can you clarify?  Does Bash have its own definition of this term?
Does it need to?

Regards,
Branden


signature.asc
Description: PGP signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread Chet Ramey

On 5/15/24 11:25 AM, Koichi Murase wrote:

2024年5月15日(水) 22:13 Chet Ramey :

On 5/14/24 4:48 PM, Koichi Murase wrote:

However, at the same time, I think the new option could be justified
from the viewpoint of language design. If we have a new namespace
``BASH_SOURCE_PATH'' for the scripts to be sourced, it would be neat
to have an option to specify the namespace exclusively.


Should these be the default semantics for BASH_SOURCE_PATH?


Even if the option is not supported, I think the default semantics
should still reference PATH and the current working directory as
fallbacks because some libraries may want to set BASH_SOURCE_PATH,


As I said, I'm not sure a library changing the value of a global user
setting like that is good practice.



`unset -v var' is not required.


That was a POSIX invention as a compromise between implementations.


Hmm. Then, is the example of `unset' irrelevant?


It means it wasn't added for orthogonality or to make the language cleaner;
it was added to allow applications to force a particular behavior in the
face of default behavior differences by existing implementations. And this
was back in 1990.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread Chet Ramey

On 5/15/24 6:27 PM, Robert Elz wrote:


All I suggested
was having a BASH_SOURCE_PATH set to the empty string mean that no path
lookup should be done, 


You can already do this; there's no reason to add these semantics to a
variable setting.


and any attempt to use a relative path (and you
can exclude ./anything or ../anything from that if you prefer - ie:


Those are not relative paths.


exclude anything which would skip the PATH-like search anyway) would
simply fail (not just return itself and be attempted relative to ".")


Is this really valuable?


   | So let's move the state of the art forward.

That isn't going to happen if you make the path searched have a BASH_*
variable name.


Maybe. We'll have to see. People will object no matter what the name is.

I have high hopes for `GLOBSORT', though.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-16 Thread Chet Ramey

On 5/15/24 11:31 AM, Koichi Murase wrote:

2024年5月15日(水) 22:21 Chet Ramey :

On 5/14/24 7:15 PM, Koichi Murase wrote:

In case we talk about different designs, I assume the following one:

* `source name' searches the files in BASH_SOURCE_PATH, PATH, and the
current working directory.


No, I'm proposing that this behavior be the default, without the option:


Yes, that's my intent. I mean "without the option" by the above `source name'.


* `source -i name', if it would be added, searches the files in
BASH_SOURCE_PATH (excluding PATH and the current working directory
unless `.' is included in BASH_SOURCE_PATH).


Maybe it was not clear to use `source name' and `source -i name' to
describe the idea. I meant I assumed the search domain being

* BASH_SOURCE_PATH + PATH + PWD without the option


It seems to me that this isn't useful. The whole reason to have and use
BASH_SOURCE_PATH is not to use PATH; a fallback to PATH if something
isn't found in BASH_SOURCE_PATH doesn't allow that. And if you're using
BASH_SOURCE_PATH, you have to set it, and if you want `.' in there,
add it. If you don't set BASH_SOURCE_PATH (from the user's perspective;
I'm not sure what I think about a loaded `library' changing the global
user settings by setting it), you get the historical behavior.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature


Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-15 Thread Robert Elz
Date:Wed, 15 May 2024 09:19:06 -0400
From:Chet Ramey 
Message-ID:  <86e381d8-dc83-4ef8-90e6-122468fa4...@case.edu>

  | Why make it different from other PATH-like variables, where a null
  | component is equivalent to "."?

Because having two ways to do the same thing is unnecessary, if "." is
meant, one can simply write "." - so having a null entry mean the same
thing adds no functionality, but means one needs to take more care when
composing PATH out of fragments, one cannot simply append ":newvalue"
to $PATH in case $PATH happened to be unset or empty.

But changing that wasn't what I actually suggested, as that would mean
either having two different PATH like scanning routines, or one which
would need to test an extra flag every time it found an empty entry (which
depending upon the implementation might alter the API).   All I suggested
was having a BASH_SOURCE_PATH set to the empty string mean that no path
lookup should be done, and any attempt to use a relative path (and you
can exclude ./anything or ../anything from that if you prefer - ie:
exclude anything which would skip the PATH-like search anyway) would
simply fail (not just return itself and be attempted relative to ".")

That's a test that can be added to the code for the dot command, before
the PATH like search starts, and only in the case where BASH_SOURCE_PATH
is set, and would be used.

  | So let's move the state of the art forward.

That isn't going to happen if you make the path searched have a BASH_*
variable name.

If you meant for PATH searches in general, that would be wildly breaking
backward compat, and I doubt you'd attempt to justify that, and nor would
I, regardless of it being an obvious improvement.

kre




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-15 Thread Koichi Murase
2024年5月15日(水) 22:21 Chet Ramey :
> On 5/14/24 7:15 PM, Koichi Murase wrote:
> > In case we talk about different designs, I assume the following one:
> >
> > * `source name' searches the files in BASH_SOURCE_PATH, PATH, and the
> > current working directory.
>
> No, I'm proposing that this behavior be the default, without the option:

Yes, that's my intent. I mean "without the option" by the above `source name'.

> > * `source -i name', if it would be added, searches the files in
> > BASH_SOURCE_PATH (excluding PATH and the current working directory
> > unless `.' is included in BASH_SOURCE_PATH).

Maybe it was not clear to use `source name' and `source -i name' to
describe the idea. I meant I assumed the search domain being

* BASH_SOURCE_PATH + PATH + PWD without the option
* BASH_SOURCE_PATH with the option (if the option is supported)



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-15 Thread Koichi Murase
2024年5月15日(水) 22:13 Chet Ramey :
> On 5/14/24 4:48 PM, Koichi Murase wrote:
> > However, at the same time, I think the new option could be justified
> > from the viewpoint of language design. If we have a new namespace
> > ``BASH_SOURCE_PATH'' for the scripts to be sourced, it would be neat
> > to have an option to specify the namespace exclusively.
>
> Should these be the default semantics for BASH_SOURCE_PATH?

Even if the option is not supported, I think the default semantics
should still reference PATH and the current working directory as
fallbacks because some libraries may want to set BASH_SOURCE_PATH,
while some other libraries (such as an old library for <= 5.2 or a
POSIX-sh library) would still assume the current behavior of the
`source' builtin searching PATH and the current working directory. If
setting BASH_SOURCE_PATH would disable PATH and the current working
directory, those libraries cannot be mixed, which seems a troublesome
restriction.

> After all, if
> you want to search the current directory, put "." into the value.

Yes, that's technically correct. However, that requires every user or
framework to set BASH_SOURCE_PATH properly as

  BASH_SOURCE_PATH=xxx:${BASH_SOURCE_PATH:-$PATH:.}

(unless without a specific reason). Otherwise, the setup can be easily
broken by a not-so-well-designed.library in a dependency.

> > `unset -v var' is not required.
>
> That was a POSIX invention as a compromise between implementations.

Hmm. Then, is the example of `unset' irrelevant?



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-15 Thread Chet Ramey

On 5/14/24 7:15 PM, Koichi Murase wrote:

In case we talk about different designs, I assume the following one:

* `source name' searches the files in BASH_SOURCE_PATH, PATH, and the
current working directory.


No, I'm proposing that this behavior be the default, without the option:


* `source -i name', if it would be added, searches the files in
BASH_SOURCE_PATH (excluding PATH and the current working directory
unless `.' is included in BASH_SOURCE_PATH).


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-15 Thread Chet Ramey

On 5/14/24 5:40 PM, Robert Elz wrote:

 Date:Tue, 14 May 2024 15:51:10 -0400
 From:Chet Ramey 
 Message-ID:  

   | What do folks think about forcing an option to enable using
   | BASH_SOURCE_PATH? Should it be required? Is it necessary? (I personally
   | think it is not.) We discussed this briefly last week but without any
   | conclusion.

I'd make it set to '' mean that only paths containing
a '/' would work for the '.' command) and:


Why make it different from other PATH-like variables, where a null
component is equivalent to "."?



I agree, but it should be expected to be defined in a profile type file
if it needs to be defined at all (a system wide one, or a user's personal one).
The default (as compiled without extra config) should be unset.


That's the idea.


But I don't think any of this is needed at all.   I agree with the OP
that having '.' use PATH is archaic, and fairly dumb. 


So let's move the state of the art forward.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-15 Thread Chet Ramey

On 5/14/24 4:48 PM, Koichi Murase wrote:


However, at the same time, I think the new option could be justified
from the viewpoint of language design. If we have a new namespace
``BASH_SOURCE_PATH'' for the scripts to be sourced, it would be neat
to have an option to specify the namespace exclusively.


Should these be the default semantics for BASH_SOURCE_PATH? After all, if
you want to search the current directory, put "." into the value.

`unset -v var' is not required. 


That was a POSIX invention as a compromise between implementations.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/




Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-14 Thread Matheus Afonso Martins Moreira
> It has been Bash's convention to allow a non-POSIX feature
> even in POSIX mode when the script explicitly requests it.

I see. I have no objections to that.
My initial implementation also worked like this.

I did it that way because someone explicitly requested that
POSIX mode not be affected in any way and this would
theoretically stop the user from sourcing a file named `-i`.

If that's not a problem then there's no reason not to
recognize the option even in POSIX mode.
I'll make it so in the v3 patch set.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-14 Thread Koichi Murase
In case we talk about different designs, I assume the following one:

* `source name' searches the files in BASH_SOURCE_PATH, PATH, and the
current working directory.
* `source -i name', if it would be added, searches the files in
BASH_SOURCE_PATH (excluding PATH and the current working directory
unless `.' is included in BASH_SOURCE_PATH).

I think it is clearer to see the patches. I attach modified patches:
0001 adds BASH_SOURCE_PATH. 0002 is an optional patch if we are to add
the command-line option to the `source' builtin.

In rebasing the patches, I noticed the stub in the latest devel push.
Should BASH_SOURCE_PATH be affected by `shopt -s sourcepath'? I think
it is more useful to be able to control PATH (through sourcepath) and
BASH_SOURCE_PATH (through whether its state is unset or empty)
separately.
From c29cf5ec2a79ef40eb09dcec3ae7402142850838 Mon Sep 17 00:00:00 2001
From: Matheus Afonso Martins Moreira 
Date: Mon, 13 May 2024 07:37:26 -0300
Subject: [PATCH 1/2] feat: support BASH_SOURCE_PATH

* shell: restrict BASH_SOURCE_PATH when appropriate

Make the BASH_SOURCE_PATH variable read-only and unsettable
when the shell is operating in restricted mode. This variable
should be restricted for the same reasons why PATH is restricted.

Modifications:

* When we are not in the POSIX mode, we consider BASH_SOURCE_PATH if
  present even without the new option `-i'.

* The new variable BASH_SOURCE_PATH has been disabled at all in the
  restricted shell since this would introduce a new loophole in
  existing setups of the restricted shells.  If the new variable were
  enabled in the restricted shells, all the existing setups would need
  to be updated to care about the variable BASH_SOURCE_PATH, which
  would be an unreasonable requirement.  In addition, considering the
  purpose of the restricted shells, we would not need to provide the
  new feature to the restricted shells.

Co-authored-by: Koichi Murase 
---
 builtins/source.def | 21 -
 1 file changed, 8 insertions(+), 13 deletions(-)

diff --git a/builtins/source.def b/builtins/source.def
index 51dddec6..4498ac9e 100644
--- a/builtins/source.def
+++ b/builtins/source.def
@@ -116,7 +116,7 @@ int
 source_builtin (WORD_LIST *list)
 {
   int result;
-  char *filename, *debug_trap, *x;
+  char *filename, *source_path, *debug_trap, *x;
 
   if (no_options (list))
 return (EX_USAGE);
@@ -143,20 +143,15 @@ source_builtin (WORD_LIST *list)
 filename = savestring (list->word->word);
   else if (absolute_pathname (list->word->word))
 filename = savestring (list->word->word);
-  else if (source_uses_path)
-{
-#if 0
-  char *spath;
+  else if (
+!posixly_correct &&
 #if defined (RESTRICTED_SHELL)
-  if (restricted == 0 && posixly_correct == 0 && (spath = path_value ("BASH_SOURCE_PATH", 1)))
-#else
-  if (posixly_correct == 0 && (spath = path_value ("BASH_SOURCE_PATH", 1)))
-#endif
-	filename = find_in_path (list->word->word, spath, FS_READABLE);
-  else
+!restricted &&
 #endif
-	filename = find_path_file (list->word->word);
-}
+(source_path = path_value ("BASH_SOURCE_PATH", 1)) != NULL)
+filename = find_in_path (list->word->word, source_path, FS_READABLE);
+  if (filename == 0 && source_uses_path)
+filename = find_path_file (list->word->word);
   if (filename == 0)
 {
   if (source_searches_cwd == 0)
-- 
2.43.0

From 172218c620adbbeca7a1dfda8eae759d38fecbc2 Mon Sep 17 00:00:00 2001
From: Matheus Afonso Martins Moreira 
Date: Mon, 13 May 2024 07:37:23 -0300
Subject: [PATCH 2/2] feat: support "source -i"

* builtins/source: parse the -i option

Passing the -i option to the source builtin
enables isolated sourcing mode which restricts
its search path to the directories defined by
the BASH_SOURCE_PATH variable. This also has
the added benefit of not touching PATH at all.

Modifications:

* If the option is explicitly supplied, we try to use BASH_SOURCE_PATH
  exclusively even in POSIX mode.  This is consistent with portable
  scripts because they are supposed not to specify the -i option.  It
  has been Bash's convention to allow a non-POSIX feature even in
  POSIX mode when the script explicitly requests it.

Co-authored-by: Koichi Murase 
---
 builtins/source.def | 62 ++---
 1 file changed, 42 insertions(+), 20 deletions(-)

diff --git a/builtins/source.def b/builtins/source.def
index 4498ac9e..251f6c02 100644
--- a/builtins/source.def
+++ b/builtins/source.def
@@ -117,9 +117,31 @@ source_builtin (WORD_LIST *list)
 {
   int result;
   char *filename, *source_path, *debug_trap, *x;
+  int opt, isolated_mode = 0;
+
+  reset_internal_getopt ();
+
+  while ((opt = internal_getopt (list, "i")) != -1)
+{
+  switch (opt)
+{
+case 'i':
+#if defined (RESTRICTED_SHELL)
+  if (restricted)
+{
+  sh_restricted ("-i");
+  return (EXECUTION_FAILURE);
+}
+#endif
+  

Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-14 Thread Matheus Afonso Martins Moreira
I agree that it is not strictly necessary.
It could be implemented without an option.
I personally have no objections to that solution.

My arguments for the option are the following:

 - It enhances compatibility
   Without the option, source behaves exactly as before.
   Existing scripts will work exactly as before even if
   the user has defined BASH_SOURCE_PATH.
   I paid special attention to the posixly correct behavior.
 - It's opt in
   Users will have to explicitly choose to use
   the new behavior of source, preventing errors.
 - It simplifies and reduces cognitive load
   Passing -i means only the directories in
   BASH_SOURCE_PATH will be searched,
   and no other direcrories. Easy to understand.
 - It's clear and explicit
   Passing -i makes it explicit to anyone
   reading the scripts that they were written
   with this new behavior in mind.

  -- Matheus



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-14 Thread Robert Elz
Date:Tue, 14 May 2024 15:51:10 -0400
From:Chet Ramey 
Message-ID:  

  | What do folks think about forcing an option to enable using
  | BASH_SOURCE_PATH? Should it be required? Is it necessary? (I personally
  | think it is not.) We discussed this briefly last week but without any
  | conclusion.

An option isn't needed.   If this were to be added at all, the right way
would be to simply if not in posix mode, then use BASH_SOURCE_PATH (if it
is set (PATH otherwise), or perhaps set and not '' - though personally if
this were me adding it, which would be unlikely, but if I were to be
convinced to do that, I'd make it set to '' mean that only paths containing
a '/' would work for the '.' command) and:

just as in another message chet.ra...@case.edu said:

  | I don't like bash providing a default for this when bash doesn't install any
  | `libraries'. Maybe it would be better to let it be something the builder or
  | distro defines.

I agree, but it should be expected to be defined in a profile type file
if it needs to be defined at all (a system wide one, or a user's personal one).
The default (as compiled without extra config) should be unset.

But I don't think any of this is needed at all.   I agree with the OP
that having '.' use PATH is archaic, and fairly dumb.   I avoid that
by simply always giving a full path to the '.' command, never a simple
name.   If I want to search for what that path should be, given a list
of directories, and a file name, then I use a shell function to find the
full path from the list and name.   But that's very rare.

Not building support for this into (any) shell means that the code to
make it work will work (or should, if written correctly) in all shells,
rather than only in one, making for much more portable scripts.

kre

ps: I particularly dislike all the changes that are "just to make the
code easier to understand" (and similar) - suggesting that kind of thing
is OK, but it is up to the maintainer to decide upon the code style, not
others.   Mixing that kind of thing in with other (substantive) code changes
is totally the wrong thing to do - makes it harder to extract the actual
changes, if something ends up not working, it can be hard to be sure whether
it is because the change wasn't installed as intended, or was always broken.





Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-14 Thread Koichi Murase
2024年5月15日(水) 4:51 Chet Ramey :
> What do folks think about forcing an option to enable using
> BASH_SOURCE_PATH? Should it be required? Is it necessary? (I personally
> think it is not.) We discussed this briefly last week but without any
> conclusion.

Either works for me (suppose the `source' builtin considers
BASH_SOURCE_PATH even without specifying the option in non-POSIX
mode). I personally think both decisions to include and not to include
the suggested option are understandable.

To talk about its necessity, I actually agree with the others. I don't
think we need it.

However, at the same time, I think the new option could be justified
from the viewpoint of language design. If we have a new namespace
``BASH_SOURCE_PATH'' for the scripts to be sourced, it would be neat
to have an option to specify the namespace exclusively. It is similar
to `unset -v'. As far as the shell script is properly designed so that
the target variable is ensured to exist, `unset var' is sufficient
because the `unset' builtin first checks the variable name and falls
back to the function name only when the variable doesn't exist. In
this sense, `unset -v var' is not required. It's not necessary, yet I
think it can be a part of a language.

--
Koichi



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-14 Thread Lawrence Velázquez
On Tue, May 14, 2024, at 3:51 PM, Chet Ramey wrote:
> On 5/13/24 6:37 AM, Matheus Afonso Martins Moreira wrote:
>> Passing the -i option to the source builtin
>> enables isolated sourcing mode which restricts
>> its search path to the directories defined by
>> the BASH_SOURCE_PATH variable. This also has
>> the added benefit of not touching PATH at all.
>
>
> What do folks think about forcing an option to enable using
> BASH_SOURCE_PATH? Should it be required? Is it necessary? (I personally
> think it is not.)

I still don't care for it.

-- 
vq



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-14 Thread Greg Wooledge
On Tue, May 14, 2024 at 03:51:10PM -0400, Chet Ramey wrote:
> On 5/13/24 6:37 AM, Matheus Afonso Martins Moreira wrote:
> > Passing the -i option to the source builtin
> > enables isolated sourcing mode which restricts
> > its search path to the directories defined by
> > the BASH_SOURCE_PATH variable. This also has
> > the added benefit of not touching PATH at all.
> 
> 
> What do folks think about forcing an option to enable using
> BASH_SOURCE_PATH? Should it be required? Is it necessary? (I personally
> think it is not.) We discussed this briefly last week but without any
> conclusion.

I think setting the BASH_SOURCE_PATH variable already expresses the
intent to use it.  There's no need for another command to enable the
variable to work.



Re: [PATCH v2 5/8] builtins/source: parse the -i option

2024-05-14 Thread Chet Ramey

On 5/13/24 6:37 AM, Matheus Afonso Martins Moreira wrote:

Passing the -i option to the source builtin
enables isolated sourcing mode which restricts
its search path to the directories defined by
the BASH_SOURCE_PATH variable. This also has
the added benefit of not touching PATH at all.



What do folks think about forcing an option to enable using
BASH_SOURCE_PATH? Should it be required? Is it necessary? (I personally
think it is not.) We discussed this briefly last week but without any
conclusion.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/