Re: [Oorexx-devel] The search order bug: a progress report, and some questions

[Josep Maria Blasco]

Hi all,

I'm copying the README.md from
https://github.com/RexxLA/rexx-repository/tree/master/ARB/standards/work-in-progress/search-order/tests/programs/TestSearchOrder
below

--- (cut) 

Test Search Order

A program to test the Search Order

Features:

   - "Call Test" and then "Call 'test'" (allows to check for
   case-sensitivity)
   - See whether search is directory-first or extension-first (special
   tailored tests for ooRexx and Regina)
   - Same (caller) directory calls
   - Current directory tests
   - PATH directory tests (PATH is modified so that it has only one
   directory -- that's enough)
   - Downward-relative directory tests (i.e., "Call 'lib/test.rex'")
   - Dot-relative directory tests (i.e., "Call './test.rex'")
   - Dotdot-relative directory tests (i.e., "Call. '../test.rex'")

To do:

   - There's a "debugLevel" variable in 'subdir/dotdotsame/same/main.rex'
   that allows control of verbosity. Change that to a program argument, expand
   and document.
   - In-program documentation.
   - Backslash-relative directory tests (Windows-only) (i.e., "Call
   '\some\path\test.rex'").
   - Drive-relative tests (Windows-only) (i.e., "Call
   'D:relative\path\test.rex'").
   - Drive-absolute tests (Windows-only) (i.e., "Call
   'D:\some\path\test.rex'").
   - The three above, but with UNC "\server\share".
   - Add support for new interpreters.

Works in

   - ooRexx (tested with 5.1.0 under Windows 11 Pro, but should work with
   other versions).
   - Regina (tested with 3.9.5 under Ubuntu 20.04 [wsl2], but should work
   with other versions).

Downloads:

   - Here
   



Installation and running the test:

   - Unzip testsearchorder.zip
   

and
   execute testsearchorder.rex in the testsearchorder folder.


--- (cut) 

You can download it here:
https://github.com/RexxLA/rexx-repository/blob/master/ARB/standards/work-in-progress/search-order/tests/programs/TestSearchOrder/testsearchorder.zip

I've summarized the results of these tests here:
https://github.com/RexxLA/rexx-repository/blob/master/ARB/standards/work-in-progress/search-order/tests/results/Regina-vs-ooRexx.md
-- I'd suggest you take some few moments to browse them.

It's very illuminating to take a look at these results and at the
comparison table.

I'd say that *the idea that, regarding the search order, "there's a Rexx
way of doing things", to be honest, is very difficult to sustain*. As
you'll see, ooRexx and Regina have few points in common, and in essential
aspects they go to opposite extremes. I don't want to copy the full
comparison chart; I'll list only some few of these aspects.

   - ooRexx has a concept of "same" (i.e., caller) directory, Regina has
   not.
   - ooRexx traverses all the possibilities *extension-first*, Regina does
   it *directory-first*.
   - ooRexx searches for a file with no extension in the last place, Regina
   does it in the first place.
   - The built-in list of extensions to search is different in ooRexx and
   Regina.
   - ooRexx allows one to specify extra (non-PATH) paths, but these go
   after the current directory (and after the application-defined path list);
   Regina puts them before the current directory.
   - ooRexx doesn't allow the user to specify extra extensions (the
   application developer can specify them); Regina controls this aspect with
   an environment variable.

Best regards,

  Josep Maria Blasco

Missatge de Josep Maria Blasco  del dia dj., 9
de febr. 2023 a les 13:02:

> Hi René,
>
> Thanks for your comments. I've been doing some tests under Regina (latest
> version, under Ubuntu). Regina exhibits some striking differences compared
> to ooRexx:
>
>- There's no concept of the "same" (or caller's directory). If you put
>a routine R in the same directory as the calling program, you can't Call R
>and expect it to work.
>- The search order is (1) REGINA_MACROS (environment variable), (2)
>current directory, (3) PATH (compare that to (1) caller's directory, (2)
>current directory, (3) application-specified path, (4) REXX_PATH, (5) PATH
>in ooRexx).
>- If the program name "contains a file path specification", then the
>search order is completely bypassed. Notice that this is much more
>stringent than ooRexx: in ooRexx, you can Call lib/my.rex and expect the
>search order to work, but in Regina the search order will be bypassed
>because the program name contains a slash.
>- This last point is *documented* in 1.4.2 (p. 15 of the 3.9.5 version
>of regina.pdf): "Note that the search algorithm to this point is ignored if
>the program name contains a file path 

Re: [Oorexx-devel] The search order bug: a progress report, and some questions

[Josep Maria Blasco]

Hi René,

Thanks for your comments. I've been doing some tests under Regina (latest
version, under Ubuntu). Regina exhibits some striking differences compared
to ooRexx:

   - There's no concept of the "same" (or caller's directory). If you put a
   routine R in the same directory as the calling program, you can't Call R
   and expect it to work.
   - The search order is (1) REGINA_MACROS (environment variable), (2)
   current directory, (3) PATH (compare that to (1) caller's directory, (2)
   current directory, (3) application-specified path, (4) REXX_PATH, (5) PATH
   in ooRexx).
   - If the program name "contains a file path specification", then the
   search order is completely bypassed. Notice that this is much more
   stringent than ooRexx: in ooRexx, you can Call lib/my.rex and expect the
   search order to work, but in Regina the search order will be bypassed
   because the program name contains a slash.
   - This last point is *documented* in 1.4.2 (p. 15 of the 3.9.5 version
   of regina.pdf): "Note that the search algorithm to this point is ignored if
   the program name contains a file path specification. eg. if "CALL .\MYPROG"
   is called, then no searching of REGINA_MACROS or PATH is done; only the
   concatenation of suffixes is carried out."
   - There is no notion of "same extension". If an extension is present,
   only this extension is searched. Otherwise, "the file name is extended by
   the extensions [1] "" (empty string), [2] ".rexx", [3] ".rex", [4] ".cmd",
   and [5]".rx" in this order. The file name case is ignored on systems that
   ignore the character case for normal file operations like DOS, Windows,
   and OS/2." Please notice that the null extension (that is, no extension)
   is the *last* one in the ooRexx search, but the *first* one in the
   Regina search.
   - The list of extensions is also different. Compare it to [1]".cls" (for
   ::requires only), [2] the same extension, the application-defined
   extensions, [3] ".rex" and [4] no extension for ooRexx.
   - There is a REGINA_SUFFIXES environment variable. If present, its
   contents are placed after the empty extension "".
   - If I read the docs correctly, I get the impression that the search
   order in Regina is *directory-first*, that is, all extensions are tried
   in a directory, then all extensions are tried again in the next directory,
   etc. ooRexx is *extension-first: *all the directories are searched for a
   given extension, then all the directories are searched again for the next
   extension, etc.

All the points except the last one have been tried with a test program that
I will upload shortly somewhere, and post the URL here. I will also try to
prepare a comparison chart and a high-level description of the different
search algorithms, so that we can compare them thoughtfully.

Re: BRexx -- Can somebody please point me to a precompiled version for
Windows, preferably the 32 bit version? I'd like to extend my test programs
to be able to work under BRexx too.

Kind regards,

  Josep Maria Blasco

Missatge de René Jansen  del dia dc., 8 de febr. 2023 a
les 12:37:

> Hi Josep,
>
> thank you for this investigation which I am sure we all have read with
> great interest.
>
> On 7 Feb 2023, at 11:35, Josep Maria Blasco 
> wrote:
>
> Once more: I don't think there's a clear, evident way to settle this
> conversation. A *decision* has to be taken. And it has to be *explained* 
> (i.e.,
> documented) and, if possible, *justified*. The last part is optional, of
> course: one can define a language as one sees fit.
>
> The weight, if any, of my contribution, is only to emphasize two things:
>
>- Other languages tackle this problem in a particular, coincident way.
>- And that way is the most economic in terms of describing the search
>procedure.
>
> This does not mean that what I am proposing should be accepted. It's only
> my point of view.
>
>
> I think this needs to be well documented (for all platforms Rexx is
> running on, with additions for the oo variants. There is a new Rexx ARB
> starting up where work can be done to isolate the different components of
> the question where things are found - I think one of the most important
> questions one encounters when trying to make something non-trivial.
>
> There is the component of history - CMS and TSO were there before the
> DOS/OS2/Unix world. There is the language philosophy angle - other
> languages make choices the might not be the Rexx way (those full of curly
> braces or significant spaces), while on some platforms (NetRexx - Java) the
> choices of the latter are like gravity.
>
> One question that comes up is if you compared the way ooRexx 5.X does this
> to Regina, Brexx or OS/2. I think one of the things an extended standard
> document could do is to help describe this and propose a standard way of
> implementing this which is straightforward on Windows/Linux(including other
> Unix like platforms like macOS).
>
> It could be argued that this 

Re: [Oorexx-devel] The search order bug: a progress report, and some questions

[René Jansen]

Hi Josep,

thank you for this investigation which I am sure we all have read with great 
interest.

> On 7 Feb 2023, at 11:35, Josep Maria Blasco  
> wrote:
> 
> Once more: I don't think there's a clear, evident way to settle this 
> conversation. A decision has to be taken. And it has to be explained (i.e., 
> documented) and, if possible, justified. The last part is optional, of 
> course: one can define a language as one sees fit.
> 
> The weight, if any, of my contribution, is only to emphasize two things:
> Other languages tackle this problem in a particular, coincident way.
> And that way is the most economic in terms of describing the search procedure.
> This does not mean that what I am proposing should be accepted. It's only my 
> point of view.
> 

I think this needs to be well documented (for all platforms Rexx is running on, 
with additions for the oo variants. There is a new Rexx ARB starting up where 
work can be done to isolate the different components of the question where 
things are found - I think one of the most important questions one encounters 
when trying to make something non-trivial.

There is the component of history - CMS and TSO were there before the 
DOS/OS2/Unix world. There is the language philosophy angle - other languages 
make choices the might not be the Rexx way (those full of curly braces or 
significant spaces), while on some platforms (NetRexx - Java) the choices of 
the latter are like gravity.

One question that comes up is if you compared the way ooRexx 5.X does this to 
Regina, Brexx or OS/2. I think one of the things an extended standard document 
could do is to help describe this and propose a standard way of implementing 
this which is straightforward on Windows/Linux(including other Unix like 
platforms like macOS).

It could be argued that this is not part of the language but of the 
implementation - but I am doubting that the documentation of a function(method) 
call or the CALL statement is complete without a specification where it finds 
what it calls. 

Economy in terms of describing - well, that is relative I think - the procedure 
for finding and overriding a BIF in VM at least requires a flowchart in the VM 
documentation (and the way Brexx does it is different) but it expresses a 
common goal of being able to override a built-in-function. In z/OS we have lpa 
(flpa, mlpa), link list, sysproc and sysexec concatenation (and the alt 
library) where a compiled Rexx program could live and I would not be able to 
economically describe that, but they are all there for a reason.

Thank you again for bringing up this discussion.

Could you mail me that zip file so I can put it somewhere where more people can 
look at it?

best regards,

René Jansen.___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] The search order bug: a progress report, and some questions

[Josep Maria Blasco]

Hello Rony,

I sent a previous reply, but it was intercepted by the SourceForge filters
because it contained a zip file attachment. SourceForge informed me about
this fact only some few hours ago (sigh). I will take the opportunity to
compose a new, more thoughtful, reply.

Missatge de Rony G. Flatscher  del dia dg., 5 de
febr. 2023 a les 19:40:

(snip)

> Not being the expert in this corner a question to you as you have
>> developed a deep insight in the meantime: while developing on json.cls
>> there are two json.cls present at the moment (Windows layout):
>>
>> C:\Program Files\ooRexx\json.cls
>> C:\Program Files\ooRexx\rexxtry.rex
>>
>> F:\work\svn\oorexx\sandbox\rony\json\json.cls
>>
>> current directory is: F:\work\svn\oorexx\sandbox\rony\json
>>
>> Now while doing interactive tests via rexxtry.rex (i.e. "C:\Program
>> Files\ooRexx\rexxtry.rex") doing a "call json.cls" will find and call
>> "C:\Program Files\ooRexx\json.cls" and *not* "json.cls" in the current
>> directory!
>>
> Sure :) Rexxtry.rex is in the "C:\Program Files\ooRexx" directory. This is
> the "same" directory of the docs, or the "parent" directory of the source
> (I'd prefer to call it the "caller's" directory, I think it's a more
> accurate description; but I disgress). "Call json.cls" is one of the cases
> where Rexx works as documented. And the search order begins... by the
> "same" directory, as per the docs. That's why the version in "C:" is called.
>
>> However, in the same rexxtry.rex session in the current directory doing a
>> "call .\json.cls" *will* resolve and call
>> "F:\work\svn\oorexx\sandbox\rony\json\json.cls": so "." will force using
>> the current directory for finding external Rexx programs.
>>
> That's exactly our bug.
>
> Hmm, maybe it is not a bug after all, but working as intended as this
> allows for overriding the default search order that starts out in the
> source directory (the directory the Rexx script got loaded and run from as
> in the case of rexxtry.rex, doing a "parse source . . path2source" would
> denote the Rexx program's source directory.)
>

I would object to that. If it were working, as you say, "as intended", then
it should have been documented. But it is not. My impression is that *this
is just a trick that works and that we are used to this trick*, to a point
that we find it natural.

The search order algorithm, by definition, imposes a certain *opacity* when
there are duplicates. A super-path list is formed by concatenating the
caller's directory, the present directory, the application-defined extra
path, and the contents of REXX_PATH and PATH. Let p_1, ..., p_n be the
component paths of this super-path list. Assume that in the search for
name.ext we have several i in [1..n] such that p_i || sep || name.ext
exists (sometimes it's a little more complicated than a simple
concatenation, but allow me, for the sake of simplicity). Let i_1, ..., i_k
be these indexes, where i_1 < i_2 <...
> If I understand you properly, the only difference between this last test
> and the previous one is the addition of ".\". This is, precisely, one of
> the cases where what Rexx does and what's in the docs *differ*. ".\"
> should mean "in *this* directory". But what does "this" mean, i.e.,
> "this" relative to what?
>
> Probably "the current directory".
>

If you were using the command line, you'd be right. For a programming
language, it's not so obvious. In any case, it's something that has to be
(1) defined and (2) documented.

Let's see what other languages do.

The gcc compiler, for example, takes "." to mean "each and every directory
specified with the -I" compiler option. Then, for example, include
"./this.h" will refer to whatever directories have been specified. Each and
every of them. "." is relative. I can provide testfiles if you want them
(but not in a zipfile, as I've had to learn).

#include 
#include "./inc.h"


int main() {

  printf("%s\n", THIS);

  return 0;

}


All that main does is to print THIS, which obviously has to be defined
somewhere else, i.e., in ./inc.h. Now, what does ./inc.h refer to? Well, it
depends on what do we specify in the -I compiler option. Let's assume that
we have two subdirectories called one and two. Now after compiling with

gcc -Ione -o main main.c,


if we type "./main", we will get

one,


assuming of course that inc.h in one #defines THIS as "one"; and after
compiling with

gcc -Itwo -o main main.c,

we will get

two


assuming, once more, the necessary and symmetrical #defines. That's what (a
compiler for) a compiled language does.

Let's see, on the other hand, what are the path calculations performed by
Python, a dynamic, interpreted language:

>>> os.getcwd()

'D:\\Dropbox'
>>> (Path("C:") / "./int.rex").resolve()
WindowsPath('C:/Users/jmblasco/int.rex')


That is, "./", relative to "C:", is "C:/Users/jmblasco/int.rex", something
relative to *the current directory of the C: drive* (which happened to be
C:\users\jmblasco), not something relative to the current 

Re: [Oorexx-devel] The search order bug: a progress report, and some questions

[Rony G. Flatscher]

Hi Josep Maria,

On 05.02.2023 15:17, Josep Maria Blasco wrote:

Missatge de Rony G. Flatscher  del dia dg., 5 de febr. 
2023 a les 13:29:


... cut ...


Not being the expert in this corner a question to you as you have developed 
a deep insight in
the meantime: while developing on json.cls there are two json.cls present 
at the moment
(Windows layout):

C:\Program Files\ooRexx\json.cls
C:\Program Files\ooRexx\rexxtry.rex

F:\work\svn\oorexx\sandbox\rony\json\json.cls

current directory is: F:\work\svn\oorexx\sandbox\rony\json

Now while doing interactive tests via rexxtry.rex (i.e. "C:\Program 
Files\ooRexx\rexxtry.rex")
doing a "call json.cls" will find and call "C:\Program 
Files\ooRexx\json.cls" and *not*
"json.cls" in the current directory!

Sure :) Rexxtry.rex is in the "C:\Program Files\ooRexx" directory. This is the "same" directory of 
the docs, or the "parent" directory of the source (I'd prefer to call it the "caller's" directory, 
I think it's a more accurate description; but I disgress). "Call json.cls" is one of the cases 
where Rexx works as documented. And the search order begins... by the "same" directory, as per the 
docs. That's why the version in "C:" is called.


However, in the same rexxtry.rex session in the current directory doing a "call 
.\json.cls"
*will* resolve and call "F:\work\svn\oorexx\sandbox\rony\json\json.cls": so 
"." will force
using the current directory for finding external Rexx programs.

That's exactly our bug.
Hmm, maybe it is not a bug after all, but working as intended as this allows for overriding the 
default search order that starts out in the source directory (the directory the Rexx script got 
loaded and run from as in the case of rexxtry.rex, doing a "parse source . . path2source" would 
denote the Rexx program's source directory.)


If I understand you properly, the only difference between this last test and the previous one is 
the addition of ".\". This is, precisely, one of the cases where what Rexx does and what's in the 
docs /differ/. ".\" should mean "in /this/ directory". But what does "this" mean, i.e., "this" 
relative to what?

Probably "the current directory".

Well, according to the docs, we have to respect the search order. And, in the first place... 
"this" should be relative to the "same" directory, that is, the very same "same" directory 
(sorry). That's the caller's directory, that is, where rexxtry resides. BUT... here's this bug we 
are talking about: since the callee's filespec begins with ".\", the search path is bypassed (it 
should not be!). This means that the filespec is resolved (more or less) like in the command line 
(to be true, the resolution mechanism provided by the SearchPath Windows API --what Rexx for 
Windows uses-- differs from the command line one, with the clear intention to get the programmers 
and users altogether completely braindamaged, I'm obliged to presume). But when you're in the 
command line (and, in this case, when you call SearchPath), "." means the /current/ directory, not 
the /same/ or caller's directory. Hence, the F: version of json.cls gets called.


This was actually the intention when doing a 'call ".\json.cls"', to override "json.cls" in 
rexxtry.rex' source directory. What would you suggest to use instead, a fully qualified path to 
".\json.cls" (which one could hardly hard code in advance)? Maybe I have myself not fully understood 
all the ramifications of what you suggest, or why you would regard 'call ".\json.cls"' resolving the 
file in the current directory would be an error (for Rexx that is).


If you had instead tried "Call C:json.cls", you'd have had the C: version called. This is maybe 
the easiest way not to get lost: "Call C:json.cls", and "Call F:json.cls". Of course, the current 
directory of C: /and/ of F: have to point to the right places.


So it seems that removing "." may have side effects that are not (yet) 
covered by the test cases?

To the contrary: if you had tried the Ubuntu version of the interpreter after having applied my 
patch (https://sourceforge.net/p/oorexx/bugs/1865/?limit=25#687f), then "Call json.cls" and "Call 
./json.cls" would have had the same effect. As one would expect, both would have called the 
version in the /same /directory. To call the version in the /current/ directory without using a 
complete, absolute path, you'd have had to resort to F:json.cls or the like :)


If possible the lookup should work the same on Unix and Windows such that a Rexx programmer would 
not have to worry on which platform the program gets run.


Personally I never use relative file paths if not absolutely necessary (and very rarely so, hence 
being more than rusty by now!). Rather than using relative paths in the Rexx programs I would adjust 
the PATH environment variable such that the desired search order is reflected via it and then start 
the Rexx program in that environment.


If you look up 

Re: [Oorexx-devel] The search order bug: a progress report, and some questions

[Josep Maria Blasco]

Missatge de Rony G. Flatscher  del dia dg., 5 de
febr. 2023 a les 13:29:

> Hi Josep Maria,
>
> thank you very much for this clear, thorough presentation of the problem
> description and the patches, although I am not the right person to assess
> it in full, yet. So hoping that once Rick and Erich get to it they can give
> authorative feedback.
>
A real pleasure. Really. I've enjoyed myself a lot. And I hope it's useful
:)

> Not being the expert in this corner a question to you as you have
> developed a deep insight in the meantime: while developing on json.cls
> there are two json.cls present at the moment (Windows layout):
>
> C:\Program Files\ooRexx\json.cls
> C:\Program Files\ooRexx\rexxtry.rex
>
> F:\work\svn\oorexx\sandbox\rony\json\json.cls
>
> current directory is: F:\work\svn\oorexx\sandbox\rony\json
>
> Now while doing interactive tests via rexxtry.rex (i.e. "C:\Program
> Files\ooRexx\rexxtry.rex") doing a "call json.cls" will find and call
> "C:\Program Files\ooRexx\json.cls" and *not* "json.cls" in the current
> directory!
>
Sure :) Rexxtry.rex is in the "C:\Program Files\ooRexx" directory. This is
the "same" directory of the docs, or the "parent" directory of the source
(I'd prefer to call it the "caller's" directory, I think it's a more
accurate description; but I disgress). "Call json.cls" is one of the cases
where Rexx works as documented. And the search order begins... by the
"same" directory, as per the docs. That's why the version in "C:" is called.

> However, in the same rexxtry.rex session in the current directory doing a
> "call .\json.cls" *will* resolve and call
> "F:\work\svn\oorexx\sandbox\rony\json\json.cls": so "." will force using
> the current directory for finding external Rexx programs.
>
That's exactly our bug.

If I understand you properly, the only difference between this last test
and the previous one is the addition of ".\". This is, precisely, one of
the cases where what Rexx does and what's in the docs *differ*. ".\" should
mean "in *this* directory". But what does "this" mean, i.e., "this"
relative to what? Well, according to the docs, we have to respect the
search order. And, in the first place... "this" should be relative to the
"same" directory, that is, the very same "same" directory (sorry). That's
the caller's directory, that is, where rexxtry resides. BUT... here's this
bug we are talking about: since the callee's filespec begins with ".\", the
search path is bypassed (it should not be!). This means that the filespec
is resolved (more or less) like in the command line (to be true, the
resolution mechanism provided by the SearchPath Windows API --what Rexx for
Windows uses-- differs from the command line one, with the clear intention
to get the programmers and users altogether completely braindamaged, I'm
obliged to presume). But when you're in the command line (and, in this
case, when you call SearchPath), "." means the *current* directory, not the
*same* or caller's directory. Hence, the F: version of json.cls gets called.

If you had instead tried "Call C:json.cls", you'd have had the C: version
called. This is maybe the easiest way not to get lost: "Call C:json.cls",
and "Call F:json.cls". Of course, the current directory of C: *and* of F:
have to point to the right places.

> So it seems that removing "." may have side effects that are not (yet)
> covered by the test cases?
>
To the contrary: if you had tried the Ubuntu version of the interpreter
after having applied my patch (
https://sourceforge.net/p/oorexx/bugs/1865/?limit=25#687f), then "Call
json.cls" and "Call ./json.cls" would have had the same effect. As one
would expect, both would have called the version in the *same *directory.
To call the version in the *current* directory without using a complete,
absolute path, you'd have had to resort to F:json.cls or the like :)

> What is your view?
>
> Best regards
>
Hope that helps,

  Josep Maria


> ---rony
>
>
> On 04.02.2023 14:27, Josep Maria Blasco wrote:
>
> *[Bringing the discussion from the RexxLA list, as per Rony's suggestion]*
>
> *Summary*
>
>- The discussion initiated in this RexxLA thread:
>
> https://groups.io/g/rexxla-members/topic/the_external_program_search/96286751
>,
>- then moved to
>https://groups.io/g/rexxla-members/topic/getting_oorexx_docs_test/96398260
>
>- and finally to
>https://groups.io/g/rexxla-members/topic/next_steps_testing_patch/96697943
>
>
>- A good description of the problem can be found here:
>https://groups.io/g/rexxla-members/message/947
>- A preliminary test can be found here:
>https://groups.io/g/rexxla-members/message/954
>- Another, quite exhaustive, review can be found here:
>https://groups.io/g/rexxla-members/message/956
>
> I've opened a bug report: https://sourceforge.net/p/oorexx/bugs/1865/ ;
> Erich observed there some interesting details about the behaviour of the
> SearchPath Windows API.
>
> I submitted a patch for the docs, as a first draft to 

Re: [Oorexx-devel] The search order bug: a progress report, and some questions

[Rony G. Flatscher]

Hi Josep Maria,

thank you very much for this clear, thorough presentation of the problem description and the 
patches, although I am not the right person to assess it in full, yet. So hoping that once Rick and 
Erich get to it they can give authorative feedback.


---

Not being the expert in this corner a question to you as you have developed a deep insight in the 
meantime: while developing on json.cls there are two json.cls present at the moment (Windows layout):


   C:\Program Files\ooRexx\json.cls
   C:\Program Files\ooRexx\rexxtry.rex

   F:\work\svn\oorexx\sandbox\rony\json\json.cls

   current directory is: F:\work\svn\oorexx\sandbox\rony\json

Now while doing interactive tests via rexxtry.rex (i.e. "C:\Program Files\ooRexx\rexxtry.rex") doing 
a "call json.cls" will find and call "C:\Program Files\ooRexx\json.cls" and *not* "json.cls" in the 
current directory!


However, in the same rexxtry.rex session in the current directory doing a "call .\json.cls" *will* 
resolve and call "F:\work\svn\oorexx\sandbox\rony\json\json.cls": so "." will force using the 
current directory for finding external Rexx programs.


So it seems that removing "." may have side effects that are not (yet) covered 
by the test cases?

What is your view?

Best regards

---rony


On 04.02.2023 14:27, Josep Maria Blasco wrote:

/[Bringing the discussion from the RexxLA list, as per Rony's suggestion]/

*_Summary_*

  * The discussion initiated in this RexxLA thread:

https://groups.io/g/rexxla-members/topic/the_external_program_search/96286751 ,
  * then moved to 
https://groups.io/g/rexxla-members/topic/getting_oorexx_docs_test/96398260
  * and finally to 
https://groups.io/g/rexxla-members/topic/next_steps_testing_patch/96697943

  * A good description of the problem can be found here:
https://groups.io/g/rexxla-members/message/947
  * A preliminary test can be found here: 
https://groups.io/g/rexxla-members/message/954
  * Another, quite exhaustive, review can be found here:
https://groups.io/g/rexxla-members/message/956

I've opened a bug report: https://sourceforge.net/p/oorexx/bugs/1865/ ; Erich observed there some 
interesting details about the behaviour of the SearchPath Windows API.


I submitted a patch for the docs, as a first draft to comment; 
https://sourceforge.net/p/oorexx/patches/218/


*_Present_*

Yesterday I submitted a patch for CALL.testGroup: https://sourceforge.net/p/oorexx/patches/219/ 
(will submit an updated version, which addresses some problems, later).


Yesterday I also had the opportunity, after some stimulus by Rony, to try my first compilation of 
the interpreter (under Ubuntu 22.04 --- wonderfully easy, btw).


*_Interesting news_*

If in platform/unix/SysFileSystem.cpp we change

return name[0] == '~' || name[0] == '/' ||
      (name[0] == '.' && name[1] == '/') ||
      (name[0] == '.' && name[1] == '.' && name[2] == '/');


and instead write only

return name[0] == '~' || name[0] == '/';


(in SysFileSystem:hasDirectory), i.e., we eliminate the tests for the "./"and "../"cases, then the 
new tests, test_search_order_4and test_search_order_5will pass (they currently don't under 5.0.0).


Additionally, _the whole test suite passes._ Tests were made using

./testOORexx.rex -X native_API


I will upload the corresponding patch soon.

*_Some comments more, and some questions_*

  * I get some impression of disorder. Maybe I should have centralized all my 
submissions in a
single place (i.e., in a RFE, for example, or in a bug report)? Currently, 
it's getting
complicated to collect all the information (that's one of the reasons why I 
am writing that
post). In the general case, what's the way to proceed? Should each of the 
different
contributions cross-reference each other?
  * The patched Unix interpreter passes all the tests, but I don't know whether 
I've inadvertently
introduced some subtle form of new bug.
  * Patching the Windows interpreter will be much more complicated, and I'm not 
certain that I
possess the required skills to tackle the job. I could try to teach myself 
C++ (never
programmed in C++ in all my life, I started with the DOS PL/I Optimizing 
Compiler, then learnt
machine language and ASSEMBLER for DOS/360 [the "D" version: it had 
WXTRNs], and then Pascal
and REXX for VM/SP -- which spoiled me completely :)) and try to program 
the required changes,
but I'm afraid this would take me several weeks. The real problem, though, 
is not time (I
don't believe we are in a hurry for a new release), but the quality of my 
hypothetical work.
I've noticed that C++ does all kind of weird tricks with pointer 
arithmetic, and I'm afraid
I'd end up by introducing a number of beginner-level bugs if I attempted to 
solve the problem
by myself. Will try it, anyway, if this is what you advise.
  * What I can produce rapidly (say in one or two days) is a pseudocode version 
(i.e., a working
Rexx 

[Oorexx-devel] The search order bug: a progress report, and some questions

[Josep Maria Blasco]

*[Bringing the discussion from the RexxLA list, as per Rony's suggestion]*

*Summary*

   - The discussion initiated in this RexxLA thread:
   https://groups.io/g/rexxla-members/topic/the_external_program_search/96286751
   ,
   - then moved to
   https://groups.io/g/rexxla-members/topic/getting_oorexx_docs_test/96398260

   - and finally to
   https://groups.io/g/rexxla-members/topic/next_steps_testing_patch/96697943


   - A good description of the problem can be found here:
   https://groups.io/g/rexxla-members/message/947
   - A preliminary test can be found here:
   https://groups.io/g/rexxla-members/message/954
   - Another, quite exhaustive, review can be found here:
   https://groups.io/g/rexxla-members/message/956

I've opened a bug report: https://sourceforge.net/p/oorexx/bugs/1865/ ;
Erich observed there some interesting details about the behaviour of the
SearchPath Windows API.

I submitted a patch for the docs, as a first draft to comment;
https://sourceforge.net/p/oorexx/patches/218/

*Present*

Yesterday I submitted a patch for CALL.testGroup:
https://sourceforge.net/p/oorexx/patches/219/ (will submit an updated
version, which addresses some problems, later).

Yesterday I also had the opportunity, after some stimulus by Rony, to try
my first compilation of the interpreter (under Ubuntu 22.04 --- wonderfully
easy, btw).

*Interesting news*

If in platform/unix/SysFileSystem.cpp we change

return name[0] == '~' || name[0] == '/' ||
  (name[0] == '.' && name[1] == '/') ||
  (name[0] == '.' && name[1] == '.' && name[2] == '/');


and instead write only

return name[0] == '~' || name[0] == '/';


(in SysFileSystem:hasDirectory), i.e., we eliminate the tests for the "./"
and "../" cases, then the new tests, test_search_order_4 and
test_search_order_5 will pass (they currently don't under 5.0.0).

Additionally, *the whole test suite passes.* Tests were made using

./testOORexx.rex -X native_API


I will upload the corresponding patch soon.

*Some comments more, and some questions*

   - I get some impression of disorder. Maybe I should have centralized all
   my submissions in a single place (i.e., in a RFE, for example, or in a bug
   report)? Currently, it's getting complicated to collect all the information
   (that's one of the reasons why I am writing that post). In the general
   case, what's the way to proceed? Should each of the different contributions
   cross-reference each other?
   - The patched Unix interpreter passes all the tests, but I don't know
   whether I've inadvertently introduced some subtle form of new bug.
   - Patching the Windows interpreter will be much more complicated, and
   I'm not certain that I possess the required skills to tackle the job. I
   could try to teach myself C++ (never programmed in C++ in all my life, I
   started with the DOS PL/I Optimizing Compiler, then learnt machine language
   and ASSEMBLER for DOS/360 [the "D" version: it had WXTRNs], and then Pascal
   and REXX for VM/SP -- which spoiled me completely :)) and try to program
   the required changes, but I'm afraid this would take me several weeks. The
   real problem, though, is not time (I don't believe we are in a hurry for a
   new release), but the quality of my hypothetical work. I've noticed that
   C++ does all kind of weird tricks with pointer arithmetic, and I'm afraid
   I'd end up by introducing a number of beginner-level bugs if I attempted to
   solve the problem by myself. Will try it, anyway, if this is what you
   advise.
   - What I can produce rapidly (say in one or two days) is a
   pseudocode version (i.e., a working Rexx program) of what has to be done in
   the Windows version. Fundamentally, the use of the SearchPath API has to
   be bypassed, and all checks have to be done "by hand" (like they are being
   done, already, in the Unix version).

  Josep Maria Blasco
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel