Re: [Oorexx-devel] Address With progress

[Rony G. Flatscher]

On 25.04.2018 18:40, Erich Steinböck wrote:
>
> Erich will have to speak for the other platforms
>
>
> Unix should be fine.  I just noticed that somehow the latest commit, fixing 
> the environment for
> spawn, didn't come through - did the commit now.
>
> For Windows there may still be work ahead. For me the new ADDRESS.testGroup 
> doesn't run.
> But then I do have a very strange issue with addressWith Debug builds: each 
> time a new rexx.exe is
> built, it will hang for a full minute (60 seconds exactly) at CreateProcess() 
> in the command
> handler (for any arbitrary command) before continuing.  This may repeat two 
> or three times, until
> it starts behaving normally again (no delay by CreateProcess).  The issue 
> will reappear as soon as
> I build a new version.
> This issue doesn't happen for Debug builds of trunk, and I've never witnessed 
> it before.
> I treid swapping back the old command handler (all other Rick's changes still 
> in place), but even
> then the issue is still there.
Your commit  also removed the 
test command handler
from Windows 
(sandbox/rick/addresswith/interpreter/platform/windows/SystemCommands.cpp), 
which
ADDRESS.testGroup seems to depend on.

---rony

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

It is in the InputRedirectory.cpp file.

Rick

On Wed, Apr 25, 2018 at 2:27 PM, Gil Barmwater 
wrote:

> OK, thanks.  I'm actually doing a redirect of input from a stream object
> but I assume the argument is the same.  And where do I look for the code
> that reads that stream object?
> Gil
>
> On 4/25/2018 2:20 PM, Rick McGuire wrote:
>
> That is the correct behavior. If you are doing the output to a stream
> object directly rather than using the STREAM resource (or USING with a
> .File object), the stream is neither opened nor closed during the
> processing. In fact, an OutputStream object need not support a close
> method, so it would be wrong to attempt it.
>
> Rick
>
> On Wed, Apr 25, 2018 at 2:15 PM, Gil Barmwater 
> wrote:
>
>> All of the builds I have attempted were Windows REL32 but I've never
>> experienced a hang at CreateProcess().  FWIW.
>>
>> I do have a question about a problem I'm seeing, however.  The scenario
>> is as follows: I've modified the test bed I wrote for the package I
>> presented at the Symposium so it would run ADDRESS ... WITH commands
>> instead.  All of my tests run except when I run two particular tests
>> consecutively.  Each test runs fine standalone.  Both tests use a file
>> called 'abc1.txt' which contains 4 lines of rexx instructions, the last
>> being 'exit'.  The first test issues > using (smallFile)>, having first done > .stream~new('abc1.txt')>.  In other words, redirecting Input to a stream
>> object.  That works as expected.  The second test issues > abc1.txt with ...> .  When run alone, it too, works as expected but if run
>> immediately after the first test, it gets > initialization: File "C:\Rexx\projects\addresswith\abc1.txt" is
>> unreadable.>  Suspecting the file may not have been closed by the first
>> test, I added a  immediately after the address ... with in
>> that test and that fixed the problem.  I attempted to find where the input
>> stream object was processed in the C++ code but got lost .  Any
>> suggestions would be appreciated.
>>
>> BTW, I saw all the work that has gone into the address.testgroup and was
>> very impressed! Nice work!
>> Gil
>>
>> On 4/25/2018 12:40 PM, Erich Steinböck wrote:
>>
>> Erich will have to speak for the other platforms
>>>
>>
>> Unix should be fine.  I just noticed that somehow the latest commit,
>> fixing the environment for spawn, didn't come through - did the commit now.
>>
>> For Windows there may still be work ahead. For me the new
>> ADDRESS.testGroup doesn't run.
>> But then I do have a very strange issue with addressWith Debug builds:
>> each time a new rexx.exe is built, it will hang for a full minute (60
>> seconds exactly) at CreateProcess() in the command handler (for any
>> arbitrary command) before continuing.  This may repeat two or three times,
>> until it starts behaving normally again (no delay by CreateProcess).  The
>> issue will reappear as soon as I build a new version.
>> This issue doesn't happen for Debug builds of trunk, and I've never
>> witnessed it before.
>> I treid swapping back the old command handler (all other Rick's changes
>> still in place), but even then the issue is still there.
>>
>>
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>
>>
>>
>> ___
>> Oorexx-devel mailing 
>> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>> --
>> Gil Barmwater
>>
>>
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Oorexx-devel mailing list
>> Oorexx-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Oorexx-devel mailing 
> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
> --
> Gil Barmwater
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on 

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

OK, thanks.  I'm actually doing a redirect of input from a stream object 
but I assume the argument is the same.  And where do I look for the code 
that reads that stream object?


Gil
On 4/25/2018 2:20 PM, Rick McGuire wrote:
That is the correct behavior. If you are doing the output to a stream 
object directly rather than using the STREAM resource (or USING with a 
.File object), the stream is neither opened nor closed during the 
processing. In fact, an OutputStream object need not support a close 
method, so it would be wrong to attempt it.


Rick

On Wed, Apr 25, 2018 at 2:15 PM, Gil Barmwater 
> wrote:


All of the builds I have attempted were Windows REL32 but I've
never experienced a hang at CreateProcess(). FWIW.

I do have a question about a problem I'm seeing, however.  The
scenario is as follows: I've modified the test bed I wrote for the
package I presented at the Symposium so it would run ADDRESS ...
WITH commands instead.  All of my tests run except when I run two
particular tests consecutively.  Each test runs fine standalone. 
Both tests use a file called 'abc1.txt' which contains 4 lines of
rexx instructions, the last being 'exit'.  The first test issues
, having
first done . In other words,
redirecting Input to a stream object. That works as expected.  The
second test issues  .  When
run alone, it too, works as expected but if run immediately after
the first test, it gets   Suspecting the file may not have been closed by the
first test, I added a  immediately after the
address ... with in that test and that fixed the problem.  I
attempted to find where the input stream object was processed in
the C++ code but got lost . Any suggestions would be
appreciated.

BTW, I saw all the work that has gone into the address.testgroup
and was very impressed! Nice work!

Gil

On 4/25/2018 12:40 PM, Erich Steinböck wrote:


Erich will have to speak for the other platforms


Unix should be fine.  I just noticed that somehow the latest
commit, fixing the environment for spawn, didn't come through -
did the commit now.

For Windows there may still be work ahead. For me the new
ADDRESS.testGroup doesn't run.
But then I do have a very strange issue with addressWith Debug
builds: each time a new rexx.exe is built, it will hang for a
full minute (60 seconds exactly) at CreateProcess() in the
command handler (for any arbitrary command) before continuing. 
This may repeat two or three times, until it starts behaving
normally again (no delay by CreateProcess). The issue will
reappear as soon as I build a new version.
This issue doesn't happen for Debug builds of trunk, and I've
never witnessed it before.
I treid swapping back the old command handler (all other Rick's
changes still in place), but even then the issue is still there.



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!http://sdm.link/slashdot


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/oorexx-devel



-- 
Gil Barmwater




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/oorexx-devel





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel


--
Gil Barmwater

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

That is the correct behavior. If you are doing the output to a stream
object directly rather than using the STREAM resource (or USING with a
.File object), the stream is neither opened nor closed during the
processing. In fact, an OutputStream object need not support a close
method, so it would be wrong to attempt it.

Rick

On Wed, Apr 25, 2018 at 2:15 PM, Gil Barmwater 
wrote:

> All of the builds I have attempted were Windows REL32 but I've never
> experienced a hang at CreateProcess().  FWIW.
>
> I do have a question about a problem I'm seeing, however.  The scenario is
> as follows: I've modified the test bed I wrote for the package I presented
> at the Symposium so it would run ADDRESS ... WITH commands instead.  All of
> my tests run except when I run two particular tests consecutively.  Each
> test runs fine standalone.  Both tests use a file called 'abc1.txt' which
> contains 4 lines of rexx instructions, the last being 'exit'.  The first
> test issues , having
> first done .  In other words,
> redirecting Input to a stream object.  That works as expected.  The second
> test issues  .  When run alone, it too,
> works as expected but if run immediately after the first test, it gets
>  "C:\Rexx\projects\addresswith\abc1.txt" is unreadable.>  Suspecting the
> file may not have been closed by the first test, I added a
>  immediately after the address ... with in that test and
> that fixed the problem.  I attempted to find where the input stream object
> was processed in the C++ code but got lost .  Any suggestions would
> be appreciated.
>
> BTW, I saw all the work that has gone into the address.testgroup and was
> very impressed! Nice work!
> Gil
>
> On 4/25/2018 12:40 PM, Erich Steinböck wrote:
>
> Erich will have to speak for the other platforms
>>
>
> Unix should be fine.  I just noticed that somehow the latest commit,
> fixing the environment for spawn, didn't come through - did the commit now.
>
> For Windows there may still be work ahead. For me the new
> ADDRESS.testGroup doesn't run.
> But then I do have a very strange issue with addressWith Debug builds:
> each time a new rexx.exe is built, it will hang for a full minute (60
> seconds exactly) at CreateProcess() in the command handler (for any
> arbitrary command) before continuing.  This may repeat two or three times,
> until it starts behaving normally again (no delay by CreateProcess).  The
> issue will reappear as soon as I build a new version.
> This issue doesn't happen for Debug builds of trunk, and I've never
> witnessed it before.
> I treid swapping back the old command handler (all other Rick's changes
> still in place), but even then the issue is still there.
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Oorexx-devel mailing 
> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
> --
> Gil Barmwater
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

All of the builds I have attempted were Windows REL32 but I've never 
experienced a hang at CreateProcess().  FWIW.


I do have a question about a problem I'm seeing, however.  The scenario 
is as follows: I've modified the test bed I wrote for the package I 
presented at the Symposium so it would run ADDRESS ... WITH commands 
instead.  All of my tests run except when I run two particular tests 
consecutively.  Each test runs fine standalone. Both tests use a file 
called 'abc1.txt' which contains 4 lines of rexx instructions, the last 
being 'exit'.  The first test issues input using (smallFile)>, having first done .stream~new('abc1.txt')>.  In other words, redirecting Input to a stream 
object.  That works as expected.  The second test issues rexx abc1.txt with ...> .  When run alone, it too, works as expected but 
if run immediately after the first test, it gets during initialization: File "C:\Rexx\projects\addresswith\abc1.txt" is 
unreadable.> Suspecting the file may not have been closed by the first 
test, I added a  immediately after the address ... with 
in that test and that fixed the problem.  I attempted to find where the 
input stream object was processed in the C++ code but got lost .  
Any suggestions would be appreciated.


BTW, I saw all the work that has gone into the address.testgroup and was 
very impressed! Nice work!


Gil
On 4/25/2018 12:40 PM, Erich Steinböck wrote:


Erich will have to speak for the other platforms


Unix should be fine.  I just noticed that somehow the latest commit, 
fixing the environment for spawn, didn't come through - did the commit 
now.


For Windows there may still be work ahead. For me the new 
ADDRESS.testGroup doesn't run.
But then I do have a very strange issue with addressWith Debug builds: 
each time a new rexx.exe is built, it will hang for a full minute (60 
seconds exactly) at CreateProcess() in the command handler (for any 
arbitrary command) before continuing.  This may repeat two or three 
times, until it starts behaving normally again (no delay by 
CreateProcess).  The issue will reappear as soon as I build a new version.
This issue doesn't happen for Debug builds of trunk, and I've never 
witnessed it before.
I treid swapping back the old command handler (all other Rick's 
changes still in place), but even then the issue is still there.



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel


--
Gil Barmwater

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

>
> Erich will have to speak for the other platforms
>

Unix should be fine.  I just noticed that somehow the latest commit, fixing
the environment for spawn, didn't come through - did the commit now.

For Windows there may still be work ahead. For me the new ADDRESS.testGroup
doesn't run.
But then I do have a very strange issue with addressWith Debug builds: each
time a new rexx.exe is built, it will hang for a full minute (60 seconds
exactly) at CreateProcess() in the command handler (for any arbitrary
command) before continuing.  This may repeat two or three times, until it
starts behaving normally again (no delay by CreateProcess).  The issue will
reappear as soon as I build a new version.
This issue doesn't happen for Debug builds of trunk, and I've never
witnessed it before.
I treid swapping back the old command handler (all other Rick's changes
still in place), but even then the issue is still there.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

The docs are not written yet, but there are lots of examples of usage in
the two test group files in the root of the source tree. In a nutshell,

address env command with input resourcei output (REPLACE|APPEND) resourceo
error (REPLACE|APPEND) resourseo

resourcei can be

STEM stemvarname
STREAM cexpr
USING cexpr

cexpr is the same sort of expression used with USE ARG default values or
the subkeyword expressions on RAISE and FORWARD.

With USING, the expression is evaluated and the source determined by the
type of the result:

1) a string. Just the string value it written to the input pipe. Useful if
you only need to send one line.
2) a stem object. Same as using the STEM option
3) a File object. Same as the STREAM option
4) an object that implements InputStream or a Monitor object. data is
obtained via LINEIN methods.
5) the object is requested as an array, and if successfully converted, the
array lines are used.

resourceo can be

STEM stemvarname
STREAM cexpr
USING cexpr

USING options are

1) a stem object
2) a File object
3) an object that implements OutputStream or a Monitor. Lineout is used
4) an object that is an OrderedCollection. lines are added using APPEND.

Special note: if the same object is used for both input and output, the
output is buffered to avoid conflicts.

Rick

On Wed, Apr 25, 2018 at 8:33 AM, Rony G. Flatscher 
wrote:

> Great!
>
> Where can one learn the available syntax?
>
> Currently, "https://sourceforge.net/projects/oorexx/files/oorexx-
> docs/5.0.0beta/rexxref.pdf"
> 
> is from 2018-03-28 and does not have the new syntax documented.
>
> ---rony
>
> On 25.04.2018 13:59, Rick McGuire wrote:
>
> Test away. I don't know of any pending work on Windows at least. Erich
> will have to speak for the other platforms.
>
> Rick
>
> On Wed, Apr 25, 2018 at 7:55 AM, Rony G. Flatscher <
> rony.flatsc...@wu.ac.at> wrote:
>
>> A question ad implementation status: is the functionality in trunk
>> already regarded to be fully
>> implemented and functional such that one can start to test it under
>> Windows, Linux and MacOSX?
>>
>> ---rony
>>
>>
>> On 21.04.2018 20:43, Gil Barmwater wrote:
>> >
>> > Yep! Got a clean build after I got ALL the updates. Thanks.
>> >
>> > Gil
>>
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rony G. Flatscher]

Great!

Where can one learn the available syntax?

Currently, 
"https://sourceforge.net/projects/oorexx/files/oorexx-docs/5.0.0beta/rexxref.pdf;
 is from
2018-03-28 and does not have the new syntax documented.

---rony


On 25.04.2018 13:59, Rick McGuire wrote:
> Test away. I don't know of any pending work on Windows at least. Erich will 
> have to speak for the
> other platforms. 
>
> Rick
>
> On Wed, Apr 25, 2018 at 7:55 AM, Rony G. Flatscher  > wrote:
>
> A question ad implementation status: is the functionality in trunk 
> already regarded to be fully
> implemented and functional such that one can start to test it under 
> Windows, Linux and MacOSX?
>
> ---rony
>
>
> On 21.04.2018 20:43, Gil Barmwater wrote:
> >
> > Yep! Got a clean build after I got ALL the updates. Thanks.
> >
> > Gil
>

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

Test away. I don't know of any pending work on Windows at least. Erich will
have to speak for the other platforms.

Rick

On Wed, Apr 25, 2018 at 7:55 AM, Rony G. Flatscher 
wrote:

> A question ad implementation status: is the functionality in trunk already
> regarded to be fully
> implemented and functional such that one can start to test it under
> Windows, Linux and MacOSX?
>
> ---rony
>
>
> On 21.04.2018 20:43, Gil Barmwater wrote:
> >
> > Yep! Got a clean build after I got ALL the updates. Thanks.
> >
> > Gil
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rony G. Flatscher]

A question ad implementation status: is the functionality in trunk already 
regarded to be fully
implemented and functional such that one can start to test it under Windows, 
Linux and MacOSX?

---rony


On 21.04.2018 20:43, Gil Barmwater wrote:
>
> Yep! Got a clean build after I got ALL the updates. Thanks.
>
> Gil


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

Yep! Got a clean build after I got ALL the updates. Thanks.

Gil
On 4/21/2018 2:09 PM, Rick McGuire wrote:
I’m guessing you ran the sun up from the wrong directory. You’re 
missing a lot of updates.


On Sat, Apr 21, 2018 at 2:06 PM Gil Barmwater > wrote:


Just finished updating to the latest sandbox revision - 11426 -
and attempted a 32-bit Windows build.  The following errors were
posted when compiling SystemCommands.cpp:

E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(179):
error C2065: 'Error_Execution_address_redirection_failed':
undeclared identifier
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(518):
error C2039: 'waitForTermination': is not a member of
'ErrorReaderThread'
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(104):
note: see declaration of 'ErrorReaderThread'
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(526):
warning C4800: 'RexxObjectPtr': forcing value to bool 'true' or
'false' (performance warning)
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(533):
error C2039: 'waitForTermination': is not a member of
'InputWriterThread'
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(75):
note: see declaration of 'InputWriterThread'
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(537):
warning C4800: 'RexxObjectPtr': forcing value to bool 'true' or
'false' (performance warning)
NMAKE : fatal error U1077: 'C:\PROGRA~2\MICROS~1.0\VC\bin\cl.exe'
: return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft
Visual Studio 14.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft
Visual Studio 14.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.

Hope these messages are helpful...

Gil
On 4/20/2018 7:19 AM, Rick McGuire wrote:

Right now, monitor objects would not work (an oversight on my
part). I'll look into adding them. Output should be easy.  Input
is sort of implicitly supported because the last step is to send
a makeArray to the object.

The reads from the pipes are done a buffer at a time and the
error output that is read on the separate thread needs to be
buffered until the process terminates, so no, they do not happen
in real time.

Rick

On Fri, Apr 20, 2018 at 7:11 AM, Rony G. Flatscher
> wrote:


On 19.04.2018 19:21, Erich Steinböck wrote:
> I just committed the Unix implementation of a redirecting system 
command handler, that
passes all
> tests in (an updated) ADDRESS.testGroup.
>
> It handles OUTPUT in the main thread, and INPUT and ERROR
in additional threads when required.
> (Handling OUTPUT in the main thread will allow Rexx code to
get concurrent progress information
> for long running commands; ERROR information becomes
available only after the ERROR thread ends).
How about, if one supplies monitor objects to be used (e.g.
one for STDIN, one for STDOUT and one
for STDERR)? Would they get messages dispatched in
"real-time" from the different threads? If so
this would allow for e.g. timestamping or analysis of the
item flows via the different standard
files and the like.

---rony


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/oorexx-devel





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!http://sdm.link/slashdot


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/oorexx-devel


-- 
Gil Barmwater



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!
http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

I’m guessing you ran the sun up from the wrong directory. You’re missing a
lot of updates.

On Sat, Apr 21, 2018 at 2:06 PM Gil Barmwater 
wrote:

> Just finished updating to the latest sandbox revision - 11426 - and
> attempted a 32-bit Windows build.  The following errors were posted when
> compiling SystemCommands.cpp:
>
> E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(179): error
> C2065: 'Error_Execution_address_redirection_failed': undeclared identifier
> E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(518): error
> C2039: 'waitForTermination': is not a member of 'ErrorReaderThread'
> E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(104): note: see
> declaration of 'ErrorReaderThread'
> E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(526): warning
> C4800: 'RexxObjectPtr': forcing value to bool 'true' or 'false'
> (performance warning)
> E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(533): error
> C2039: 'waitForTermination': is not a member of 'InputWriterThread'
> E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(75): note: see
> declaration of 'InputWriterThread'
> E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(537): warning
> C4800: 'RexxObjectPtr': forcing value to bool 'true' or 'false'
> (performance warning)
> NMAKE : fatal error U1077: 'C:\PROGRA~2\MICROS~1.0\VC\bin\cl.exe' : return
> code '0x2'
> Stop.
> NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual
> Studio 14.0\VC\BIN\nmake.exe"' : return code '0x2'
> Stop.
> NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual
> Studio 14.0\VC\BIN\nmake.exe"' : return code '0x2'
> Stop.
> Hope these messages are helpful...
>
> Gil
> On 4/20/2018 7:19 AM, Rick McGuire wrote:
>
> Right now, monitor objects would not work (an oversight on my part). I'll
> look into adding them. Output should be easy.  Input is sort of implicitly
> supported because the last step is to send a makeArray to the object.
>
> The reads from the pipes are done a buffer at a time and the error output
> that is read on the separate thread needs to be buffered until the process
> terminates, so no, they do not happen in real time.
>
> Rick
>
> On Fri, Apr 20, 2018 at 7:11 AM, Rony G. Flatscher <
> rony.flatsc...@wu.ac.at> wrote:
>
>>
>> On 19.04.2018 19:21, Erich Steinböck wrote:
>> > I just committed the Unix implementation of a redirecting system
>> command handler, that passes all
>> > tests in (an updated) ADDRESS.testGroup.
>> >
>> > It handles OUTPUT in the main thread, and INPUT and ERROR in additional
>> threads when required.
>> > (Handling OUTPUT in the main thread will allow Rexx code to get
>> concurrent progress information
>> > for long running commands; ERROR information becomes available only
>> after the ERROR thread ends).
>> How about, if one supplies monitor objects to be used (e.g. one for
>> STDIN, one for STDOUT and one
>> for STDERR)? Would they get messages dispatched in "real-time" from the
>> different threads? If so
>> this would allow for e.g. timestamping or analysis of the item flows via
>> the different standard
>> files and the like.
>>
>> ---rony
>>
>>
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Oorexx-devel mailing list
>> Oorexx-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Oorexx-devel mailing 
> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
> --
> Gil Barmwater
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

Just finished updating to the latest sandbox revision - 11426 - and 
attempted a 32-bit Windows build.  The following errors were posted when 
compiling SystemCommands.cpp:


E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(179): error 
C2065: 'Error_Execution_address_redirection_failed': undeclared identifier
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(518): error 
C2039: 'waitForTermination': is not a member of 'ErrorReaderThread'
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(104): note: 
see declaration of 'ErrorReaderThread'
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(526): warning 
C4800: 'RexxObjectPtr': forcing value to bool 'true' or 'false' 
(performance warning)
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(533): error 
C2039: 'waitForTermination': is not a member of 'InputWriterThread'
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(75): note: 
see declaration of 'InputWriterThread'
E:\Sandbox\interpreter\platform\windows\SystemCommands.cpp(537): warning 
C4800: 'RexxObjectPtr': forcing value to bool 'true' or 'false' 
(performance warning)
NMAKE : fatal error U1077: 'C:\PROGRA~2\MICROS~1.0\VC\bin\cl.exe' : 
return code '0x2'

Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual 
Studio 14.0\VC\BIN\nmake.exe"' : return code '0x2'

Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual 
Studio 14.0\VC\BIN\nmake.exe"' : return code '0x2'

Stop.

Hope these messages are helpful...

Gil
On 4/20/2018 7:19 AM, Rick McGuire wrote:
Right now, monitor objects would not work (an oversight on my part). 
I'll look into adding them. Output should be easy.  Input is sort of 
implicitly supported because the last step is to send a makeArray to 
the object.


The reads from the pipes are done a buffer at a time and the error 
output that is read on the separate thread needs to be buffered until 
the process terminates, so no, they do not happen in real time.


Rick

On Fri, Apr 20, 2018 at 7:11 AM, Rony G. Flatscher 
> wrote:



On 19.04.2018 19:21, Erich Steinböck wrote:
> I just committed the Unix implementation of a redirecting system command 
handler, that
passes all
> tests in (an updated) ADDRESS.testGroup.
>
> It handles OUTPUT in the main thread, and INPUT and ERROR in
additional threads when required.
> (Handling OUTPUT in the main thread will allow Rexx code to get
concurrent progress information
> for long running commands; ERROR information becomes available
only after the ERROR thread ends).
How about, if one supplies monitor objects to be used (e.g. one
for STDIN, one for STDOUT and one
for STDERR)? Would they get messages dispatched in "real-time"
from the different threads? If so
this would allow for e.g. timestamping or analysis of the item
flows via the different standard
files and the like.

---rony


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/oorexx-devel





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!http://sdm.link/slashdot


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel


--
Gil Barmwater

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

>
> Does that mean that in theory it also should work on Android?
>

No, not at this time.
I found one source mentioning that the posix_spawn* family of calls would
become available with Android P
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

>
> Unable to reproduce this on windows
>
CreateInterpreter() runs into a segmentation fault if there is no
environment.  Seems to be when for primitiveSearchName(BASEIMAGE,..)
getenv("PATH") returns NULL.
We had no environment in spawned commands because I thought if the
appropriate arg was set to NULL, inheriting the parent's environment would
be default.  I committed a change that supplies the parent environment
string array.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

Right now, monitor objects would not work (an oversight on my part). I'll
look into adding them. Output should be easy.  Input is sort of implicitly
supported because the last step is to send a makeArray to the object.

The reads from the pipes are done a buffer at a time and the error output
that is read on the separate thread needs to be buffered until the process
terminates, so no, they do not happen in real time.

Rick

On Fri, Apr 20, 2018 at 7:11 AM, Rony G. Flatscher 
wrote:

>
> On 19.04.2018 19:21, Erich Steinböck wrote:
> > I just committed the Unix implementation of a redirecting system command
> handler, that passes all
> > tests in (an updated) ADDRESS.testGroup.
> >
> > It handles OUTPUT in the main thread, and INPUT and ERROR in additional
> threads when required.
> > (Handling OUTPUT in the main thread will allow Rexx code to get
> concurrent progress information
> > for long running commands; ERROR information becomes available only
> after the ERROR thread ends).
> How about, if one supplies monitor objects to be used (e.g. one for STDIN,
> one for STDOUT and one
> for STDERR)? Would they get messages dispatched in "real-time" from the
> different threads? If so
> this would allow for e.g. timestamping or analysis of the item flows via
> the different standard
> files and the like.
>
> ---rony
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rony G. Flatscher]

On 19.04.2018 19:21, Erich Steinböck wrote:
> I just committed the Unix implementation of a redirecting system command 
> handler, that passes all
> tests in (an updated) ADDRESS.testGroup.
>
> It handles OUTPUT in the main thread, and INPUT and ERROR in additional 
> threads when required.
> (Handling OUTPUT in the main thread will allow Rexx code to get concurrent 
> progress information
> for long running commands; ERROR information becomes available only after the 
> ERROR thread ends).
How about, if one supplies monitor objects to be used (e.g. one for STDIN, one 
for STDOUT and one
for STDERR)? Would they get messages dispatched in "real-time" from the 
different threads? If so
this would allow for e.g. timestamping or analysis of the item flows via the 
different standard
files and the like.

---rony

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rony G. Flatscher]

On 19.04.2018 19:21, Erich Steinböck wrote:
> I just committed the Unix implementation of a redirecting system command 
> handler, that passes all
> tests in (an updated) ADDRESS.testGroup.
>
> It handles OUTPUT in the main thread, and INPUT and ERROR in additional 
> threads when required.
> (Handling OUTPUT in the main thread will allow Rexx code to get concurrent 
> progress information
> for long running commands; ERROR information becomes available only after the 
> ERROR thread ends).
>
> Redirection is available for all Unix environments - with the update we now 
> have sh, bsh, csh,
> bash, ksh, tcsh, zsh.
> Redirection is also available for the direct, no-shell environment (now 
> called "path" instead of
> "noshell" which is the same name as Regina uses)
Does that mean that in theory it also should work on Android? You mentioned in 
one of your posts,
that Android's Linux might pose a problem.

---rony

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

On Thu, Apr 19, 2018 at 1:21 PM, Erich Steinböck  wrote:

> I just committed the Unix implementation of a redirecting system command
> handler, that passes all tests in (an updated) ADDRESS.testGroup.
>
> It handles OUTPUT in the main thread, and INPUT and ERROR in additional
> threads when required. (Handling OUTPUT in the main thread will allow Rexx
> code to get concurrent progress information for long running commands;
> ERROR information becomes available only after the ERROR thread ends).
>
> Redirection is available for all Unix environments - with the update we
> now have sh, bsh, csh, bash, ksh, tcsh, zsh.
> Redirection is also available for the direct, no-shell environment (now
> called "path" instead of "noshell" which is the same name as Regina uses)
>
> Known issues (=segfaults)
> This segfaults, but I have no idea why.  doesn't have to exist.
> rexx -e "'rexx '"
>

Unable to reproduce this on windows.

>
> This segfaults (I confused the empty() method with the delete() method )
> rexx -e "q = .RexxQueue~new('qfault'); q~delete; address '' 'echo 123'
> with output using (q)"
>

Just committed a fix for this one.

>
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

I just committed the Unix implementation of a redirecting system command
handler, that passes all tests in (an updated) ADDRESS.testGroup.

It handles OUTPUT in the main thread, and INPUT and ERROR in additional
threads when required. (Handling OUTPUT in the main thread will allow Rexx
code to get concurrent progress information for long running commands;
ERROR information becomes available only after the ERROR thread ends).

Redirection is available for all Unix environments - with the update we now
have sh, bsh, csh, bash, ksh, tcsh, zsh.
Redirection is also available for the direct, no-shell environment (now
called "path" instead of "noshell" which is the same name as Regina uses)

Known issues (=segfaults)
This segfaults, but I have no idea why.  doesn't have to exist.
rexx -e "'rexx '"

This segfaults (I confused the empty() method with the delete() method )
rexx -e "q = .RexxQueue~new('qfault'); q~delete; address '' 'echo 123' with
output using (q)"
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

Yep, it probably should. I'll take a look at it.

Rick

On Mon, Apr 16, 2018 at 10:04 AM, Erich Steinböck <
erich.steinbo...@gmail.com> wrote:

> Should this print "line" twice?  It currently prints two null strings.
>
> ~~~
> stem. = "line"
> stem.0 = 2
> address "" "cat" with input using ( stem. )"
> ~~~
>
> When using explicit assignments, this works as expected
> stem.1 = "line"; stem.2 = "line"
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

I added a waitForTermination() method to SysThread...pretty trivial to do.
The unix version hasn't even been compiled yet.

Rick

On Thu, Apr 5, 2018 at 5:58 PM, Erich Steinböck 
wrote:

> I have a working redirecting context on Ubuntu (no tests run yet).
> But I guess I need to think a little more about the blocking issue.
> What we're doing here is, fork a child (which runs the command) and
> connect us to the child through at most three pipes. A stdin pipe, a stdout
> pipe, and an error pipe. We write to the stdin pipe, which the child then
> reads, and the child writes to both the stdout and the error pipe, which we
> than will read back in.
>
> Now pipes are something that the OS provides and it does some buffering
> for us: you can write stuff to a pipe, before the child consumes it. But
> it's just a small buffer, 4K or 64K. After that the pipe may block (or
> break, which I may have experienced)
>
> So if we're eager to send stuff through the stdin pipe for the child to
> use it, and the child is slow, or, worse, doesn't even expect anything on
> stdin, the pipe blocks and we're stalled. In the meantime we could do a
> little reading on the output or error pipe, because the child may be slow,
> because it itself has already filled up, say, its output pipe and reading
> that would be a relief (break the stall), but I'm not sure how to actually
> do this.
>
> When do we write, when read? Which pipes? How large pipe read/write blocks
> do we use? Do we sleep a little bit in between?
> I also need to figure out how to do non-blocking reads.
>
>
> We don't yet have ADDRESS WITH tests for a real Unix redirecting command
> context yet, have we?
>
>
> On Wed, Apr 4, 2018 at 4:04 PM, Erich Steinböck <
> erich.steinbo...@gmail.com> wrote:
>
>> From the description in https://blog.uxul.de/e?e=mt-160 (very bottom) it
>> seems "trivial" to do.
>> Not a Linux guru at all, but I'd give it a try if no one else volunteers.
>>
>> The blog also contains a discussion of an interesting quirk with fork(),
>> which is why the sample code uses POSIX spawn to execute the command.
>> POSIX spawn seems to be available on all the platforms ooRexx supports.
>> Not on Android though.
>>
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

We haven't needed it yet anywhere, but it looks fairly trivial to add a
waitForTermination() method to the SysThread class. Unfortunately, I'm in
the middle of migrating to a new PC, and neither my old PC or the new one
is really functional at this time.

Rick

On Fri, Apr 13, 2018 at 6:40 PM, Erich Steinböck  wrote:

>
>>- How do I wait for SysThreads to terminate? Or check if they're
>>still running?
>>
>> For the Windows version, I called terminate on the thread object that
>> will clean things up.
>>
>
> When terminate() returns, I know that dispatch() has finished?
> I believe I need to wait for dispatch() to finish, or how else do I know
> when the main thread can start to retrieve the data from the error thread
> (see next paragraph)
>
>
>>
>>>- Can I raise an exception from within the dispatch() method (if I
>>>have the appropriate exit context available, of course)? I'm asking,
>>>because your code delays raising the exception until after terminate()
>>>
>>> No, you can't. The context objects are tied to the thread making the
>> call to the handler. You also cannot call the ioContext methods from the
>> Error thread. It would have to hold any data read and then have the main
>> thread handle it at the end.
>>
>
> OK, I see, thanks
>
>
>>>- My (yet completely untested) code issues a message "terminate
>>>called after throwing an instance of 'NativeActivation*'" - haven't ever
>>>seen that before (I of course assume this due to my own bugs
>>>
>>> You must have tried to raise an error on other than the main thread.
>> There's no backing ooRexx code there to handle the error.  All callbacks to
>> the interpreter need to happen on the main thread.
>>
>
> Yes, that's what I did ..
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

On Fri, Apr 13, 2018 at 6:40 PM, Erich Steinböck  wrote:

>
>>- How do I wait for SysThreads to terminate? Or check if they're
>>still running?
>>
>> For the Windows version, I called terminate on the thread object that
>> will clean things up.
>>
>
> When terminate() returns, I know that dispatch() has finished?
> I believe I need to wait for dispatch() to finish, or how else do I know
> when the main thread can start to retrieve the data from the error thread
> (see next paragraph)
>

The easiest way would probably be to add a semaphore to the SysThread
subclass that dispatch can post when it finishes.

>
>
>>
>>>- Can I raise an exception from within the dispatch() method (if I
>>>have the appropriate exit context available, of course)? I'm asking,
>>>because your code delays raising the exception until after terminate()
>>>
>>> No, you can't. The context objects are tied to the thread making the
>> call to the handler. You also cannot call the ioContext methods from the
>> Error thread. It would have to hold any data read and then have the main
>> thread handle it at the end.
>>
>
> OK, I see, thanks
>
>
>>>- My (yet completely untested) code issues a message "terminate
>>>called after throwing an instance of 'NativeActivation*'" - haven't ever
>>>seen that before (I of course assume this due to my own bugs
>>>
>>> You must have tried to raise an error on other than the main thread.
>> There's no backing ooRexx code there to handle the error.  All callbacks to
>> the interpreter need to happen on the main thread.
>>
>
> Yes, that's what I did ..
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

>
>
>- How do I wait for SysThreads to terminate? Or check if they're still
>running?
>
> For the Windows version, I called terminate on the thread object that will
> clean things up.
>

When terminate() returns, I know that dispatch() has finished?
I believe I need to wait for dispatch() to finish, or how else do I know
when the main thread can start to retrieve the data from the error thread
(see next paragraph)


>
>>- Can I raise an exception from within the dispatch() method (if I
>>have the appropriate exit context available, of course)? I'm asking,
>>because your code delays raising the exception until after terminate()
>>
>> No, you can't. The context objects are tied to the thread making the call
> to the handler. You also cannot call the ioContext methods from the Error
> thread. It would have to hold any data read and then have the main thread
> handle it at the end.
>

OK, I see, thanks


>>- My (yet completely untested) code issues a message "terminate
>>called after throwing an instance of 'NativeActivation*'" - haven't ever
>>seen that before (I of course assume this due to my own bugs
>>
>> You must have tried to raise an error on other than the main thread.
> There's no backing ooRexx code there to handle the error.  All callbacks to
> the interpreter need to happen on the main thread.
>

Yes, that's what I did ..
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

On Fri, Apr 13, 2018 at 6:00 PM, Erich Steinböck  wrote:

> What I did for Windows with the separate thread should be easily adapted
>> for Linux
>
> I've rewritten the Linux handler with separate threads for INPUT and ERROR
> (if requested), while handling OUTPUT in the main thread.
> I'm just starting the first tests, and I've got a few questions.
>
>- How do I wait for SysThreads to terminate? Or check if they're still
>running?
>
> For the Windows version, I called terminate on the thread object that will
clean things up.


>
>- Can I raise an exception from within the dispatch() method (if I
>have the appropriate exit context available, of course)? I'm asking,
>because your code delays raising the exception until after terminate()
>
> No, you can't. The context objects are tied to the thread making the call
to the handler. You also cannot call the ioContext methods from the Error
thread. It would have to hold any data read and then have the main thread
handle it at the end.


>- An io context just provides those new redirection functions, and not
>on top everything what an exit context provides, right?
>
> Correct

>
>- My (yet completely untested) code issues a message "terminate called
>after throwing an instance of 'NativeActivation*'" - haven't ever seen that
>before (I of course assume this due to my own bugs)
>
> You must have tried to raise an error on other than the main thread.
There's no backing ooRexx code there to handle the error.  All callbacks to
the interpreter need to happen on the main thread.

Rick

>
>-
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

>
> What I did for Windows with the separate thread should be easily adapted
> for Linux

I've rewritten the Linux handler with separate threads for INPUT and ERROR
(if requested), while handling OUTPUT in the main thread.
I'm just starting the first tests, and I've got a few questions.

   - How do I wait for SysThreads to terminate? Or check if they're still
   running?
   - Can I raise an exception from within the dispatch() method (if I have
   the appropriate exit context available, of course)? I'm asking, because
   your code delays raising the exception until after terminate()
   - An io context just provides those new redirection functions, and not
   on top everything what an exit context provides, right?
   - My (yet completely untested) code issues a message "terminate called
   after throwing an instance of 'NativeActivation*'" - haven't ever seen that
   before (I of course assume this due to my own bugs)
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

If you haven't already done so, the code I added has 
somecontext->RaiseCondition() method calls that should be reviewed for 
their appropriateness.


Gil
On 4/13/2018 11:57 AM, Rick McGuire wrote:
ERROR and FAILURE conditions are related to the execution of the 
command itself. Other sorts of failures in the redirection should 
probably be SYNTAX condition, probably in the Error_Execution category.


Rick

On Fri, Apr 13, 2018 at 11:37 AM, Erich Steinböck 
> wrote:


would the RC .. be the value from the executed command itself
.. (rather than from "rxqueue")?

Yes, the rc from the command.
With this implementationrxqueue isn't involved at all, the
redirection is done through Windows/Unix pipes.

Q:
With redirection we will have (potentially many, but still rare)
additional FAILURE situations. Currently a FAILURE is probably
always due to an unknown command. We have documented that for
FAILURE the descriptive string is the command that was scheduled
to run. In case of redirection-related failures, should we provide
more detailed error information in the ADDITIONAL object
(currently FAILURE does not have an ADDITIONAL object). If so,
what kind of object should we return?  An array of strings,
similar to the SYNTAX ADDITIONAL object? E. g. "input", "error
creating pipe"?  Or is that too much detail?  Just a SYNTAX
"resources exhausted"?


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/oorexx-devel





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel


--
Gil Barmwater

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

ERROR and FAILURE conditions are related to the execution of the command
itself. Other sorts of failures in the redirection should probably be
SYNTAX condition, probably in the Error_Execution category.

Rick

On Fri, Apr 13, 2018 at 11:37 AM, Erich Steinböck <
erich.steinbo...@gmail.com> wrote:

> would the RC .. be the value from the executed command itself .. (rather
>> than from "rxqueue")?
>>
> Yes, the rc from the command.
> With this implementation rxqueue isn't involved at all, the redirection
> is done through Windows/Unix pipes.
>
> Q:
> With redirection we will have (potentially many, but still rare)
> additional FAILURE situations. Currently a FAILURE is probably always due
> to an unknown command. We have documented that for FAILURE the descriptive
> string is the command that was scheduled to run. In case of
> redirection-related failures, should we provide more detailed error
> information in the ADDITIONAL object (currently FAILURE does not have an
> ADDITIONAL object). If so, what kind of object should we return?  An array
> of strings, similar to the SYNTAX ADDITIONAL object? E. g. "input", "error
> creating pipe"?  Or is that too much detail?  Just a SYNTAX "resources
> exhausted"?
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

>
> would the RC .. be the value from the executed command itself .. (rather
> than from "rxqueue")?
>
Yes, the rc from the command.
With this implementation rxqueue isn't involved at all, the redirection is
done through Windows/Unix pipes.

Q:
With redirection we will have (potentially many, but still rare) additional
FAILURE situations. Currently a FAILURE is probably always due to an
unknown command. We have documented that for FAILURE the descriptive string
is the command that was scheduled to run. In case of redirection-related
failures, should we provide more detailed error information in the
ADDITIONAL object (currently FAILURE does not have an ADDITIONAL object).
If so, what kind of object should we return?  An array of strings, similar
to the SYNTAX ADDITIONAL object? E. g. "input", "error creating pipe"?  Or
is that too much detail?  Just a SYNTAX "resources exhausted"?
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rony G. Flatscher]

On 13.04.2018 17:11, Rick McGuire wrote:
> The RC value is handled exactly the same as a command without redirection 
> since we're doing the
> piping rather than the command shell.
super, this is really great!

---rony

>
>
> On Fri, Apr 13, 2018 at 11:05 AM, Rony G. Flatscher  > wrote:
>
> Just a question ad "RC" (return code): would the RC value in Rexx upon 
> return from the command
> be the value from the executed command itself in all of these scenarios 
> (rather than from
> "rxqueue")?
>
> ---rony
>
>
> On 13.04.2018 13:12, Rick McGuire wrote:
>> I hate it when code compiles and runs on the first attempt :-) Just 
>> moved the pipe reading
>> code to before wait for process termination and the big data test case 
>> started working.
>> Erich, What I did for Windows with the separate thread should be easily 
>> adapted for Linux. 
>>
>> Rick
>>
>> On Fri, Apr 13, 2018 at 6:58 AM, Rick McGuire > > wrote:
>>
>> Ok, I have things working with a separate thread filling the input 
>> pipe, but I'm able to
>> get a hang with a simple example (see below). As I suspected, the 
>> problem is caused by
>> waiting for the process to complete before reading from the output 
>> pipe. The command
>> process stalls when the pipe gets filled and waits until data is 
>> read from the pipe. So,
>> the process never terminates because it is waiting for the pipe to 
>> be cleared out. The
>> attached programs work fine with much smaller amounts of data. 
>>
>> test.rex 
>>
>> a1=.array~new(10)
>> a1~fill('This is a test')
>>
>> addresscommand'rexx flipit.rex'withinputusing(a1) outputstema.
>> saya.0
>>
>> flipit.rex
>>
>> signalonnotready
>>
>> loopforever
>> sayreverse(linein())
>> end
>>
>> notready:
>>
>> Rick
>>
>> On Thu, Apr 12, 2018 at 7:59 PM, Gil Barmwater 
>> > > wrote:
>>
>> You are correct on the placement of the APPEND|REPLACE keywords. 
>>  I somehow misread
>> the BNF :-( .  As for the required parenthesis around the 
>> variable name, this will be
>> a difference, I suspect, between ooRexx and Regina that we need 
>> to clearly document,
>> probably by including an example showing that usage.  And I have 
>> no problem with the
>> required dot after stem objects (as in using (foo.)) but 'output 
>> stem s' seems to me
>> unambiguous as the keyword 'stem' indicates the name 's' is the 
>> name of a stem. Just
>> my opinion.
>>
>> I access the pipes the same way the sample code I found did, 
>> with the ReadFile() and
>> WriteFile() functions.  As I said earlier, I hadn't tested large 
>> amounts of
>> redirected output data so the problem didn't arise during my 
>> development and
>> testing.  The sample code didn't actually do a wait so that was 
>> something I added
>> without thinking about the full pipe scenario.  I believe a loop 
>> with a timed wait
>> and pipe reads (and buffered output writes) might address this 
>> but I haven't had time
>> to experiment with it yet.  However, even that approach won't 
>> cover the case where
>> the code blocks writing to the input pipe since we wouldn't get 
>> to the wait loop
>> until the input data is all written.  Sounds like the thread 
>> approach might be the
>> most straightforward way to go.
>>
>> Gil
>>
>> On 4/12/2018 5:39 PM, Rick McGuire wrote:
>>> It looks like spinning off a thread to do this is pretty easy 
>>> using the classes we
>>> already have. Gil, what commands have you been using to test 
>>> both ends of the pipe?
>>>
>>> Also, I notice that the code waits for the process to complete 
>>> before reading the
>>> output pipes. Shouldn't it be reading before the process 
>>> completes? Otherwise, a
>>> command that creates a large amount of output will fill the 
>>> pipe up. 
>>>
>>> Rick
>>>
>>> On Thu, Apr 12, 2018 at 5:03 PM, Rick McGuire 
>>> >> > wrote:
>>>
>>> I decided to post a question to StackOverflow to see if 
>>> anybody has an
>>> suggestions. I'm sort of thinking the solution might be to 
>>> spin off a thread to
>>> handle the writes to the buffer so that the main thread can 
>>> immediately start
>>> emptying the output pipes. I think I'll try taking a stab 
>>> at this for the
>>> Windows version. 

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

The RC value is handled exactly the same as a command without redirection
since we're doing the piping rather than the command shell.

Rick

On Fri, Apr 13, 2018 at 11:05 AM, Rony G. Flatscher  wrote:

> Just a question ad "RC" (return code): would the RC value in Rexx upon
> return from the command be the value from the executed command itself in
> all of these scenarios (rather than from "rxqueue")?
>
> ---rony
>
> On 13.04.2018 13:12, Rick McGuire wrote:
>
> I hate it when code compiles and runs on the first attempt :-) Just moved
> the pipe reading code to before wait for process termination and the big
> data test case started working. Erich, What I did for Windows with the
> separate thread should be easily adapted for Linux.
>
> Rick
>
> On Fri, Apr 13, 2018 at 6:58 AM, Rick McGuire 
> wrote:
>
>> Ok, I have things working with a separate thread filling the input pipe,
>> but I'm able to get a hang with a simple example (see below). As I
>> suspected, the problem is caused by waiting for the process to complete
>> before reading from the output pipe. The command process stalls when the
>> pipe gets filled and waits until data is read from the pipe. So, the
>> process never terminates because it is waiting for the pipe to be cleared
>> out. The attached programs work fine with much smaller amounts of data.
>>
>> test.rex
>>
>> a1 = .array~new(10)a1~fill('This is a test')
>> address command 'rexx flipit.rex' with input using (a1) output stem a.say a.0
>>
>> flipit.rex
>>
>> signal on notready
>> loop forever   say reverse(linein())end
>> notready:
>>
>> Rick
>>
>> On Thu, Apr 12, 2018 at 7:59 PM, Gil Barmwater 
>> wrote:
>>
>>> You are correct on the placement of the APPEND|REPLACE keywords.  I
>>> somehow misread the BNF :-( .  As for the required parenthesis around
>>> the variable name, this will be a difference, I suspect, between ooRexx and
>>> Regina that we need to clearly document, probably by including an example
>>> showing that usage.  And I have no problem with the required dot after stem
>>> objects (as in using (foo.)) but 'output stem s' seems to me unambiguous as
>>> the keyword 'stem' indicates the name 's' is the name of a stem. Just my
>>> opinion.
>>>
>>> I access the pipes the same way the sample code I found did, with the
>>> ReadFile() and WriteFile() functions.  As I said earlier, I hadn't tested
>>> large amounts of redirected output data so the problem didn't arise during
>>> my development and testing.  The sample code didn't actually do a wait so
>>> that was something I added without thinking about the full pipe scenario.
>>> I believe a loop with a timed wait and pipe reads (and buffered output
>>> writes) might address this but I haven't had time to experiment with it
>>> yet.  However, even that approach won't cover the case where the code
>>> blocks writing to the input pipe since we wouldn't get to the wait loop
>>> until the input data is all written.  Sounds like the thread approach might
>>> be the most straightforward way to go.
>>> Gil
>>>
>>> On 4/12/2018 5:39 PM, Rick McGuire wrote:
>>>
>>> It looks like spinning off a thread to do this is pretty easy using the
>>> classes we already have. Gil, what commands have you been using to test
>>> both ends of the pipe?
>>>
>>> Also, I notice that the code waits for the process to complete before
>>> reading the output pipes. Shouldn't it be reading before the process
>>> completes? Otherwise, a command that creates a large amount of output will
>>> fill the pipe up.
>>>
>>> Rick
>>>
>>> On Thu, Apr 12, 2018 at 5:03 PM, Rick McGuire 
>>> wrote:
>>>
 I decided to post a question to StackOverflow to see if anybody has an
 suggestions. I'm sort of thinking the solution might be to spin off a
 thread to handle the writes to the buffer so that the main thread can
 immediately start emptying the output pipes. I think I'll try taking a stab
 at this for the Windows version.

 Rick

 On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater  wrote:

> Found and fixed the problem and have been through all my tests that I
> developed for my proof of concept package.  In the process, I found 
> several
> things I wasn't expecting.
>
> 1) The parser seems to have reversed the expected positions of the
> target type - STEM, STREAM, or USING - and the "mode" - APPEND or REPLACE.
> I believe the syntax per the ANSI standard should be e.g. WITH OUTPUT STEM
> APPEND someStem. and not WITH OUTPUT APPEND STEM someStem.
>
> 2) There seems to be a requirement for the stream/stem name to be
> either a literal - "abc.txt" - or if it is a variable, enclosed in
> parenthesis - (someFile).  I expected the () for USING but not for STEM or
> STREAM.
>
> 3) My package allowed for stem "names" to be specified with or without

Re: [Oorexx-devel] Address With progress

[Rony G. Flatscher]

Just a question ad "RC" (return code): would the RC value in Rexx upon return 
from the command be
the value from the executed command itself in all of these scenarios (rather 
than from "rxqueue")?

---rony


On 13.04.2018 13:12, Rick McGuire wrote:
> I hate it when code compiles and runs on the first attempt :-) Just moved the 
> pipe reading code to
> before wait for process termination and the big data test case started 
> working. Erich, What I did
> for Windows with the separate thread should be easily adapted for Linux. 
>
> Rick
>
> On Fri, Apr 13, 2018 at 6:58 AM, Rick McGuire  > wrote:
>
> Ok, I have things working with a separate thread filling the input pipe, 
> but I'm able to get a
> hang with a simple example (see below). As I suspected, the problem is 
> caused by waiting for
> the process to complete before reading from the output pipe. The command 
> process stalls when
> the pipe gets filled and waits until data is read from the pipe. So, the 
> process never
> terminates because it is waiting for the pipe to be cleared out. The 
> attached programs work
> fine with much smaller amounts of data. 
>
> test.rex 
>
> a1=.array~new(10)
> a1~fill('This is a test')
>
> addresscommand'rexx flipit.rex'withinputusing(a1) outputstema.
> saya.0
>
> flipit.rex
>
> signalonnotready
>
> loopforever
> sayreverse(linein())
> end
>
> notready:
>
> Rick
>
> On Thu, Apr 12, 2018 at 7:59 PM, Gil Barmwater  > wrote:
>
> You are correct on the placement of the APPEND|REPLACE keywords.  I 
> somehow misread the
> BNF :-( .  As for the required parenthesis around the variable name, 
> this will be a
> difference, I suspect, between ooRexx and Regina that we need to 
> clearly document,
> probably by including an example showing that usage.  And I have no 
> problem with the
> required dot after stem objects (as in using (foo.)) but 'output stem 
> s' seems to me
> unambiguous as the keyword 'stem' indicates the name 's' is the name 
> of a stem. Just my
> opinion.
>
> I access the pipes the same way the sample code I found did, with the 
> ReadFile() and
> WriteFile() functions.  As I said earlier, I hadn't tested large 
> amounts of redirected
> output data so the problem didn't arise during my development and 
> testing.  The sample
> code didn't actually do a wait so that was something I added without 
> thinking about the
> full pipe scenario.  I believe a loop with a timed wait and pipe 
> reads (and buffered
> output writes) might address this but I haven't had time to 
> experiment with it yet. 
> However, even that approach won't cover the case where the code 
> blocks writing to the
> input pipe since we wouldn't get to the wait loop until the input 
> data is all written. 
> Sounds like the thread approach might be the most straightforward way 
> to go.
>
> Gil
>
> On 4/12/2018 5:39 PM, Rick McGuire wrote:
>> It looks like spinning off a thread to do this is pretty easy using 
>> the classes we
>> already have. Gil, what commands have you been using to test both 
>> ends of the pipe?
>>
>> Also, I notice that the code waits for the process to complete 
>> before reading the output
>> pipes. Shouldn't it be reading before the process completes? 
>> Otherwise, a command that
>> creates a large amount of output will fill the pipe up. 
>>
>> Rick
>>
>> On Thu, Apr 12, 2018 at 5:03 PM, Rick McGuire > > wrote:
>>
>> I decided to post a question to StackOverflow to see if anybody 
>> has an suggestions.
>> I'm sort of thinking the solution might be to spin off a thread 
>> to handle the writes
>> to the buffer so that the main thread can immediately start 
>> emptying the output
>> pipes. I think I'll try taking a stab at this for the Windows 
>> version. 
>>
>> Rick
>>
>> On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater 
>> > > wrote:
>>
>> Found and fixed the problem and have been through all my 
>> tests that I developed
>> for my proof of concept package.  In the process, I found 
>> several things I wasn't
>> expecting.
>>
>> 1) The parser seems to have reversed the expected positions 
>> of the target type -
>> STEM, STREAM, or USING - and the "mode" - APPEND or REPLACE. 
>>  I believe the
>> syntax per the ANSI standard should be e.g. WITH OUTPUT STEM 
>> APPEND someStem. and
>> not WITH OUTPUT APPEND 

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

I hate it when code compiles and runs on the first attempt :-) Just moved
the pipe reading code to before wait for process termination and the big
data test case started working. Erich, What I did for Windows with the
separate thread should be easily adapted for Linux.

Rick

On Fri, Apr 13, 2018 at 6:58 AM, Rick McGuire  wrote:

> Ok, I have things working with a separate thread filling the input pipe,
> but I'm able to get a hang with a simple example (see below). As I
> suspected, the problem is caused by waiting for the process to complete
> before reading from the output pipe. The command process stalls when the
> pipe gets filled and waits until data is read from the pipe. So, the
> process never terminates because it is waiting for the pipe to be cleared
> out. The attached programs work fine with much smaller amounts of data.
>
> test.rex
>
> a1 = .array~new(10)a1~fill('This is a test')
> address command 'rexx flipit.rex' with input using (a1) output stem a.say a.0
>
> flipit.rex
>
> signal on notready
> loop forever   say reverse(linein())end
> notready:
>
> Rick
>
> On Thu, Apr 12, 2018 at 7:59 PM, Gil Barmwater 
> wrote:
>
>> You are correct on the placement of the APPEND|REPLACE keywords.  I
>> somehow misread the BNF :-( .  As for the required parenthesis around
>> the variable name, this will be a difference, I suspect, between ooRexx and
>> Regina that we need to clearly document, probably by including an example
>> showing that usage.  And I have no problem with the required dot after stem
>> objects (as in using (foo.)) but 'output stem s' seems to me unambiguous as
>> the keyword 'stem' indicates the name 's' is the name of a stem. Just my
>> opinion.
>>
>> I access the pipes the same way the sample code I found did, with the
>> ReadFile() and WriteFile() functions.  As I said earlier, I hadn't tested
>> large amounts of redirected output data so the problem didn't arise during
>> my development and testing.  The sample code didn't actually do a wait so
>> that was something I added without thinking about the full pipe scenario.
>> I believe a loop with a timed wait and pipe reads (and buffered output
>> writes) might address this but I haven't had time to experiment with it
>> yet.  However, even that approach won't cover the case where the code
>> blocks writing to the input pipe since we wouldn't get to the wait loop
>> until the input data is all written.  Sounds like the thread approach might
>> be the most straightforward way to go.
>> Gil
>>
>> On 4/12/2018 5:39 PM, Rick McGuire wrote:
>>
>> It looks like spinning off a thread to do this is pretty easy using the
>> classes we already have. Gil, what commands have you been using to test
>> both ends of the pipe?
>>
>> Also, I notice that the code waits for the process to complete before
>> reading the output pipes. Shouldn't it be reading before the process
>> completes? Otherwise, a command that creates a large amount of output will
>> fill the pipe up.
>>
>> Rick
>>
>> On Thu, Apr 12, 2018 at 5:03 PM, Rick McGuire 
>> wrote:
>>
>>> I decided to post a question to StackOverflow to see if anybody has an
>>> suggestions. I'm sort of thinking the solution might be to spin off a
>>> thread to handle the writes to the buffer so that the main thread can
>>> immediately start emptying the output pipes. I think I'll try taking a stab
>>> at this for the Windows version.
>>>
>>> Rick
>>>
>>> On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater 
>>> wrote:
>>>
 Found and fixed the problem and have been through all my tests that I
 developed for my proof of concept package.  In the process, I found several
 things I wasn't expecting.

 1) The parser seems to have reversed the expected positions of the
 target type - STEM, STREAM, or USING - and the "mode" - APPEND or REPLACE.
 I believe the syntax per the ANSI standard should be e.g. WITH OUTPUT STEM
 APPEND someStem. and not WITH OUTPUT APPEND STEM someStem.

 2) There seems to be a requirement for the stream/stem name to be
 either a literal - "abc.txt" - or if it is a variable, enclosed in
 parenthesis - (someFile).  I expected the () for USING but not for STEM or
 STREAM.

 3) My package allowed for stem "names" to be specified with or without
 the trailing . but my testing shows the trailing . is required or an error
 message is generated.  I'm OK with that as long as it is documented.

 4) I get an error (unable to open ... for output) when I specify

 output stream "abc2.txt" error replace stream "abc2.txt"
 which was testing the redirection of both output and error to the same
 stream.

 5) Using a non-existent file for redirected input generates an error
 (unable to open ... for reading) while my package treated this as a null
 file.  Probably not a problem as the error message would 

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

Ok, I have things working with a separate thread filling the input pipe,
but I'm able to get a hang with a simple example (see below). As I
suspected, the problem is caused by waiting for the process to complete
before reading from the output pipe. The command process stalls when the
pipe gets filled and waits until data is read from the pipe. So, the
process never terminates because it is waiting for the pipe to be cleared
out. The attached programs work fine with much smaller amounts of data.

test.rex

a1 = .array~new(10)a1~fill('This is a test')
address command 'rexx flipit.rex' with input using (a1) output stem a.say a.0

flipit.rex

signal on notready
loop forever   say reverse(linein())end
notready:

Rick

On Thu, Apr 12, 2018 at 7:59 PM, Gil Barmwater 
wrote:

> You are correct on the placement of the APPEND|REPLACE keywords.  I
> somehow misread the BNF :-( .  As for the required parenthesis around the
> variable name, this will be a difference, I suspect, between ooRexx and
> Regina that we need to clearly document, probably by including an example
> showing that usage.  And I have no problem with the required dot after stem
> objects (as in using (foo.)) but 'output stem s' seems to me unambiguous as
> the keyword 'stem' indicates the name 's' is the name of a stem. Just my
> opinion.
>
> I access the pipes the same way the sample code I found did, with the
> ReadFile() and WriteFile() functions.  As I said earlier, I hadn't tested
> large amounts of redirected output data so the problem didn't arise during
> my development and testing.  The sample code didn't actually do a wait so
> that was something I added without thinking about the full pipe scenario.
> I believe a loop with a timed wait and pipe reads (and buffered output
> writes) might address this but I haven't had time to experiment with it
> yet.  However, even that approach won't cover the case where the code
> blocks writing to the input pipe since we wouldn't get to the wait loop
> until the input data is all written.  Sounds like the thread approach might
> be the most straightforward way to go.
> Gil
>
> On 4/12/2018 5:39 PM, Rick McGuire wrote:
>
> It looks like spinning off a thread to do this is pretty easy using the
> classes we already have. Gil, what commands have you been using to test
> both ends of the pipe?
>
> Also, I notice that the code waits for the process to complete before
> reading the output pipes. Shouldn't it be reading before the process
> completes? Otherwise, a command that creates a large amount of output will
> fill the pipe up.
>
> Rick
>
> On Thu, Apr 12, 2018 at 5:03 PM, Rick McGuire 
> wrote:
>
>> I decided to post a question to StackOverflow to see if anybody has an
>> suggestions. I'm sort of thinking the solution might be to spin off a
>> thread to handle the writes to the buffer so that the main thread can
>> immediately start emptying the output pipes. I think I'll try taking a stab
>> at this for the Windows version.
>>
>> Rick
>>
>> On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater 
>> wrote:
>>
>>> Found and fixed the problem and have been through all my tests that I
>>> developed for my proof of concept package.  In the process, I found several
>>> things I wasn't expecting.
>>>
>>> 1) The parser seems to have reversed the expected positions of the
>>> target type - STEM, STREAM, or USING - and the "mode" - APPEND or REPLACE.
>>> I believe the syntax per the ANSI standard should be e.g. WITH OUTPUT STEM
>>> APPEND someStem. and not WITH OUTPUT APPEND STEM someStem.
>>>
>>> 2) There seems to be a requirement for the stream/stem name to be either
>>> a literal - "abc.txt" - or if it is a variable, enclosed in parenthesis -
>>> (someFile).  I expected the () for USING but not for STEM or STREAM.
>>>
>>> 3) My package allowed for stem "names" to be specified with or without
>>> the trailing . but my testing shows the trailing . is required or an error
>>> message is generated.  I'm OK with that as long as it is documented.
>>>
>>> 4) I get an error (unable to open ... for output) when I specify
>>>
>>> output stream "abc2.txt" error replace stream "abc2.txt"
>>> which was testing the redirection of both output and error to the same
>>> stream.
>>>
>>> 5) Using a non-existent file for redirected input generates an error
>>> (unable to open ... for reading) while my package treated this as a null
>>> file.  Probably not a problem as the error message would flag a file name
>>> typo :-)
>>>
>>> As for checking in the code, I am not a committer so I plan to generate
>>> a patch file and attach it to RFE 4.
>>>
>>> One other item that I discovered is that I can get a deadlock in Windows
>>> as well.  I had not tested sending a lot of data to redirected output but
>>> when I did, my package locked up, presumably due to the output pipe being
>>> full.  I plan to experiment with ways around this but will generate the
>>> patch 

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

You are correct on the placement of the APPEND|REPLACE keywords. I 
somehow misread the BNF :-( .  As for the required parenthesis around 
the variable name, this will be a difference, I suspect, between ooRexx 
and Regina that we need to clearly document, probably by including an 
example showing that usage.  And I have no problem with the required dot 
after stem objects (as in using (foo.)) but 'output stem s' seems to me 
unambiguous as the keyword 'stem' indicates the name 's' is the name of 
a stem. Just my opinion.


I access the pipes the same way the sample code I found did, with the 
ReadFile() and WriteFile() functions.  As I said earlier, I hadn't 
tested large amounts of redirected output data so the problem didn't 
arise during my development and testing.  The sample code didn't 
actually do a wait so that was something I added without thinking about 
the full pipe scenario.  I believe a loop with a timed wait and pipe 
reads (and buffered output writes) might address this but I haven't had 
time to experiment with it yet.  However, even that approach won't cover 
the case where the code blocks writing to the input pipe since we 
wouldn't get to the wait loop until the input data is all written.  
Sounds like the thread approach might be the most straightforward way to 
go.


Gil
On 4/12/2018 5:39 PM, Rick McGuire wrote:
It looks like spinning off a thread to do this is pretty easy using 
the classes we already have. Gil, what commands have you been using to 
test both ends of the pipe?


Also, I notice that the code waits for the process to complete before 
reading the output pipes. Shouldn't it be reading before the process 
completes? Otherwise, a command that creates a large amount of output 
will fill the pipe up.


Rick

On Thu, Apr 12, 2018 at 5:03 PM, Rick McGuire > wrote:


I decided to post a question to StackOverflow to see if anybody
has an suggestions. I'm sort of thinking the solution might be to
spin off a thread to handle the writes to the buffer so that the
main thread can immediately start emptying the output pipes. I
think I'll try taking a stab at this for the Windows version.

Rick

On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater
> wrote:

Found and fixed the problem and have been through all my tests
that I developed for my proof of concept package.  In the
process, I found several things I wasn't expecting.

1) The parser seems to have reversed the expected positions of
the target type - STEM, STREAM, or USING - and the "mode" -
APPEND or REPLACE.  I believe the syntax per the ANSI standard
should be e.g. WITH OUTPUT STEM APPEND someStem. and not WITH
OUTPUT APPEND STEM someStem.

2) There seems to be a requirement for the stream/stem name to
be either a literal - "abc.txt" - or if it is a variable,
enclosed in parenthesis - (someFile).  I expected the () for
USING but not for STEM or STREAM.

3) My package allowed for stem "names" to be specified with or
without the trailing . but my testing shows the trailing . is
required or an error message is generated.  I'm OK with that
as long as it is documented.

4) I get an error (unable to open ... for output) when I specify

output stream "abc2.txt" error replace stream "abc2.txt"

which was testing the redirection of both output and error to
the same stream.

5) Using a non-existent file for redirected input generates an
error (unable to open ... for reading) while my package
treated this as a null file.  Probably not a problem as the
error message would flag a file name typo :-)

As for checking in the code, I am not a committer so I plan to
generate a patch file and attach it to RFE 4.

One other item that I discovered is that I can get a deadlock
in Windows as well.  I had not tested sending a lot of data to
redirected output but when I did, my package locked up,
presumably due to the output pipe being full.  I plan to
experiment with ways around this but will generate the patch
file with the code as it stands now.

Gil

On 4/11/2018 2:32 PM, Rick McGuire wrote:


On Wed, Apr 11, 2018 at 2:26 PM Gil Barmwater
> wrote:

Just a quick update on my progress so far.  I've gotten
all the code in
(hopefully) the right places and I have a clean build. 
My initial tests
are working but I've just hit my first problem which I
need to debug.
Hopefully, it will not be too much longer before I'm done.

checking the code in as it is is also an option...it gets
   

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

It looks like spinning off a thread to do this is pretty easy using the
classes we already have. Gil, what commands have you been using to test
both ends of the pipe?

Also, I notice that the code waits for the process to complete before
reading the output pipes. Shouldn't it be reading before the process
completes? Otherwise, a command that creates a large amount of output will
fill the pipe up.

Rick

On Thu, Apr 12, 2018 at 5:03 PM, Rick McGuire  wrote:

> I decided to post a question to StackOverflow to see if anybody has an
> suggestions. I'm sort of thinking the solution might be to spin off a
> thread to handle the writes to the buffer so that the main thread can
> immediately start emptying the output pipes. I think I'll try taking a stab
> at this for the Windows version.
>
> Rick
>
> On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater 
> wrote:
>
>> Found and fixed the problem and have been through all my tests that I
>> developed for my proof of concept package.  In the process, I found several
>> things I wasn't expecting.
>>
>> 1) The parser seems to have reversed the expected positions of the target
>> type - STEM, STREAM, or USING - and the "mode" - APPEND or REPLACE.  I
>> believe the syntax per the ANSI standard should be e.g. WITH OUTPUT STEM
>> APPEND someStem. and not WITH OUTPUT APPEND STEM someStem.
>>
>> 2) There seems to be a requirement for the stream/stem name to be either
>> a literal - "abc.txt" - or if it is a variable, enclosed in parenthesis -
>> (someFile).  I expected the () for USING but not for STEM or STREAM.
>>
>> 3) My package allowed for stem "names" to be specified with or without
>> the trailing . but my testing shows the trailing . is required or an error
>> message is generated.  I'm OK with that as long as it is documented.
>>
>> 4) I get an error (unable to open ... for output) when I specify
>>
>> output stream "abc2.txt" error replace stream "abc2.txt"
>> which was testing the redirection of both output and error to the same
>> stream.
>>
>> 5) Using a non-existent file for redirected input generates an error
>> (unable to open ... for reading) while my package treated this as a null
>> file.  Probably not a problem as the error message would flag a file name
>> typo :-)
>>
>> As for checking in the code, I am not a committer so I plan to generate a
>> patch file and attach it to RFE 4.
>>
>> One other item that I discovered is that I can get a deadlock in Windows
>> as well.  I had not tested sending a lot of data to redirected output but
>> when I did, my package locked up, presumably due to the output pipe being
>> full.  I plan to experiment with ways around this but will generate the
>> patch file with the code as it stands now.
>>
>> Gil
>>
>> On 4/11/2018 2:32 PM, Rick McGuire wrote:
>>
>>
>> On Wed, Apr 11, 2018 at 2:26 PM Gil Barmwater 
>> wrote:
>>
>>> Just a quick update on my progress so far.  I've gotten all the code in
>>> (hopefully) the right places and I have a clean build.  My initial tests
>>> are working but I've just hit my first problem which I need to debug.
>>> Hopefully, it will not be too much longer before I'm done.
>>
>> checking the code in as it is is also an option...it gets more eyeballs
>> on the problem
>>
>> Rick
>>
>>>
>>>
>>> --
>>>
>>> --
>>> Gil Barmwater
>>>
>>>
>>> 
>>> --
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> ___
>>> Oorexx-devel mailing list
>>> Oorexx-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>>
>>
>>
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>
>>
>>
>> ___
>> Oorexx-devel mailing 
>> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>> --
>> Gil Barmwater
>>
>>
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Oorexx-devel mailing list
>> Oorexx-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

I decided to post a question to StackOverflow to see if anybody has an
suggestions. I'm sort of thinking the solution might be to spin off a
thread to handle the writes to the buffer so that the main thread can
immediately start emptying the output pipes. I think I'll try taking a stab
at this for the Windows version.

Rick

On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater 
wrote:

> Found and fixed the problem and have been through all my tests that I
> developed for my proof of concept package.  In the process, I found several
> things I wasn't expecting.
>
> 1) The parser seems to have reversed the expected positions of the target
> type - STEM, STREAM, or USING - and the "mode" - APPEND or REPLACE.  I
> believe the syntax per the ANSI standard should be e.g. WITH OUTPUT STEM
> APPEND someStem. and not WITH OUTPUT APPEND STEM someStem.
>
> 2) There seems to be a requirement for the stream/stem name to be either a
> literal - "abc.txt" - or if it is a variable, enclosed in parenthesis -
> (someFile).  I expected the () for USING but not for STEM or STREAM.
>
> 3) My package allowed for stem "names" to be specified with or without the
> trailing . but my testing shows the trailing . is required or an error
> message is generated.  I'm OK with that as long as it is documented.
>
> 4) I get an error (unable to open ... for output) when I specify
>
> output stream "abc2.txt" error replace stream "abc2.txt"
> which was testing the redirection of both output and error to the same
> stream.
>
> 5) Using a non-existent file for redirected input generates an error
> (unable to open ... for reading) while my package treated this as a null
> file.  Probably not a problem as the error message would flag a file name
> typo :-)
>
> As for checking in the code, I am not a committer so I plan to generate a
> patch file and attach it to RFE 4.
>
> One other item that I discovered is that I can get a deadlock in Windows
> as well.  I had not tested sending a lot of data to redirected output but
> when I did, my package locked up, presumably due to the output pipe being
> full.  I plan to experiment with ways around this but will generate the
> patch file with the code as it stands now.
>
> Gil
>
> On 4/11/2018 2:32 PM, Rick McGuire wrote:
>
>
> On Wed, Apr 11, 2018 at 2:26 PM Gil Barmwater 
> wrote:
>
>> Just a quick update on my progress so far.  I've gotten all the code in
>> (hopefully) the right places and I have a clean build.  My initial tests
>> are working but I've just hit my first problem which I need to debug.
>> Hopefully, it will not be too much longer before I'm done.
>
> checking the code in as it is is also an option...it gets more eyeballs on
> the problem
>
> Rick
>
>>
>>
>> --
>>
>> --
>> Gil Barmwater
>>
>>
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Oorexx-devel mailing list
>> Oorexx-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Oorexx-devel mailing 
> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
> --
> Gil Barmwater
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater 
wrote:

> Found and fixed the problem and have been through all my tests that I
> developed for my proof of concept package.  In the process, I found several
> things I wasn't expecting.
>
> 1) The parser seems to have reversed the expected positions of the target
> type - STEM, STREAM, or USING - and the "mode" - APPEND or REPLACE.  I
> believe the syntax per the ANSI standard should be e.g. WITH OUTPUT STEM
> APPEND someStem. and not WITH OUTPUT APPEND STEM someStem.
>
> 2) There seems to be a requirement for the stream/stem name to be either a
> literal - "abc.txt" - or if it is a variable, enclosed in parenthesis -
> (someFile).  I expected the () for USING but not for STEM or STREAM.
>
> 3) My package allowed for stem "names" to be specified with or without the
> trailing . but my testing shows the trailing . is required or an error
> message is generated.  I'm OK with that as long as it is documented.
>
> 4) I get an error (unable to open ... for output) when I specify
>
> output stream "abc2.txt" error replace stream "abc2.txt"
> which was testing the redirection of both output and error to the same
> stream.
>
>
Problem now fixed.

Rick



> 5) Using a non-existent file for redirected input generates an error
> (unable to open ... for reading) while my package treated this as a null
> file.  Probably not a problem as the error message would flag a file name
> typo :-)
>
> As for checking in the code, I am not a committer so I plan to generate a
> patch file and attach it to RFE 4.
>
> One other item that I discovered is that I can get a deadlock in Windows
> as well.  I had not tested sending a lot of data to redirected output but
> when I did, my package locked up, presumably due to the output pipe being
> full.  I plan to experiment with ways around this but will generate the
> patch file with the code as it stands now.
>
> Gil
>
> On 4/11/2018 2:32 PM, Rick McGuire wrote:
>
>
> On Wed, Apr 11, 2018 at 2:26 PM Gil Barmwater 
> wrote:
>
>> Just a quick update on my progress so far.  I've gotten all the code in
>> (hopefully) the right places and I have a clean build.  My initial tests
>> are working but I've just hit my first problem which I need to debug.
>> Hopefully, it will not be too much longer before I'm done.
>
> checking the code in as it is is also an option...it gets more eyeballs on
> the problem
>
> Rick
>
>>
>>
>> --
>>
>> --
>> Gil Barmwater
>>
>>
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Oorexx-devel mailing list
>> Oorexx-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Oorexx-devel mailing 
> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
> --
> Gil Barmwater
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

Grrr, hit the wrong key again...

On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater 
wrote:

> Found and fixed the problem and have been through all my tests that I
> developed for my proof of concept package.  In the process, I found several
> things I wasn't expecting.
>
> 1) The parser seems to have reversed the expected positions of the target
> type - STEM, STREAM, or USING - and the "mode" - APPEND or REPLACE.  I
> believe the syntax per the ANSI standard should be e.g. WITH OUTPUT STEM
> APPEND someStem. and not WITH OUTPUT APPEND STEM someStem.
>
> 2) There seems to be a requirement for the stream/stem name to be either a
> literal - "abc.txt" - or if it is a variable, enclosed in parenthesis -
> (someFile).  I expected the () for USING but not for STEM or STREAM.
>
> 3) My package allowed for stem "names" to be specified with or without the
> trailing . but my testing shows the trailing . is required or an error
> message is generated.  I'm OK with that as long as it is documented.
>
> 4) I get an error (unable to open ... for output) when I specify
>
> output stream "abc2.txt" error replace stream "abc2.txt"
> which was testing the redirection of both output and error to the same
> stream.
>
> 5) Using a non-existent file for redirected input generates an error
> (unable to open ... for reading) while my package treated this as a null
> file.  Probably not a problem as the error message would flag a file name
> typo :-)
>

I think this is better behavior than failing silently.


>
> As for checking in the code, I am not a committer so I plan to generate a
> patch file and attach it to RFE 4.
>

Oh, I thought you were a committer. Attaching a patch also works.


>
> One other item that I discovered is that I can get a deadlock in Windows
> as well.  I had not tested sending a lot of data to redirected output but
> when I did, my package locked up, presumably due to the output pipe being
> full.  I plan to experiment with ways around this but will generate the
> patch file with the code as it stands now.
>
> Gil
>
> On 4/11/2018 2:32 PM, Rick McGuire wrote:
>
>
> On Wed, Apr 11, 2018 at 2:26 PM Gil Barmwater 
> wrote:
>
>> Just a quick update on my progress so far.  I've gotten all the code in
>> (hopefully) the right places and I have a clean build.  My initial tests
>> are working but I've just hit my first problem which I need to debug.
>> Hopefully, it will not be too much longer before I'm done.
>
> checking the code in as it is is also an option...it gets more eyeballs on
> the problem
>
> Rick
>
>>
>>
>> --
>>
>> --
>> Gil Barmwater
>>
>>
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Oorexx-devel mailing list
>> Oorexx-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Oorexx-devel mailing 
> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
> --
> Gil Barmwater
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

On Thu, Apr 12, 2018 at 2:56 PM, Gil Barmwater 
wrote:

> Found and fixed the problem and have been through all my tests that I
> developed for my proof of concept package.  In the process, I found several
> things I wasn't expecting.
>
> 1) The parser seems to have reversed the expected positions of the target
> type - STEM, STREAM, or USING - and the "mode" - APPEND or REPLACE.  I
> believe the syntax per the ANSI standard should be e.g. WITH OUTPUT STEM
> APPEND someStem. and not WITH OUTPUT APPEND STEM someStem.
>
No, the syntax diagrams on page 35 of the ANSI spec specify that APPEND and
REPLACE preceed the resource specification.



> 2) There seems to be a requirement for the stream/stem name to be either a
> literal - "abc.txt" - or if it is a variable, enclosed in parenthesis -
> (someFile).  I expected the () for USING but not for STEM or STREAM.
>
That is the standard ooRexx convention for instructions that can have
multiple keywords, for example, RAISE or FORWARD. This syntax convention
avoids conflicts with new keywords being added.


> 3) My package allowed for stem "names" to be specified with or without the
> trailing . but my testing shows the trailing . is required or an error
> message is generated.  I'm OK with that as long as it is documented.
>

Again, the specification says it is a stem. variable name. No period, it is
NOT a stem.


> 4) I get an error (unable to open ... for output) when I specify
>
> output stream "abc2.txt" error replace stream "abc2.txt"
> which was testing the redirection of both output and error to the same
> stream.
>


Hmmm, this should work...I even have some tests unit tests that test this.
I'll take a look at it.

>
> 5) Using a non-existent file for redirected input generates an error
> (unable to open ... for reading) while my package treated this as a null
> file.  Probably not a problem as the error message would flag a file name
> typo :-)
>
> As for checking in the code, I am not a committer so I plan to generate a
> patch file and attach it to RFE 4.
>
> One other item that I discovered is that I can get a deadlock in Windows
> as well.  I had not tested sending a lot of data to redirected output but
> when I did, my package locked up, presumably due to the output pipe being
> full.  I plan to experiment with ways around this but will generate the
> patch file with the code as it stands now.
>
> Gil
>
> On 4/11/2018 2:32 PM, Rick McGuire wrote:
>
>
> On Wed, Apr 11, 2018 at 2:26 PM Gil Barmwater 
> wrote:
>
>> Just a quick update on my progress so far.  I've gotten all the code in
>> (hopefully) the right places and I have a clean build.  My initial tests
>> are working but I've just hit my first problem which I need to debug.
>> Hopefully, it will not be too much longer before I'm done.
>
> checking the code in as it is is also an option...it gets more eyeballs on
> the problem
>
> Rick
>
>>
>>
>> --
>>
>> --
>> Gil Barmwater
>>
>>
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Oorexx-devel mailing list
>> Oorexx-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Oorexx-devel mailing 
> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
> --
> Gil Barmwater
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

Found and fixed the problem and have been through all my tests that I 
developed for my proof of concept package.  In the process, I found 
several things I wasn't expecting.


1) The parser seems to have reversed the expected positions of the 
target type - STEM, STREAM, or USING - and the "mode" - APPEND or 
REPLACE.  I believe the syntax per the ANSI standard should be e.g. WITH 
OUTPUT STEM APPEND someStem. and not WITH OUTPUT APPEND STEM someStem.


2) There seems to be a requirement for the stream/stem name to be either 
a literal - "abc.txt" - or if it is a variable, enclosed in parenthesis 
- (someFile).  I expected the () for USING but not for STEM or STREAM.


3) My package allowed for stem "names" to be specified with or without 
the trailing . but my testing shows the trailing . is required or an 
error message is generated.  I'm OK with that as long as it is documented.


4) I get an error (unable to open ... for output) when I specify

output stream "abc2.txt" error replace stream "abc2.txt"

which was testing the redirection of both output and error to the same 
stream.


5) Using a non-existent file for redirected input generates an error 
(unable to open ... for reading) while my package treated this as a null 
file.  Probably not a problem as the error message would flag a file 
name typo :-)


As for checking in the code, I am not a committer so I plan to generate 
a patch file and attach it to RFE 4.


One other item that I discovered is that I can get a deadlock in Windows 
as well.  I had not tested sending a lot of data to redirected output 
but when I did, my package locked up, presumably due to the output pipe 
being full.  I plan to experiment with ways around this but will 
generate the patch file with the code as it stands now.


Gil
On 4/11/2018 2:32 PM, Rick McGuire wrote:


On Wed, Apr 11, 2018 at 2:26 PM Gil Barmwater > wrote:


Just a quick update on my progress so far.  I've gotten all the
code in
(hopefully) the right places and I have a clean build.  My initial
tests
are working but I've just hit my first problem which I need to debug.
Hopefully, it will not be too much longer before I'm done.

checking the code in as it is is also an option...it gets more 
eyeballs on the problem


Rick



-- 

-- 
Gil Barmwater




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/oorexx-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!http://sdm.link/slashdot


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel


--
Gil Barmwater

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

These sound like good questions to ask on StackOverflow.

Rick

On Thu, Apr 12, 2018 at 2:37 AM, Erich Steinböck  wrote:

> checking the code in as it is is also an option...it gets more eyeballs on
>> the problem
>>
> I checked in the code for a Unix redirecting command handler.
> It's still work-in-progress, not only because of some minor issues (e. g.
> Ctrl+C handling), but because some severe restrictions, that I know no
> solutions for.
>
> 1) in a scenario, where we feed redirected input to a command that really
> doesn't consume it (e. g. echo), the command may finish before we can
> pipe all stdin input.  The spawned child exits, closing its stdin pipe read
> end.  On my test system, Ubuntu 16.04, the following write() operation
> then, instead of failing with an errno indication, will exit the program.
> No message, no segmentation fault, no indication at all.
> I'm trying circumventing this issue by not closing the parent stdin pipe
> read end (which normally isn't used at all and is suggested to be closed in
> most sample code).  This fixes the "exit" issue, but now, when the child
> finishes, further writes to the pipe don't give an error indication (as it
> is still valid because the parent read end is open) which leads to an
> endless loop.  This might be fixed if the handler checked whether the
> commandchild was still alive upon each loop (adding another complication
> that I wouldn't expect to be necessary)
>
> 2) when piping more than 64K (the default pipe size on Linux) to cat by
> redirecting stdin and stdout cat typically issues an error an output will
> miss lines.  I've left debug trace statements in the handler to show the
> issue.
>
> Feeding 660 lines with 100 characters each (plus a newline)
> the handler sees the correct 0 total bytes, writes them in 4K blocks
> (the size of PIP_BUF which ensures we have atomic writes)
> at 64K the pipe is full, tha handler sees an EAGAIN error, tries to read
> from stdout (nothing there yet), waits a little and loops.
> loops #2 and #3 see the full stdin pipe, and an empty stdout pipe, but now
> cat complains: "cat: -: Resource temporarily unavailable"
> in loop #4 the handler is able to write the remaining bytes to stdin, and
> also stdout can be read now, but not all expected output lines are there
>
> With less than 64K input to stdin, things seem to work fine.
>
> ~~~
> $ ./rexx -e "a = .Array~new; address io 'cat' with input using
> (.Array~new(arg(1))~fill('-'~copies(100))) output stem o.; say o.0 o.1"
> 660
> io args /bin/sh -c cat
> io redirection requested
> io input redirection requested
> io output redirection requested
> io spawned command
> io input feed total buffer 0 bytes
> io loop #1, input:1 output:1 error:0
> io input feed write offset 0, length 4096, written 4096 bytes
> io input feed write offset 4096, length 4096, written 4096 bytes
> io input feed write offset 8192, length 4096, written 4096 bytes
> io input feed write offset 12288, length 4096, written 4096 bytes
> io input feed write offset 16384, length 4096, written 4096 bytes
> io input feed write offset 20480, length 4096, written 4096 bytes
> io input feed write offset 24576, length 4096, written 4096 bytes
> io input feed write offset 28672, length 4096, written 4096 bytes
> io input feed write offset 32768, length 4096, written 4096 bytes
> io input feed write offset 36864, length 4096, written 4096 bytes
> io input feed write offset 40960, length 4096, written 4096 bytes
> io input feed write offset 45056, length 4096, written 4096 bytes
> io input feed write offset 49152, length 4096, written 4096 bytes
> io input feed write offset 53248, length 4096, written 4096 bytes
> io input feed write offset 57344, length 4096, written 4096 bytes
> io input feed write offset 61440, length 4096, written 4096 bytes
> io input feed write offset 65536, length 1124, written -1 bytes
> io input feed write failure Resource temporarily unavailable
> io output buffer -1 bytes read
> io output retry: Resource temporarily unavailable
> io loop wait 250 usec done
> io loop #2, input:1 output:1 error:0
> io input feed write offset 65536, length 1124, written -1 bytes
> io input feed write failure Resource temporarily unavailable
> io output buffer -1 bytes read
> io output retry: Resource temporarily unavailable
> io loop wait 250 usec done
> io loop #3, input:1 output:1 error:0
> io input feed write offset 65536, length 1124, written -1 bytes
> io input feed write failure Resource temporarily unavailable
> io output buffer -1 bytes read
> io output retry: Resource temporarily unavailable
> cat: -: Resource temporarily unavailable
> io loop wait 250 usec done
> io loop #4, input:1 output:1 error:0
> io input feed write offset 65536, length 1124, written 1124 bytes
> io input feed closed
> io output buffer 4096 bytes read
> io output buffer 4096 bytes read
> io output buffer 4096 bytes read
> io output buffer 4096 bytes read
> io output buffer 

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

>
> checking the code in as it is is also an option...it gets more eyeballs on
> the problem
>
I checked in the code for a Unix redirecting command handler.
It's still work-in-progress, not only because of some minor issues (e. g.
Ctrl+C handling), but because some severe restrictions, that I know no
solutions for.

1) in a scenario, where we feed redirected input to a command that really
doesn't consume it (e. g. echo), the command may finish before we can pipe
all stdin input.  The spawned child exits, closing its stdin pipe read
end.  On my test system, Ubuntu 16.04, the following write() operation
then, instead of failing with an errno indication, will exit the program.
No message, no segmentation fault, no indication at all.
I'm trying circumventing this issue by not closing the parent stdin pipe
read end (which normally isn't used at all and is suggested to be closed in
most sample code).  This fixes the "exit" issue, but now, when the child
finishes, further writes to the pipe don't give an error indication (as it
is still valid because the parent read end is open) which leads to an
endless loop.  This might be fixed if the handler checked whether the
commandchild was still alive upon each loop (adding another complication
that I wouldn't expect to be necessary)

2) when piping more than 64K (the default pipe size on Linux) to cat by
redirecting stdin and stdout cat typically issues an error an output will
miss lines.  I've left debug trace statements in the handler to show the
issue.

Feeding 660 lines with 100 characters each (plus a newline)
the handler sees the correct 0 total bytes, writes them in 4K blocks
(the size of PIP_BUF which ensures we have atomic writes)
at 64K the pipe is full, tha handler sees an EAGAIN error, tries to read
from stdout (nothing there yet), waits a little and loops.
loops #2 and #3 see the full stdin pipe, and an empty stdout pipe, but now
cat complains: "cat: -: Resource temporarily unavailable"
in loop #4 the handler is able to write the remaining bytes to stdin, and
also stdout can be read now, but not all expected output lines are there

With less than 64K input to stdin, things seem to work fine.

~~~
$ ./rexx -e "a = .Array~new; address io 'cat' with input using
(.Array~new(arg(1))~fill('-'~copies(100))) output stem o.; say o.0 o.1" 660
io args /bin/sh -c cat
io redirection requested
io input redirection requested
io output redirection requested
io spawned command
io input feed total buffer 0 bytes
io loop #1, input:1 output:1 error:0
io input feed write offset 0, length 4096, written 4096 bytes
io input feed write offset 4096, length 4096, written 4096 bytes
io input feed write offset 8192, length 4096, written 4096 bytes
io input feed write offset 12288, length 4096, written 4096 bytes
io input feed write offset 16384, length 4096, written 4096 bytes
io input feed write offset 20480, length 4096, written 4096 bytes
io input feed write offset 24576, length 4096, written 4096 bytes
io input feed write offset 28672, length 4096, written 4096 bytes
io input feed write offset 32768, length 4096, written 4096 bytes
io input feed write offset 36864, length 4096, written 4096 bytes
io input feed write offset 40960, length 4096, written 4096 bytes
io input feed write offset 45056, length 4096, written 4096 bytes
io input feed write offset 49152, length 4096, written 4096 bytes
io input feed write offset 53248, length 4096, written 4096 bytes
io input feed write offset 57344, length 4096, written 4096 bytes
io input feed write offset 61440, length 4096, written 4096 bytes
io input feed write offset 65536, length 1124, written -1 bytes
io input feed write failure Resource temporarily unavailable
io output buffer -1 bytes read
io output retry: Resource temporarily unavailable
io loop wait 250 usec done
io loop #2, input:1 output:1 error:0
io input feed write offset 65536, length 1124, written -1 bytes
io input feed write failure Resource temporarily unavailable
io output buffer -1 bytes read
io output retry: Resource temporarily unavailable
io loop wait 250 usec done
io loop #3, input:1 output:1 error:0
io input feed write offset 65536, length 1124, written -1 bytes
io input feed write failure Resource temporarily unavailable
io output buffer -1 bytes read
io output retry: Resource temporarily unavailable
cat: -: Resource temporarily unavailable
io loop wait 250 usec done
io loop #4, input:1 output:1 error:0
io input feed write offset 65536, length 1124, written 1124 bytes
io input feed closed
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output buffer 4096 bytes read
io output 

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

On Wed, Apr 11, 2018 at 2:26 PM Gil Barmwater 
wrote:

> Just a quick update on my progress so far.  I've gotten all the code in
> (hopefully) the right places and I have a clean build.  My initial tests
> are working but I've just hit my first problem which I need to debug.
> Hopefully, it will not be too much longer before I'm done.

checking the code in as it is is also an option...it gets more eyeballs on
the problem

Rick

>
>
> --
>
> --
> Gil Barmwater
>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

Just a quick update on my progress so far.  I've gotten all the code in 
(hopefully) the right places and I have a clean build.  My initial tests 
are working but I've just hit my first problem which I need to debug.  
Hopefully, it will not be too much longer before I'm done.


--

--
Gil Barmwater


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

Yes, but Windows uses crlf makers for file i/o. Just \n is a *ix thing.
However, if just a new line is expected in this situation on windows, I can
simplify my code.

In any event, I just committed the ReadInputBuffer change...currently
returning crlf linends on Windows.

Rick

On Fri, Apr 6, 2018 at 11:09 AM, Gil Barmwater 
wrote:

> Thanks for the quick reply.  I may wait for the ReadInputBuffer() as it
> may make it more like my package code which takes an array, converts it to
> a string with embedded linends, and writes it in chunks to the pipe.
> On the subject of the "proper" linends, I would think that since this is
> Windows-specific code, you wouldn't need to use the platform EOL marker.
>
> Gil
>
> On 4/6/2018 10:56 AM, Rick McGuire wrote:
>
> Gil,
>
> ReadInput returns just the data for the next line, with no line
> terminator. You can include the line terminator by doing a second write for
> just the terminator.
>
> I am also working on a ReadInputBuffer() function that will allow you to
> obtain the entire redirected output as a single buffer. Your question about
> the linends does raise a question though. I've written this code to use the
> platform EOL marker (i.e., CRLF on Windows). Should I be using '\n'
> everywhere?
>
> Rick
>
> On Fri, Apr 6, 2018 at 10:36 AM, Gil Barmwater 
> wrote:
>
>> Rick,
>>
>> Thanks for that info.  I agree that we should see how it plays out with
>> the default pipe buffer sizes and address it later if we encounter problems
>> in practice.
>>
>> I'm working on the code to write the data to the input pipe.  I've looked
>> at the testCommandHandler() and found the ReadInput() method (haven't done
>> an SVN update recently so you may have changed that) but wasn't sure what
>> the interface was.  It appears to read a line at a time but does it return
>> the \n as part of the data?  I need to include the \n characters when I
>> write the data to the pipe.  And is there a maximum size for a line of
>> input?
>>
>> Gil
>> On 4/6/2018 9:43 AM, Rick McGuire wrote:
>>
>> Gil,
>>
>> Been doing a little reading on the Windows Pipe and I/O functions. and it
>> appears you could also encounter a problem, though in a slightly different
>> way. If you are still writing data to the pipe and the buffer is full,
>> WriteFile() will block until the other end of the pipe reads enough data
>> from the pipe to complete your write. Unfortunately, the other process may
>> have already filled the output pipe and is sitting in the same WriteFile
>> wait situation waiting for you to read from the output pipe.
>>
>> It is possible to create a non-blocking pipe by using CreateNamedPipe()
>> rather than CreatePipe(), but I'm not sure this fixes the problem since the
>> non-blocking behavior applies to both read and write operations. This means
>> that the other process will see unexpected errors if either pipe's buffer
>> is full, and likely terminate the command. I suspect for now, we may need
>> to see how this plays out in practice. I see it is possible to specify a
>> buffer size on the CreatePipe() call. Perhaps creating a larger output
>> buffer would prevent the other process from filling it before the input
>> data is all written.
>>
>> Rick
>>
>> On Thu, Apr 5, 2018 at 8:29 PM, Gil Barmwater 
>> wrote:
>>
>>> Let me start by saying that I know next to nothing about *ix but perhaps
>>> the sequence in Windows may shed some light here.  The example I used for
>>> my package, and which is the basis of what I'm doing for ADDRESS...WITH on
>>> Windows, does the following: 1) create the pipes and pass the handles to
>>> the child process at creation, 2) with both processes now executing
>>> asynchronously, the main process writes all the data to the input pipe,
>>> possibly waiting should it fill the pipe, and closes it, 3) the child
>>> process reads the data, waiting if necessary, until it reaches the end and
>>> writes the data to the output and error pipes, 4) the main process,
>>> meanwhile, having finished writing the input pipe data, waits for the child
>>> process to terminate, 5) the child process finishes writing the output and
>>> error pipe data, closes them and terminates, and 6) the main process
>>> resumes after the child process has terminated, and reads the output and
>>> error pipes.  I don't see a blocking problem on the input side but there
>>> could be one on the output side if the pipes fill up and the OS can't
>>> "extend" them in some way.  Then the child will block, unable to write (and
>>> therefore to complete and terminate) and the main program will wait
>>> forever.  I'm hoping the OS can detect that somehow and notify one or both
>>> processes but without lots of research, I don't know the answer.
>>> Gil
>>>
>>> On 4/5/2018 5:58 PM, Erich Steinböck wrote:
>>>
>>> I have a working redirecting context on Ubuntu (no tests run yet).
>>> But I guess I need to think a little more about the 

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

Thanks for the quick reply.  I may wait for the ReadInputBuffer() as it 
may make it more like my package code which takes an array, converts it 
to a string with embedded linends, and writes it in chunks to the pipe.


On the subject of the "proper" linends, I would think that since this is 
Windows-specific code, you wouldn't need to use the platform EOL marker.


Gil
On 4/6/2018 10:56 AM, Rick McGuire wrote:

Gil,

ReadInput returns just the data for the next line, with no line 
terminator. You can include the line terminator by doing a second 
write for just the terminator.


I am also working on a ReadInputBuffer() function that will allow you 
to obtain the entire redirected output as a single buffer. Your 
question about the linends does raise a question though. I've written 
this code to use the platform EOL marker (i.e., CRLF on Windows). 
Should I be using '\n' everywhere?


Rick

On Fri, Apr 6, 2018 at 10:36 AM, Gil Barmwater 
> wrote:


Rick,

Thanks for that info.  I agree that we should see how it plays out
with the default pipe buffer sizes and address it later if we
encounter problems in practice.

I'm working on the code to write the data to the input pipe.  I've
looked at the testCommandHandler() and found the ReadInput()
method (haven't done an SVN update recently so you may have
changed that) but wasn't sure what the interface was.  It appears
to read a line at a time but does it return the \n as part of the
data?  I need to include the \n characters when I write the data
to the pipe.  And is there a maximum size for a line of input?

Gil

On 4/6/2018 9:43 AM, Rick McGuire wrote:

Gil,

Been doing a little reading on the Windows Pipe and I/O
functions. and it appears you could also encounter a problem,
though in a slightly different way. If you are still writing data
to the pipe and the buffer is full, WriteFile() will block until
the other end of the pipe reads enough data from the pipe to
complete your write. Unfortunately, the other process may have
already filled the output pipe and is sitting in the same
WriteFile wait situation waiting for you to read from the output
pipe.

It is possible to create a non-blocking pipe by using
CreateNamedPipe() rather than CreatePipe(), but I'm not sure this
fixes the problem since the non-blocking behavior applies to both
read and write operations. This means that the other process will
see unexpected errors if either pipe's buffer is full, and likely
terminate the command. I suspect for now, we may need to see how
this plays out in practice. I see it is possible to specify a
buffer size on the CreatePipe() call. Perhaps creating a larger
output buffer would prevent the other process from filling it
before the input data is all written.

Rick

On Thu, Apr 5, 2018 at 8:29 PM, Gil Barmwater
> wrote:

Let me start by saying that I know next to nothing about *ix
but perhaps the sequence in Windows may shed some light
here.  The example I used for my package, and which is the
basis of what I'm doing for ADDRESS...WITH on Windows, does
the following: 1) create the pipes and pass the handles to
the child process at creation, 2) with both processes now
executing asynchronously, the main process writes all the
data to the input pipe, possibly waiting should it fill the
pipe, and closes it, 3) the child process reads the data,
waiting if necessary, until it reaches the end and writes the
data to the output and error pipes, 4) the main process,
meanwhile, having finished writing the input pipe data, waits
for the child process to terminate, 5) the child process
finishes writing the output and error pipe data, closes them
and terminates, and 6) the main process resumes after the
child process has terminated, and reads the output and error
pipes.  I don't see a blocking problem on the input side but
there could be one on the output side if the pipes fill up
and the OS can't "extend" them in some way. Then the child
will block, unable to write (and therefore to complete and
terminate) and the main program will wait forever. I'm hoping
the OS can detect that somehow and notify one or both
processes but without lots of research, I don't know the answer.

Gil

On 4/5/2018 5:58 PM, Erich Steinböck wrote:

I have a working redirecting context on Ubuntu (no tests run
yet).
But I guess I need to think a little more about the blocking
issue.
What we're doing here is, fork a child (which runs the
command) and connect us to the child through at most three
pipes. A 

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

Gil,

ReadInput returns just the data for the next line, with no line terminator.
You can include the line terminator by doing a second write for just the
terminator.

I am also working on a ReadInputBuffer() function that will allow you to
obtain the entire redirected output as a single buffer. Your question about
the linends does raise a question though. I've written this code to use the
platform EOL marker (i.e., CRLF on Windows). Should I be using '\n'
everywhere?

Rick

On Fri, Apr 6, 2018 at 10:36 AM, Gil Barmwater 
wrote:

> Rick,
>
> Thanks for that info.  I agree that we should see how it plays out with
> the default pipe buffer sizes and address it later if we encounter problems
> in practice.
>
> I'm working on the code to write the data to the input pipe.  I've looked
> at the testCommandHandler() and found the ReadInput() method (haven't done
> an SVN update recently so you may have changed that) but wasn't sure what
> the interface was.  It appears to read a line at a time but does it return
> the \n as part of the data?  I need to include the \n characters when I
> write the data to the pipe.  And is there a maximum size for a line of
> input?
>
> Gil
> On 4/6/2018 9:43 AM, Rick McGuire wrote:
>
> Gil,
>
> Been doing a little reading on the Windows Pipe and I/O functions. and it
> appears you could also encounter a problem, though in a slightly different
> way. If you are still writing data to the pipe and the buffer is full,
> WriteFile() will block until the other end of the pipe reads enough data
> from the pipe to complete your write. Unfortunately, the other process may
> have already filled the output pipe and is sitting in the same WriteFile
> wait situation waiting for you to read from the output pipe.
>
> It is possible to create a non-blocking pipe by using CreateNamedPipe()
> rather than CreatePipe(), but I'm not sure this fixes the problem since the
> non-blocking behavior applies to both read and write operations. This means
> that the other process will see unexpected errors if either pipe's buffer
> is full, and likely terminate the command. I suspect for now, we may need
> to see how this plays out in practice. I see it is possible to specify a
> buffer size on the CreatePipe() call. Perhaps creating a larger output
> buffer would prevent the other process from filling it before the input
> data is all written.
>
> Rick
>
> On Thu, Apr 5, 2018 at 8:29 PM, Gil Barmwater 
> wrote:
>
>> Let me start by saying that I know next to nothing about *ix but perhaps
>> the sequence in Windows may shed some light here.  The example I used for
>> my package, and which is the basis of what I'm doing for ADDRESS...WITH on
>> Windows, does the following: 1) create the pipes and pass the handles to
>> the child process at creation, 2) with both processes now executing
>> asynchronously, the main process writes all the data to the input pipe,
>> possibly waiting should it fill the pipe, and closes it, 3) the child
>> process reads the data, waiting if necessary, until it reaches the end and
>> writes the data to the output and error pipes, 4) the main process,
>> meanwhile, having finished writing the input pipe data, waits for the child
>> process to terminate, 5) the child process finishes writing the output and
>> error pipe data, closes them and terminates, and 6) the main process
>> resumes after the child process has terminated, and reads the output and
>> error pipes.  I don't see a blocking problem on the input side but there
>> could be one on the output side if the pipes fill up and the OS can't
>> "extend" them in some way.  Then the child will block, unable to write (and
>> therefore to complete and terminate) and the main program will wait
>> forever.  I'm hoping the OS can detect that somehow and notify one or both
>> processes but without lots of research, I don't know the answer.
>> Gil
>>
>> On 4/5/2018 5:58 PM, Erich Steinböck wrote:
>>
>> I have a working redirecting context on Ubuntu (no tests run yet).
>> But I guess I need to think a little more about the blocking issue.
>> What we're doing here is, fork a child (which runs the command) and
>> connect us to the child through at most three pipes. A stdin pipe, a stdout
>> pipe, and an error pipe. We write to the stdin pipe, which the child then
>> reads, and the child writes to both the stdout and the error pipe, which we
>> than will read back in.
>>
>> Now pipes are something that the OS provides and it does some buffering
>> for us: you can write stuff to a pipe, before the child consumes it. But
>> it's just a small buffer, 4K or 64K. After that the pipe may block (or
>> break, which I may have experienced)
>>
>> So if we're eager to send stuff through the stdin pipe for the child to
>> use it, and the child is slow, or, worse, doesn't even expect anything on
>> stdin, the pipe blocks and we're stalled. In the meantime we could do a
>> little reading on the output or 

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

Rick,

Thanks for that info.  I agree that we should see how it plays out with 
the default pipe buffer sizes and address it later if we encounter 
problems in practice.


I'm working on the code to write the data to the input pipe. I've looked 
at the testCommandHandler() and found the ReadInput() method (haven't 
done an SVN update recently so you may have changed that) but wasn't 
sure what the interface was.  It appears to read a line at a time but 
does it return the \n as part of the data?  I need to include the \n 
characters when I write the data to the pipe.  And is there a maximum 
size for a line of input?


Gil

On 4/6/2018 9:43 AM, Rick McGuire wrote:

Gil,

Been doing a little reading on the Windows Pipe and I/O functions. and 
it appears you could also encounter a problem, though in a slightly 
different way. If you are still writing data to the pipe and the 
buffer is full, WriteFile() will block until the other end of the pipe 
reads enough data from the pipe to complete your write. Unfortunately, 
the other process may have already filled the output pipe and is 
sitting in the same WriteFile wait situation waiting for you to read 
from the output pipe.


It is possible to create a non-blocking pipe by using 
CreateNamedPipe() rather than CreatePipe(), but I'm not sure this 
fixes the problem since the non-blocking behavior applies to both read 
and write operations. This means that the other process will see 
unexpected errors if either pipe's buffer is full, and likely 
terminate the command. I suspect for now, we may need to see how this 
plays out in practice. I see it is possible to specify a buffer size 
on the CreatePipe() call. Perhaps creating a larger output buffer 
would prevent the other process from filling it before the input data 
is all written.


Rick

On Thu, Apr 5, 2018 at 8:29 PM, Gil Barmwater > wrote:


Let me start by saying that I know next to nothing about *ix but
perhaps the sequence in Windows may shed some light here.  The
example I used for my package, and which is the basis of what I'm
doing for ADDRESS...WITH on Windows, does the following: 1) create
the pipes and pass the handles to the child process at creation,
2) with both processes now executing asynchronously, the main
process writes all the data to the input pipe, possibly waiting
should it fill the pipe, and closes it, 3) the child process reads
the data, waiting if necessary, until it reaches the end and
writes the data to the output and error pipes, 4) the main
process, meanwhile, having finished writing the input pipe data,
waits for the child process to terminate, 5) the child process
finishes writing the output and error pipe data, closes them and
terminates, and 6) the main process resumes after the child
process has terminated, and reads the output and error pipes.  I
don't see a blocking problem on the input side but there could be
one on the output side if the pipes fill up and the OS can't
"extend" them in some way.  Then the child will block, unable to
write (and therefore to complete and terminate) and the main
program will wait forever.  I'm hoping the OS can detect that
somehow and notify one or both processes but without lots of
research, I don't know the answer.

Gil

On 4/5/2018 5:58 PM, Erich Steinböck wrote:

I have a working redirecting context on Ubuntu (no tests run yet).
But I guess I need to think a little more about the blocking issue.
What we're doing here is, fork a child (which runs the command)
and connect us to the child through at most three pipes. A stdin
pipe, a stdout pipe, and an error pipe. We write to the stdin
pipe, which the child then reads, and the child writes to both
the stdout and the error pipe, which we than will read back in.

Now pipes are something that the OS provides and it does some
buffering for us: you can write stuff to a pipe, before the child
consumes it. But it's just a small buffer, 4K or 64K. After that
the pipe may block (or break, which I may have experienced)

So if we're eager to send stuff through the stdin pipe for the
child to use it, and the child is slow, or, worse, doesn't even
expect anything on stdin, the pipe blocks and we're stalled. In
the meantime we could do a little reading on the output or error
pipe, because the child may be slow, because it itself has
already filled up, say, its output pipe and reading that would be
a relief (break the stall), but I'm not sure how to actually do this.

When do we write, when read? Which pipes? How large pipe
read/write blocks do we use? Do we sleep a little bit in between?
I also need to figure out how to do non-blocking reads.


We don't yet have ADDRESS WITH tests for a real Unix redirecting
command context yet, have we?


On Wed, Apr 4, 

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

Gil,

Been doing a little reading on the Windows Pipe and I/O functions. and it
appears you could also encounter a problem, though in a slightly different
way. If you are still writing data to the pipe and the buffer is full,
WriteFile() will block until the other end of the pipe reads enough data
from the pipe to complete your write. Unfortunately, the other process may
have already filled the output pipe and is sitting in the same WriteFile
wait situation waiting for you to read from the output pipe.

It is possible to create a non-blocking pipe by using CreateNamedPipe()
rather than CreatePipe(), but I'm not sure this fixes the problem since the
non-blocking behavior applies to both read and write operations. This means
that the other process will see unexpected errors if either pipe's buffer
is full, and likely terminate the command. I suspect for now, we may need
to see how this plays out in practice. I see it is possible to specify a
buffer size on the CreatePipe() call. Perhaps creating a larger output
buffer would prevent the other process from filling it before the input
data is all written.

Rick

On Thu, Apr 5, 2018 at 8:29 PM, Gil Barmwater 
wrote:

> Let me start by saying that I know next to nothing about *ix but perhaps
> the sequence in Windows may shed some light here.  The example I used for
> my package, and which is the basis of what I'm doing for ADDRESS...WITH on
> Windows, does the following: 1) create the pipes and pass the handles to
> the child process at creation, 2) with both processes now executing
> asynchronously, the main process writes all the data to the input pipe,
> possibly waiting should it fill the pipe, and closes it, 3) the child
> process reads the data, waiting if necessary, until it reaches the end and
> writes the data to the output and error pipes, 4) the main process,
> meanwhile, having finished writing the input pipe data, waits for the child
> process to terminate, 5) the child process finishes writing the output and
> error pipe data, closes them and terminates, and 6) the main process
> resumes after the child process has terminated, and reads the output and
> error pipes.  I don't see a blocking problem on the input side but there
> could be one on the output side if the pipes fill up and the OS can't
> "extend" them in some way.  Then the child will block, unable to write (and
> therefore to complete and terminate) and the main program will wait
> forever.  I'm hoping the OS can detect that somehow and notify one or both
> processes but without lots of research, I don't know the answer.
> Gil
>
> On 4/5/2018 5:58 PM, Erich Steinböck wrote:
>
> I have a working redirecting context on Ubuntu (no tests run yet).
> But I guess I need to think a little more about the blocking issue.
> What we're doing here is, fork a child (which runs the command) and
> connect us to the child through at most three pipes. A stdin pipe, a stdout
> pipe, and an error pipe. We write to the stdin pipe, which the child then
> reads, and the child writes to both the stdout and the error pipe, which we
> than will read back in.
>
> Now pipes are something that the OS provides and it does some buffering
> for us: you can write stuff to a pipe, before the child consumes it. But
> it's just a small buffer, 4K or 64K. After that the pipe may block (or
> break, which I may have experienced)
>
> So if we're eager to send stuff through the stdin pipe for the child to
> use it, and the child is slow, or, worse, doesn't even expect anything on
> stdin, the pipe blocks and we're stalled. In the meantime we could do a
> little reading on the output or error pipe, because the child may be slow,
> because it itself has already filled up, say, its output pipe and reading
> that would be a relief (break the stall), but I'm not sure how to actually
> do this.
>
> When do we write, when read? Which pipes? How large pipe read/write blocks
> do we use? Do we sleep a little bit in between?
> I also need to figure out how to do non-blocking reads.
>
>
> We don't yet have ADDRESS WITH tests for a real Unix redirecting command
> context yet, have we?
>
>
> On Wed, Apr 4, 2018 at 4:04 PM, Erich Steinböck <
> erich.steinbo...@gmail.com> wrote:
>
>> From the description in https://blog.uxul.de/e?e=mt-160 (very bottom) it
>> seems "trivial" to do.
>> Not a Linux guru at all, but I'd give it a try if no one else volunteers.
>>
>> The blog also contains a discussion of an interesting quirk with fork(),
>> which is why the sample code uses POSIX spawn to execute the command.
>> POSIX spawn seems to be available on all the platforms ooRexx supports.
>> Not on Android though.
>>
>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Oorexx-devel mailing 
> 

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

I also found this:

https://linux.die.net/man/2/pipe2

which allows you to create the pipe as non blocking.

Rick

On Thu, Apr 5, 2018 at 5:58 PM, Erich Steinböck 
wrote:

> I have a working redirecting context on Ubuntu (no tests run yet).
> But I guess I need to think a little more about the blocking issue.
> What we're doing here is, fork a child (which runs the command) and
> connect us to the child through at most three pipes. A stdin pipe, a stdout
> pipe, and an error pipe. We write to the stdin pipe, which the child then
> reads, and the child writes to both the stdout and the error pipe, which we
> than will read back in.
>
> Now pipes are something that the OS provides and it does some buffering
> for us: you can write stuff to a pipe, before the child consumes it. But
> it's just a small buffer, 4K or 64K. After that the pipe may block (or
> break, which I may have experienced)
>
> So if we're eager to send stuff through the stdin pipe for the child to
> use it, and the child is slow, or, worse, doesn't even expect anything on
> stdin, the pipe blocks and we're stalled. In the meantime we could do a
> little reading on the output or error pipe, because the child may be slow,
> because it itself has already filled up, say, its output pipe and reading
> that would be a relief (break the stall), but I'm not sure how to actually
> do this.
>
> When do we write, when read? Which pipes? How large pipe read/write blocks
> do we use? Do we sleep a little bit in between?
> I also need to figure out how to do non-blocking reads.
>
>
> We don't yet have ADDRESS WITH tests for a real Unix redirecting command
> context yet, have we?
>
>
> On Wed, Apr 4, 2018 at 4:04 PM, Erich Steinböck <
> erich.steinbo...@gmail.com> wrote:
>
>> From the description in https://blog.uxul.de/e?e=mt-160 (very bottom) it
>> seems "trivial" to do.
>> Not a Linux guru at all, but I'd give it a try if no one else volunteers.
>>
>> The blog also contains a discussion of an interesting quirk with fork(),
>> which is why the sample code uses POSIX spawn to execute the command.
>> POSIX spawn seems to be available on all the platforms ooRexx supports.
>> Not on Android though.
>>
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

On Thu, Apr 5, 2018 at 8:29 PM, Gil Barmwater 
wrote:

> Let me start by saying that I know next to nothing about *ix but perhaps
> the sequence in Windows may shed some light here.  The example I used for
> my package, and which is the basis of what I'm doing for ADDRESS...WITH on
> Windows, does the following: 1) create the pipes and pass the handles to
> the child process at creation, 2) with both processes now executing
> asynchronously, the main process writes all the data to the input pipe,
> possibly waiting should it fill the pipe, and closes it, 3) the child
> process reads the data, waiting if necessary, until it reaches the end and
> writes the data to the output and error pipes, 4) the main process,
> meanwhile, having finished writing the input pipe data, waits for the child
> process to terminate, 5) the child process finishes writing the output and
> error pipe data, closes them and terminates, and 6) the main process
> resumes after the child process has terminated, and reads the output and
> error pipes.  I don't see a blocking problem on the input side but there
> could be one on the output side if the pipes fill up and the OS can't
> "extend" them in some way.  Then the child will block, unable to write (and
> therefore to complete and terminate) and the main program will wait
> forever.  I'm hoping the OS can detect that somehow and notify one or both
> processes but without lots of research, I don't know the answer.
>

It sounds like that's the basics of what's going on with the Linux version
as well. However, on Linux the pipes are fairly limited in size. So Erich's
situation is basically this:

1) He writes as much data to the input pipe, but manages to write the data
faster than the process is consuming it, so he gets an error indicating the
pipe is full.
2) He can't just wait a little bit and try again, because the other process
might have the same problem on the output pipes.
3) If he switches to reading from the output pipes to clear those out, the
other process will resume reading from the input pipe because it has not
been closed yet, it will block.
4) Meanwhile, Erich is also blocked because he's also looking for data on a
pipe.

A non-blocking read would fix that problem, since it would allow him to
switch back-and-forth between reading and writing without creating a stall
situation. Hopefully, Windows will handle this for out, but I'm pretty sure
non-blocking reads are possible on Windows.

Rick




> Gil
>
> On 4/5/2018 5:58 PM, Erich Steinböck wrote:
>
> I have a working redirecting context on Ubuntu (no tests run yet).
> But I guess I need to think a little more about the blocking issue.
> What we're doing here is, fork a child (which runs the command) and
> connect us to the child through at most three pipes. A stdin pipe, a stdout
> pipe, and an error pipe. We write to the stdin pipe, which the child then
> reads, and the child writes to both the stdout and the error pipe, which we
> than will read back in.
>
> Now pipes are something that the OS provides and it does some buffering
> for us: you can write stuff to a pipe, before the child consumes it. But
> it's just a small buffer, 4K or 64K. After that the pipe may block (or
> break, which I may have experienced)
>
> So if we're eager to send stuff through the stdin pipe for the child to
> use it, and the child is slow, or, worse, doesn't even expect anything on
> stdin, the pipe blocks and we're stalled. In the meantime we could do a
> little reading on the output or error pipe, because the child may be slow,
> because it itself has already filled up, say, its output pipe and reading
> that would be a relief (break the stall), but I'm not sure how to actually
> do this.
>
> When do we write, when read? Which pipes? How large pipe read/write blocks
> do we use? Do we sleep a little bit in between?
> I also need to figure out how to do non-blocking reads.
>
>
> We don't yet have ADDRESS WITH tests for a real Unix redirecting command
> context yet, have we?
>
>
> On Wed, Apr 4, 2018 at 4:04 PM, Erich Steinböck <
> erich.steinbo...@gmail.com> wrote:
>
>> From the description in https://blog.uxul.de/e?e=mt-160 (very bottom) it
>> seems "trivial" to do.
>> Not a Linux guru at all, but I'd give it a try if no one else volunteers.
>>
>> The blog also contains a discussion of an interesting quirk with fork(),
>> which is why the sample code uses POSIX spawn to execute the command.
>> POSIX spawn seems to be available on all the platforms ooRexx supports.
>> Not on Android though.
>>
>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Oorexx-devel mailing 
> listOorexx-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
> --
> Gil Barmwater
>
>
> 

Re: [Oorexx-devel] Address With progress

[Gil Barmwater]

Let me start by saying that I know next to nothing about *ix but perhaps 
the sequence in Windows may shed some light here.  The example I used 
for my package, and which is the basis of what I'm doing for 
ADDRESS...WITH on Windows, does the following: 1) create the pipes and 
pass the handles to the child process at creation, 2) with both 
processes now executing asynchronously, the main process writes all the 
data to the input pipe, possibly waiting should it fill the pipe, and 
closes it, 3) the child process reads the data, waiting if necessary, 
until it reaches the end and writes the data to the output and error 
pipes, 4) the main process, meanwhile, having finished writing the input 
pipe data, waits for the child process to terminate, 5) the child 
process finishes writing the output and error pipe data, closes them and 
terminates, and 6) the main process resumes after the child process has 
terminated, and reads the output and error pipes.  I don't see a 
blocking problem on the input side but there could be one on the output 
side if the pipes fill up and the OS can't "extend" them in some way.  
Then the child will block, unable to write (and therefore to complete 
and terminate) and the main program will wait forever.  I'm hoping the 
OS can detect that somehow and notify one or both processes but without 
lots of research, I don't know the answer.


Gil
On 4/5/2018 5:58 PM, Erich Steinböck wrote:

I have a working redirecting context on Ubuntu (no tests run yet).
But I guess I need to think a little more about the blocking issue.
What we're doing here is, fork a child (which runs the command) and 
connect us to the child through at most three pipes. A stdin pipe, a 
stdout pipe, and an error pipe. We write to the stdin pipe, which the 
child then reads, and the child writes to both the stdout and the 
error pipe, which we than will read back in.


Now pipes are something that the OS provides and it does some 
buffering for us: you can write stuff to a pipe, before the child 
consumes it. But it's just a small buffer, 4K or 64K. After that the 
pipe may block (or break, which I may have experienced)


So if we're eager to send stuff through the stdin pipe for the child 
to use it, and the child is slow, or, worse, doesn't even expect 
anything on stdin, the pipe blocks and we're stalled. In the meantime 
we could do a little reading on the output or error pipe, because the 
child may be slow, because it itself has already filled up, say, its 
output pipe and reading that would be a relief (break the stall), but 
I'm not sure how to actually do this.


When do we write, when read? Which pipes? How large pipe read/write 
blocks do we use? Do we sleep a little bit in between?

I also need to figure out how to do non-blocking reads.


We don't yet have ADDRESS WITH tests for a real Unix redirecting 
command context yet, have we?



On Wed, Apr 4, 2018 at 4:04 PM, Erich Steinböck 
> wrote:


From the description in https://blog.uxul.de/e?e=mt-160
 (very bottom) it seems "trivial"
to do.
Not a Linux guru at all, but I'd give it a try if no one else
volunteers.

The blog also contains a discussion of an interesting quirk with
fork(), which is why the sample code uses POSIX spawn to execute
the command.
POSIX spawn seems to be available on all the platforms ooRexx
supports. Not on Android though.




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!http://sdm.link/slashdot


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel


--
Gil Barmwater

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

This article seems to describe the exact situation you are dealing with:

https://www.geeksforgeeks.org/non-blocking-io-with-pipes-in-c/

Rick

On Thu, Apr 5, 2018 at 5:58 PM, Erich Steinböck 
wrote:

> I have a working redirecting context on Ubuntu (no tests run yet).
> But I guess I need to think a little more about the blocking issue.
> What we're doing here is, fork a child (which runs the command) and
> connect us to the child through at most three pipes. A stdin pipe, a stdout
> pipe, and an error pipe. We write to the stdin pipe, which the child then
> reads, and the child writes to both the stdout and the error pipe, which we
> than will read back in.
>
> Now pipes are something that the OS provides and it does some buffering
> for us: you can write stuff to a pipe, before the child consumes it. But
> it's just a small buffer, 4K or 64K. After that the pipe may block (or
> break, which I may have experienced)
>
> So if we're eager to send stuff through the stdin pipe for the child to
> use it, and the child is slow, or, worse, doesn't even expect anything on
> stdin, the pipe blocks and we're stalled. In the meantime we could do a
> little reading on the output or error pipe, because the child may be slow,
> because it itself has already filled up, say, its output pipe and reading
> that would be a relief (break the stall), but I'm not sure how to actually
> do this.
>
> When do we write, when read? Which pipes? How large pipe read/write blocks
> do we use? Do we sleep a little bit in between?
> I also need to figure out how to do non-blocking reads.
>
>
> We don't yet have ADDRESS WITH tests for a real Unix redirecting command
> context yet, have we?
>
>
> On Wed, Apr 4, 2018 at 4:04 PM, Erich Steinböck <
> erich.steinbo...@gmail.com> wrote:
>
>> From the description in https://blog.uxul.de/e?e=mt-160 (very bottom) it
>> seems "trivial" to do.
>> Not a Linux guru at all, but I'd give it a try if no one else volunteers.
>>
>> The blog also contains a discussion of an interesting quirk with fork(),
>> which is why the sample code uses POSIX spawn to execute the command.
>> POSIX spawn seems to be available on all the platforms ooRexx supports.
>> Not on Android though.
>>
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

On Thu, Apr 5, 2018 at 5:58 PM, Erich Steinböck 
wrote:

> I have a working redirecting context on Ubuntu (no tests run yet).
> But I guess I need to think a little more about the blocking issue.
> What we're doing here is, fork a child (which runs the command) and
> connect us to the child through at most three pipes. A stdin pipe, a stdout
> pipe, and an error pipe. We write to the stdin pipe, which the child then
> reads, and the child writes to both the stdout and the error pipe, which we
> than will read back in.
>
> Now pipes are something that the OS provides and it does some buffering
> for us: you can write stuff to a pipe, before the child consumes it. But
> it's just a small buffer, 4K or 64K. After that the pipe may block (or
> break, which I may have experienced)
>
> So if we're eager to send stuff through the stdin pipe for the child to
> use it, and the child is slow, or, worse, doesn't even expect anything on
> stdin, the pipe blocks and we're stalled. In the meantime we could do a
> little reading on the output or error pipe, because the child may be slow,
> because it itself has already filled up, say, its output pipe and reading
> that would be a relief (break the stall), but I'm not sure how to actually
> do this.
>

I believe it is possible to increase the size of the pipe.

https://stackoverflow.com/questions/32449332/how-to-change-a-pipe-size-in-linux

Absent a non-blocking read, I'm not sure how this should be handled.


>
> When do we write, when read? Which pipes? How large pipe read/write blocks
> do we use? Do we sleep a little bit in between?
> I also need to figure out how to do non-blocking reads.
>
>
> We don't yet have ADDRESS WITH tests for a real Unix redirecting command
> context yet, have we?
>
Unless Mark Hessling has something, we're in new territory here.

Rick


>
>
> On Wed, Apr 4, 2018 at 4:04 PM, Erich Steinböck <
> erich.steinbo...@gmail.com> wrote:
>
>> From the description in https://blog.uxul.de/e?e=mt-160 (very bottom) it
>> seems "trivial" to do.
>> Not a Linux guru at all, but I'd give it a try if no one else volunteers.
>>
>> The blog also contains a discussion of an interesting quirk with fork(),
>> which is why the sample code uses POSIX spawn to execute the command.
>> POSIX spawn seems to be available on all the platforms ooRexx supports.
>> Not on Android though.
>>
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Rick McGuire]

On Thu, Apr 5, 2018 at 5:58 PM, Erich Steinböck 
wrote:

> I have a working redirecting context on Ubuntu (no tests run yet).
> But I guess I need to think a little more about the blocking issue.
> What we're doing here is, fork a child (which runs the command) and
> connect us to the child through at most three pipes. A stdin pipe, a stdout
> pipe, and an error pipe. We write to the stdin pipe, which the child then
> reads, and the child writes to both the stdout and the error pipe, which we
> than will read back in.
>
> Now pipes are something that the OS provides and it does some buffering
> for us: you can write stuff to a pipe, before the child consumes it. But
> it's just a small buffer, 4K or 64K. After that the pipe may block (or
> break, which I may have experienced)
>
> So if we're eager to send stuff through the stdin pipe for the child to
> use it, and the child is slow, or, worse, doesn't even expect anything on
> stdin, the pipe blocks and we're stalled. In the meantime we could do a
> little reading on the output or error pipe, because the child may be slow,
> because it itself has already filled up, say, its output pipe and reading
> that would be a relief (break the stall), but I'm not sure how to actually
> do this.
>
> When do we write, when read? Which pipes? How large pipe read/write blocks
> do we use? Do we sleep a little bit in between?
> I also need to figure out how to do non-blocking reads.
>

I believe it is possible to increase the buffer size of the pipe.

>
>
> We don't yet have ADDRESS WITH tests for a real Unix redirecting command
> context yet, have we?
>
>
> On Wed, Apr 4, 2018 at 4:04 PM, Erich Steinböck <
> erich.steinbo...@gmail.com> wrote:
>
>> From the description in https://blog.uxul.de/e?e=mt-160 (very bottom) it
>> seems "trivial" to do.
>> Not a Linux guru at all, but I'd give it a try if no one else volunteers.
>>
>> The blog also contains a discussion of an interesting quirk with fork(),
>> which is why the sample code uses POSIX spawn to execute the command.
>> POSIX spawn seems to be available on all the platforms ooRexx supports.
>> Not on Android though.
>>
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

I have a working redirecting context on Ubuntu (no tests run yet).
But I guess I need to think a little more about the blocking issue.
What we're doing here is, fork a child (which runs the command) and connect
us to the child through at most three pipes. A stdin pipe, a stdout pipe,
and an error pipe. We write to the stdin pipe, which the child then reads,
and the child writes to both the stdout and the error pipe, which we than
will read back in.

Now pipes are something that the OS provides and it does some buffering for
us: you can write stuff to a pipe, before the child consumes it. But it's
just a small buffer, 4K or 64K. After that the pipe may block (or break,
which I may have experienced)

So if we're eager to send stuff through the stdin pipe for the child to use
it, and the child is slow, or, worse, doesn't even expect anything on
stdin, the pipe blocks and we're stalled. In the meantime we could do a
little reading on the output or error pipe, because the child may be slow,
because it itself has already filled up, say, its output pipe and reading
that would be a relief (break the stall), but I'm not sure how to actually
do this.

When do we write, when read? Which pipes? How large pipe read/write blocks
do we use? Do we sleep a little bit in between?
I also need to figure out how to do non-blocking reads.


We don't yet have ADDRESS WITH tests for a real Unix redirecting command
context yet, have we?


On Wed, Apr 4, 2018 at 4:04 PM, Erich Steinböck 
wrote:

> From the description in https://blog.uxul.de/e?e=mt-160 (very bottom) it
> seems "trivial" to do.
> Not a Linux guru at all, but I'd give it a try if no one else volunteers.
>
> The blog also contains a discussion of an interesting quirk with fork(),
> which is why the sample code uses POSIX spawn to execute the command.
> POSIX spawn seems to be available on all the platforms ooRexx supports.
> Not on Android though.
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Address With progress

[Erich Steinböck]

>From the description in https://blog.uxul.de/e?e=mt-160 (very bottom) it
seems "trivial" to do.
Not a Linux guru at all, but I'd give it a try if no one else volunteers.

The blog also contains a discussion of an interesting quirk with fork(),
which is why the sample code uses POSIX spawn to execute the command.
POSIX spawn seems to be available on all the platforms ooRexx supports. Not
on Android though.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

[Oorexx-devel] Address With progress

[Rick McGuire]

I still have to write a bunch of unit tests for the global Address With
setup and overrides, but all of the base redirection mechanisms are now
passing the unit tests. The missing piece right now is the redirection in
the system handlers.

Any Linux gurus out there willing to take a look at how to do the unix
piping?

Rick
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel