Re: [Oorexx-devel] Ad compiled/tokenized Rexx code (rexxc) ...

[Rony G. Flatscher]

On 09.04.2019 17:38, Moritz Hoffmann wrote:
> Well, the Java scripting framework will always use a Reader, which decodes 
> the input according to
> the current Charset. I don't see a way to circumvent this, either by not 
> using a Reader instance
> or by injecting a custom, non-adapting Charset. I agree with Rick that this 
> is not a problem that
> ooRexx should solve. Instead, I'd base64-encode the scripts, and decode them 
> before passing them
> to the interpreter. Base64 will survive any reasonable Java byte decoding 
> magic.

The reason why it is a request (and not an error report ;) ) is that so far the 
binary
representation has not caused a problem, because only Rexx would create and 
later read the
self-produced tokenized code (knowing its internal structure and the binary 
data part).

Now, that Rexx scripts get deployed in an environment that only processes text 
and not binary data
(in this case the Java scripting frameworks) and there are use-cases (whenever 
java.io.Reader gets
triggered in the scripting frameworks) that cannot be safely intercepted by 
BSF4ooRexx, which
enables ooRexx to be hosted by and deployed from Java (applications) and also 
making all of the Java
world available to ooRexx.

As this uncovered problem lies in the binary representation of the tokenized 
Rexx code produced by
rexxc which in this use case turns out to be actually (and unexpectedly) a 
quite fragile solution it
would nevertheless be solvable by having rexxc (or some postprocessing utility) 
store it in form of
a pure text rendering. Hence this request addressed to ooRexx.

---

Ad Rick's idea to create a utiltity in BSF4ooRexx to recode the rexxc-produced 
binary data in an
"immune" textual form (basically coding the binary data with c2x or base64) and 
change BSF4ooRexx
internal (native) code to deal with this situation has the following 
implications:

  * the 'rexxc' generated and postprocessed tokenized format is then not 
compatible with Rexx anymore,
  * hence submitting such a non-compatible text-encoded version of the 
tokenized format would not be
processable by Rexx,

  * every Rexx programmer who wishes to use Rexx as a scripting language for 
Java applications or
uses BSF4ooRexx to create demanding applications in pure ooRexx using Java 
classes for missing
ooRexx features (like portable GUIs, but also IPV6-socket connections or 
ssl-connections, etc.)
would need to know that for sheltering their Rexx code they must

  o run 'rexxc' on each script,
  o plus must not forget to run the BSF4ooRexx postprocessing script 
applied on those freshly
produced 'rexxc' tokenized scripts,
  o deploy the postprocessed versions (*not* the original and Rexx 
compatible 'rexxc' tokenized
version)

  * now take into account that application systems need to be maintained in a 
constant manner
possibly by different programmers, each maintenance change causes the need 
to 'rexxc' and in
addition to apply the non-ooRexx utility to postprocess the tokenized code 
to turn it into pure
7-bit text; adding another step, making it more cumbersome and therefore 
over time mostlikely
more error-prone.

If it was possible to encode the tokenized form rexxc currently produces in an 
agreed upon format as
text then ooRexx could process such a version by converting it back into its 
binary form, and
proceed with the resulting data as today. In this scenario one would not create 
a non-compatible
version of a tokenized Rexx program, but rather one that ooRexx would be able 
to execute without the
help of any other tool.

This is just thinking loud about the situation and the idea of creating a 
non-ooRexx compatible
tokenized version of Rexx programs in the file system.
(Still have to think about it.)

---rony


>
> On Tue, Apr 9, 2019 at 3:34 PM Rony G. Flatscher  > wrote:
>
> On 09.04.2019 15:13, Rick McGuire wrote:
> > And it doesn't make sense to put the onus on solving this on the 
> interpreter. If it is
> desired to
> > store binary code in an inherently text-based interface, then 
> BSF4ooRexx should handle this be
> > including a utility to perform the transform and then decoding the 
> format before passing it
> to the
> > interpreter.
>
> Well, please tell IBM, Oracle and the OpenJDK community which devised and 
> applied all of the Java
> scripting frameworks in the past twenty years such that they expect 
> scripts to be supplied as text
> only, not as binary data.
>
> If a Java program employs e.g. the "javax.script" framework and it is 
> supplied the name of a Rexx
> program file, the "javax.script" framework will use some "java.io.Reader" 
> to read the script from
> the file. In the case of rexxc-tokenized Rexx code the Reader will 
> inadvertently destroy the
> binary
> data due to possible codepage translations well before BSF4ooRexx 
> receives the (then 

Re: [Oorexx-devel] Ad compiled/tokenized Rexx code (rexxc) ...

[Moritz Hoffmann]

Well, the Java scripting framework will always use a Reader, which decodes
the input according to the current Charset. I don't see a way to circumvent
this, either by not using a Reader instance or by injecting a custom,
non-adapting Charset. I agree with Rick that this is not a problem that
ooRexx should solve. Instead, I'd base64-encode the scripts, and decode
them before passing them to the interpreter. Base64 will survive any
reasonable Java byte decoding magic.

Cheers,
Moritz

On Tue, Apr 9, 2019 at 3:34 PM Rony G. Flatscher 
wrote:

> On 09.04.2019 15:13, Rick McGuire wrote:
> > And it doesn't make sense to put the onus on solving this on the
> interpreter. If it is desired to
> > store binary code in an inherently text-based interface, then BSF4ooRexx
> should handle this be
> > including a utility to perform the transform and then decoding the
> format before passing it to the
> > interpreter.
>
> Well, please tell IBM, Oracle and the OpenJDK community which devised and
> applied all of the Java
> scripting frameworks in the past twenty years such that they expect
> scripts to be supplied as text
> only, not as binary data.
>
> If a Java program employs e.g. the "javax.script" framework and it is
> supplied the name of a Rexx
> program file, the "javax.script" framework will use some "java.io.Reader"
> to read the script from
> the file. In the case of rexxc-tokenized Rexx code the Reader will
> inadvertently destroy the binary
> data due to possible codepage translations well before BSF4ooRexx receives
> the (then ruined binary)
> script data.
>
> There is nothing, BSF4ooRexx could do here as it exploits/applies and
> depends on the Java scripting
> frameworks, unfortunately.
>
> Hence asking ooRexx to the rescue;  suggesting to add a single step before
> storing the rexxc
> produced binary data by turning the binary data into a hexadecimal string
> and store that lossless
> rendering;  and upon receiving a tokenized Rexx program adding a single
> reverse step to turn the
> hexadecimal string losslessly back into the needed binary form for
> unflatening.
>
> ---rony
>
>
>
>
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>


-- 
Moritz Hoffmann;
http://antiguru.de/
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Ad compiled/tokenized Rexx code (rexxc) ...

[Rick McGuire]

Seriously, did you not not even read me email. This is totally solvable in
BSFooRexx by storing it a non-binary format defined by BSFooRexx that the
BSFooRexx framework can recognize as being a binary format that needs to be
decoded before passing it to the interpreter.  Provide a little utility
that can transform the compiled program into that format before hand.

Rick

On Tue, Apr 9, 2019 at 9:34 AM Rony G. Flatscher 
wrote:

> On 09.04.2019 15:13, Rick McGuire wrote:
> > And it doesn't make sense to put the onus on solving this on the
> interpreter. If it is desired to
> > store binary code in an inherently text-based interface, then BSF4ooRexx
> should handle this be
> > including a utility to perform the transform and then decoding the
> format before passing it to the
> > interpreter.
>
> Well, please tell IBM, Oracle and the OpenJDK community which devised and
> applied all of the Java
> scripting frameworks in the past twenty years such that they expect
> scripts to be supplied as text
> only, not as binary data.
>
> If a Java program employs e.g. the "javax.script" framework and it is
> supplied the name of a Rexx
> program file, the "javax.script" framework will use some "java.io.Reader"
> to read the script from
> the file. In the case of rexxc-tokenized Rexx code the Reader will
> inadvertently destroy the binary
> data due to possible codepage translations well before BSF4ooRexx receives
> the (then ruined binary)
> script data.
>
> There is nothing, BSF4ooRexx could do here as it exploits/applies and
> depends on the Java scripting
> frameworks, unfortunately.
>
> Hence asking ooRexx to the rescue;  suggesting to add a single step before
> storing the rexxc
> produced binary data by turning the binary data into a hexadecimal string
> and store that lossless
> rendering;  and upon receiving a tokenized Rexx program adding a single
> reverse step to turn the
> hexadecimal string losslessly back into the needed binary form for
> unflatening.
>
> ---rony
>
>
>
>
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Ad compiled/tokenized Rexx code (rexxc) ...

[Rony G. Flatscher]

On 09.04.2019 14:56, Michael Lueck wrote:
> Rony G. Flatscher wrote:
>>
>> However,  since he started to use "rexxc" to compile/tokenize all of the 
>> Rexx source files that
>> comprise his elaborate application to shelter his code, he has started to 
>> run into abends/errors
>> 13.1. 
>
>
> "sheltering code" is the only reason he chose to compile/tokenize his code... 
> there is no
> requirement to do so when using BSF4ooRexx?

Indeed "sheltering code" is the only (and quite understandable) reason.

BSF4ooRexx depends on the Java scripting frameworks from IBM, Oracle and 
OpenJDK, which expect
script code to be supplied as text and not as binary data (which unfortunately 
causes this problem).

---rony




___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Ad compiled/tokenized Rexx code (rexxc) ...

[Rick McGuire]

And it doesn't make sense to put the onus on solving this on the
interpreter. If it is desired to store binary code in an inherently
text-based interface, then BSF4ooRexx should handle this be including a
utility to perform the transform and then decoding the format before
passing it to the interpreter.

Rick

On Tue, Apr 9, 2019 at 9:10 AM Michael Lueck 
wrote:

> Greetings Rony,
>
>
> Rony G. Flatscher wrote:
> >
> > However,  since he started to use "rexxc" to compile/tokenize all of the
> Rexx source files that comprise his elaborate application to shelter his
> code, he has started to run into abends/errors 13.1.
>
>
> "sheltering code" is the only reason he chose to compile/tokenize his
> code... there is no requirement to do so when using BSF4ooRexx?
>
> While tokenizing ooRexx code seems appealing, I have seen time after time
> where tokenized ooRexx code fails to cooperate. I personally have
> "staunchly" decided against ever tokenizing any of my ooRexx
> code.
>
> Thus asking if it is a requirement I have somehow missed that "Programs
> utilizing / touching BSF4ooRexx MUST be tokenized."
>
> I am thankful,
>
> --
> Michael Lueck
> Lueck Data Systems
> http://www.lueckdatasystems.com/
>
>
> ___
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Ad compiled/tokenized Rexx code (rexxc) ...

[Michael Lueck]

Greetings Rony,


Rony G. Flatscher wrote:


However,  since he started to use "rexxc" to compile/tokenize all of the Rexx source files that comprise his elaborate application to shelter his code, he has started to run into abends/errors 13.1. 



"sheltering code" is the only reason he chose to compile/tokenize his code... 
there is no requirement to do so when using BSF4ooRexx?

While tokenizing ooRexx code seems appealing, I have seen time after time where tokenized ooRexx code fails to cooperate. I personally have "staunchly" decided against ever tokenizing any of my ooRexx 
code.


Thus asking if it is a requirement I have somehow missed that "Programs utilizing / 
touching BSF4ooRexx MUST be tokenized."

I am thankful,

--
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Ad compiled/tokenized Rexx code (rexxc) ...

[Rony G. Flatscher]

On 09.04.2019 14:32, Rony G. Flatscher wrote:
... cut ...

> Unfortunately, running rexxc-tokenized/compiled scripts via the Java 
> scripting framework does not
> work as apprehended in my last post and causes the following runtime error:
>
> G:\test\oorexx\rexxc>rexxj test.rexx
> RexxDispatcher.java: Throwable of type 
> 'org.rexxla.bsf.engines.rexx.RexxException' thrown while invoking Rexx:
> getLocalizedMessage(): [BSF4ooRexx/routine/jniRexxRunProgram(), error 9:
> Rexx traceback line not available from the Rexx condition object (Rexx 
> condition may have been caused by a call or message from Java to Rexx)
> Error 3 running  line 0:  Failure during initialization.
> Error 3.903:  Program "G:\test\oorexx\rexxc\test.rexx" cannot be run by 
> this version of the REXX interpreter.]
>
> (The error text within the square brackets gets created in native code from 
> the Rexx condition
> object. The 'getLocalizedMessage():' right before the opening square bracket 
> seems to come from
> ooRexx, it is not used in the native code.)

... cut ...

The "getLocalizedMessage()" does *not* come from ooRexx, it comes from one of 
the Java programs.

---rony

___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

Re: [Oorexx-devel] Ad compiled/tokenized Rexx code (rexxc) ...

[Rony G. Flatscher]

 fixes NewRoutine(), 
LoadPackageFromDate() and also
the Routine's class method newFile() when employed with 
rexxc-tokenized/compiled code, thank you
very much!

---

Unfortunately, running rexxc-tokenized/compiled scripts via the Java scripting 
framework does not
work as apprehended in my last post and causes the following runtime error:

G:\test\oorexx\rexxc>rexxj test.rexx
RexxDispatcher.java: Throwable of type 
'org.rexxla.bsf.engines.rexx.RexxException' thrown while invoking Rexx:
getLocalizedMessage(): [BSF4ooRexx/routine/jniRexxRunProgram(), error 9:
Rexx traceback line not available from the Rexx condition object (Rexx 
condition may have been caused by a call or message from Java to Rexx)
Error 3 running  line 0:  Failure during initialization.
Error 3.903:  Program "G:\test\oorexx\rexxc\test.rexx" cannot be run by 
this version of the REXX interpreter.]

(The error text within the square brackets gets created in native code from the 
Rexx condition
object. The 'getLocalizedMessage():' right before the opening square bracket 
seems to come from
ooRexx, it is not used in the native code.)

The reason for this runtime error with rexxc-tokenized/compiled code is caused 
by the implicitly
employed Java Reader object for reading the content of a file, changing the 
tokenized/compiled
binary data by applying codepage translations. (There are no Java based 
scripting frameworks that
would allow using a Stream for reading binary script data, unfortunately.)

---rony


On 07.04.2019 15:33, Rony G. Flatscher wrote:
> ... cut ...
>
> ---
>
> *Unfortunately*, there is one principal huge problem left with 
> compiled/tokenized Rexx code, that
> is beyond BSF4ooRexx' ability to solve for good: BSF4ooRexx uses two Java 
> scripting frameworks,
> one originating from IBM (DeveloperWorks) which got handed over to the ASF 
> named "BSF" (Bean
> Scripting Framework), one originating from Java itself which got introduced 
> later as the "Java
> scripting framework" in Java 1.6/6.0 (package named "javax.script", the URL 
> to the package
> documentation 
> ).
>
> BSF4ooRexx implements under the name "RexxScript" the javax.script scripting 
> framework, which
> makes it e.g. possible for JavaFX to denote the programming language "rexx" 
> in the FXML files that
> define the GUI. The FXMLLoader class will therefore be able to create an 
> ooRexx interpreter with
> the help of "javax.script" and execute the code in any given files or Rexx 
> code supplied verbaitm
> in event attributes of JavaFX GUI elements.
>
> The Java documentation for the javax.script.Engine interface that must be 
> (and has been)
> implemented can be found here:
> . 
> Executing ooRexx code
> from Java will be done with one of the eval() methods which either expect a 
> /java.lang.String/ or
> a /java.io.Reader/ object that represents the code to execute.
>
> A Java /Reader /will expect plain text to be read/and applies codepage 
> translations/ if necessary!
> This means that a /Reader /object if reading compiled/tokenized Rexx code 
> might transform the
> binary data! With other words, it is quite likely that compiled/tokenized 
> (binary data) Rexx code
> read by Java from a file with a /Reader/ gets destroyed by applying 
> mistakingly code-page
> translations to that binary data! :-((
>
> ---
>
> A maybe feasible solution could be to apply c2x() to the binary data that 
> results when applying
> "rexxc" and store that data right after the eye-catcher string "/**/@REXX" 
> that indicates that the
> remaining data represents the compiled/tokenized version of a Rexx program. 
> Upon loading, that
> hexadecimal string representing the binary data could be changed back to the 
> original binary
> representation with x2c() before processing it.
>
> This way all the Java defined String/Reader based interfaces would keep 
> working and there would be
> no risk whatsoever, that automatically applied codepage translations could 
> ruin/destroy the binary
> data as that data would be turned into plain text by c2x()!
>
> I would file a short RFE to this effect, but wanted to give "the long story" 
> (trying to explain
> the background and why it would be very important for allowing the Java 
> scripting frameworks to be
> applied safely to compiled/tokenized Rexx code as well) here.
>
> ---rony
>

___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel

[Oorexx-devel] Ad compiled/tokenized Rexx code (rexxc) ...

[Rony G. Flatscher]

In the past days I got problem reports from a professional developer who 
exclusively uses ooRexx and
for his rather complex GUIs he uses the portable JavaFX libraries via 
BSF4ooRexx. Everything has
worked out fine it seems, using ooRexx 5.0beta and BSF4ooRexx.

However,  since he started to use "rexxc" to compile/tokenize all of the Rexx 
source files that
comprise his elaborate application to shelter his code, he has started to run 
into abends/errors
13.1. This was the kick-off to analyze the code paths of BSF4ooRexx and getting 
to the ooRexx native
APIs NewRoutine() and LoadPackageFromData() that currently cause that error. 
Once these native APIs
become able to process compiled/tokenized Rexx code like LoadPackage(), these 
errors will go away.

---

*Unfortunately*, there is one principal huge problem left with 
compiled/tokenized Rexx code, that is
beyond BSF4ooRexx' ability to solve for good: BSF4ooRexx uses two Java 
scripting frameworks, one
originating from IBM (DeveloperWorks) which got handed over to the ASF named 
"BSF" (Bean Scripting
Framework), one originating from Java itself which got introduced later as the 
"Java scripting
framework" in Java 1.6/6.0 (package named "javax.script", the URL to the 
package documentation
).

BSF4ooRexx implements under the name "RexxScript" the javax.script scripting 
framework, which makes
it e.g. possible for JavaFX to denote the programming language "rexx" in the 
FXML files that define
the GUI. The FXMLLoader class will therefore be able to create an ooRexx 
interpreter with the help
of "javax.script" and execute the code in any given files or Rexx code supplied 
verbaitm in event
attributes of JavaFX GUI elements.

The Java documentation for the javax.script.Engine interface that must be (and 
has been) implemented
can be found here: 
.
Executing ooRexx code from Java will be done with one of the eval() methods 
which either expect a
/java.lang.String/ or a /java.io.Reader/ object that represents the code to 
execute.

A Java /Reader /will expect plain text to be read/and applies codepage 
translations/ if necessary!
This means that a /Reader /object if reading compiled/tokenized Rexx code might 
transform the binary
data! With other words, it is quite likely that compiled/tokenized (binary 
data) Rexx code read by
Java from a file with a /Reader/ gets destroyed by applying mistakingly 
code-page translations to
that binary data! :-((

---

A maybe feasible solution could be to apply c2x() to the binary data that 
results when applying
"rexxc" and store that data right after the eye-catcher string "/**/@REXX" that 
indicates that the
remaining data represents the compiled/tokenized version of a Rexx program. 
Upon loading, that
hexadecimal string representing the binary data could be changed back to the 
original binary
representation with x2c() before processing it.

This way all the Java defined String/Reader based interfaces would keep working 
and there would be
no risk whatsoever, that automatically applied codepage translations could 
ruin/destroy the binary
data as that data would be turned into plain text by c2x()!

I would file a short RFE to this effect, but wanted to give "the long story" 
(trying to explain the
background and why it would be very important for allowing the Java scripting 
frameworks to be
applied safely to compiled/tokenized Rexx code as well) here.

---rony


___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel