hansva opened a new issue, #9439:
URL: https://github.com/apache/hop/issues/9439

   ### What needs to happen?
   
   The Script transform does two things for every row that it only needs to do 
once per run. The first affects every script language, the second only Groovy.
   
   1. **Output field types are looked up per field per row.** 
`getValueFromScript` calls `field.getHopType()`, which searches the plugin 
registry for the type name, for every output field of every row. In a profile 
this was 11% of the transform's CPU.
   2. **Groovy re-creates the script for every row.** The transform calls 
`compiledScript.eval(context)` per row. Groovy's JSR-223 engine then creates a 
new script object, a closure for each of its methods and a metaclass for that 
one object, so Groovy has to link every call in the script again. This costs 
about 5 microseconds per row, about 60% of the transform's time on a two-line 
script.
   
   ### Proposed changes
   
   **1. All languages: resolve the output field types once**
   
   On the first row, store the Hop type of each output field in an `int[]` next 
to `replaceIndex`, and use it in `getValueFromScript`. No behavior changes.
   
   **2. Groovy only: run the compiled script directly**
   
   - Compile the row script once to a Groovy script class.
   - For each row, create a new instance of that class over one binding that 
reads and writes the transform's bindings, and call `run()`.
   - Do not reuse one instance for all rows: a script with an `@Field` variable 
must get a freshly initialised field for every row, as it does today.
   
   The JSR-223 engine provides three things that must keep working:
   
   - **Functions of the start script can be called from the row script and the 
end script.** Run the start and end scripts the same way and give the scripts a 
base class that passes an unknown method on to the other scripts.
   - **The `context` variable.** Bind it once.
   - **`println` writes to the context's writer.** Bind `out` to it.
   
   JavaScript and Python stay on the JSR-223 path. Their engines do not 
re-create the script per row: measured the same way, Python evaluates 1.67 
million rows per second and JavaScript 369,000, against 199,000 for Groovy.
   
   ### Results must not change
   
   Change 2 replaces engine behavior with our own code, so it only goes in when 
these give the same output before and after:
   
   | Script | Expected |
   |---|---|
   | Sets output variables from input fields | same values |
   | Calls a function defined in the start script | works |
   | Changes a variable that the start script set | value carries over to the 
next row |
   | Defines and calls its own function | works |
   | Uses an `@Field` variable | initialised again for every row |
   | Uses `context`, `transform`, `row`, `rowMeta`, `previousRow`, `rowNumber` 
| same values |
   | Keeps a closure in a variable between rows | works |
   | Sets `pipeline_status` to skip, abort or fail | same effect |
   | Throws an exception or has a syntax error | the transform fails with the 
script error |
   | Uses the script result as an output field | same value |
   
   The Script samples and integration tests must also give the same output.
   
   If any of these cannot be kept, drop change 2 and ship change 1 alone.
   
   ### Expected gain
   
   - Change 1: about 10% for every script language. From the profile share, not 
measured after a fix.
   - Change 2: about 2.4x for Groovy on benchmark B256 (2 million rows, 
two-line script, 116,000 rows per second today). Calculated from the measured 
cost of the evaluation, not measured in Hop. Outside Hop, the same script runs 
at 199,000 rows per second through JSR-223 and 7.9 million with a new instance 
per row run directly.
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to