hansva opened a new issue, #9439: URL: https://github.com/apache/hop/issues/9439
### What needs to happen? The Script transform does two things for every row that it only needs to do once per run. The first affects every script language, the second only Groovy. 1. **Output field types are looked up per field per row.** `getValueFromScript` calls `field.getHopType()`, which searches the plugin registry for the type name, for every output field of every row. In a profile this was 11% of the transform's CPU. 2. **Groovy re-creates the script for every row.** The transform calls `compiledScript.eval(context)` per row. Groovy's JSR-223 engine then creates a new script object, a closure for each of its methods and a metaclass for that one object, so Groovy has to link every call in the script again. This costs about 5 microseconds per row, about 60% of the transform's time on a two-line script. ### Proposed changes **1. All languages: resolve the output field types once** On the first row, store the Hop type of each output field in an `int[]` next to `replaceIndex`, and use it in `getValueFromScript`. No behavior changes. **2. Groovy only: run the compiled script directly** - Compile the row script once to a Groovy script class. - For each row, create a new instance of that class over one binding that reads and writes the transform's bindings, and call `run()`. - Do not reuse one instance for all rows: a script with an `@Field` variable must get a freshly initialised field for every row, as it does today. The JSR-223 engine provides three things that must keep working: - **Functions of the start script can be called from the row script and the end script.** Run the start and end scripts the same way and give the scripts a base class that passes an unknown method on to the other scripts. - **The `context` variable.** Bind it once. - **`println` writes to the context's writer.** Bind `out` to it. JavaScript and Python stay on the JSR-223 path. Their engines do not re-create the script per row: measured the same way, Python evaluates 1.67 million rows per second and JavaScript 369,000, against 199,000 for Groovy. ### Results must not change Change 2 replaces engine behavior with our own code, so it only goes in when these give the same output before and after: | Script | Expected | |---|---| | Sets output variables from input fields | same values | | Calls a function defined in the start script | works | | Changes a variable that the start script set | value carries over to the next row | | Defines and calls its own function | works | | Uses an `@Field` variable | initialised again for every row | | Uses `context`, `transform`, `row`, `rowMeta`, `previousRow`, `rowNumber` | same values | | Keeps a closure in a variable between rows | works | | Sets `pipeline_status` to skip, abort or fail | same effect | | Throws an exception or has a syntax error | the transform fails with the script error | | Uses the script result as an output field | same value | The Script samples and integration tests must also give the same output. If any of these cannot be kept, drop change 2 and ship change 1 alone. ### Expected gain - Change 1: about 10% for every script language. From the profile share, not measured after a fix. - Change 2: about 2.4x for Groovy on benchmark B256 (2 million rows, two-line script, 116,000 rows per second today). Calculated from the measured cost of the evaluation, not measured in Hop. Outside Hop, the same script runs at 199,000 rows per second through JSR-223 and 7.9 million with a new instance per row run directly. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
