hansva opened a new issue, #9455:
URL: https://github.com/apache/hop/issues/9455

   ### Apache Hop version?
   
   2.20.0
   
   ### Java version?
   
   21
   
   ### Operating system
   
   Other
   
   ### What happened?
   
   Four small performance fixes, each in one transform. None changes what the 
transform produces.
   
   **1. Standardize phone number resolves its settings for every row**
   
   For every row and every configured field, `processRow()` resolves the input, 
output, country, number type and validity field names, the number format and 
the default country, looks the fields up in the row by name, and parses the 
name of the number format. About 15% of the transform's CPU in a profile.
   
   Resolve all of it once, on the first row: the names, the field positions, 
the format and the default region per configured field. An input or country 
field that does not exist is reported on the first row, as it is today.
   
   **2. Load file content in memory resolves the file it already has**
   
   `getFileContent()` calls `getFileBinaryContent(data.file.toString(), 
variables)`, which resolves the file again from its name and opens it. The 
transform already holds the file object and knows its size. About 11% of its 
CPU.
   
   Read the content from `data.file` directly, into a buffer of the known size. 
The public static `getFileBinaryContent(String, IVariables)` stays for other 
callers.
   
   **3. Avro output works out the path of every field for every row**
   
   `getRecord()` strips and splits the Avro name of each output field and looks 
the field up in the schema by name, for every row. About 8% of CPU and 21% of 
the allocations.
   
   Work out the layout once, on the first row: for each record, the schema 
positions of its fields and its nested records. Per row, only put the values 
in. The layout comes from the same walk over the fields as today, so nested 
records and union schemas keep working.
   
   **4. Metadata injection polls for the template with a 50 ms sleep**
   
   After starting the template pipeline, the transform loops: copy the line 
counters, sleep 50 ms, check again. A template that finishes right after a 
check is noticed up to 50 ms later. One long run does not feel that, but a 
pipeline that injects a small template for every row or file pays it every 
time: about 25 seconds per 1,000 injections.
   
   Wait on the template's finished event with a 50 ms timeout instead of 
sleeping. The transform then goes on as soon as the template is done, still 
copies the line counters while it runs, and still reacts to a stop.
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to