hansva opened a new issue, #9455: URL: https://github.com/apache/hop/issues/9455
### Apache Hop version? 2.20.0 ### Java version? 21 ### Operating system Other ### What happened? Four small performance fixes, each in one transform. None changes what the transform produces. **1. Standardize phone number resolves its settings for every row** For every row and every configured field, `processRow()` resolves the input, output, country, number type and validity field names, the number format and the default country, looks the fields up in the row by name, and parses the name of the number format. About 15% of the transform's CPU in a profile. Resolve all of it once, on the first row: the names, the field positions, the format and the default region per configured field. An input or country field that does not exist is reported on the first row, as it is today. **2. Load file content in memory resolves the file it already has** `getFileContent()` calls `getFileBinaryContent(data.file.toString(), variables)`, which resolves the file again from its name and opens it. The transform already holds the file object and knows its size. About 11% of its CPU. Read the content from `data.file` directly, into a buffer of the known size. The public static `getFileBinaryContent(String, IVariables)` stays for other callers. **3. Avro output works out the path of every field for every row** `getRecord()` strips and splits the Avro name of each output field and looks the field up in the schema by name, for every row. About 8% of CPU and 21% of the allocations. Work out the layout once, on the first row: for each record, the schema positions of its fields and its nested records. Per row, only put the values in. The layout comes from the same walk over the fields as today, so nested records and union schemas keep working. **4. Metadata injection polls for the template with a 50 ms sleep** After starting the template pipeline, the transform loops: copy the line counters, sleep 50 ms, check again. A template that finishes right after a check is noticed up to 50 ms later. One long run does not feel that, but a pipeline that injects a small template for every row or file pays it every time: about 25 seconds per 1,000 injections. Wait on the template's finished event with a 50 ms timeout instead of sleeping. The transform then goes on as soon as the template is done, still copies the line counters while it runs, and still reacts to a stop. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
