hansva opened a new issue, #9445: URL: https://github.com/apache/hop/issues/9445
### What needs to happen? The Pipeline executor transform converts every field of every incoming row to a String, in `PipelineExecutor.processRow()`. These values are only needed when a child pipeline is started, as the values of the parameters that are mapped to a field. With a group size of 1,000 rows, 999 of every 1,000 conversions are thrown away; with wide rows, dates and numbers, the conversion is a large part of the transform's own work. ### Proposed change Convert the fields only when a child pipeline is executed, from the same row that is used today: - grouping by size or by time, and a new file name from a field: the current row; - grouping by a field, and the end of the input: the last row in the group buffer (`getLastIncomingFieldValues()`, unchanged). `processRow()` no longer builds the list for every row; the three places that pass it to `executePipeline(...)` build it at that moment. Nothing else changes: the parameter values, the order in which a field, a static value and an inherited variable are tried, the retry handling and the rows that go to the child pipeline stay the same. ### Measured gain Prototype against `apache/main`, both built from the same source, 2 million rows with 11 fields, one child pipeline per 1,000 rows, 5 runs each: | | Rows/s | Median | |---|---|---| | Before | 75k-89k | 87k | | After | 96k-110k | 100k | About 1.15x. With a group size of 1 there is no gain, because a child pipeline starts for every row. ### Issue Priority Priority: 2 ### Issue Component Component: API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
