hansva opened a new issue, #9445:
URL: https://github.com/apache/hop/issues/9445

   ### What needs to happen?
   
   The Pipeline executor transform converts every field of every incoming row 
to a String, in `PipelineExecutor.processRow()`. These values are only needed 
when a child pipeline is started, as the values of the parameters that are 
mapped to a field. With a group size of 1,000 rows, 999 of every 1,000 
conversions are thrown away; with wide rows, dates and numbers, the conversion 
is a large part of the transform's own work.
   
   ### Proposed change
   
   Convert the fields only when a child pipeline is executed, from the same row 
that is used today:
   
   - grouping by size or by time, and a new file name from a field: the current 
row;
   - grouping by a field, and the end of the input: the last row in the group 
buffer (`getLastIncomingFieldValues()`, unchanged).
   
   `processRow()` no longer builds the list for every row; the three places 
that pass it to `executePipeline(...)` build it at that moment.
   
   Nothing else changes: the parameter values, the order in which a field, a 
static value and an inherited variable are tried, the retry handling and the 
rows that go to the child pipeline stay the same.
   
   ### Measured gain
   
   Prototype against `apache/main`, both built from the same source, 2 million 
rows with 11 fields, one child pipeline per 1,000 rows, 5 runs each:
   
   | | Rows/s | Median |
   |---|---|---|
   | Before | 75k-89k | 87k |
   | After | 96k-110k | 100k |
   
   About 1.15x. With a group size of 1 there is no gain, because a child 
pipeline starts for every row.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: API


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to