dmainou commented on issue #6340: URL: https://github.com/apache/hop/issues/6340#issuecomment-3787974748
Ok, sorry it took me some days but here is the reproduction. I created 2 pipelines that are otherise the same. one creates a column "this_pipeline" with value "a" and the other "b" They receive parameters from above Executor and Parameter_passed. With the executor being the pipeline that triggered them e.g. a pipeline with 10 copies of the transform executor vs 1. and the parameter that we passed a/b <img width="939" height="556" alt="Image" src="https://github.com/user-attachments/assets/3cbba08f-1c4e-4d4a-8365-1b7e32f55b59"; /> The pipelines with the executor are identical other than the number of copies that are started for the pipeline executor transform. <img width="1254" height="772" alt="Image" src="https://github.com/user-attachments/assets/0d4b636c-ce53-4a65-bd2d-15487798d3ea"; /> The output is a csv where the data is appended where the file name is the value of the "executor" parameter. <img width="696" height="52" alt="Image" src="https://github.com/user-attachments/assets/016d00eb-f924-47b3-9f45-6d2be8458535"; /> Upon review of the outputs we can clearly observe that the pipeline that execute the logic for all rows was pipeline "a" as opposed to pipeline "b" for "b" rows. <img width="863" height="311" alt="Image" src="https://github.com/user-attachments/assets/86796863-557c-4ec2-b8fb-2ee0a206a654"; /> Therefore, if you have logic that needs execution that is specific to the type of data you are receiving you are currently constrained to a single lane. It would be great if we could launch multiple copies to gain some speed. [pipeline_executor.zip](https://github.com/user-attachments/files/24812389/pipeline_executor.zip) Thank you for your help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
