Hi Dev, I came across few requirements for For each. I would like to discuss these before i create few JIRA tasks.
1. Currently For each take the array list and run jobs based on array size without caring about the Array size. On grid resources it may not be optimal to run lot of jobs in parallel and some time we can run into Airavata software limits also. My idea is if we can make For each configurable with number of jobs in parallel limits. We can split the jobs in batches based on limit defined by workflow composer. 2. There can be a case if few jobs fail while submission (connection failure or other middleware failure) and rest of the jobs submit fine. We need to find a way to handle such failures. In case of connection we can retry job submission but if resubmission fails, we need to either cancel rest of the jobs or decide to accept partial results. 3. Handle partial application failures. If i ran 20 jobs and only one did not produce results and if gateway is ok with results of 19 jobs and can run 1 remaining job as a separate process. We can have some attribute to contract to be strict check or not with few other attributes based on advise. 4. Show some information about the number of jobs running/completed/waiting on the node. Thanks Raminder
