Startrekzky commented on issue #2499: URL: https://github.com/apache/incubator-devlake/issues/2499#issuecomment-1281003183
> > > @asterix314 Thanks for the feedback, it makes sense to me, @Startrekzky what do you think? > > > > > > I'm not sure, may I know what specific problem happened while you were collecting data from multiple GitHub repos? Thanks. @asterix314 > > @Startrekzky I used the "2D list" API to collect a few hundred GitHub projects into DevLake. The stages ran in sequence. But when one stage fails, the whole pipeline stalls. There are many reasons for one stage to fail/stall: > > * miss-spelt project names > * rate limit exceeded > * etc. > > As a result, during the few days of the pipeline's execution, I found myself having to repeatedly return to the config-ui page to check if any stage failed and resume the pipeline manually. It would be much more efficient to automatically skip failed stages, and let the user intervene only at the end, when no executable stages are left unprocessed. Thanks for the use case, I'll think about it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
