Dev-iL commented on issue #60387: URL: https://github.com/apache/airflow/issues/60387#issuecomment-3742897376
@Arunodoy18 That does sound like it aligns with @potiuk's [plan of action](https://github.com/apache/airflow/pull/60270#issuecomment-3729119122): > ...instead of running git fetch by the task, we could add extra command to the executin API - so that task can ask supervisor to fetch the commit. This could solve all the problems and only require read permission to the bare repository in order to checkout the commit by task after it's been fetched by the supervisor. That has another advantage as well, we could potentially serialize those requests or batch them in supervisor and tha would avoid another quite likely potential issue where multiple tasks of the same dag run are running in parallel (which will pretty much always happen if we have mapped tasks) and asking to fetch the repository at the same time - this will work but it is not optimal - potentially many git conections opened, rate limiting might start playing a role etc. So leaving all the git fetch operations to the supervisor seems like a good idea. > > That however I think will require a bit more coupling between the bundle interface and execution api - because this is not exclusively a problem of GitBundle only - other bundles might have very similar problems and we should solve it in a generic way so that there is a generic task -> supervisor ("get bundle version NNN") has to be added to the execution API. > > But I see that as the best long-term option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
