jedcunningham commented on issue #27476: URL: https://github.com/apache/airflow/issues/27476#issuecomment-1310888233
> Have you ever had a case or seen a single user complaining about "efficiency" of their git server? Yes, though not directly "efficiency" - GitHub does rate limiting. And yes, a local mirror is an excellent solution to that. > HTTP request per 10 seconds by default and you can make the period longer. We default to 60s (which is a mistake imo, PR incoming!). You can't go beyond "0" without some risk of task failures from being out of sync. Acceptable for some, not others. Going to a high interval, say hourly, is really asking for trouble. That's what I mean when I say, unless you are on a single LocalExecutor, you really don't have a safe way to turn down the sync frequency naively (e.g. without a mirror, and even then, Airflow needs to poll the mirror very frequently and the mirror sync interval turned down). Ultimately, Airflow expects the DAGs to be consistent across the environment. All of these different options introduce risk that that won't be the case. There are tradeoffs all over the place here. > But when you already use Git to store your files, and you want to use GitSync IMHO adding persistency only make things worse, and never better IMHO. Generally, yes, but I'm not ready to say "never". There are definitely places where NFS is better supported/funded by multiple orders of magnitude over the git service - git just may not be built to be hit real time from everywhere across the org. Letting NFS take the brunt of it and syncing from git infrequently _can_ make sense. Don't get me wrong though, I'm not going to die on this hill. I'm more of a "bake it into the image" + KubernetesExecutor guy anyways 🤷♂️. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
