stefankeidel commented on PR #46237: URL: https://github.com/apache/airflow/pull/46237#issuecomment-2623567271
> Hi, looks good! > > I would love to understand why it makes a difference if it runs every 15 minutes or every hour (for example)? > > How does this affect the worker performance? I personally wouldn't think it'd affect worker performance at all, since all they do is write to that file system. If anything it could see it improving because the volumes are not busy listing files all the time. But then again, I'm not super familiar with all the different types of setups Airflow can be run in. In our case, extensive pruning on that shared volume was a) a major performance hog - we're dealing with Azure Blob Storage as backend, which is fairly flow to do a recursive `find` looking for last modified times and b) a major cost factor (`ListBlob` transactions get pricey). And our Airflow setup is not that large, maybe ~100 DAGs or so. I broke one of the helm chart unit tests in the process here it seems. Will be attempting to deal with that this morning, but I need a working local setup first :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
