aeroyorch commented on PR #69774: URL: https://github.com/apache/airflow/pull/69774#issuecomment-5278456938
> Do you mind giving more context as to why we are doing this. If you use case is "LRU evicts my hot dags too aggressively", tweaking the settings for your use case is probably better. > > It looks like we are introducing back the memory leak this bounded cache was fixing in the first place. I don't think we should allow unbouded cache there. Hi Pierre, sure. I don't think this brings back the leak. Today `cache_size=0` is a plain `dict` with no eviction at all. With this change, `cache_size=0` + `cache_ttl>0` creates a real `TTLCache`. That evicts exactly what grows and keeps the hot ones. Defaults don't change. Tuning the size doesn't really help here: the scheduler cycles through all active `dag_version_id`s, so any cap below the active set evicts each key just before it's needed again. @kaxil raised this in the #69007 review and @potiuk asked for TTL eviction without a size cap in `DBDagBag` as a first step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
