victormacaubas commented on PR #73004: URL: https://github.com/apache/airflow/pull/73004#issuecomment-6777806383
We hit both of these right after upgrading from 2.11.2 to 3.3.2 (apache-airflow-providers-amazon 9.36.0, watchtower 3.4.0, Python 3.12, CeleryExecutor on ECS Fargate, CloudWatch remote logging). Neither leak happened on 2.11.2. Triggerer: memory went from ~47% to ~75% of a 1 GiB task after the upgrade and kept climbing at about 2 MiB/h. After ~29h the supervisor process had 209 threads, all idle. 193 of them were created after startup, spread evenly at ~6.7/h, which matches how often our triggers start and finish. Open file descriptors stayed flat at 31. Celery workers (worker_concurrency = 12): after ~32h, thread counts across the 12 pool processes ranged from 7 to 514. RssAnon grew linearly with thread count, ~327 kB per thread (194 MiB at 7 threads, 356 MiB at 514). That's ~1,660 leaked threads and ~530 MiB on a single worker, consistent with the numbers in https://github.com/apache/airflow/pull/73004#issuecomment-6577601794. For now we've turned off remote logging on the triggerer only. Once the task is done, the UI doesn't read the .trigger.* stream anyway (#70317), so we don't lose much. For workers we're planning to set worker_max_tasks_per_child until a fix ships. +1 to this PR. Since it shows up on both the triggerer and the workers, it would be great to get it reviewed and into a provider release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
