mtraynham edited a comment on issue #21026: URL: https://github.com/apache/airflow/issues/21026#issuecomment-1054673312
@potiuk I've tested Airflow 2.2.4 and still see this issue with the recommended healthcheck.test, https://github.com/apache/airflow/blob/958860fcd7c9ecdf60b7ebeef4397b348835c8db/docs/apache-airflow/start/docker-compose.yaml I've taken the `docker-compose.yml` file from above and replaced references of `2.2.3` with `2.2.4`. Also using the included script to collect service stats, from above, I've found the following: | Date | Container | CPU Percent | Mem Usage | Mem Percent | |------------------------|-----------------------------------------------------------|-------------|---------------------|-------------| | 2022-02-28T16:59:44UTC | airflow_worker_no_healthcheck.1.fyihqxhf8hb574ivnz7uflpbz | 2.08% | 1.123GiB / 14.91GiB | 7.53% | | 2022-02-28T16:59:44UTC | airflow_worker_healthcheck.1.rq48arboysa4b8fqn6i5jb7ad | 0.43% | 1.139GiB / 14.91GiB | 7.64% | | 2022-02-28T21:00:41UTC | airflow_worker_no_healthcheck.1.fyihqxhf8hb574ivnz7uflpbz | 0.75% | 1.123GiB / 14.91GiB | 7.54% | | 2022-02-28T21:00:41UTC | airflow_worker_healthcheck.1.rq48arboysa4b8fqn6i5jb7ad | 88.68% | 1.319GiB / 14.91GiB | 8.85% | Over a period of 4 hours, memory on the container with the health check has increased ~180 MB in usage, or about 45 MB per hour, which is what I observed before. Over the course of say 3 days that eventually becomes ~3.24 GB of unclaimed memory. For reference, the test environment is: Airflow 2.2.4 - Celery Executor 1 Webserver, 1 Scheduler, 2 Workers (1 with health check) Ubuntu 20.04.3 LTS Linux host1 5.4.0-88-generic 99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Docker version 20.10.12, build e91ed57 I've included the dataset as CSV. Every now and then memory spikes and some is reclaimed, but it still keeps trickling up unlike the service without a healthcheck. [stats_airflow.csv](https://github.com/apache/airflow/files/8156896/stats_airflow.csv) Is there potentially a different health check the worker service could use instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
