alkismavridis commented on issue #56641: URL: https://github.com/apache/airflow/issues/56641#issuecomment-3479139319
Hi. Plot twist here 🫤 After running it for couple of days, seems like memory keeps increasing - it just does so slower. Screenshot is 7 days of data (with reduced parallelism to 16): <img width="1487" height="302" alt="Image" src="https://github.com/user-attachments/assets/f2710fbd-3afa-4741-aeb7-29d06fc2b18f"; /> smem: ``` $ smem -k PID User Command Swap USS PSS RSS 1 default /usr/bin/dumb-init -- /entr 0 92.0K 120.0K 1.2M 117071 default bash 0 572.0K 809.0K 3.6M 24603 default bash 0 568.0K 812.0K 3.5M 46027 default bash 0 568.0K 812.0K 3.7M 43790 default bash 0 572.0K 828.0K 3.6M 49584 default bash 0 580.0K 835.0K 3.6M 1712 default bash 0 608.0K 846.0K 3.8M 117078 default /usr/bin/python3 /usr/bin/s 0 13.5M 13.6M 16.0M 36073 default airflow worker -- 019a375e- 0 44.4M 75.2M 186.6M 2337 default airflow worker -- LocalExec 0 56.2M 86.8M 199.8M 1628 default airflow worker -- LocalExec 0 38.2M 90.0M 202.2M 32 default airflow worker -- LocalExec 0 51.1M 90.2M 189.3M 26 default /usr/python/bin/python3.12 0 83.8M 90.4M 149.3M 31 default airflow worker -- LocalExec 0 52.9M 91.9M 191.2M 731 default airflow worker -- LocalExec 0 54.7M 93.5M 193.2M 732 default airflow worker -- LocalExec 0 54.9M 93.7M 193.9M 2363 default airflow worker -- LocalExec 0 68.8M 94.4M 201.0M 114189 default airflow worker -- 019a47df- 0 43.0M 95.2M 207.6M 1760 default airflow worker -- LocalExec 0 63.7M 99.7M 197.9M 1765 default airflow worker -- LocalExec 0 63.7M 99.7M 198.3M 25 default airflow serve-logs 0 97.6M 104.3M 165.4M 2338 default airflow worker -- LocalExec 0 104.5M 112.8M 185.5M 1987 default airflow worker -- LocalExec 0 130.6M 134.2M 199.8M 2112 default airflow worker -- LocalExec 0 132.5M 136.1M 198.9M 1823 default airflow worker -- LocalExec 0 134.8M 138.1M 198.5M 2778 default airflow worker -- LocalExec 0 137.7M 140.8M 200.4M 1812 default airflow worker -- LocalExec 0 138.4M 141.7M 203.1M 7 default /usr/python/bin/python3.12 0 139.1M 143.9M 203.8M 2679 default airflow worker -- LocalExec 0 143.1M 145.7M 197.3M ``` As you see, the USS of some workers has reached 140MB levels. I am still not sure if there is a maximum and what that would be. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
