potiuk commented on PR #63: URL: https://github.com/apache/airflow-ci-infra/pull/63#issuecomment-1827650095
> > IMHO it should be ok to reduce it to 60% (31.2Gb), but let's start with 10% of extra memory for tests, then we can check if we need more memory > > See above - K8S tests take way more of docker filesystem (this is also the reason why we cannot run more than 3 of them in parallel). I also am not sure if that changes anything (it's more of an exercise here to show what we have) - I believe `tmpfs` will only take as memory as it actually uses, so I am not sure if we are gaining much by decreasing the limits :) BTW. This actually made me think that possibly we could see if we could have the k8s tests themselves optimized. We are using kind and deploying the clusters in their "default" configuration, which is - I guess - kind of not really "development" setting but something in-between development and production (usually those come with default settings that are suitable for low-production setting as well) . But our tests are not very demanding (except setting up the cluster and deploying Airflow we merely run a few pods and several DAGs). So maybe we could somehow optimize the settings of the kind clusters we have to take less of `/var/lib/docker`. Nothing really immediate but if we do - we could possibly run more of them in parallel. I've done similar exercise in the past to decrease memory used by Postgres/MySQL (and I was able to save 50% of memory they used by using low settings that were enough for our tests). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
