potiuk commented on PR #28207: URL: https://github.com/apache/airflow/pull/28207#issuecomment-1341822398
This one is based on #28209 (so #28209 should be merged first) - and it should deal with the memory issues we had in public runners (it should also speed up our tests): 1) Unit tests now do not have "Integration" as test type - so regular "matrix" tests do not run integration tests. Much less memory used for those jobs in general. 2) We have a new command in breeze `breeze testing integration-tests` that allows to run integration tests very easyily - for example `breeze testing integration-tests --integration mongo` will run mongo db tests, `breeze testing integration-tests --integration all` will run all integration tests 3) I have removed the old openldap integration (it turned out it is not used any more) 4) We are using the breeze command in CI in a separate job to run integration tests 5) In Self-hosted runners we run all integration tests for Postgres at once with all integrations enabled (because memory!) 5) In Self-hosted runners we also additionaly run all tests for MySQL 7) In Public runners we run the integration tests sequentially (cassandra, mongo, pinot, celery, (trino + kerberos) and we run `breeze stop` between each integration test type - this way we only run one/two docker-compose integrations at a time and we stop/reclaim memory after each integration test type -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
