Hello everyone, I worked intensively on the new operators and also made some improvements and reviews of PRs to existing ones. I noticed that sometimes a code which looks valid and passing all tests finally didn't work with the service. It could be small mistake which passed uncatched or some external service made with time some breaking changes to its API. The most reliable way to verify is operator really working is to run system test for it.
There is a great Breeze environment which allows to do this is easy way. However creating Google Cloud or AWS project, configuring it to work it with local Breeze environment may be time consuming and error prone. Running system tests also costs money. Due to this reasons system tests that are not being run frequently may become problematic to run. In my opinion all of that makes system tests little bit abandoned however I see a lot of potential benefits of running system tests in scheduled manner. The most important one is that it allow to catch problems with the operators before they will reported by the users. Long time ago Jarek created AIP-4 [1][2] which describes automation of system tests. I wanted to refresh it and due to this https://github.com/PolideaInternal/airflow-system-tests repository was published. It describes how to setup Google Cloud Platform project and consists of configuration which allows to run tests in Breeze within Google Cloud Build. Additional scripts allows to trigger bunch of system tests with one command (they could be triggered periodically by scheduler). It is not perfect, still need some work, but by using it I was able to run several tests related to GCP once a week and I am looking for a way to do it within community. I would appreciate hearing your opinion on this. Best regards Tobiasz https://github.com/PolideaInternal/airflow-system-tests [1] https://github.com/apache/airflow/issues/10835 [2] https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+Automation+of+System+Tests+for+external+systems
