mik-laj opened a new issue #11657: URL: https://github.com/apache/airflow/issues/11657
Hello, # Overview Currently, Helm Chart for Airflow has two types of tests: ([Learn the best practices of Helm Chart testing](https://blog.gruntwork.io/automated-testing-for-kubernetes-and-helm-charts-using-terratest-a4ddc4e67344)) - **Template testing (unit testing)**: these tests render the templates against various input values. These types of tests let we verify that the template rendered the expected resources in the manner you intended. These tests are fast to execute and can catch syntactic errors of your template - **Integration testing**: these tests take the rendered templates and deploy them to a real Kubernetes cluster. We can then verify the deployed infrastructure works as intended by hitting the endpoints or querying Kubernetes for the resources. These tests closely resemble an actual deployment and give us a close approximation of how it might behave when we are ready to push the chart to production. In each case, we use a different framework: - For template testing, we use [Helm-unittest](https://github.com/lrills/helm-unittest) - For integration testing, we use bash scripts and Pytest + kind cluster # Start reading here Today I would like to talk about template testing. In my opinion, using helm-unittest is not a very good solution. - This framework is not very popular, so any contributor who wants to make a change must read the documentation first to add new tests. (People are lazy by nature, so they just don't add these tests). - Another problem is the form - the YAML file. The YAML file is not very flexible and makes it very difficult to write more unusual tests. It would be more flexible to express the tests in a more normal programming language. During the discussion with @dimberman , I prepared a small POC that shows how we can test the Helm Chart with Pytest + Python. Here is link: https://github.com/apache/airflow/pull/11533#discussion_r504998933 ```python import subprocess import unittest from tempfile import NamedTemporaryFile import yaml from typing import Dict, Optional OBJECT_COUNT_IN_BASIC_DEPLOYMENT = 22 class TestBaseChartTest(unittest.TestCase): def render_chart(self, name, values: Optional[Dict] = None): values = values or {} with NamedTemporaryFile() as tmp_file: content = yaml.dump(values) tmp_file.write(content.encode()) print(content) tmp_file.flush() templates = subprocess.check_output(["helm", "template", name, "..", '--values', tmp_file.name]) k8s_objects = yaml.load_all(templates) k8s_objects = [k8s_object for k8s_object in k8s_objects if k8s_object] import pprint pprint.pprint(k8s_objects) return k8s_objects def test_basic_deployments(self): k8s_objects = self.render_chart("TEST-BASIC", {"chart": {'metadata': 'AA'}}) list_of_kind_names_tuples = [ (k8s_object['kind'], k8s_object['metadata']['name']) for k8s_object in k8s_objects ] self.assertEqual( list_of_kind_names_tuples, [ ('ServiceAccount', 'TEST-BASIC-scheduler'), ('ServiceAccount', 'TEST-BASIC-webserver'), ('ServiceAccount', 'TEST-BASIC-worker'), ('Secret', 'TEST-BASIC-postgresql'), ('Secret', 'TEST-BASIC-airflow-metadata'), ('Secret', 'TEST-BASIC-airflow-result-backend'), ('ConfigMap', 'TEST-BASIC-airflow-config'), ('ClusterRole', 'TEST-BASIC-pod-launcher-role'), ('ClusterRoleBinding', 'TEST-BASIC-pod-launcher-rolebinding'), ('Service', 'TEST-BASIC-postgresql-headless'), ('Service', 'TEST-BASIC-postgresql'), ('Service', 'TEST-BASIC-statsd'), ('Service', 'TEST-BASIC-webserver'), ('Deployment', 'TEST-BASIC-scheduler'), ('Deployment', 'TEST-BASIC-statsd'), ('Deployment', 'TEST-BASIC-webserver'), ('StatefulSet', 'TEST-BASIC-postgresql'), ('Secret', 'TEST-BASIC-fernet-key'), ('Secret', 'TEST-BASIC-redis-password'), ('Secret', 'TEST-BASIC-broker-url'), ('Job', 'TEST-BASIC-create-user'), ('Job', 'TEST-BASIC-run-airflow-migrations') ] ) self.assertEqual(OBJECT_COUNT_IN_BASIC_DEPLOYMENT, len(k8s_objects)) def test_basic_deployment_without_default_users(self): k8s_objects = self.render_chart("TEST-BASIC", {"webserver": {'defaultUser': {'enabled': False}}}) list_of_kind_names_tuples = [ (k8s_object['kind'], k8s_object['metadata']['name']) for k8s_object in k8s_objects ] self.assertNotIn(('Job', 'TEST-BASIC-create-user'), list_of_kind_names_tuples) ``` Now there is a question for the community. What do you think about it? Should we migrate all tests to Pytestt? Is this the only reason contributors don't add unit tests? What else can we do to encourage contributors to write tests? Best regards, Kamil BreguĊa ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
