potiuk commented on code in PR #59: URL: https://github.com/apache/airflow-ci-infra/pull/59#discussion_r1738926738
########## README.rst: ########## @@ -17,3 +17,129 @@ CI Infrastructure for Apache Airflow ==================================== + +Apache Airflow uses Github Actions as its Continuous Integration (CI) tool. When Airflow contributors open a pull +request, Github Actions runs a series of tests on Apache organization runners. However, due to the limited resources +of these runners, the tests could take a significant amount of time, sometimes up to 2 hours. To improve efficiency +for active committers, a separate set of runners is available for them. These CI runs are executed on self-hosted +runners deployed on an Amazon Elastic Kubernetes Service (EKS) cluster. + +This repository contains the necessary scripts and configuration files for managing the CI infrastructure +of Apache Airflow. + +.. _aws_configuration: + +Access to the AWS Account +------------------------- +Access to the AWS account is primarily restricted to the PMC members. However, as a committer, you can request access +to the account if you believe it is necessary for your contributions. To request access, please inquire on the +`#internal-airflow-ci-cd`_ Slack channel. + +.. _#internal-airflow-ci-cd: https://apache-airflow.slack.com/archives/C015SLQF059 + + +Once you have access to the AWS account, you can use the `awscli`_ to interact with the account. To configure the CLI, +you need to follow the steps below: + +1. Get your `AWS access key ID`_ and secret access key from the AWS IAM console. + +2. Run `aws configure` and enter the credentials when prompted:: + + aws configure --profile airflow + + You will be asked to provide the access and secret keys, the region, and the output format. + For the region, you should choose `us-east-2`, and for the output format, you could skip it to use the default value. + +3. Update kubeconfig to access the EKS cluster:: + + AWS_PROFILE=airflow aws eks --region us-east-2 update-kubeconfig --name airflow --alias airflow + + This command will update the kubeconfig file located at `~/.kube/config` to include the EKS cluster. + You can verify that the cluster is added by running `kubectl get nodes`. + + +.. _awscli: https://aws.amazon.com/cli/ +.. _AWS access key ID: https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html + + +Airflow CI infrastructure architecture +-------------------------------------- +.. image:: Airflow_CI.png + :width: 100% + :align: center + :alt: Airflow CI infrastructure architecture + +Manage the infrastructure resources +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We use Terraform to manage the EKS cluster and other AWS resources, and Helm (with helmfile) to manage the +Kubernetes resources inside the cluster. + +For Terraform, you can find the detailed documentation here: `Terraform Documentation <terraform/README.rst>`_ Review Comment: For the future.... Should we switch to OpenToFU (following the Hashicorp drama) ? https://opentofu.org/ Would be a nice small project on it's own I guess for someone to try after we make it works.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
