mik-laj commented on a change in pull request #16571: URL: https://github.com/apache/airflow/pull/16571#discussion_r659253383
########## File path: docs/apache-airflow-providers-amazon/operators/eks.rst ########## @@ -0,0 +1,265 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Amazon Elastic Kubernetes Service (EKS) Operators +================================================= + +`Amazon Elastic Kubernetes Service (Amazon EKS) <https://aws.amazon.com/eks/>`__ is a managed service +that makes it easy for you to run Kubernetes on AWS without needing to stand up or maintain your own +Kubernetes control plane. Kubernetes is an open-source system for automating the deployment, scaling, +and management of containerized applications. + +.. contents:: + :depth: 1 + :local: + +Prerequisite Tasks +------------------ + +.. include:: _partials/prerequisite_tasks.rst + +Overview +-------- + +Airflow to Amazon Elastic Kubernetes Service (EKS) integration provides Operators to create and +interact with the EKS clusters and compute infrastructure. + + - :class:`~airflow.providers.amazon.aws.operators.eks` + +4 example_dags are provided which showcase these operators in action. + + - example_eks_create_cluster.py + - example_eks_create_cluster_with_nodegroup.py + - example_eks_create_nodegroup.py + - example_eks_pod_operator.py Review comment: ```suggestion - ``example_eks_create_cluster.py`` - ``example_eks_create_cluster_with_nodegroup.py`` - ``example_eks_create_nodegroup.py`` - ``example_eks_pod_operator.py`` ``` To avoid spelling check errors We do not try to teach specific examples in these guides, so we do not have to focus on them. On the other hand, when a user is interested, each example has a link that takes it to the source code. ########## File path: airflow/providers/amazon/aws/operators/eks.py ########## @@ -0,0 +1,737 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review comment: All fields created in ctor. ########## File path: airflow/providers/amazon/aws/operators/eks.py ########## @@ -0,0 +1,737 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review comment: All fields created in ctor. We should opt-out approach use. ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") Review comment: How to pass the credentials to AWS CLI when we have Airflow installed at alternative cloud provider or on-permise? In the case of ``GKEStartPodOperator``, we use [the ``GoogleBaseHook.provide_authorized_gcloud`` method](https://github.com/apache/airflow/blob/2625007c8aeca9ed98dea361ba13c2622482d71f/airflow/providers/google/common/hooks/base_google.py#L483) to pass credential from Airflow to Cloud SDK. ########## File path: airflow/providers/amazon/aws/operators/eks.py ########## @@ -0,0 +1,737 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# pylint: disable=invalid-name +"""This module contains Amazon EKS operators.""" +import json +import os +from datetime import datetime +from time import sleep +from typing import Dict, List, Optional + +from boto3 import Session + +from airflow.models import BaseOperator +from airflow.providers.amazon.aws.hooks.eks import DEFAULT_PAGINATION_TOKEN, DEFAULT_RESULTS_PER_PAGE, EKSHook +from airflow.providers.amazon.aws.utils.eks_kube_config import ( + DEFAULT_CONTEXT_NAME, + DEFAULT_KUBE_CONFIG_PATH, + DEFAULT_NAMESPACE_NAME, + DEFAULT_POD_USERNAME, + generate_config_file, +) +from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator + +CHECK_INTERVAL_SECONDS = 15 +TIMEOUT_SECONDS = 25 * 60 +CONN_ID = "eks" +REGION = Session().region_name +DEFAULT_COMPUTE_TYPE = 'nodegroup' +DEFAULT_NODEGROUP_NAME_SUFFIX = '-nodegroup' +DEFAULT_POD_NAME = 'pod' +KUBE_CONFIG_ENV_VAR = 'KUBECONFIG' + + +class EKSCreateClusterOperator(BaseOperator): + """ + Creates an Amazon EKS Cluster control plane. + + Optionally, can also create the supporting compute architecture: + If argument 'compute' is provided with a value of 'nodegroup', will also attempt to create an Amazon + EKS Managed Nodegroup for the cluster. See EKSCreateNodegroupOperator documentation for requirements. + + .. seealso:: + For more information on how to use this operator, take a look at the guide: + :ref:`howto/operator:EKSCreateClusterOperator` + + :param cluster_name: The unique name to give to your Amazon EKS Cluster. + :type cluster_name: str + :param cluster_role_arn: The Amazon Resource Name (ARN) of the IAM role that provides permissions for the + Kubernetes control plane to make calls to AWS API operations on your behalf. + :type cluster_role_arn: str + :param resources_vpc_config: The VPC configuration used by the cluster control plane. + :type resources_vpc_config: Dict + :param compute: The type of compute architecture to generate along with the cluster. + Defaults to 'nodegroup' to generate an EKS Managed Nodegroup. + :type compute: str + :param aws_conn_id: The Airflow connection used for AWS credentials. + If this is None or empty then the default boto3 behaviour is used. If + running Airflow in a distributed manner and aws_conn_id is None or + empty, then the default boto3 configuration would be used (and must be + maintained on each worker node). + :type aws_conn_id: str + + If 'compute' is 'nodegroup', the following are required: + + :param nodegroup_name: The unique name to give your EKS Managed Nodegroup. + :type nodegroup_name: str + :param nodegroup_role_arn: The Amazon Resource Name (ARN) of the IAM role to associate + with the EKS Managed Nodegroup. + :type nodegroup_role_arn: str + + """ + + def __init__( + self, + cluster_name: str, + cluster_role_arn: str, + resources_vpc_config: Dict, + nodegroup_name: Optional[str] = None, + nodegroup_role_arn: Optional[str] = None, + compute: Optional[str] = DEFAULT_COMPUTE_TYPE, + conn_id: Optional[str] = CONN_ID, + region: Optional[str] = REGION, + **kwargs, + ) -> None: + super().__init__(**kwargs) + self.clusterName = cluster_name + self.clusterRoleArn = cluster_role_arn + self.resourcesVpcConfig = resources_vpc_config + self.compute = compute + self.conn_id = conn_id + self.region = region + + if self.compute == 'nodegroup': + self.nodegroupName = nodegroup_name or self.clusterName + DEFAULT_NODEGROUP_NAME_SUFFIX + if nodegroup_role_arn: + self.nodegroupRoleArn = nodegroup_role_arn + else: + message = "Creating an EKS Managed Nodegroup requires nodegroup_role_arn to be passed in." + self.log.error(message) + raise AttributeError(message) + + def execute(self, context): + eks_hook = EKSHook( + aws_conn_id=self.conn_id, + region_name=self.region, + ) + + eks_hook.create_cluster( + name=self.clusterName, + roleArn=self.clusterRoleArn, + resourcesVpcConfig=self.resourcesVpcConfig, + ) + + if self.compute is not None: + self.log.info("Waiting for EKS Cluster to provision. This will take some time.") + + countdown = TIMEOUT_SECONDS + while eks_hook.get_cluster_state(clusterName=self.clusterName) != "ACTIVE": + if countdown >= CHECK_INTERVAL_SECONDS: + countdown -= CHECK_INTERVAL_SECONDS + self.log.info( + "Waiting for cluster to start. Checking again in %d seconds", CHECK_INTERVAL_SECONDS + ) + sleep(CHECK_INTERVAL_SECONDS) + else: + message = "Cluster is still inactive after the allocated time limit. Aborting." + self.log.error(message) + raise RuntimeError(message) + + if self.compute == 'nodegroup': + eks_hook.create_nodegroup( + clusterName=self.clusterName, + nodegroupName=self.nodegroupName, + subnets=self.resourcesVpcConfig.get('subnetIds'), + nodeRole=self.nodegroupRoleArn, + ) + + +class EKSCreateNodegroupOperator(BaseOperator): + """ + Creates am Amazon EKS Managed Nodegroup for an existing Amazon EKS Cluster. + + .. seealso:: + For more information on how to use this operator, take a look at the guide: + :ref:`howto/operator:EKSCreateNodegroupOperator` + + :param cluster_name: The name of the Amazon EKS Cluster to create the managed nodegroup in. + :type cluster_name: str + :param nodegroup_name: The unique name to give your managed nodegroup. + :type nodegroup_name: str + :param nodegroup_subnets: + The subnets to use for the Auto Scaling group that is created for the managed nodegroup. + :type nodegroup_subnets: List[str] + :param nodegroup_role_arn: + The Amazon Resource Name (ARN) of the IAM role to associate with the managed nodegroup. + :type nodegroup_role_arn: str + :param aws_conn_id: The Airflow connection used for AWS credentials. + If this is None or empty then the default boto3 behaviour is used. If + running Airflow in a distributed manner and aws_conn_id is None or + empty, then the default boto3 configuration would be used (and must be + maintained on each worker node). + :type aws_conn_id: str + + """ + + def __init__( + self, + cluster_name: str, + nodegroup_subnets: List[str], + nodegroup_role_arn: str, + nodegroup_name: Optional[str], + conn_id: Optional[str] = CONN_ID, + region: Optional[str] = REGION, + **kwargs, + ) -> None: + super().__init__(**kwargs) + self.clusterName = cluster_name + self.nodegroupSubnets = nodegroup_subnets + self.nodegroupRoleArn = nodegroup_role_arn + self.nodegroupName = nodegroup_name or cluster_name + datetime.now().strftime("%Y%m%d_%H%M%S") + self.conn_id = conn_id + self.region = region + + def execute(self, context): + eks_hook = EKSHook( + aws_conn_id=self.conn_id, + region_name=self.region, + ) + + return eks_hook.create_nodegroup( + clusterName=self.clusterName, + nodegroupName=self.nodegroupName, + subnets=self.nodegroupSubnets, + nodeRole=self.nodegroupRoleArn, + ) + + +class EKSDeleteClusterOperator(BaseOperator): + """ + Deletes the Amazon EKS Cluster control plane and all nodegroups attached to it. + + .. seealso:: + For more information on how to use this operator, take a look at the guide: + :ref:`howto/operator:EKSDeleteClusterOperator` + + :param cluster_name: The name of the Amazon EKS Cluster to delete. + :type cluster_name: str + :param aws_conn_id: The Airflow connection used for AWS credentials. + If this is None or empty then the default boto3 behaviour is used. If + running Airflow in a distributed manner and aws_conn_id is None or + empty, then the default boto3 configuration would be used (and must be + maintained on each worker node). + :type aws_conn_id: str + + """ + + def __init__( + self, cluster_name: str, conn_id: Optional[str] = CONN_ID, region: Optional[str] = REGION, **kwargs + ) -> None: + super().__init__(**kwargs) + self.clusterName = cluster_name + self.conn_id = conn_id + self.region = region + + def execute(self, context): + eks_hook = EKSHook( + aws_conn_id=self.conn_id, + region_name=self.region, + ) + + nodegroups = eks_hook.list_nodegroups(clusterName=self.clusterName).get('nodegroups') + nodegroup_count = len(nodegroups) + if nodegroup_count > 0: + self.log.info( + "A cluster can not be deleted with attached nodegroups. Deleting %d nodegroups.", + nodegroup_count, + ) + for group in nodegroups: + eks_hook.delete_nodegroup(clusterName=self.clusterName, nodegroupName=group) + + # Scaling up the timeout based on the number of nodegroups that are being processed. + additional_seconds = 5 * 60 + countdown = TIMEOUT_SECONDS + (nodegroup_count * additional_seconds) + while len(eks_hook.list_nodegroups(clusterName=self.clusterName).get('nodegroups')) > 0: + if countdown >= CHECK_INTERVAL_SECONDS: + countdown -= CHECK_INTERVAL_SECONDS + sleep(CHECK_INTERVAL_SECONDS) + self.log.info( + "Waiting for the remaining %s nodegroups to delete. Checking again in %d seconds.", + nodegroup_count, + CHECK_INTERVAL_SECONDS, + ) + else: + message = "Nodegroups are still inactive after the allocated time limit. Aborting." + self.log.error(message) + raise RuntimeError(message) + + self.log.info("No nodegroups remain, deleting cluster.") + return eks_hook.delete_cluster(name=self.clusterName) + + +class EKSDeleteNodegroupOperator(BaseOperator): + """ + Deletes an Amazon EKS Nodegroup from an Amazon EKS Cluster. + + .. seealso:: + For more information on how to use this operator, take a look at the guide: + :ref:`howto/operator:EKSDeleteNodegroupOperator` + + :param cluster_name: The name of the Amazon EKS Cluster that is associated with your nodegroup. + :type cluster_name: str + :param nodegroup_name: The name of the nodegroup to delete. + :type nodegroup_name: str + :param aws_conn_id: The Airflow connection used for AWS credentials. + If this is None or empty then the default boto3 behaviour is used. If + running Airflow in a distributed manner and aws_conn_id is None or + empty, then the default boto3 configuration would be used (and must be + maintained on each worker node). + :type aws_conn_id: str + + """ + + def __init__( + self, + cluster_name: str, + nodegroup_name: str, + conn_id: Optional[str] = CONN_ID, + region: Optional[str] = REGION, + **kwargs, + ) -> None: + super().__init__(**kwargs) + self.clusterName = cluster_name + self.nodegroupName = nodegroup_name + self.conn_id = conn_id + self.region = region + + def execute(self, context): + eks_hook = EKSHook( + aws_conn_id=self.conn_id, + region_name=self.region, + ) + + return eks_hook.delete_nodegroup(clusterName=self.clusterName, nodegroupName=self.nodegroupName) + + +class EKSDescribeAllClustersOperator(BaseOperator): + """ + Describes all Amazon EKS Clusters in your AWS account. + + :param max_results: The maximum number of results to return. + :type max_results: int + :param next_token: The nextToken value returned from a previous paginated execution. Review comment: When defining a DAG, the user should define all tasks. Here, however, there is a case where the number of tasks is not known until the first page is downloaded, and therefore cannot be defined in Airflow. In the case of Google, we've always fetched all items from all pages. This way, the user could access all elements and did not have to define dynamic DAGs. ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") + if installed is None: + message = ( + "AWS CLI version 2 must be installed on the worker. See: " + "https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html"; + ) + print(message) + raise UnmetDependency(message) + + # Set up the client + session = boto3.Session(region_name=aws_region, profile_name=aws_profile) + eks_client = session.client("eks") + + # get cluster details + cluster = eks_client.describe_cluster(name=eks_cluster_name) + cluster_cert = cluster["cluster"]["certificateAuthority"]["data"] + cluster_ep = cluster["cluster"]["endpoint"] + + # build the cluster config hash + cli_args = [ + "--region", + aws_region, + "eks", + "get-token", + "--cluster-name", + eks_cluster_name, + ] + if role_arn: + cli_args.extend(["--role-arn", role_arn]) Review comment: > the lesser evil was assuming that someone using an AWS service would be able to install an AWS tool. For me, this is very problematic for three reasons: - Passing credentials from Airflow to AWS CLI. - AWS CLI is a native library because it cannot install the latest version via PIP, so in many cases it will be very difficult to use. - Official Docker image doesn't have preinstalled AWS CLI. ########## File path: docs/apache-airflow-providers-amazon/operators/eks.rst ########## @@ -0,0 +1,265 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Amazon Elastic Kubernetes Service (EKS) Operators +================================================= + +`Amazon Elastic Kubernetes Service (Amazon EKS) <https://aws.amazon.com/eks/>`__ is a managed service +that makes it easy for you to run Kubernetes on AWS without needing to stand up or maintain your own +Kubernetes control plane. Kubernetes is an open-source system for automating the deployment, scaling, +and management of containerized applications. + +.. contents:: + :depth: 1 + :local: + +Prerequisite Tasks +------------------ + +.. include:: _partials/prerequisite_tasks.rst + +Overview +-------- + +Airflow to Amazon Elastic Kubernetes Service (EKS) integration provides Operators to create and +interact with the EKS clusters and compute infrastructure. + + - :class:`~airflow.providers.amazon.aws.operators.eks` + +4 example_dags are provided which showcase these operators in action. + + - example_eks_create_cluster.py + - example_eks_create_cluster_with_nodegroup.py + - example_eks_create_nodegroup.py + - example_eks_pod_operator.py Review comment: Yes. It ignores any code literal or code block. ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") Review comment: This is important because you probably used the default credentials from the CLI in Breeze, as you were logged in there before. In a production environment, these credentials will be provided by Airflow e.g. via the Secret backend, So AWS CLI must be able to share the same credentials. Some Airflow users have a lot of AWS credentials for different accounts and we should be able to use them correctly in this operator as well. The user should not be aware in any way that a CLI is being used to get the token. ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") + if installed is None: + message = ( + "AWS CLI version 2 must be installed on the worker. See: " + "https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html"; + ) + print(message) + raise UnmetDependency(message) + + # Set up the client + session = boto3.Session(region_name=aws_region, profile_name=aws_profile) + eks_client = session.client("eks") + + # get cluster details + cluster = eks_client.describe_cluster(name=eks_cluster_name) + cluster_cert = cluster["cluster"]["certificateAuthority"]["data"] + cluster_ep = cluster["cluster"]["endpoint"] + + # build the cluster config hash + cli_args = [ + "--region", + aws_region, + "eks", + "get-token", + "--cluster-name", + eks_cluster_name, + ] + if role_arn: + cli_args.extend(["--role-arn", role_arn]) Review comment: For more inspiration, you can also look at [aws-cli/awscli/customizations/eks/get_token.py](https://github.com/aws/aws-cli/blob/develop/awscli/customizations/eks/get_token.py), [kubergrunt eks token](https://github.com/gruntwork-io/kubergrunt), [kubernetes-sigs/aws-iam-authenticator](https://github.com/kubernetes-sigs/aws-iam-authenticator) ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") + if installed is None: + message = ( + "AWS CLI version 2 must be installed on the worker. See: " + "https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html"; + ) + print(message) + raise UnmetDependency(message) + + # Set up the client + session = boto3.Session(region_name=aws_region, profile_name=aws_profile) + eks_client = session.client("eks") + + # get cluster details + cluster = eks_client.describe_cluster(name=eks_cluster_name) + cluster_cert = cluster["cluster"]["certificateAuthority"]["data"] + cluster_ep = cluster["cluster"]["endpoint"] + + # build the cluster config hash + cli_args = [ + "--region", + aws_region, + "eks", + "get-token", + "--cluster-name", + eks_cluster_name, + ] + if role_arn: + cli_args.extend(["--role-arn", role_arn]) Review comment: No. Gcloud is not preinstalled. We have a guide that explains how to add `gcloud` to a Docker image. See: http://airflow.apache.org/docs/docker-stack/recipes.html#google-cloud-sdk-installation ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") + if installed is None: + message = ( + "AWS CLI version 2 must be installed on the worker. See: " + "https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html"; + ) + print(message) + raise UnmetDependency(message) + + # Set up the client + session = boto3.Session(region_name=aws_region, profile_name=aws_profile) + eks_client = session.client("eks") + + # get cluster details + cluster = eks_client.describe_cluster(name=eks_cluster_name) + cluster_cert = cluster["cluster"]["certificateAuthority"]["data"] + cluster_ep = cluster["cluster"]["endpoint"] + + # build the cluster config hash + cli_args = [ + "--region", + aws_region, + "eks", + "get-token", + "--cluster-name", + eks_cluster_name, + ] + if role_arn: + cli_args.extend(["--role-arn", role_arn]) Review comment: The KubernetesEngine operator requires an update. It was added when I didn't know a better way to authenticate. it should use google access-token for authentication: https://github.com/apache/airflow/pull/16571#discussion_r659255470 If you want, you can keep a dependency on AWS CLI (though I'd prefer not), but you definitely should pass credentials from Airflow to AWS CLI and that's the main issue. Now, this operator cannot be used on Cloud Composer, GKE, Astronomer, or many other environments that do not have set up the default AWS credentials in AWS CLI. ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") + if installed is None: + message = ( + "AWS CLI version 2 must be installed on the worker. See: " + "https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html"; + ) + print(message) + raise UnmetDependency(message) + + # Set up the client + session = boto3.Session(region_name=aws_region, profile_name=aws_profile) + eks_client = session.client("eks") + + # get cluster details + cluster = eks_client.describe_cluster(name=eks_cluster_name) + cluster_cert = cluster["cluster"]["certificateAuthority"]["data"] + cluster_ep = cluster["cluster"]["endpoint"] + + # build the cluster config hash + cli_args = [ + "--region", + aws_region, + "eks", + "get-token", + "--cluster-name", + eks_cluster_name, + ] + if role_arn: + cli_args.extend(["--role-arn", role_arn]) Review comment: The best way to integrate is when the token generation code would also use the Airflow hook because then we would have the greatest certainty that all credentials were passed from Airflow correctly. ;-) ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") + if installed is None: + message = ( + "AWS CLI version 2 must be installed on the worker. See: " + "https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html"; + ) + print(message) + raise UnmetDependency(message) + + # Set up the client + session = boto3.Session(region_name=aws_region, profile_name=aws_profile) + eks_client = session.client("eks") + + # get cluster details + cluster = eks_client.describe_cluster(name=eks_cluster_name) + cluster_cert = cluster["cluster"]["certificateAuthority"]["data"] + cluster_ep = cluster["cluster"]["endpoint"] + + # build the cluster config hash + cli_args = [ + "--region", + aws_region, + "eks", + "get-token", + "--cluster-name", + eks_cluster_name, + ] + if role_arn: + cli_args.extend(["--role-arn", role_arn]) Review comment: > that is asking for a problem, later IMHO. I agree that this could bring unexpected problems in the future, but having to install AWS CLI by users and integrating Airflow with AWS CLI itself can be very difficult in edge cases too. We do not have a perfect solution here. The authorization method rarely changes compared to the product API, so I believe it will be enough stable approach. Especially since it is described in the AWS documentation. > Amazon EKS uses IAM to provide authentication to your Kubernetes cluster through [the AWS IAM authenticator for Kubernetes](https://github.com/kubernetes-sigs/aws-iam-authenticator). or > This can be used as an alternative to the aws-iam-authenticator. https://docs.aws.amazon.com/cli/latest/reference/eks/get-token.html In the aws-iam-authenticator documentation, we have the code snippet I cited above. https://github.com/kubernetes-sigs/aws-iam-authenticator#api-authorization-from-outside-a-cluster ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") + if installed is None: + message = ( + "AWS CLI version 2 must be installed on the worker. See: " + "https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html"; + ) + print(message) + raise UnmetDependency(message) + + # Set up the client + session = boto3.Session(region_name=aws_region, profile_name=aws_profile) + eks_client = session.client("eks") + + # get cluster details + cluster = eks_client.describe_cluster(name=eks_cluster_name) + cluster_cert = cluster["cluster"]["certificateAuthority"]["data"] + cluster_ep = cluster["cluster"]["endpoint"] + + # build the cluster config hash + cli_args = [ + "--region", + aws_region, + "eks", + "get-token", + "--cluster-name", + eks_cluster_name, + ] + if role_arn: + cli_args.extend(["--role-arn", role_arn]) Review comment: > that is asking for a problem, later IMHO. I agree that this could bring unexpected problems in the future, but having to install AWS CLI by users and integrating Airflow with AWS CLI itself can be very difficult in edge cases too. We do not have a perfect solution here. The authorization method rarely changes compared to the product API, so I believe it will be enough stable approach. Especially since it is described in the AWS documentation. > Amazon EKS uses IAM to provide authentication to your Kubernetes cluster through [the AWS IAM authenticator for Kubernetes](https://github.com/kubernetes-sigs/aws-iam-authenticator). https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html or > This can be used as an alternative to the aws-iam-authenticator. https://docs.aws.amazon.com/cli/latest/reference/eks/get-token.html In the aws-iam-authenticator documentation, we have the code snippet I cited above. https://github.com/kubernetes-sigs/aws-iam-authenticator#api-authorization-from-outside-a-cluster ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") + if installed is None: + message = ( + "AWS CLI version 2 must be installed on the worker. See: " + "https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html"; + ) + print(message) + raise UnmetDependency(message) + + # Set up the client + session = boto3.Session(region_name=aws_region, profile_name=aws_profile) + eks_client = session.client("eks") + + # get cluster details + cluster = eks_client.describe_cluster(name=eks_cluster_name) + cluster_cert = cluster["cluster"]["certificateAuthority"]["data"] + cluster_ep = cluster["cluster"]["endpoint"] + + # build the cluster config hash + cli_args = [ + "--region", + aws_region, + "eks", + "get-token", + "--cluster-name", + eks_cluster_name, + ] + if role_arn: + cli_args.extend(["--role-arn", role_arn]) Review comment: If you are concerned about maintaining this code, why not add it to AWS SDK for Python? It could have been helpful for other people as well. Have you thought to create an internal ticket on this topic? ########## File path: airflow/providers/amazon/aws/utils/eks_kube_config.py ########## @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import os +from shutil import which +from typing import Optional + +import boto3 +import yaml + +HOME = os.environ.get('HOME', '/tmp') +DEFAULT_KUBE_CONFIG_FILENAME = 'config' +DEFAULT_KUBE_CONFIG_PATH = str(os.path.join(HOME, '/.kube/', DEFAULT_KUBE_CONFIG_FILENAME)) +DEFAULT_CONTEXT_NAME = 'aws' +DEFAULT_NAMESPACE_NAME = 'default' +DEFAULT_POD_USERNAME = 'aws' + + +def generate_config_file( + eks_cluster_name: str, + eks_namespace_name: str, + aws_profile: Optional[str], + kube_config_file_location: Optional[str] = DEFAULT_KUBE_CONFIG_PATH, + pod_username: Optional[str] = DEFAULT_POD_USERNAME, + pod_context: Optional[str] = DEFAULT_CONTEXT_NAME, + role_arn: Optional[str] = None, + aws_region: Optional[str] = None, +) -> None: + """ + Writes the kubeconfig file given an EKS Cluster name, AWS region, and file path. + + :param eks_cluster_name: The name of the cluster to create the EKS Managed Nodegroup in. + :type eks_cluster_name: str + :param eks_namespace_name: The namespace to run within kubernetes. + :type eks_namespace_name: str + :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. + :type aws_profile: str + :param kube_config_file_location: Path to save the generated kube_config file to. + :type kube_config_file_location: str + :param pod_username: The username under which to execute the pod. + :type pod_username: str + :param pod_context: The name of the context access parameters to use. + :type pod_context: str + :param role_arn: The Amazon Resource Name (ARN) of the IAM role to associate with your nodegroup. + :type role_arn: str + :param aws_region: The name of the AWS Region the EKS Cluster resides in. + :type aws_region: str + """ + installed = which("aws") + if installed is None: + message = ( + "AWS CLI version 2 must be installed on the worker. See: " + "https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html"; + ) + print(message) + raise UnmetDependency(message) + + # Set up the client + session = boto3.Session(region_name=aws_region, profile_name=aws_profile) + eks_client = session.client("eks") + + # get cluster details + cluster = eks_client.describe_cluster(name=eks_cluster_name) + cluster_cert = cluster["cluster"]["certificateAuthority"]["data"] + cluster_ep = cluster["cluster"]["endpoint"] + + # build the cluster config hash + cli_args = [ + "--region", + aws_region, + "eks", + "get-token", + "--cluster-name", + eks_cluster_name, + ] + if role_arn: + cli_args.extend(["--role-arn", role_arn]) Review comment: If you are concerned about maintaining this code, why not add it to AWS SDK for Python? It could have been helpful for other people as well. Have you thought to create an internal ticket on this topic? We may not solve this problem now, but we may hope to improve it in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
