assapin commented on a change in pull request #11: URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r600206432
########## File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md ########## @@ -0,0 +1,160 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required bgit y applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Install Liminal in Airflow +* [Workflow](#workflow) +* [Prerequisites](#prerequisites) + * [Supported Distributions](#supported-distributions) +* [Get Started](#Get-Started) + * [Scripted Installation](#scripted-installation) + * [Manual Installation](#manual-installation) +* [References and other resources](#references-and-other-resources) + +## Workflow +Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS. +The process of liminal is emphasized on the following diagram: + + Review comment: we shouldn't use this as -is, it contains many issues which are not related to the topic; I can help you prepare a diagram with only the relevant parts ########## File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md ########## @@ -0,0 +1,160 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required bgit y applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Install Liminal in Airflow +* [Workflow](#workflow) +* [Prerequisites](#prerequisites) + * [Supported Distributions](#supported-distributions) +* [Get Started](#Get-Started) + * [Scripted Installation](#scripted-installation) + * [Manual Installation](#manual-installation) +* [References and other resources](#references-and-other-resources) + +## Workflow +Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS. +The process of liminal is emphasized on the following diagram: + + + +## Prerequisites +Before you begin, ensure you have met the following requirements: + +* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed +* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed +* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file +* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation] +* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc] +* Make sure that the [example repository][liminal-getting-started-project] is your workspace + +### Supported Distributions + +|Distribution | Versions | +|-|-| +|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 | +|[Airflow Helm Chart][airflowChart] | 7.14.3 | + +#### You may need to add the Repo + +```sh +helm repo add airflow-stable https://airflow-helm.github.io/charts +helm repo update +``` + +## Get Started + +### Scripted Installation + +[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow. +* Move to the [example repository project][liminal-getting-started-project] +* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it + + + +### Manual Installation + +Here are a couple of steps of how to install liminal in Airflow on Kubernetes: +* [Find Airflow components](#Find-Airflow-pod-names) +* [Install liminal in the components](#Install-liminal-in-Airflow-pods) +* [Setting up liminal](#Setting-up-liminal) + +#### Find Airflow pod names: + +```sh +echo -n "Please enter a namespace: " +read namespace +components=("web" "scheduler" "worker") +namespace=${namespace} +podNames=() +for component in ${components[@]} +do + podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n" + +done + +echo -e "The following Airflow pods are:\n${podNames}" +``` +#### Install liminal in Airflow pods: + +```sh +IFS=$'\n'|''; set -f; podNames=( ${podNames} ) +webPodName=${podNames[${#podNames[@]} - 1]} + +for podName in `echo -e ${podNames[@]}` +do + echo "Installing liminal in pod: ${podName}" + kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal' +done +``` + +#### Setting up liminal: Review comment: the title should be "Deploying Liminal Yamls into Airflow" ########## File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md ########## @@ -0,0 +1,160 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required bgit y applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Install Liminal in Airflow +* [Workflow](#workflow) +* [Prerequisites](#prerequisites) + * [Supported Distributions](#supported-distributions) +* [Get Started](#Get-Started) + * [Scripted Installation](#scripted-installation) + * [Manual Installation](#manual-installation) +* [References and other resources](#references-and-other-resources) + +## Workflow +Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS. +The process of liminal is emphasized on the following diagram: + + + +## Prerequisites +Before you begin, ensure you have met the following requirements: + +* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed +* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed +* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file +* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation] +* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc] +* Make sure that the [example repository][liminal-getting-started-project] is your workspace + +### Supported Distributions + +|Distribution | Versions | +|-|-| +|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 | +|[Airflow Helm Chart][airflowChart] | 7.14.3 | + +#### You may need to add the Repo + +```sh +helm repo add airflow-stable https://airflow-helm.github.io/charts +helm repo update +``` + +## Get Started + +### Scripted Installation + +[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow. +* Move to the [example repository project][liminal-getting-started-project] +* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it + + + +### Manual Installation + +Here are a couple of steps of how to install liminal in Airflow on Kubernetes: +* [Find Airflow components](#Find-Airflow-pod-names) +* [Install liminal in the components](#Install-liminal-in-Airflow-pods) +* [Setting up liminal](#Setting-up-liminal) + +#### Find Airflow pod names: + +```sh +echo -n "Please enter a namespace: " +read namespace +components=("web" "scheduler" "worker") +namespace=${namespace} +podNames=() +for component in ${components[@]} +do + podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n" + +done + +echo -e "The following Airflow pods are:\n${podNames}" +``` +#### Install liminal in Airflow pods: + +```sh +IFS=$'\n'|''; set -f; podNames=( ${podNames} ) +webPodName=${podNames[${#podNames[@]} - 1]} + +for podName in `echo -e ${podNames[@]}` +do + echo "Installing liminal in pod: ${podName}" + kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal' +done +``` + +#### Setting up liminal: +There are a couple of ways to deploy Yamls in Airflow. +1. Mount your Amazon EFS file system on a Linux instance: + ``` + sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /opt/airflow/dags + ``` +2. Use gitsync function in order to sync Yamls in the Amazon EFS file system. In this case, we chose to use a kubernetes job: Review comment: I am not sure it's necessary to get into this suggestion. Since it does essentially what we describe the deployment box does just using a git action... If you really want to contrast our proposal with a different approach, you can mention (without code) that some people deploy into airflow in pulls from git from inside airflow etc - but that's not our recommended way. ########## File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md ########## @@ -0,0 +1,122 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required bgit y applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Install Liminal in Airflow +* [Prerequisites](#prerequisites) + * [Supported Distributions](#supported-distributions) +* [Get Started](#Get-Started) + * [Scripted Installation](#scripted-installation) + * [Manual Installation](#manual-installation) +* [References and other resources](#references-and-other-resources) + +## Prerequisites +Before you begin, ensure you have met the following requirements: + +* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed +* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed +* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file +* You have Airflow on Kubernetes with AWS EFS +* You have already created [custom DAGs][custom-dag] following the liminal [Getting Started Documentation][liminalGetStarted-doc] +* Make sure that the [example repository][liminal-getting-started-project] is your workspace + +### Supported Distributions + +|Distribution | Versions | +|-|-| +|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 | +|[Airflow Helm Chart][airflowChart] | 7.14.3 | + +#### You may need to add the Repo + +```sh +helm repo add airflow-stable https://airflow-helm.github.io/charts +helm repo update +``` + +## Get Started + +### Scripted Installation + +[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow. +* Move to the [example repository project][liminal-getting-started-project] +* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it + + + +### Manual Installation + +Here are a couple of steps of how to install liminal in Airflow on Kubernetes: +* [Find Airflow components](#Find-Airflow-pod-names) +* [Install liminal in the components](#Install-liminal-in-Airflow-pods) +* [Setting up liminal](#Setting-up-liminal) + +#### Find Airflow pod names: + +```sh +echo -n "Please enter a namespace: " +read namespace +components=("web" "scheduler" "worker") +namespace=${namespace} +podNames=() +for component in ${components[@]} +do + podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n" + +done + +echo -e "The following Airflow pods are:\n${podNames}" +``` +#### Install liminal in Airflow pods: Review comment: so in practice, given that we assume airflow is already installed, this is the best practice to add a new package later on? ########## File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md ########## @@ -0,0 +1,160 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required bgit y applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Install Liminal in Airflow +* [Workflow](#workflow) +* [Prerequisites](#prerequisites) + * [Supported Distributions](#supported-distributions) +* [Get Started](#Get-Started) + * [Scripted Installation](#scripted-installation) + * [Manual Installation](#manual-installation) +* [References and other resources](#references-and-other-resources) + +## Workflow +Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS. Review comment: To deploy and run liminal on airflow, we need to complete two tasks: 1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server) 2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags) The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them. The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs. ########## File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md ########## @@ -0,0 +1,160 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required bgit y applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Install Liminal in Airflow +* [Workflow](#workflow) +* [Prerequisites](#prerequisites) + * [Supported Distributions](#supported-distributions) +* [Get Started](#Get-Started) + * [Scripted Installation](#scripted-installation) + * [Manual Installation](#manual-installation) +* [References and other resources](#references-and-other-resources) + +## Workflow +Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS. +The process of liminal is emphasized on the following diagram: + + + +## Prerequisites +Before you begin, ensure you have met the following requirements: + +* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed +* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed +* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file +* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation] +* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc] +* Make sure that the [example repository][liminal-getting-started-project] is your workspace + +### Supported Distributions + +|Distribution | Versions | +|-|-| +|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 | +|[Airflow Helm Chart][airflowChart] | 7.14.3 | + +#### You may need to add the Repo + +```sh +helm repo add airflow-stable https://airflow-helm.github.io/charts +helm repo update +``` + +## Get Started + +### Scripted Installation + +[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow. +* Move to the [example repository project][liminal-getting-started-project] +* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it + + + +### Manual Installation + +Here are a couple of steps of how to install liminal in Airflow on Kubernetes: +* [Find Airflow components](#Find-Airflow-pod-names) +* [Install liminal in the components](#Install-liminal-in-Airflow-pods) +* [Setting up liminal](#Setting-up-liminal) + +#### Find Airflow pod names: + +```sh +echo -n "Please enter a namespace: " +read namespace +components=("web" "scheduler" "worker") +namespace=${namespace} +podNames=() +for component in ${components[@]} +do + podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n" + +done + +echo -e "The following Airflow pods are:\n${podNames}" +``` +#### Install liminal in Airflow pods: + +```sh +IFS=$'\n'|''; set -f; podNames=( ${podNames} ) +webPodName=${podNames[${#podNames[@]} - 1]} + +for podName in `echo -e ${podNames[@]}` +do + echo "Installing liminal in pod: ${podName}" + kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal' +done +``` + +#### Setting up liminal: +There are a couple of ways to deploy Yamls in Airflow. Review comment: I suggest to repeat the message: Now that Airflow contains the liminal package, we need to deploy the actual Yamls (DAGs) into airflow - by placing the Yaml files in each components` /opt/airflow/dags folder. There are multiple ways to achieve this. Our recommended approach is to use EFS as a shared folder containing the Liminal Yamls. To this end, you will need to perform the deployment from a (separate) linux box, which can get hold of the Liminal Yamls and deploy them into the EFS - which we shall name "the deployment box". One time setup: 1. Create an EFS file system 2. Mount the EFS file system onto a deployment box ```sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /mnt/efs``` 2. Create a folder on an EFS drive. This will be our target for deployment ```mkdir -p /mnt/efs/opt/airflow/liminal_home``` 3. set ```export LIMINAL_HOME=/mnt/efs/opt/airflow/liminal_home``` 4. Follow the guide <<<<add a good link>>>> for deploying Airflow on K8S, with EFS mount. Point the pods' PV to the right EFS folder, like so: ``` volumeMounts: - name: efs-data mountPath: /opt/airflow/dags ``` where efs-data is a PVC pointing to ```<EFS_ID>:/opt/airflow/liminal_home``` Once this setup is done (one time), you have: 1. A deployment box which can deploy Liminal to an EFS folder 2. Airflow pods which will pick up the Yamls from that folder automatically. Each time you want to deploy the Liminal Yamls from the deployment box: ```liminal deploy <<<path to liminal user code>>>``` ########## File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md ########## @@ -0,0 +1,160 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required bgit y applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Install Liminal in Airflow +* [Workflow](#workflow) +* [Prerequisites](#prerequisites) + * [Supported Distributions](#supported-distributions) +* [Get Started](#Get-Started) + * [Scripted Installation](#scripted-installation) + * [Manual Installation](#manual-installation) +* [References and other resources](#references-and-other-resources) + +## Workflow +Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS. +The process of liminal is emphasized on the following diagram: + + + +## Prerequisites +Before you begin, ensure you have met the following requirements: + +* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed +* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed +* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file +* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation] Review comment: this post is actually not good in my opinion. I believe the recommended way to deploy efs today is to use csi; in this post they for some reason set up an NFS server pod, not sure what for... Essentially I think we can write our own post in a matter of 2-3 hours which will be better. ########## File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md ########## @@ -0,0 +1,160 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required bgit y applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Install Liminal in Airflow +* [Workflow](#workflow) +* [Prerequisites](#prerequisites) + * [Supported Distributions](#supported-distributions) +* [Get Started](#Get-Started) + * [Scripted Installation](#scripted-installation) + * [Manual Installation](#manual-installation) +* [References and other resources](#references-and-other-resources) + +## Workflow +Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS. +The process of liminal is emphasized on the following diagram: + + + +## Prerequisites +Before you begin, ensure you have met the following requirements: + +* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed +* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed +* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file +* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation] +* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc] +* Make sure that the [example repository][liminal-getting-started-project] is your workspace + +### Supported Distributions + +|Distribution | Versions | +|-|-| +|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 | +|[Airflow Helm Chart][airflowChart] | 7.14.3 | + +#### You may need to add the Repo + +```sh +helm repo add airflow-stable https://airflow-helm.github.io/charts +helm repo update +``` + +## Get Started + +### Scripted Installation + +[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow. +* Move to the [example repository project][liminal-getting-started-project] +* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it + + + +### Manual Installation + +Here are a couple of steps of how to install liminal in Airflow on Kubernetes: +* [Find Airflow components](#Find-Airflow-pod-names) +* [Install liminal in the components](#Install-liminal-in-Airflow-pods) +* [Setting up liminal](#Setting-up-liminal) + +#### Find Airflow pod names: + +```sh +echo -n "Please enter a namespace: " +read namespace +components=("web" "scheduler" "worker") +namespace=${namespace} +podNames=() +for component in ${components[@]} +do + podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n" + +done + +echo -e "The following Airflow pods are:\n${podNames}" +``` +#### Install liminal in Airflow pods: + +```sh +IFS=$'\n'|''; set -f; podNames=( ${podNames} ) +webPodName=${podNames[${#podNames[@]} - 1]} + +for podName in `echo -e ${podNames[@]}` +do + echo "Installing liminal in pod: ${podName}" + kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal' +done +``` + +#### Setting up liminal: +There are a couple of ways to deploy Yamls in Airflow. +1. Mount your Amazon EFS file system on a Linux instance: + ``` + sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /opt/airflow/dags + ``` +2. Use gitsync function in order to sync Yamls in the Amazon EFS file system. In this case, we chose to use a kubernetes job: + ``` + apiVersion: batch/v1 + kind: Job + metadata: + namespace: production + name: airflower-production-git + spec: + activeDeadlineSeconds: 60 + backoffLimit: 1 + template: + spec: + containers: + - name: git + image: alpine/git + command: ["/bin/sh","-c"] + args: ["cd /dags; git fetch && git reset --hard origin/master"] + volumeMounts: + - mountPath: /dags + name: dags + volumes: + - name: dags + persistentVolumeClaim: + claimName: airflow-production + restartPolicy: Never + ``` +3. Deploy the Yamls by using kubectl in order to copy the Yamls to the mounted file system. In this case, we chose to use kubectl to copy the Yamls to the mounted EFS: Review comment: Again, in terms of completeness - you are right, it can also be done in this way. however we need to focus on our recommended way, for which we asked the user to set up EFS you can mention it in the previous section in passing: There are other ways to deploy into Aiflow. You can poll a git repo from inside airflow and copy the files in pull. Alternatively, you can do something similar to how we installed liminal on the pods, and iterate on them, copying the file into each. However, to use native Liminal tools, we recommend the following approach.... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
