Daniel, Thanks for responding. As you’re working on this, here’s something to think about…
We are running AF on AWS and we use EFS as our file system for dags/plugins/logs. Ideally, we would just create a single EFS instance and have 3 directories, one each for dags/plugins/logs. To do this, we would just need 1 mount in the pod and the AF config would point to the locations of the folders. This is exactly what we do for the webserver and scheduler pods. Currently, there MUST be 2 mounts, one for dags and one for logs (plugins is missing). This forces us to have 2 EFS instances, one for logs and one for dags. Having multiple EFS instances isn’t the end of the world, but in an ideal world, we wouldn’t need it. Thanks, Jim From: Daniel Imberman <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, February 26, 2020 at 6:57 AM To: "[email protected]" <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: Custom Plugins with the Kubernetes Executor... Hi Jim, This is a good catch, thank you for bringing this to my attention! I’ve created a Jira ticket and should hopefully be able to get to it this week. I will also push to have it added to the next release of airflow. As a stop-gap will you be able to bake your plugins into your docker image? https://issues.apache.org/jira/browse/AIRFLOW-6923 via Newton Mail<https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2> On Tue, Feb 25, 2020 at 3:45 PM, Jim Majure <[email protected]> wrote: I am attempting to use the Kubernetes executor and custom plugins and it doesn’t seem to work. The Kubernetes configuration section allows one to specify the location of the dags and the logs, but not the plugins. Are plugins expected to be deployed in the base docker container that hosts airflow? [kubernetes] airflow_configmap = {{ include "ah-airflow.fullname" . }}-configmap worker_container_repository = {{ .Values.image.repository }} worker_container_tag = {{ .Values.image.tag }} worker_container_image_pull_policy = {{ .Values.image.pullPolicy }} delete_worker_pods = True dags_in_image = False dags_volume_claim = {{ include "ah-airflow.fullname" . }}-efs-pvc-dags dags_volume_subpath = dags logs_volume_claim = {{ include "ah-airflow.fullname" . }}-efs-pvc-logs logs_volume_subpath = logs dags_volume_host = logs_volume_host = in_cluster = True namespace = default gcp_service_account_keys = Jim Majure| Principal Machine Learning Engineer aurishealth.com<http://aurishealth.com/> | 150 Shoreline Dr. | Redwood City, CA | 94065<webextlink://150%20Shoreline%20Dr.%20|%20 Redwood%20City,%20CA%20|%20 94065> (515) 829-0667 [signature_181900226]
