barrywhart commented on code in PR #25829:
URL: https://github.com/apache/airflow/pull/25829#discussion_r954085725
##########
airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py:
##########
@@ -409,6 +425,12 @@ def extract_xcom(self, pod: k8s.V1Pod):
self.log.info("xcom result: \n%s", result)
return json.loads(result)
+ def pre_execute(self, context):
+ if isinstance(self.env_vars, str):
+ self.env_vars =
convert_env_vars(ast.literal_eval(self.env_vars.strip()))
Review Comment:
A bit more about our use case: I'm an MLOps engineer. We provide a platform
to make data science development and deployment easy for data scientists. Data
scientists don't write DAGs or other Airflow code. Instead, every PR build for
every data science repo runs a single, standard Airflow DAG that is highly
flexible. We call it the MLOps experiment pipeline. To define environment
variables required by Kubernetes pod(s) in this workflow, data scientists
update a simple list of key-value pairs in an `.env` file. The CI/CD job passes
these variables as part of the `dag_run.conf` JSON structure.
This is the architecture of the MLOps experiment pipeline:
Here are a couple of typical `.env` files from two different repos:
1. Simple project
```
SCRIPT=train.py
GOOGLE_APPLICATION_CREDENTIALS="/root/.config/gcloud/application_default_credentials.json"
```
2. Complex project
```
PREDICTION_DATE=2022-05-18
SCRIPT=evaluate.py
MODEL_NAME=multiembedder
RECOMMENDER_NAME=multiembedder_sku_copurchase
RUN_ID=2022.08.11.10.57.07-beta
#MULTIEMBEDDER_ID=large-model
COPURCHASE_CANDIDATES_BY_CAT_ID=2022.08.02.13.40.13-beta
COVIEW_CANDIDATES_BY_CAT_ID=2022.08.02.13.53.21-beta
#RUN_ID=temp-1234
#SCRIPT=monitor.py
#STRATEGY=sku__copurchase
```
Note that the exact same DAG supports both projects, one with 2 variables
and one with around 10 variables. This flexibility is vital for our data
science users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
