barrywhart commented on code in PR #25829:
URL: https://github.com/apache/airflow/pull/25829#discussion_r954085725
##########
airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py:
##########
@@ -409,6 +425,12 @@ def extract_xcom(self, pod: k8s.V1Pod):
self.log.info("xcom result: \n%s", result)
return json.loads(result)
+ def pre_execute(self, context):
+ if isinstance(self.env_vars, str):
+ self.env_vars =
convert_env_vars(ast.literal_eval(self.env_vars.strip()))
Review Comment:
A bit more about our use case: I'm an MLOps engineer. We provide a platform
to make data science development and deployment easy for data scientists. Data
scientists don't write DAGs or other Airflow code. Instead, every PR build for
every data science repo runs a single, standard Airflow DAG that is highly
flexible. To define environment variables required by the Kubernetes pod, data
scientists update a simple list of key-value pairs in an `.env` file. The CI/CD
job passes these variables as part of the `dag_run.conf` JSON structure.
This is the architecture of the MLOps experiment pipeline:
Here are a couple of typical `.env` files from two different repos:
1. Simple project
```
SCRIPT=train.py
GOOGLE_APPLICATION_CREDENTIALS="/root/.config/gcloud/application_default_credentials.json"
```
2. Complex project
```
PREDICTION_DATE=2022-05-18
SCRIPT=evaluate.py
MODEL_NAME=multiembedder
RECOMMENDER_NAME=multiembedder_sku_copurchase
RUN_ID=2022.08.11.10.57.07-beta
#MULTIEMBEDDER_ID=large-model
COPURCHASE_CANDIDATES_BY_CAT_ID=2022.08.02.13.40.13-beta
COVIEW_CANDIDATES_BY_CAT_ID=2022.08.02.13.53.21-beta
#RUN_ID=temp-1234
#SCRIPT=monitor.py
#STRATEGY=sku__copurchase
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
