tseruga opened a new issue, #26045: URL: https://github.com/apache/airflow/issues/26045
### Description Proposal: Allow jobs deployed by the helm chart to be created without assuming that they will use the same environments as the pods they are deployed alongside. Ways to accomplish this: - Add new boolean values for the jobs, allowing the user to opt-out of using the environment provided by the `custom_airflow_environment` helper, and instead provide their own environment definition catered to these jobs. - Allow the user to specify the environment on a per-resource level, failing back to the deployment-wide `custom_airflow_environment` if per-resource environments aren't provided. ### Use case/motivation The `custom_airflow_environment` helper is used in the creation of all Airflow pods and jobs (e.g. `run-airflow-migrations` and `create-user-job` jobs). There is no option to specify environments on a per-resource level, forcing all resources to share the same environment variables. For pods, this is reasonable behavior as they should share environments. For jobs (which are immutable once the resource is created), this creates a difficult situation where the environment cannot be altered once the helm chart is deployed if using tools such as flux or argo. In practice this means that environments across the entire deployment cannot be changed without doing a full uninstall and reinstall of the helm chart. These jobs accomplish very small, focused tasks when compared to the pods that they share their environments with - and I believe that they should not be forced to share the same environment variables, as it's very unlikely that they require the same environment definitions. References to this helper in the job templates: https://github.com/apache/airflow/blob/main/chart/templates/jobs/migrate-database-job.yaml#L100 https://github.com/apache/airflow/blob/main/chart/templates/jobs/create-user-job.yaml#L98 Note that there is no way to avoid the `custom_airflow_environment` helper from being applied to these templates. ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
