potiuk commented on PR #36294: URL: https://github.com/apache/airflow/pull/36294#issuecomment-1862496330
I am not sure if that is really needed as a feature. That sounds like pretty orthogonal to Airflow "functionality" as application and more of a generic nature. This looks like a completely independent feature of the chart that does not interact with other components of the chart and I see no particular reason why we should add the whole complexity of those to manage them in our chart. I would rather focus on telling users how they can easily add this and similar kinds of "extensions" to Airflow chart without having to implement it as feature and without having to carry a burden of maintaining those by the Airflow community. The problem with such "features" as chart is that people expect that everything will be "configurable" for them as a black box - say you will want to add - for example - anti-affinity to your cronJob to not run at the same machine as your scheduler or things like that - which K8S has plenty of. Surely you can define template in the way that allows you to add any resource option, but when it gets more and more complex and flexible, you either go the route where you literally copy&paste K8S resource definition to your values.yml (which requires you to understand resource definition of K8S as a user) or you introduce a parallel yaml definition of those resources in your values.yaml - so basically your custom "resource" definition. I think K8S resource definition is "good enough" as an abstraction layer and when you manage k8s application - even via chart - you need to understand those and learn it, and be able to write your own definition. For me Helm Chart is not a black box - it's much more of a "gray box" where managing and customizing you helm applications requires you to understand resource definitions. So (with such a long intro) I think it would be great to explain how you can do similar things and show this as an example but purely as a documentation, not as a feature of the chart. Not many people are aware but there is a very easy (albeit a bit counter-intuitive) way of extending the charts by making your own chart and adding the chart you want to extend as a sub-chart. https://helm.sh/docs/chart_template_guide/subcharts_and_globals/ In this particular case you can easily create your own chart, make airflow a sub-chart of it and add such cron job (or many of thsoe) as a resource in your custom chart. This makes it very easy to maintain and it's usually way simpler to do because you would not have to make it generic, you could hard-code your resources and definition of cron-job of yours in a much simpler way, and you could add any kind of labeling, scheme, extra resources etc. etc. And it has the extra flexibility of being able to do anything else the same way. It's not limiting the users in any way, they can usually do it much faster and easier. I believe that If you convert this into a documentation example of how you can extend Airlfow chart and add your own custom functionality (cron job being a good example of this) in https://airflow.apache.org/docs/helm-chart/stable/production-guide.html - that would be a much better approach to make it possible for Airflow Chart users. They willl get not only an explanation on how can they add cron jobs but also a segway to be able to add other things they miss. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
