kaxil commented on issue #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file URL: https://github.com/apache/airflow/pull/6230#issuecomment-573270938 > > > > That's a tough one as that would break a LOT of DAGs, that said it would be nice to have some class typing as it's hard to know off-hand what configs you can set. > > > > @ashb @potiuk @nuclearpinguin @kaxil What do you guys think about this? Maybe we have an `executor_config_dict` argument for easy backporting and slowly deprecate? > > > > > > > > > It should be doable to detect a dict and "upgrade" it in the same argument. It would be nice if we supported both for a few releases - we _could_ just say F-it and need an update for Airflow 2.0, but I'd like us to avoid any hard breaking changes to DAGs without a very good reason (and us regretting our past code is not a good enough reason for me 😄 ) > > > > > > ooo I like that idea. We can do type checking/deprecation warnings for the first few versions and slowly wean people off without needing extraneous parameters. > > I think we can create a separate operator. This will significantly simplify all the code because we will not have to support two configurations at the same time in the same operator. We can then simplify the migration to the new operator by printing the code to be copied. > #6375 > > We already want to use a similar approach with GCP operators and others available only in Airflow 2.0. We create operators that have a lot of changes and only support Python 3. The operators will be available as a separate package, but they can be used in Airflow 1.10, so users will have two operator variants available. Airflow 2.0 will abandon previous operator variants and only the latest ones will be available. In Airflow 2.0 they will be available in the same place as the operators backported in Airflow 1.10 > Here is more information: > https://lists.apache.org/thread.html/bdc1a070295121782e2e054ae62cac363ef84540b48f750916eae88a%40%3Cdev.airflow.apache.org%3E That is a good idea too. `KubernetesPodYamlOperator` 👍 sounds good to me. I am fine either way whatever we choose too. But if using the same operator breaks backwards compatibility, then we might want to think for a separate operator like Kamil suggested
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
