kaxil commented on issue #44352: URL: https://github.com/apache/airflow/issues/44352#issuecomment-2532577200
@potiuk 's options here: https://lists.apache.org/thread/8jtgr1m072lgsy4tl3km9ptb216vqjds > we have three > possible options to tell our users:. > > 1) either copy to workers only the configuration that is needed (but which > ones are needed and how to separate those config entries in the docs?) > 2) or NO configuration should be present in the worker - and all > configuration that is needed should be passed by Task SDK to the worker > 3) or we assess that task and dag file processors do not need any > configuration at all > > I think if we assess that 3) is the case - there is no other option but to > remove conf. That would also be really nice as we would not have to have > "conf" reading code in TaskSDK at all. > But if we think that tasks and dag parsing will actually need some > configuration, then we can go either 1) or 2) route > > I think option 2) is much better from less confusion, security and "single > source of truth" point of view. And if we go for 2) then `namespace = > conf.get("kubernetes", "NAMESPACE")` will not work because configuration > will not be present there and in this case what would make more sense is to > leave "conf" as part of the context but only pass the "needed" > configuration there. > If we go for option 1) then likely yes we could remove it from context. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
