kaxil commented on issue #51545: URL: https://github.com/apache/airflow/issues/51545#issuecomment-2980465968
>I think you misunderstood my description I think only task-sdk should be client and it should be "all" you need to run client. Even if it means duplication of some code (serialization) - and you can make duplication smartly by symbolic linking or copying relevant code between distributions in pre-commit. Yup onboard with that. We might not need to even duplicate the serialization module between server and client but will need to prove/POC that - I might have created a GH issue for it. Agreed on Providers relying on Task SDK alone apart from the Executors one. Can make it independent too but just need to think through the BaseExecutor interface which is used in the Scheduler too. --- >I also think the only reason we should split "airflow-core" (and that's the motivation I have) is to shard dependencies - and make different part of the system (schduler, api-server, triggerer ) use potentially different set of dependencies - especially api_server should be split from the rest. So introducing "common" is for me the way how you can have common utils used by all those separate "distributions" to have common utls like logging, conf, etc. etc. - but all of them should always come from the same "commit" when installed togetther (i.e. should be pinned to the same version that they were tested it in "main" and "v3_*" branch). Could we maybe tackle and discuss that separately if you agree? The primary reason for me requesting that is by keeping current "core" as Server and Task SDK as client -- once we truly decouple them. We can achieve one user goal for Independent upgrades for Server & client. That from the release POV, won't be much different than how we do it today since they are separate packages. Splitting API-server and Scheduler and other packages would be more involved and we can do that separately in my humble opinion, if you agree. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
