potiuk commented on issue #51545: URL: https://github.com/apache/airflow/issues/51545#issuecomment-2980467860
And yes - I think we want to split packages for different reasons :) - that's where confusion comes from. you want to split "client vs. server" - but there introducing "common distribution" (even serialization) is going to create a lot of problems - mostly because conflicting dependencies that will inevitably happen when we try to have "common" package working with separately versioned "client" and "server". So I think we should do everything to make "task-sdk" (client) fully standalone and separate and not sharing any "installable" code with api_server (server). They might share some of the code by duplication, but they should never "import " common code from a shared location. That's a recipe for disaster. So my motivation for splitting the "core" is completely different. If you want to have one binary covering "api_server", "scheduler" (and let's not forget - "triggerer", and "dag processor") - then it makes almost 0 sense to have "common" distribution. It only makes sense IMHO if we decide to have several distributions - each for each of the "components" and each of those distributions will use a subset of "common" and have a separate resulting set of dependencies it should rely on. Other than that "task.sdk" should never use that "common code". One of the problems you are going to have is "triggerer" and "dag processor". Because effectively both of them have to have both "core" parts and "task-sdk" parts installed. If you want to version task-sdk separately and "core" separately and have a common code that will be used by both - and you will try to have "triggerer" and "dag processor" to effectively use: * latesst core * some version of commmon * whatever version of task-sdk - identical and compatible with the "worker" task-sdk. This is going to spin out of control very very very quickly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
