potiuk commented on issue #51545: URL: https://github.com/apache/airflow/issues/51545#issuecomment-2980231385
And following what we already do with providers - (but the other way round) we should be running complete suite of tests of all previously released `task-sdk` versions that we want to supoprt with the new "airflow core" suite of packages and run the tests for those different versions with the "main" version of airflow-core` suite of packages. This is the only way we can actually be more-or-less sure we have not broken something accidentally. There should be a suite of tests on Task-SDK that actually run against running "api-server" locally - where `api-server" code comes from "main" but "task-sdk" and "task-sdk-tests" come from past versions. This is also why "task-sdk" should be completely standalone package. And it should be everything that is needed for example by the celerly worker to run tasks, or everything that should be installed in container image for "kubernetes" executor or everything needed to be installed by "edge executor". Generally providers shoudl only ever import from task.sdk and never import anything from other airflow distributions. > In my head, separate providers and executors means we'd have distributions like apache-airflow-providers-cncf-kubernetes and apache-airflow-executor-kubernetes. Naming pattern not fixed, but something like that. Yes. And that's fine "apacha-aurflow-executor-cncf-kubernetes" can still depend on the code from "apache-airflow-providers-cncf-kubernetes". Which will depend only (and exclusively) on task-sdk. This means that yes - you should install "task-sdk" and "apache-airflow-providers-cncf-kuernetes" (transitively) on scheduler - but this is IMHO completelhy no problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
