potiuk commented on issue #51545:
URL: https://github.com/apache/airflow/issues/51545#issuecomment-2979760380
Also few opinionated answers from my side
* Should the Providers with executors be separated into providers to that
they can be included on Server side?
Yes
* How many packages is optimal? Trade-off between focus and complexity
One common package with all the "small" utils that can be used, likely with
some optional dependencies for utils that might or might not be needed. Usually
the complexity and overhead is with dependencies that given "common util" bring
- not with the code of the util itself.
I'd say `airflow-*`: `common-util`, `scheduler`, `api-server`, `triggerer`,
`task-sdk` (for all kinds of workers) is probably best approach. And
"apache-airflow" as meta-data package installing all of them together.
* What naming convention is clearest? For users and developers
I'd vote for `apache-airflow-*` from the above list .
* How to handle backward compatibility? Migration path for existing users
I think we should stick with releasing all those packages together in the
same version (except task-sdk that should be always backwards compatible -
until Airlfow 4. This will help with testing - because we will not introduce
matrix of versions. At the same time package split will allow us to have far
less dependencies in different packages.
* What about provider dependencies? Which packages should providers depend
on?
`task-sdk` only. Task-sdk should not depend on common-util. There some
duplication is likely but we should handle it at pre-commit level where task
sdk code (serialization) is copied to common util for example)
* Testing strategy? How to test package combinations efficiently
There should not be combinations. All packages should have the same version
and should be released together - even if there are no changes in particular
distribution. They should have `==` pinned dependencies to other
`apache-airflow` things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
