potiuk commented on PR #60472: URL: https://github.com/apache/airflow/pull/60472#issuecomment-3798846524
> 1 - When we run` uv sync` or` uv run`, we only install the dev dependency group. How will the required and optional dependencies be upgraded in that case? No - we install required dependencies **and** `dev` group. No optional deps unless you also specify `--extra` flag. This is what we are doing here, when we want some optional devs to by also added for development always - we just add them as `current-distribution[extra]` to `dev` group so that always when dev group is updated by `uv sync` extra deps are also updated. > 2 - I have read this - Why do we do this? Do we group the optional SQLAlchemy dependencies together with task-sdk, and the required SQLAlchemy dependency with airflow-core, or is it not possible to separate them since SQLAlchemy is required? So far everything we install requires `airflow-core` - which bings sqlalchemy in, but shortly - for Airflow 3.2, `airflow-core` and `airflow-task-sdk` will be completely independent from each other and `task-sdk` will not have `sqlalchemy` as dependency. So far `sqlalchemy` was simply always installed through provider -> task-sdk-> airflow-core -> sqlalchemy. Therefor each provider that expects sqlalchmy will have to bring it in as a dependnecy (either optional or required - depending how they use. There are few other dependency "knots" that we will have to unentangle - with executors, and ui plugins - in order to make airflow distributions more isolated, independent and only have dependencies they needed - mostly because in the future, different components of airflow will be installable separately - with different sets of dependencies - different image will be needed for airlfow-core, different (possbibly) for airflow-ui, different for task-sdk, also multi-team feature will allo w different teams to have different set of providers and dependencies installed in their workers. This is a "north" star of our isolation that I also explained in more detail in this discussion at the devlist https://lists.apache.org/thread/xwspbh8wfqcyovpznqh415mbs227qdoq > 3. What’s the meaning of workspace = true? Does it mean we can install it from PyPI? (I noticed )Most modules (like task-sdk and providers) have workspace = true, but airflowctl does not, since we don’t install it from PyPI Also airflow-ctl has "workspace=true" - whrere do you get it does not have it ? Maybe we missed it somewhere? Some older version did not have it yes - but I think we added it since. Look at https://docs.astral.sh/uv/concepts/projects/workspaces/. Workspace is a way to bring together several distributions that are otherwise separate distributions - to make sure that they all can be installed together with common set of dependencies. Apart of the isolation above we also have to support the case where airflow is installed from a single image - where you can run all components: scheduier, api-server, workers (with providers), triggerer etc. -> and we have to be able to make sure that all those distributions that we want to install together potentially - can in fact be installed - i.e. they have no conflicting dependencies. This is what `uv sync` in the root airflow directory does (this is the workspace root) - it actually takes all the distributions and attempts to install all of them and find the right set of dependencies for all of them that are non-conflicting. Also when distributions are in the same workspace, you can refer to them by their name as dependency , and rather than installing them from `PyPI` when you run `uv sync` during local development, you install them from sources in the same workspace. This way when `provider-airbyte` uses `task-sdk` as dependency, when you go to the folder of the `provider-airbyte` and you run `uv sync`, then `uv` will install `task-sdk` from the local `task-sdk` source folder in airflow repo that is part of the workspace, but it will install `airbyte-api` from `PyPI` because it cannot find it in local workspace. > 4. I don’t exactly remember, but when we changed airflow-core/src/airflow, you mentioned that we could only achieve this by using UV and workspaces. Is this related to the workspace = true option? Not sure what you are asking for here and what changing `airflow-core/src/airflow` might mean - but I recommend you to read the `workspace` docs in `uv`. You can also read my series of blogs about the monorepo/workspace - there are 4 parts starting with https://medium.com/apache-airflow/modern-python-monorepo-for-apache-airflow-part-1-1fe84863e1e1 - link to remaining 3 parts in there, I am also (this Saturday) giving a talk about it at FOSDEM https://fosdem.org/2026/schedule/event/WE7NHM-modern-python-monorepo-apache-airflow/ which should be publishded in a few weeks. Also we are recording a "Talk Python To Me" podcast with @amoghrajesh on 10th of Feb about it https://talkpython.fm/ and there will be podcast recording available (but also livestream will likely be there - more info closer to the time on slack and devlist). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
