-1 binding (btw Vincent, your vote is binding in this vote 🎉)
On Fri, Jul 21, 2023 at 7:52 AM Beck, Vincent <vincb...@amazon.com.invalid> wrote: > -1 (non-binding) > > On 2023-07-21, 5:46 AM, "Michał Modras" <michalmod...@google.com.inva > <mailto:michalmod...@google.com.inva>LID> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > -1 (non-binding) > > > Dask is more niche than Celery or K8s - let's err on the side of making > Airflow core dependencies more light-weight and not preinstall it. > > > On Fri, Jul 21, 2023 at 10:57 AM Pankaj Koti > <pankaj.k...@astronomer.io.inva <mailto:pankaj.k...@astronomer.io.inva>lid> > wrote: > > > > 0 (non-binding) > > > > I agree with most of what has been discussed in the thread > > https://lists.apache.org/thread/0d9x4kl7hc2qzvho2mbdf35ohn5w12l6 < > https://lists.apache.org/thread/0d9x4kl7hc2qzvho2mbdf35ohn5w12l6> > > and this email thread. > > > > I did a check on Airflow discussions ( > > https://github.com/apache/airflow/discussions?discussions_q=dask+ < > https://github.com/apache/airflow/discussions?discussions_q=dask+>) > > and Airflow issues( > > https://github.com/apache/airflow/issues?q=is%3Aissue+dask+ < > https://github.com/apache/airflow/issues?q=is%3Aissue+dask+>) > > and upon reading their descriptions, see that there are some users using > > the Dask executor although I personally have never tried it. > > > > If we decide to not pre-install it, in my opinion, a detailed and > top-level > > news fragment / Changelog would help us here which mentions it to install > > the provider additionally in case they would like to continue using it. > > > > Regards, > > > > > > > > Pankaj Koti > > > > *Senior Software Engineer, *OSS Engineering Team. > > Location: Pune, India > > > > Timezone: Indian Standard Time (IST) > > > > Email: pankaj.k...@astronomer.io <mailto:pankaj.k...@astronomer.io> > > > > Mobile: +91 9730079985 <+91%2097300%2079985> > > > > > > On Fri, Jul 21, 2023 at 2:20 PM Jarek Potiuk <ja...@potiuk.com <mailto: > ja...@potiuk.com>> wrote: > > > > > > One possible solution could be allowing users to instruct Airflow not > > to > > > pre-install executors providers by passing an argument via > > > `--install-option` during pip install, and add install them explicitly > by > > > providing the extras. > > > > > > Yes it would be nice. Unfortunately the dependency mechanisms of > > > Python (and `pip`) are not as flexible. You cannot individually remove > > > requirements, and having requirements is the only way you can have > > > "install also that other package" feature (extras is the mechanism to > > > install optional dependencies, but it's not something we should ask > > > regular users to do > > > > > > > > > > > > On Fri, Jul 21, 2023 at 10:46 AM Jarek Potiuk <ja...@potiuk.com > <mailto:ja...@potiuk.com>> wrote: > > > > > > > > And just to clarify: The voting will last till Tuesday 25 July 20023 > > > 11am CST. > > > > > > > > On Fri, Jul 21, 2023 at 10:43 AM Jarek Potiuk <ja...@potiuk.com > <mailto:ja...@potiuk.com>> > > wrote: > > > > > > > > > > Clarification: Ash is right (I totally forgot about it). > > > > > > > > > > We do have dask in > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html#core-airflow-extras > < > https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html#core-airflow-extras > > > > > > > and it even explicitly mentions it: > > > > > > > > > > * "dask pip install 'apache-airflow[dask]' DaskExecutor" > > > > > > > > > > So technically speaking you **should** use the dask extra to > install > > > > > distributed, dask and cloudpickle package - all needed to run the > > dask > > > > > executor. And we will keep it this way - it will just move from > "core > > > > > extras" to "providers". Actually it will move to "deprecated" and > we > > > > > will alias "dask" it with the new "daskexecutor", because the name > > > > > "apache-airflow-providers-dask" is not allowed by PyPI (I guess the > > > > > -dask suffix is considered potentially harmful for typosquatting) > so > > I > > > > > reserved "apache-airflow-providers-daskexecutor". > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > On Fri, Jul 21, 2023 at 10:34 AM Hussein Awala <huss...@awala.fr > <mailto:huss...@awala.fr>> > > > wrote: > > > > > > > > > > > > > based on the premise that your had to install the `dask` extra > in > > > the > > > > > > first place to get dask module of the right version > > > > > > > > > > > > After reviewing the Dask Executor > > > > > > < > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/dask.html > < > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/dask.html > > > > > > > > > > > > section in Airflow documentation, I didn't find any requirement > or > > > > > > recommendation to add the `dask` extra when using it. So, > including > > > the > > > > > > dependencies explicitly in the requirements file should suffice > to > > > use the > > > > > > executor. This means that this change might potentially break > > > existing > > > > > > setups for some users when they upgrade Airflow version, which > > > contradicts > > > > > > the Airflow deprecation policy. > > > > > > > > > > > > If the premise holds true for Dask, it should also apply to > Celery > > > and > > > > > > Kubernetes. Nevertheless, we decided to pre-install their > providers > > > after > > > > > > the migration. > > > > > > > > > > > > However, this executor isn't as popular as the others, and as an > > > Airflow > > > > > > user, I prefer not to have Dask dependencies in my environment > > when I > > > > > > install Airflow with the Celery executor. This helps avoid > > conflicts > > > and > > > > > > enables me to upgrade libraries independently from their > > > constraints. One > > > > > > possible solution could be allowing users to instruct Airflow not > > to > > > > > > pre-install executors providers by passing an argument via > > > > > > `--install-option` during pip install, and add install them > > > explicitly by > > > > > > providing the extras. > > > > > > > > > > > > Since the flexibility of our deprecation policy isn't entirely > > clear > > > to me, > > > > > > I'm neutral on this matter, with a vote of +0. > > > > > > > > > > > > On Fri, Jul 21, 2023 at 9:10 AM Elad Kalif <elad...@apache.org > <mailto:elad...@apache.org>> > > > wrote: > > > > > > > > > > > > > I agree with Ash > > > > > > > -1 as well. > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 21, 2023 at 9:29 AM Ash Berlin-Taylor < > > a...@apache.org <mailto:a...@apache.org>> > > > wrote: > > > > > > > > > > > > > > > -1 - based on the premise that your had to install the `dask` > > > extra in > > > > > > > the > > > > > > > > first place to get dask module of the right version, so if we > > > make the > > > > > > > > existing extra depends on the new provider then it's good > > enough. > > > > > > > > > > > > > > > > On 21 July 2023 06:22:28 BST, Jarek Potiuk <ja...@potiuk.com > <mailto:ja...@potiuk.com>> > > > wrote: > > > > > > > > >Q: Do we want to pre-install Dask provider in Airflow 2.7.0 > > > (with Dask > > > > > > > > >executor) once separated ? > > > > > > > > > > > > > > > > > >Discussion was here: > > > > > > > > > > > > https://lists.apache.org/thread/0d9x4kl7hc2qzvho2mbdf35ohn5w12l6 < > https://lists.apache.org/thread/0d9x4kl7hc2qzvho2mbdf35ohn5w12l6> > > > > > > > > > > > > > > > > > >Please vote: > > > > > > > > > > > > > > > > > >* +1 -> yes, we want to have dask provider preinstalled > > > > > > > > >* -1 -> no, it's fine to make it optional > > > > > > > > >* 0 -> no opinion > > > > > > > > > > > > > > > > > >Consider it my -1: I think we should NOT preinstall Dask > > > provider. > > > > > > > > > > > > > > > > > >Voting guidelines here. This is really a "procedural" matter > > > rather > > > > > > > > >than code modification: > > > > > > > > > > > > > > > > > >> Votes on procedural issues follow the common format of > > > majority rule > > > > > > > > unless otherwise stated. That is, if there are more > favourable > > > votes than > > > > > > > > unfavourable ones, the issue is considered to have passed -- > > > regardless > > > > > > > of > > > > > > > > the number of votes in each category. (If the number of votes > > > seems too > > > > > > > > small to be representative of a community consensus, the > issue > > is > > > > > > > typically > > > > > > > > not pursued. > > > > > > > > > > > > > > > > > >In this case committers have binding votes but other > community > > > members > > > > > > > > >are encouraged to state their non-binding votes as well. > > > > > > > > > > > > > > > > > >https://www.apache.org/foundation/voting.html < > https://www.apache.org/foundation/voting.html> > > > > > > > > > > > > > > > > > > > > >--------------------------------------------------------------------- > > > > > > > > >To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > <mailto:dev-unsubscr...@airflow.apache.org> > > > > > > > > >For additional commands, e-mail: > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org> > > > For additional commands, e-mail: dev-h...@airflow.apache.org <mailto: > dev-h...@airflow.apache.org> > > > > > > > > > > > >
