The tests in CI were failing - you can see the failures here: https://travis-ci.org/potiuk/airflow/jobs/647748452
Regarding the non-implicit packages and sequence of installation: from my tests it does not depend on the sequence of the installation. I manually filtered "airflow" package from installation in the provider's package. This means that provider's package does not contain "airflow/__init__.py". But to be on the safe side, I also added "apache-airflow~1.10.*" as dependency to the "apache-airlfow-providers" package. It's impossible to install "providers" before "apache-airlfow" from 1.10. series because it is a dependency. I tested both cases that are valid and both work fine: - installing "apache-airflow" first and then "apache-airflow-providers" (this way you can install "providers" for any 1.10.* version of apache-airflow - installing just "apacher-airflow-providers" on a clean system - it automatically pulls and installs "apache-airflow==1.10.9" when installed J. On Sun, Feb 9, 2020 at 10:11 PM Ash Berlin-Taylor <[email protected]> wrote: > And also how it failed when making it an implicit package? I did do some > quick tests before suggesting it earlier in this thread and pytest and mypy > ran seemingly without error (but I did only do very quick tests) > > My understanding is that come pip install time without implicit packages > that the order in which modules get installed can change/break the > behaviour. > -a > On Feb 9 2020, at 9:09 pm, Kamil Breguła <[email protected]> > wrote: > > Can you show some code snippet? I am curious about this solution. > > > > On Sun, Feb 9, 2020 at 10:06 PM Jarek Potiuk <[email protected]> > wrote: > > > > > > I have bad news and good news :) > > > Bad news - just making "providers" package an implicit one does not > work - > > > when you mix implicit and explicit packages there are some problems. > Tools > > > like pytest, mypy, pylint do not handle this well and think that > "google", > > > "amazon" are top-level packages when analysing the sources. > > > > > > Good news - I managed to get providers packages for backporting work > > > without having to change "providers" to implicit package. It was pretty > > > much just a matter of filtering found packages in setup.py - I am > sending > > > a PR shortly to enable that. I also managed to use breeze environment > to > > > test the backported packages with system tests (at least for GCP > operators > > > where we have system test). > > > > > > We can therefore close the subject of implicit packages. We do not need > > > them for now. Later we might want to convert to implicit packages but > this > > > is something that might never happen. > > > > > > J, > > > > > > > > > > > > > > > On Wed, Feb 5, 2020 at 2:40 PM Jarek Potiuk <[email protected]> > > > wrote: > > > > > > > Yep. Learning from the bigger move I will see how small is the > "smaller" > > > > one I would love to see and can make informed decision. > > > > > > > > On Wed, Feb 5, 2020 at 1:35 PM Ash Berlin-Taylor <[email protected]> > wrote: > > > > > Ah right okay, not an impossible task then. > > > > > Still, I am very strongly in favour of the smallest change for the > goal > > > > > (which is to allow back-ported providers to be installed on > 1.10.x) - both > > > > > cos then most things are more normal, including code under a > provider, and > > > > > then that way we don't have to fight our tools. > > > > > On Feb 5 2020, at 12:08 pm, Jarek Potiuk <[email protected] > > > > > > > wrote: > > > > > > > > > > > > > > If we make every namespace under airflow.* implict, then we > have to > > > > > ensure > > > > > > > that our modules do not clash with _any and every_ possible > top level > > > > > > > package that exists in PyPi, because we cannot know what might > be > > > > > > > > > > > > > > > > installed > > > > > > > by our users. That seems like an impossible feat. Have I > > > > > > > > > > > > > > > > mis-understood? > > > > > > > If I haven't mis-understoond then I this is I think a strong > argument > > > > > > > > > > > > > > > > for > > > > > > > _not_ making more than we absolutely have to an implicit > namepsace. > > > > > > > > > > > > > > > > > > > Nope. This is a problem for MyPy and Pytest only so mostly for > our > > > > > tooling. > > > > > > And only if our own modules import packages with the same name > as the > > > > > > module. So we are in full control here. > > > > > > > > > > > > > > > > > > > > > > > > > > -a > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > -- > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
