Ah BTW. I just noticed that for some reason I pasted an old PR earlier in the thread :(. This is the one with requirements.txt I am talking about: https://github.com/apache/airflow/pull/7730
On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Nope. Not blocking. I can work with my branch just requirements.txt is > enough for that :) > > I think the problem with semver is that it is loosely followed - we had a > number of breakages in the past with minor version upgrades :(. > > J. > > > > On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > >> Thanks for the detailed explanation Jarek. >> >> How about we have an upper limit for all our dependencies, example instead >> of "google-cloud-storage>=1.16", we have >> "google-cloud-storage>=1.16,<2.0" ? >> >> If a dependency breaks compatibility in minor versions, we can't do >> anything about it but if they follow SemVer, we should be safe and the >> first-time installers would have a non-breaking package. WDYT? >> >> Btw I hope this is not blocking you in building a production image as I >> think requirements.txt is solving that? Please let me know if it is >> blocking. >> >> PS: I am also just dumping my ideas to solve this issue. Love to hear what >> others think too. >> >> Regards, >> Kaxil >> >> >> >> >> >> On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <jarek.pot...@polidea.com> >> wrote: >> >> > I think we have similar understanding. But let me just clarify because I >> > think we think about we think about solving two different problems >> > My proposal is not solving all problems with dependencies - quite the >> > contrary, I want to solve just one specific "repeatability" problem - >> read >> > on :).. >> > >> > 1. A potential source of confusion: using "-pinned" for installation >> but >> > > using "non-pinned" for DAG development. >> > > >> > >> > This could be confusing indeed - but they are the same in fact - >> > just deps might be different over time. >> > >> > 2. Most of the users would still try to install "apache-airflow" >> package >> > > that might have been broken for example because of a dependency >> > release, >> > > either way, we would still have to suggest them to use "pinned" >> > version >> > > >> > >> > True. I thought we might describe it in the README and make it >> prominently >> > explained. Usually people look at the readme in PyPI when they are >> > installing >> > stuff and it does not work: https://pypi.org/project/apache-airflow/. >> > >> > Also - we could of course explain how to use requirements.txt from the >> > released >> > version when they are installing it. That would be an extra friction >> point >> > though >> > and maybe having "always installable" version of airflow is a better >> > choice. >> > >> > 3. If they install "pinned" version, it is no longer a library again, >> > > that is users won't be able to use new NumPy release or matplotlib >> for >> > > example. In which case we are just circling back to the same >> problem, >> > > "either we risk broken package" while releasing or we risk >> potentially >> > > incompatible versions. >> > > >> > >> > Yep. But maybe it's just a question of naming. Maybe even we could name >> > this package differently to indicate that this version is a way to >> quickly >> > install >> > airflow but not to do any serious development with it. >> > >> > So speaking about THE problem I want to solve with the >> > requirements.txt and apache-airflow-pinned package: >> > >> > I really only want to solve "first-time-user" experience here - nothing >> > more. I >> > definitely do not want to replace the current installation method for >> > experienced >> > users - for them using --constraint requirements.txt is exactly what >> they >> > need. >> > The only problem I am trying to solve with that is "repeatability" of >> > installation. >> > >> > Maybe "apache-airflow-quickinstall" or something like that would be >> better >> > than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or >> > something like that. I think about it as a "flavour" of ariflow rather >> than >> > anything else. I even originally implemented it as [pinned] extra where >> I >> > pinned all requirements. Unfortunately I found that if you have >> > main requirement without limits, adding the same requirement as extra >> with >> > == does not make it pinned :(. That was my original plan. >> > >> > >> > > Btw I have been on "we should have pinned dependency" camp as Airflow >> > > should definitely install without breaking since day-1 but I think a >> > > separate "-pinned" package won't solve that issue. >> > > >> > >> > Ah yeah we went the same route. I do not think we can solve the >> > "library vs. app" problem easily. This is a bit of "eat-and-have-cake" >> > at the same time. I know people have problems >> > with conflicting dependencies when they are trying to install libraries >> > with different requirements. And I am not even trying to solve that >> > problem now. Not even close. This requires some other solution >> > (for example separate virtualenvs with different dependencies >> > build from wheels on per-task basis). But that's something much further >> > in the future (if at all). >> > >> > >> > > >> > > WDYT? Also please do let me know if I have misunderstood something >> > > (definitely possible :D). >> > > >> > > Regards, >> > > Kaxil >> > >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
