Yesterday we had another master breakage - this time from elasticsearch releasing MINOR version 7.6 breaking our builds (not it was MINOR version so should be compatible .... it was not for us). I fixed it quickly yesterday by limiting it to < 7.6 but for me - this is quite clear that trying to rely on SemVer being followed by others is a futile effort (at least in python's world).
The theory is nice, but it breaks in practice. And it's not really a fault of the library maintainers. It's simply sometimes not so easy to see how your APIs are used - and in Python, you cannot prevent using stuff that you think is an internal detail. This is what happened in elasticsearch case yesterday - apparently, our plugin was using an "internal" API unknowingly and some parameters from that API were dropped during refactoring of elasticsearch library. My observation (it's anecdotal though) is that the COVID-19 situation made people have more time, fewer distractions, fewer things to do, and we have higher frequency of OSS packages being released recently so we should protect a bit from more often breakages. I think learning from yesterday is: * we should merge the requirements.txt solution quickly to prevent further breakages (I am reading and testing it now) - I think everyone agrees it's good to have it * I think we can continue discussing whether apache-airflow-pinned package should be released or not. I can leave the code building the package but we can decide about it after some more discussion Does it sound good? J On Fri, Mar 20, 2020 at 2:47 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > And rebased it right now and fixed automated requirements update. > > On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >> Ah BTW. I just noticed that for some reason I pasted an old PR earlier in >> the thread :(. >> This is the one with requirements.txt I am talking about: >> https://github.com/apache/airflow/pull/7730 >> >> On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk <jarek.pot...@polidea.com> >> wrote: >> >>> Nope. Not blocking. I can work with my branch just requirements.txt is >>> enough for that :) >>> >>> I think the problem with semver is that it is loosely followed - we had >>> a number of breakages in the past with minor version upgrades :(. >>> >>> J. >>> >>> >>> >>> On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik <kaxiln...@gmail.com> wrote: >>> >>>> Thanks for the detailed explanation Jarek. >>>> >>>> How about we have an upper limit for all our dependencies, example >>>> instead >>>> of "google-cloud-storage>=1.16", we have >>>> "google-cloud-storage>=1.16,<2.0" ? >>>> >>>> If a dependency breaks compatibility in minor versions, we can't do >>>> anything about it but if they follow SemVer, we should be safe and the >>>> first-time installers would have a non-breaking package. WDYT? >>>> >>>> Btw I hope this is not blocking you in building a production image as I >>>> think requirements.txt is solving that? Please let me know if it is >>>> blocking. >>>> >>>> PS: I am also just dumping my ideas to solve this issue. Love to hear >>>> what >>>> others think too. >>>> >>>> Regards, >>>> Kaxil >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Mar 19, 2020 at 2:43 PM Jarek Potiuk <jarek.pot...@polidea.com> >>>> wrote: >>>> >>>> > I think we have similar understanding. But let me just clarify >>>> because I >>>> > think we think about we think about solving two different problems >>>> > My proposal is not solving all problems with dependencies - quite the >>>> > contrary, I want to solve just one specific "repeatability" problem - >>>> read >>>> > on :).. >>>> > >>>> > 1. A potential source of confusion: using "-pinned" for >>>> installation but >>>> > > using "non-pinned" for DAG development. >>>> > > >>>> > >>>> > This could be confusing indeed - but they are the same in fact - >>>> > just deps might be different over time. >>>> > >>>> > 2. Most of the users would still try to install "apache-airflow" >>>> package >>>> > > that might have been broken for example because of a dependency >>>> > release, >>>> > > either way, we would still have to suggest them to use "pinned" >>>> > version >>>> > > >>>> > >>>> > True. I thought we might describe it in the README and make it >>>> prominently >>>> > explained. Usually people look at the readme in PyPI when they are >>>> > installing >>>> > stuff and it does not work: https://pypi.org/project/apache-airflow/. >>>> > >>>> > Also - we could of course explain how to use requirements.txt from the >>>> > released >>>> > version when they are installing it. That would be an extra friction >>>> point >>>> > though >>>> > and maybe having "always installable" version of airflow is a better >>>> > choice. >>>> > >>>> > 3. If they install "pinned" version, it is no longer a library >>>> again, >>>> > > that is users won't be able to use new NumPy release or >>>> matplotlib for >>>> > > example. In which case we are just circling back to the same >>>> problem, >>>> > > "either we risk broken package" while releasing or we risk >>>> potentially >>>> > > incompatible versions. >>>> > > >>>> > >>>> > Yep. But maybe it's just a question of naming. Maybe even we could >>>> name >>>> > this package differently to indicate that this version is a way to >>>> quickly >>>> > install >>>> > airflow but not to do any serious development with it. >>>> > >>>> > So speaking about THE problem I want to solve with the >>>> > requirements.txt and apache-airflow-pinned package: >>>> > >>>> > I really only want to solve "first-time-user" experience here - >>>> nothing >>>> > more. I >>>> > definitely do not want to replace the current installation method for >>>> > experienced >>>> > users - for them using --constraint requirements.txt is exactly what >>>> they >>>> > need. >>>> > The only problem I am trying to solve with that is "repeatability" of >>>> > installation. >>>> > >>>> > Maybe "apache-airflow-quickinstall" or something like that would be >>>> better >>>> > than "apache-airflow-pinned" or "apache-airflow-repeatable-install" or >>>> > something like that. I think about it as a "flavour" of ariflow >>>> rather than >>>> > anything else. I even originally implemented it as [pinned] extra >>>> where I >>>> > pinned all requirements. Unfortunately I found that if you have >>>> > main requirement without limits, adding the same requirement as extra >>>> with >>>> > == does not make it pinned :(. That was my original plan. >>>> > >>>> > >>>> > > Btw I have been on "we should have pinned dependency" camp as >>>> Airflow >>>> > > should definitely install without breaking since day-1 but I think a >>>> > > separate "-pinned" package won't solve that issue. >>>> > > >>>> > >>>> > Ah yeah we went the same route. I do not think we can solve the >>>> > "library vs. app" problem easily. This is a bit of "eat-and-have-cake" >>>> > at the same time. I know people have problems >>>> > with conflicting dependencies when they are trying to install >>>> libraries >>>> > with different requirements. And I am not even trying to solve that >>>> > problem now. Not even close. This requires some other solution >>>> > (for example separate virtualenvs with different dependencies >>>> > build from wheels on per-task basis). But that's something much >>>> further >>>> > in the future (if at all). >>>> > >>>> > >>>> > > >>>> > > WDYT? Also please do let me know if I have misunderstood something >>>> > > (definitely possible :D). >>>> > > >>>> > > Regards, >>>> > > Kaxil >>>> > >>>> >>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
