Thanks for explaining, Matthias. That's particularly concerning with Qt, as it's core to so many projects and prides itself on being cross-platform. I still think conda (using pinned versions) is worth trying.
Cheers! Ari On Sun, Aug 2, 2020 at 4:04 AM Matthias Kuhn <[email protected]> wrote: > Hi Ari, all > > I can follow your reasoning. Various discussions have taken place about > this already. > > Currently it's also a question of feasibility. Windows and macOS have > completely different dependency build chains / repositories. Synchronizing > those within their current completely separate frameworks will be hard to > achieve. (other OS'es come with their own set of dependencies already, so > we only can ignore them - or could build complete packages including all > deps in the future if we do have a standard set). > > I had a look at conda-forge in the past months to evaluate if it could be > considered fit for a reference build system. One precaution I had from the > start was the rolling release dependency hell we had experienced with > homebrew. It still needs to be proven if we get away with pinning. The > major roadblock I hit was already updating Qt. I tried to lift it to the > latest version with no success (and nobody else succeeded either despite > constant efforts in the last 4 months since that -- because of CI > limitations / build issues). If we would have a proposal how to continue > here (custom hosted CI with a responsible person taking care of it?), I am > open to explore this further. > > In summary, the current process we have works good enough, with the > occasional troubles of varying dependency versions you mention. Changing it > would be risky but possible and it is a major effort for which we currently > lack a widely accepted plan, resources and/or (long-term committed) > volunteers to do so. > > For your particular case here, since there are no standard versions of > dependencies, conda might be as good as any other guess. Or you go for the > installer of your main target platform instead. > > Best regards > > Matthias > > > On 8/2/20 8:49 AM, Ari Meyer wrote: > > Obrigado, Alexandre, for all the details. :-) I guess I'm still baffled > by, "Conda-forge community always tries to use the most recent version of > all dependencies." It just seems like a "recipe" for disaster -- just > built on assumptions. Maybe you get lucky and everything just happens to > work. Or maybe you get slightly less lucky and encounter breakages > immediately, forcing you to fiddle with the build until you find some > combination that seems to work, if you can't pin the versions. Or maybe > you get unlucky and everything builds and *seems* to work based on the > extent of your testing, but your testing may have important gaps. > > In my previous contract I was using PyViz, which was rebranded as > HoloViz. As that set of libraries was immature and constantly in flux, and > we wanted to take advantage of new features/fixes, our policy was to go > with the latest releases. I wasn't comfortable with this, esp. after > enduring many problems, but we just dealt with bugs and regressions as they > came up. QGIS, though, is much more mature, after more than a decade since > 1.0 was released (2009 according to Wiki). I assumed, therefore, that they > wouldn't trust such a hit-or-miss strategy with dependencies. But maybe > I'm just being a worry wart. ;-) In any case, I'm going to be hard-pressed > to sell conda to my team if the dependencies are out of sync with those of > the installers. > > On Fri, Jul 31, 2020 at 8:49 AM Alexandre Neto <[email protected]> > wrote: > >> Hi, >> >> I will take this opportunity to give a brief explanation of how conda / >> conda-forge works. At least what I was able to understand so far. >> >> Full disclaimer: I know very little about building and maintaining >> packages and installers, including the QGIS ones, but I try to trigger the >> update of QGIS versions on Conda-forge. >> >> *Conda* - an open source package management system and environment >> management system developed mainly by anaconda. It provides tools to build, >> install and run packages, and setup environments. One interesting feature >> is that, in conda, the same building recipe can be used to build the >> packages for Windows, Mac and Linux. Normally, to build a package, you use >> prebuild packages of its dependencies which are available in conda online >> channels. This doesn't mean that the same versions of the dependencies are >> used for all platforms, as some dependency may not be available or doesn't >> work for a specific platform. >> >> *Conda-forge* - a community that maintains a big amount of conda recipes >> for many open source packages, which are stored in github repositories. >> Those are called feedstocks. Among them, there is a qgis-feedstock and also >> feedstocks for almost all of its dependencies. Collaboration to keep those >> recipes is normally very welcome. >> >> https://conda-forge.org/feedstocks/ >> >> https://github.com/conda-forge/qgis-feedstock >> >> All conda-forge recipes (each feedstock github branch) are built >> automatically using Azure Pipelines and uploaded to the conda-forge channel >> in Anaconda cloud, making it easily deployed for installation. QGIS has a >> master (stable) and a LTR branch. >> >> *Anaconda cloud* is a repository of prebuilt packages, which is >> organized in channels. Anyone can have a channel and upload their locally >> built packages so that others can use them. If you search for QGIS in >> Anaconda cloud you will find several channels with several versions >> available. >> >> Answering Ari question: >> >> On Fri, Jul 31, 2020 at 6:15 AM Ari Meyer <[email protected]> wrote: >> >>> @Greg: I found that many of the exact dependencies were not even >>> available through the main channels. So if the QGIS devs compile/build >>> against such versions, I'm not sure there's an easy way to even specify >>> such a conda recipe, unless all those dependencies are also made available >>> somehow. I didn't expect that those versions would not even be there with >>> the others for the various libraries. Could this imply that the library >>> developers don't want those versions to be used? Not sure. >>> >> >> I can only speak for Conda Forge and mainly QGIS feedstock. I normally >> try to keep the recipes updated for every single new version of QGIS and >> QGIS LTR, but I may forget to do so. Then, sometimes the recipes don't work >> out of the box for a certain patch release of QGIS, the conda-forge folks >> try to fix them, but meanwhile, a new patch release is out, so we may just >> abandon that version recipe and move on. I expect the same may happen with >> other packages... There is no intention of excluding versions for any >> reason. >> >> Regarding the version of the dependencies. Conda-forge community always >> tries to use the most recent version of all dependencies. Sometimes that >> doesn't work, and one needs to pin the version of a dependency. Also, my >> idea was to keep the LTR main dependencies pinned during all its lifecycle, >> to avoid regressions. >> >> *Cool things about QGIS on conda (and conda-forge):* >> - One single transparent recipe to build QGIS in all three main platforms; >> - As users, to install one or several packages for a certain workflow, >> you normally create an environment to isolate the install, which allows >> having several versions of the same software in parallel in your machine. >> Nevertheless, the packages shared by different environments/QGIS versions >> are only downloaded once and don't take more space in the disk. Symbolic >> links are used. >> - Once set, you can share an environment with others, making it perfect >> for collaborative development. >> - Updating QGIS is like osfeo4w in all platforms, only the necessary >> packages are downloads and install. >> - Using Conda-forge infrastructure, you only need to update the recipe, >> the build and upload to anaconda cloud are done in azure pipelines >> "automagically". >> - It's a great way to use QGIS PyQGIS and the command line tools (the >> reason why I try to update QGIS versions in conda-forge) >> >> *Caveats of conda-forge:* >> - As QGIS project, we have no direct control over dependencies used on >> the master version, as the community has the goal of using the most recent >> dependency versions that is possible. The same is not necessarily true for >> LTR, where I feel we would be allowed to control and pin the dependencies >> - Currently, there are some dependencies missing. For example, QtWebkit, >> which limits QGIS functionality, including some nice plugins like >> DataPlotly, and resource sharing plugin. >> >> It seems to me that, if the QGIS project wanted to use conda as their >> main building/packaging system, we would need to: >> - Keep a fork of conda-forge recipes, and upload the result to a >> dedicated channel, probably even outside anaconda cloud. >> - We could use dependencies available on conda-forge or other channels, >> but we would need to be ready to maintain abandoned packages that greatly >> impact QGIS, like QtWebkit; >> - Recreate the installers (at least Windows and Mac) to use conda tools >> for downloading, installing, and updating packages. >> >> Last bit, when I was working for Boundless, Larry Shaffer was trying to >> do all this mentioned above, but then the company was bought by Planet... >> Not sure what is the status of this work now. >> >> https://github.com/osgeo-forge >> >> Thanks, Sorry for the long post. >> >> Alexandre Neto >> >> >> >> > _______________________________________________ > QGIS-Developer mailing [email protected] > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer > >
_______________________________________________ QGIS-Developer mailing list [email protected] List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
