Good to hear, Alexandre. Your efforts are much appreciated! Ari On Sun, Aug 2, 2020 at 5:13 PM Alexandre Neto <[email protected]> wrote:
> Hi, > > Following the discussion on QGIS dependencies on conda-forge, I had a chat > with one of conda-forge most active maintainers. > > It seems that the conda-forge community is aware of the dangers of > building only bleeding edge versions of the packages. There's a CFEP (same > as our QEP) to improve the situation. > > https://hackmd.io/N1hoJGJBSqGTFd83pxCyYA > > Nevertheless, regarding the QGIS packages, and because they are not > dependencies of any other packages, I was told that it would be ok to > create local pinning configuration files for both our stable and LTR > branches. This means we could have the control of the most important > dependencies. I will try to mimic osgeo4w for 3.16 onwards, and make sure > the pinning is not changed during the LTR life cycle. > > This does not fix the issue with no one maintaining the QtWebKit package, > but I don't think no one really worked actively on it. > > My intentions are not to replace our packaging system (poor me), that, > like Mathias said, works fine for now. I will just try to keep QGIS > available on conda, as it seems useful for some folks, and it doesn't > really take me too much time. > > Thanks, > > Alexandre Neto > > A domingo, 2/08/2020, 10:04, Matthias Kuhn <[email protected]> escreveu: > >> Hi Ari, all >> >> I can follow your reasoning. Various discussions have taken place about >> this already. >> >> Currently it's also a question of feasibility. Windows and macOS have >> completely different dependency build chains / repositories. Synchronizing >> those within their current completely separate frameworks will be hard to >> achieve. (other OS'es come with their own set of dependencies already, so >> we only can ignore them - or could build complete packages including all >> deps in the future if we do have a standard set). >> >> I had a look at conda-forge in the past months to evaluate if it could be >> considered fit for a reference build system. One precaution I had from the >> start was the rolling release dependency hell we had experienced with >> homebrew. It still needs to be proven if we get away with pinning. The >> major roadblock I hit was already updating Qt. I tried to lift it to the >> latest version with no success (and nobody else succeeded either despite >> constant efforts in the last 4 months since that -- because of CI >> limitations / build issues). If we would have a proposal how to continue >> here (custom hosted CI with a responsible person taking care of it?), I am >> open to explore this further. >> >> In summary, the current process we have works good enough, with the >> occasional troubles of varying dependency versions you mention. Changing it >> would be risky but possible and it is a major effort for which we currently >> lack a widely accepted plan, resources and/or (long-term committed) >> volunteers to do so. >> >> For your particular case here, since there are no standard versions of >> dependencies, conda might be as good as any other guess. Or you go for the >> installer of your main target platform instead. >> >> Best regards >> >> Matthias >> >> >> On 8/2/20 8:49 AM, Ari Meyer wrote: >> >> Obrigado, Alexandre, for all the details. :-) I guess I'm still baffled >> by, "Conda-forge community always tries to use the most recent version of >> all dependencies." It just seems like a "recipe" for disaster -- just >> built on assumptions. Maybe you get lucky and everything just happens to >> work. Or maybe you get slightly less lucky and encounter breakages >> immediately, forcing you to fiddle with the build until you find some >> combination that seems to work, if you can't pin the versions. Or maybe >> you get unlucky and everything builds and *seems* to work based on the >> extent of your testing, but your testing may have important gaps. >> >> In my previous contract I was using PyViz, which was rebranded as >> HoloViz. As that set of libraries was immature and constantly in flux, and >> we wanted to take advantage of new features/fixes, our policy was to go >> with the latest releases. I wasn't comfortable with this, esp. after >> enduring many problems, but we just dealt with bugs and regressions as they >> came up. QGIS, though, is much more mature, after more than a decade since >> 1.0 was released (2009 according to Wiki). I assumed, therefore, that they >> wouldn't trust such a hit-or-miss strategy with dependencies. But maybe >> I'm just being a worry wart. ;-) In any case, I'm going to be hard-pressed >> to sell conda to my team if the dependencies are out of sync with those of >> the installers. >> >> On Fri, Jul 31, 2020 at 8:49 AM Alexandre Neto <[email protected]> >> wrote: >> >>> Hi, >>> >>> I will take this opportunity to give a brief explanation of how conda / >>> conda-forge works. At least what I was able to understand so far. >>> >>> Full disclaimer: I know very little about building and maintaining >>> packages and installers, including the QGIS ones, but I try to trigger the >>> update of QGIS versions on Conda-forge. >>> >>> *Conda* - an open source package management system and environment >>> management system developed mainly by anaconda. It provides tools to build, >>> install and run packages, and setup environments. One interesting feature >>> is that, in conda, the same building recipe can be used to build the >>> packages for Windows, Mac and Linux. Normally, to build a package, you use >>> prebuild packages of its dependencies which are available in conda online >>> channels. This doesn't mean that the same versions of the dependencies are >>> used for all platforms, as some dependency may not be available or doesn't >>> work for a specific platform. >>> >>> *Conda-forge* - a community that maintains a big amount of conda >>> recipes for many open source packages, which are stored in github >>> repositories. Those are called feedstocks. Among them, there is a >>> qgis-feedstock and also feedstocks for almost all of its dependencies. >>> Collaboration to keep those recipes is normally very welcome. >>> >>> https://conda-forge.org/feedstocks/ >>> >>> https://github.com/conda-forge/qgis-feedstock >>> >>> All conda-forge recipes (each feedstock github branch) are built >>> automatically using Azure Pipelines and uploaded to the conda-forge channel >>> in Anaconda cloud, making it easily deployed for installation. QGIS has a >>> master (stable) and a LTR branch. >>> >>> *Anaconda cloud* is a repository of prebuilt packages, which is >>> organized in channels. Anyone can have a channel and upload their locally >>> built packages so that others can use them. If you search for QGIS in >>> Anaconda cloud you will find several channels with several versions >>> available. >>> >>> Answering Ari question: >>> >>> On Fri, Jul 31, 2020 at 6:15 AM Ari Meyer <[email protected]> wrote: >>> >>>> @Greg: I found that many of the exact dependencies were not even >>>> available through the main channels. So if the QGIS devs compile/build >>>> against such versions, I'm not sure there's an easy way to even specify >>>> such a conda recipe, unless all those dependencies are also made available >>>> somehow. I didn't expect that those versions would not even be there with >>>> the others for the various libraries. Could this imply that the library >>>> developers don't want those versions to be used? Not sure. >>>> >>> >>> I can only speak for Conda Forge and mainly QGIS feedstock. I normally >>> try to keep the recipes updated for every single new version of QGIS and >>> QGIS LTR, but I may forget to do so. Then, sometimes the recipes don't work >>> out of the box for a certain patch release of QGIS, the conda-forge folks >>> try to fix them, but meanwhile, a new patch release is out, so we may just >>> abandon that version recipe and move on. I expect the same may happen with >>> other packages... There is no intention of excluding versions for any >>> reason. >>> >>> Regarding the version of the dependencies. Conda-forge community always >>> tries to use the most recent version of all dependencies. Sometimes that >>> doesn't work, and one needs to pin the version of a dependency. Also, my >>> idea was to keep the LTR main dependencies pinned during all its lifecycle, >>> to avoid regressions. >>> >>> *Cool things about QGIS on conda (and conda-forge):* >>> - One single transparent recipe to build QGIS in all three main >>> platforms; >>> - As users, to install one or several packages for a certain workflow, >>> you normally create an environment to isolate the install, which allows >>> having several versions of the same software in parallel in your machine. >>> Nevertheless, the packages shared by different environments/QGIS versions >>> are only downloaded once and don't take more space in the disk. Symbolic >>> links are used. >>> - Once set, you can share an environment with others, making it perfect >>> for collaborative development. >>> - Updating QGIS is like osfeo4w in all platforms, only the necessary >>> packages are downloads and install. >>> - Using Conda-forge infrastructure, you only need to update the recipe, >>> the build and upload to anaconda cloud are done in azure pipelines >>> "automagically". >>> - It's a great way to use QGIS PyQGIS and the command line tools (the >>> reason why I try to update QGIS versions in conda-forge) >>> >>> *Caveats of conda-forge:* >>> - As QGIS project, we have no direct control over dependencies used on >>> the master version, as the community has the goal of using the most recent >>> dependency versions that is possible. The same is not necessarily true for >>> LTR, where I feel we would be allowed to control and pin the dependencies >>> - Currently, there are some dependencies missing. For example, QtWebkit, >>> which limits QGIS functionality, including some nice plugins like >>> DataPlotly, and resource sharing plugin. >>> >>> It seems to me that, if the QGIS project wanted to use conda as their >>> main building/packaging system, we would need to: >>> - Keep a fork of conda-forge recipes, and upload the result to a >>> dedicated channel, probably even outside anaconda cloud. >>> - We could use dependencies available on conda-forge or other channels, >>> but we would need to be ready to maintain abandoned packages that greatly >>> impact QGIS, like QtWebkit; >>> - Recreate the installers (at least Windows and Mac) to use conda tools >>> for downloading, installing, and updating packages. >>> >>> Last bit, when I was working for Boundless, Larry Shaffer was trying to >>> do all this mentioned above, but then the company was bought by Planet... >>> Not sure what is the status of this work now. >>> >>> https://github.com/osgeo-forge >>> >>> Thanks, Sorry for the long post. >>> >>> Alexandre Neto >>> >>> >>> >>> >> _______________________________________________ >> QGIS-Developer mailing [email protected] >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer >> >>
_______________________________________________ QGIS-Developer mailing list [email protected] List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
