Hi Ari, all
I can follow your reasoning. Various discussions have taken place about
this already.
Currently it's also a question of feasibility. Windows and macOS have
completely different dependency build chains / repositories.
Synchronizing those within their current completely separate frameworks
will be hard to achieve. (other OS'es come with their own set of
dependencies already, so we only can ignore them - or could build
complete packages including all deps in the future if we do have a
standard set).
I had a look at conda-forge in the past months to evaluate if it could
be considered fit for a reference build system. One precaution I had
from the start was the rolling release dependency hell we had
experienced with homebrew. It still needs to be proven if we get away
with pinning. The major roadblock I hit was already updating Qt. I tried
to lift it to the latest version with no success (and nobody else
succeeded either despite constant efforts in the last 4 months since
that -- because of CI limitations / build issues). If we would have a
proposal how to continue here (custom hosted CI with a responsible
person taking care of it?), I am open to explore this further.
In summary, the current process we have works good enough, with the
occasional troubles of varying dependency versions you mention. Changing
it would be risky but possible and it is a major effort for which we
currently lack a widely accepted plan, resources and/or (long-term
committed) volunteers to do so.
For your particular case here, since there are no standard versions of
dependencies, conda might be as good as any other guess. Or you go for
the installer of your main target platform instead.
Best regards
Matthias
On 8/2/20 8:49 AM, Ari Meyer wrote:
Obrigado, Alexandre, for all the details. :-) I guess I'm still
baffled by, "Conda-forge community always tries to use the most recent
version of all dependencies." It just seems like a "recipe" for
disaster -- just built on assumptions. Maybe you get lucky and
everything just happens to work. Or maybe you get slightly less lucky
and encounter breakages immediately, forcing you to fiddle with the
build until you find some combination that seems to work, if you can't
pin the versions. Or maybe you get unlucky and everything builds and
*seems* to work based on the extent of your testing, but your testing
may have important gaps.
In my previous contract I was using PyViz, which was rebranded as
HoloViz. As that set of libraries was immature and constantly in
flux, and we wanted to take advantage of new features/fixes, our
policy was to go with the latest releases. I wasn't comfortable with
this, esp. after enduring many problems, but we just dealt with bugs
and regressions as they came up. QGIS, though, is much more mature,
after more than a decade since 1.0 was released (2009 according to
Wiki). I assumed, therefore, that they wouldn't trust such a
hit-or-miss strategy with dependencies. But maybe I'm just being a
worry wart. ;-) In any case, I'm going to be hard-pressed to sell
conda to my team if the dependencies are out of sync with those of the
installers.
On Fri, Jul 31, 2020 at 8:49 AM Alexandre Neto <[email protected]
<mailto:[email protected]>> wrote:
Hi,
I will take this opportunity to give a brief explanation of how
conda / conda-forge works. At least what I was able to understand
so far.
Full disclaimer: I know very little about building and maintaining
packages and installers, including the QGIS ones, but I try to
trigger the update of QGIS versions on Conda-forge.
*Conda* - an open source package management system and environment
management system developed mainly by anaconda. It provides tools
to build, install and run packages, and setup environments. One
interesting feature is that, in conda, the same building recipe
can be used to build the packages for Windows, Mac and Linux.
Normally, to build a package, you use prebuild packages of its
dependencies which are available in conda online channels. This
doesn't mean that the same versions of the dependencies are used
for all platforms, as some dependency may not be available or
doesn't work for a specific platform.
*Conda-forge* - a community that maintains a big amount of conda
recipes for many open source packages, which are stored in github
repositories. Those are called feedstocks. Among them, there is a
qgis-feedstock and also feedstocks for almost all of
its dependencies. Collaboration to keep those recipes is normally
very welcome.
https://conda-forge.org/feedstocks/
https://github.com/conda-forge/qgis-feedstock
All conda-forge recipes (each feedstock github branch) are built
automatically using Azure Pipelines and uploaded to the
conda-forge channel in Anaconda cloud, making it easily deployed
for installation. QGIS has a master (stable) and a LTR branch.
*Anaconda cloud* is a repository of prebuilt packages, which is
organized in channels. Anyone can have a channel and upload their
locally built packages so that others can use them. If you search
for QGIS in Anaconda cloud you will find several channels with
several versions available.
Answering Ari question:
On Fri, Jul 31, 2020 at 6:15 AM Ari Meyer <[email protected]
<mailto:[email protected]>> wrote:
@Greg: I found that many of the exact dependencies were not
even available through the main channels. So if the QGIS devs
compile/build against such versions, I'm not sure there's an
easy way to even specify such a conda recipe, unless all those
dependencies are also made available somehow. I didn't expect
that those versions would not even be there with the others
for the various libraries. Could this imply that the library
developers don't want those versions to be used? Not sure.
I can only speak for Conda Forge and mainly QGIS feedstock. I
normally try to keep the recipes updated for every single new
version of QGIS and QGIS LTR, but I may forget to do so. Then,
sometimes the recipes don't work out of the box for a certain
patch release of QGIS, the conda-forge folks try to fix them, but
meanwhile, a new patch release is out, so we may just abandon that
version recipe and move on. I expect the same may happen with
other packages... There is no intention of excluding versions for
any reason.
Regarding the version of the dependencies. Conda-forge community
always tries to use the most recent version of all dependencies.
Sometimes that doesn't work, and one needs to pin the version of a
dependency. Also, my idea was to keep the LTR main dependencies
pinned during all its lifecycle, to avoid regressions.
*Cool things about QGIS on conda (and conda-forge):*
- One single transparent recipe to build QGIS in all three main
platforms;
- As users, to install one or several packages for a certain
workflow, you normally create an environment to isolate the
install, which allows having several versions of the same software
in parallel in your machine. Nevertheless, the packages shared by
different environments/QGIS versions are only downloaded once and
don't take more space in the disk. Symbolic links are used.
- Once set, you can share an environment with others, making it
perfect for collaborative development.
- Updating QGIS is like osfeo4w in all platforms, only the
necessary packages are downloads and install.
- Using Conda-forge infrastructure, you only need to update the
recipe, the build and upload to anaconda cloud are done in azure
pipelines "automagically".
- It's a great way to use QGIS PyQGIS and the command line tools
(the reason why I try to update QGIS versions in conda-forge)
*Caveats of conda-forge:*
- As QGIS project, we have no direct control over dependencies
used on the master version, as the community has the goal of using
the most recent dependency versions that is possible. The same is
not necessarily true for LTR, where I feel we would be allowed to
control and pin the dependencies
- Currently, there are some dependencies missing. For example,
QtWebkit, which limits QGIS functionality, including some nice
plugins like DataPlotly, and resource sharing plugin.
It seems to me that, if the QGIS project wanted to use conda as
their main building/packaging system, we would need to:
- Keep a fork of conda-forge recipes, and upload the result to a
dedicated channel, probably even outside anaconda cloud.
- We could use dependencies available on conda-forge or other
channels, but we would need to be ready to maintain abandoned
packages that greatly impact QGIS, like QtWebkit;
- Recreate the installers (at least Windows and Mac) to use conda
tools for downloading, installing, and updating packages.
Last bit, when I was working for Boundless, Larry Shaffer was
trying to do all this mentioned above, but then the company was
bought by Planet... Not sure what is the status of this work now.
https://github.com/osgeo-forge
Thanks, Sorry for the long post.
Alexandre Neto
_______________________________________________
QGIS-Developer mailing list
[email protected]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
QGIS-Developer mailing list
[email protected]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
