Re: [QGIS-Developer] Some thought on LTR

[Even Rouault]

> And at this stage in the life of the LTR I just
> don't see the risks of regressions outweighing anything but
> crash/corruption fixes.

"Stage" is the keyword here. Number of software vendors have generally several 
stages during the lifetime of the product they support, generally a first one 
which is the polishing of the release where they can even add new features, 
another one where they only backport (critical) bug fixes, and a final one 
where they only backport security related bug fixes.

So, maybe we could have a 2 phase process:
- First 6 months: backport of bug fixes with the same criteria as we do 
currently
- Last 6 months: only critical bug fixes (crashes, data loss)

(or 9 months / 3 months, whatever...)

But if we're really in that mode, that should not only apply to QGIS but to 
its dependent libraries as well. Basically this would mean that for OSGeo4W, 
you should have 2 distinct sets for the dependencies: one frozen (or with very 
controlled changes) for the LTR, and a living one for the new versions.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS WFS3 Server temporal intervals

[Even Rouault]

On vendredi 18 octobre 2019 12:28:12 CEST Alessandro Pasotti wrote:
> Hi,
> 
> I was thinking about how to better implement this feature for QGIS server
> (but it does not need to be restricted to the server).
> 
> The WFS3 specs covers pretty much all use cases: you can have features with
> a single date/datetime temporal property and a set of date/datetime
> properties, the latter case can (not "must"!)  be used to define one (or
> many) date/datetime intervals.
> 
> Now, the specs say that it's up to the server to decide if only a single
> property has to be considered when querying or if multiple properties have
> to be combined together to create one (or many) intervals. But since we
> code the server it means that it's up to us to decide how to handle it.
> 
> Note that the conditions need to be AND(ed) and unless properties values
> are equal this does make sense only when multiple properties are combined
> to form intervals.
> 
> So, the question is how to model this in QGIS?
> 
> Since this information may spread over multiple fields, I think it should
> go into the QgsVectorLayer class (or some separate temporal settings
> companion class/struct).

Hi Alessandro,

Are you thinking to scenarios like a layer would have an "eventStart" and 
"eventEnd" timestamps and you would want them to be considered together when 
an incoming request with a datetime=request_begin/request_end interval 
arrives, so as to only select features whose eventStart <= request_end and 
eventEnd >= requestbegin ?

Your above proposal makes sense, but involves complication. I think it might 
be wise to wait a bit before designing this, beause filtering is going to be 
the main topic of the next OAPIF Hackathon in November (
https://www.opengeospatial.org/events/191105apisprint ), so an extension with 
more complex/complete filtering (probably similar to what FES offers) is 
likely to be created. So I can imagine that it would offer the possibility to 
explicitly filter on multiple datetime properties. Yet that wouldn't solve the 
issue with the datetime filtering capability of OAPIF core, but perhaps for 
core, we could adopt a simple solution, like use the first datetime field of 
the layer by default, unless the user (QGIS server admin) selects another one. 
And if more complex behaviour is desired, then OAPIF clients would use the 
filter extension to be able to combine filters on several fields.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS projects - can they be licensed and/or copyrighted ?

[Even Rouault]

On mercredi 30 octobre 2019 12:31:45 CET Bo Victor Thomsen wrote:
> Hi list -
> 
> During a university lecture about QGIS and licensing (GPL) I got a
> question about QGIS projects.
> 
> Can a project be copyrighted or licensed differently from QGIS ? We are
> not talking about plugins - they are always under the same license af
> QGIS, namely GPL.
> 
> But what about the QGIS projects ? Same license or copyrigt af QGIS
> itself. Or what ??

The GPL license mostly sets constraints about redistribution, not use. So 
whatever your produce with QGIS, data, projects, styles, etc you can decide 
freely what you do with it: public domain, open data, or whatever license you 
see fit (*)

This is covered by the official GPL FAQ:
https://www.gnu.org/licenses/gpl-faq.en.html#GPLOutput
https://www.gnu.org/licenses/gpl-faq.en.html#WhatCaseIsOutputGPL

Even

(*) if your projects deals with other people data, then the constraints of 
that data might have consequence on your project, but that doesn't come from 
QGIS itself.

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS projects - can they be licensed and/or copyrighted ?

[Even Rouault]

> > A very small nibble: What about python code embedded in the project ?
> 
> That is a good question where I don't know the answer. It might be
> subject to the GPL license - but not sure. I would be interested to hear
> other people's opinion on this particular case.

Huh, that becomes involved... In all logic, conveying code through a .py file 
or a .qgz doesn't change the fact that it is code, so I'd expect licensing 
rules to also apply to it. So if that Python code uses QGIS API, it is 
probably to be under the GPL (but that only affects those code snippets, not 
the rest of the project).

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Can't edot OGC API Features connection

[Even Rouault]

On vendredi 1 novembre 2019 11:49:11 CET Rahkonen Jukka (MML) wrote:
> Hi,
> 
> It seems to me that if I edit the settings of the OGC API Features
> connection the changes do not have any effect but I must delete and
> re-create the connection instead.

If you modify it from the Browser right ? I can also reproduce that too, with 
it WFS, or other datasources like WMS. Probably worth filing a bug about that 
if there isn't already one.
If you use the Data Source Manager, changes are properly taken into account.
> 
> Also, I see a message "The download limit has been reached. You may want to
> check the 'Only request features overlapping the view extent' option to be
> able to zoom in to fetch all data" but I can't see such option in the user
> interface.

It is available from the Data Source Manager. At the bottom of the right part 
of the window when you select the WFS/OAPIF category. 
I'm surprised you see this message:
- normally it should appear only if you define a maximum number of features to 
download
- and the "Only request features overlapping the view extent" is supposed to 
be the default, but maybe you unchecked it in a previous session.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] How to encode blobs in (Geo)JSON?

[Even Rouault]

On jeudi 7 novembre 2019 17:40:08 CET Alessandro Pasotti wrote:
> Hi,
> 
> I'm running into an issue with OAPIF and blob/binary fields, the
> QgsJsonExporter does not currently handle that kind of field and just
> convert them to string (whatever that means).
> 
> This is clearly wrong in case the blob does not contain ascii or UTF-8, I
> think the way to go might be to base64 the blob but I'd like to know if
> anyone has a better idea.

Initially, I was going to suggest perhaps to wrap it inside an object with an 
explicit base64 member

like:

"my_binary_property": {
"base64": ""
}

But after looking at
https://json-schema.org/latest/json-schema-validation.html#rfc.section.8.3
https://json-schema.org/latest/json-schema-validation.html#rfc.section.8.6

perhaps just putting it at top level ( "my_binary_property": 
"base64contenthere" } is fine as you can tag this string property with 
"contentEncoding": "base64"

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] How to encode blobs in (Geo)JSON?

[Even Rouault]

> Are you talking about the schema ?

yes,"contentEncoding": "base64" would be for the JSON Schema

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] How to encode blobs in (Geo)JSON?

[Even Rouault]

> If I'm not mistaken, we are still in the design phase for the schema 

In OAPIF ? Well, not really. rel=describedBy is a possibility already offered 
by the core.
Used by a few implementations:
- JSON schema for:
http://databio.spacebel.be/eo-features/collections/datasets
- XML schema for:
   
http://ows.geo-solutions.it/geoserver/ne/ogc/features/collections/countries50m
   
http://www.pvretano.com/cubewerx/cubeserv/default/wfs/3.0.0/framework/collections/UNINCORPORATED_PL

What is in discussion is a summary endpoint to present queryable, sortable, 
etc.. properties:
https://github.com/opengeospatial/ogcapi-features/pull/287

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] How to encode blobs in (Geo)JSON?

[Even Rouault]

> Thanks for the links,  I suppose that the quickest way for QGIS Server will
> be to forward to WFS XML schema like geoserver does.
> 
> JSON schema is waaay harder to implement from scratches.

Yes, it is non trivial. That said for a flat structure, you probably might 
have most of it more or less predefined, and then insert the field definition 
at the right place.

> 
> Will GDAL be able to use describedBy in the future?

pretty much the outcome of the hackaton for me :
https://github.com/OSGeo/gdal/pull/1997


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Problems loading and using GDALinfo on hdf5 files (only on Windows)

[Even Rouault]

> So this means it must be a windows \
> OSGeo4W thing, no?

Yes, Windows thing. Already reported at
https://github.com/OSGeo/gdal/issues/1428

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Delaying 3.10.1?

[Even Rouault]

On vendredi 22 novembre 2019 08:43:23 CET Jürgen E. Fischer wrote:
> Hi Nyall,
> 
> On Fri, 22. Nov 2019 at 17:17:07 +1000, Nyall Dawson wrote:
> > But I'd love a couple of days to keep digging, if that's at all
> > possible
> Black friday?  Ok.

Trying to reproduce on my side in isolated way with use of only PROJ-only API

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] LTR management [was Re: Delaying 3.10.1?]

[Even Rouault]

> QGIS LTR in Windows has also switched from proj 5.x and gdal 2.x to
> proj 6.x and gdal 3.x which has resulted in some new bugs.

I see 2 different situations:

- 3.10 was intended to work with GDAL 3 & PROJ 6 (work started with 3.8 if I 
remember). OSGeo4W received a late upgrade to them because GRASS 7.8 wasn't 
ready yet for them (no offense to the GRASS team !), but devs have been able 
to work against GDAL 3 & PROJ 6, so that wasn't unknown territory. Probably 
that 3.10.0 could have been delayed while OSGeo4W hadn't switched to GDAL 3/
PROJ 6, but it wasn't clear when GRASS would be out. So overall how the 
situation was handled doesn't seem that bad for a release exercising new major 
dependencies.

- I do agree that the switch to GDAL 3 & PROJ 6 for a near end-of-life 3.4 LTR 
was unfortunate (but more than the timing, that does not seem appropriate for 
a release which was not designed originally to work with them) and I felt -0 
on that.

> It will be additional work for package managers and more resources, but it
> will be good to have some guidelines on LTR and PRs dependencies for
> packaging.

Each LTR should ideally have its own set of dependencies, allowing controlled 
changes of them when needed (adopting a new bugfix version might be OK), and 
allowing the current version to use more aggressively new versions of the 
dependencies. That said, I'm not a packager, and don't know the effort & cost 
associated, but that's clearly an area where the project could/should invest. 
Similar situation is likely to happen again in the future.

I'm also wondering if there shouldn't be a vote, from developers, PSC, bug 
triagers, testers or probably a mix of them forming a representative body, to 
adjudicate:
- dependency upgrades
- decision of releasing a new version (most other OSGeo projects I'm familiar 
with rely on a PSC formal vote to issue releases (*)).

Or one might consider a mixed approach to have a good compromise of agility vs 
tighter control:
- use time-based approach, as done currently, for non-LTR versions.
- formally approve the release of LTR versions, and important engineering 
decisions that affect them, as there are the ones with the most user exposure. 
Sometime ago I also suggested to possibly consider 2 phases in a LTR life-
cycle: first half where quite "aggressive" backporting is accepted (if it 
doesn't break API, etc..), second half where a much more conservative approach 
is taken. It is rather obvious that a .0, .1 needs more stabilization than a .
12 or a .13

Just food for thought :-)

Even

(*) even bugfix ones, which is admidetly sometimes a bit overkill

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] LTR management [was Re: Delaying 3.10.1?]

[Even Rouault]

On samedi 23 novembre 2019 12:23:12 CET Paolo Cavallini wrote:
> Hi Tim,
> 
> Il 23/11/19 11:22, Tim Sutton ha scritto:
> > For windows at least we could ‘freeze’ 3.4.x by pointing them to use the
> > standalone installers but then we have an LTR that is not getting bug
> > fixes. As I read Jürgen’s roadmap [1] we still have 4 more bug fix
> > releases of 3.4 before 3.10 becomes LTR, which is quite a long time to
> > have our ’stable’ version out there with no bug fixes.
> 
> yes, that's what I meant. It's a kind of bad hack, so as Bas also
> pointed out the proper solution is just to go ahead and fix the bugs.
> I feel bad about people wasting their time to fix this compatibility bug
> that will be useful just for a relatively short period, however.

I'm pretty sure there will be problems with CRS handling with QGIS 3.4 + 
GDAL3/PROJ6 that will *not* be fixable, due to intended changes of behaviour 
in GDAL3/PROJ6, particularly the export of CRS definition to PROJ strings, 
that QGIS 3.4 still uses it, that is deprecated now and works only in a 
degraded mode regarding export datum information (retrospectively, I should 
probably have completely remove exportToProj4() to make that obvious). QGIS 
3.4 to work properly with GDAL3/PROJ6 would require backporting all the PROJ6 
specific work started with 3.8, which is another big no no.

For example just seing that srs.db in 3.4.13 package has for OSGB36:
parameters (String) = +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 
+x_0=40 +y_0=-10 +ellps=airy +units=m +no_defs
where GDAL 2 based versions have an extra 
+towgs84=446.448,-125.157,542.06,0.15,0.247,0.842,-20.489 which is 
essential to make things work properly in a PROJ4 string oriented workflow as 
used by 3.4. I'v just tried with a GPKG in WGS84 with a single point in 
England, and a conversion of it to OSGB36 (with ogr2ogr of GDAL 3 or GDAL 2): 
when loaded in 3.4.13 based on GDAL 3, the points are distant of 130 m (which 
is the missing datum shift). Whereas when loaded with 3.4.13 based on GDAL 2 
(or 3.10 with GDAL 3), they overlay properly.

So people dealing with datums != WGS84 should better stick with 3.4.12 on 
Windows currently, if wanting to stay in the 3.4 series.

QGIS 3.4 combined with GDAL3/PROJ6 is a *design* bug, not a regular bug that 
can be fixed.

I don't see why QGIS 3.4 on OSGeo4W using a GDAL 2.4.3 + PROJ 5.2 
wouldn't be technically achievable. Isn't there a gdal and gdal-dev package in 
it, several QGIS variants already ? So why not a gdal2_4 ? Well, I guess that 
doing that now would require creating a grass package that depends on gdal2_4 
etc. So yes, that might be painful to do at that stage. Dunno...

But for the future, I can imagine we could have:
qgis3_10 depending on gdal3 (or possibly even gdal3_0 ?) and proj6 and 
qt_whatever_we_use_currently
And qgis3_12 could decide to upgrade one of its dependencies without impacting 
qgis3_10.

Whatever a hot fix, or a long term solution (pun intended :-)) along the above 
lines or better is chosen, it would seem to me as a very appropriate use of 
funds of the project to investigate what is possible.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] LTR management [was Re: Delaying 3.10.1?]

[Even Rouault]

On samedi 23 novembre 2019 17:44:15 CET Jürgen E. Fischer wrote:
> Hi Even,
> 
> On Sat, 23. Nov 2019 at 17:36:52 +0100, Even Rouault wrote:
> > But for the future, I can imagine we could have:
> > qgis3_10 depending on gdal3 (or possibly even gdal3_0 ?) and proj6 and
> > qt_whatever_we_use_currently
> > And qgis3_12 could decide to upgrade one of its dependencies without
> > impacting qgis3_10.
> 
> Yes, that's the route I might have followed if 3.4 didn't (claim to) support
> PROJ 6.

Actually, I *think* that QGIS 3.4 + GDAL 2.4 + PROJ 6 would probably work OK, 
because QGIS 3.4 would still use the legacy proj_api.h. The issue was more 
related to the upgrade to GDAL 3 changing how PROJ.4 strings are exported, and 
thus in practice requiring QGIS to be updated to use the new proj.h 6 API.

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] LTR management [was Re: Delaying 3.10.1?]

[Even Rouault]

> I'd also like to extend Tim's thanks here and publicly state my
> appreciation to Jürgen for an extremely difficult, thankless job which
> you've done SO well, SO consistently and for SO long.

I "third" this appreciation !

> Regarding 3.4: One option we should explore is just freezing the
> Windows releases at 3.4.13

That would rather be 3.4.12 to have the last built based on GDAL 2.4 + PROJ 5, 
no ? At least for the standalone installer

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Delaying 3.10.1?

[Even Rouault]

> Ok, thanks to a **marathon** debugging effort from Even, we have a fix
> inbound -- https://github.com/OSGeo/PROJ/pull/1746

Now merged in PROJ master.
Cherry-picked in PROJ 6.2 branch per
https://github.com/OSGeo/PROJ/commit/bce4b158ab5f7d146de8e8fc98df4612dc8c2c9e

> 
> Looks like we'll need to cherry-pick the PROJ fix into the osgeo4w
> build in order to resolve the blocking issue in QGIS.

I would also suggest applying commit
https://github.com/OSGeo/gdal/commit/bec5fbd4fbf5d69a8386b920f0dd6d951eceb117
(which applies to GDAL 3.0 branch)
on top of the 3.0.2 release
This fixes a subtle multi-threading bug that causes memory corruption and
could likely be the source of crashes at random times

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] Fwd: Call for review on RFC 4 text: Remote access to grids and GeoTIFF grids

[Even Rouault]

Hi,

the below can be of interest for QGIS in the future.

Particularly the part that describes a capability to access grids stored 
remotely:
https://github.com/rouault/PROJ/blob/rfc4_remote_and_geotiff_grid/docs/source/community/rfc/rfc-4.rst#network-access-to-grids

and more specifically I'd be interested in hearing if the proposed callback 
mechanism
intended to make QGIS able to plug its networking mechanisms into PROJ looks 
reasonable.

Even

--  Forwarded Message  --

Subject: Call for review on RFC 4 text: Remote access to grids and GeoTIFF grids
Date: lundi 25 novembre 2019, 13:46:37 CET
From: Even Rouault 
To: p...@lists.osgeo.org

Hi,

As announced last week, you'll find a RFC describing two new capabilities,
remote access to grids and a GeoTIFF-based format for grids, in
https://github.com/OSGeo/PROJ/pull/1747

You can get an (almost correct) preview of it by going through
https://github.com/rouault/PROJ/blob/rfc4_remote_and_geotiff_grid/docs/source/community/rfc/rfc-4.rst
(some links will not work in this preview: expected given that this rendering 
doesn't go through Sphinx)

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-
-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] LTR management [was Re: Delaying 3.10.1?]

[Even Rouault]

> I think the issues are deeper then the crashes/projection failures
> fixed by the GDAL/proj cherry-picked commits.

Yes, actually QGIS 3.4 should not be affected by the PROJ fix, because it  
uses the old pj_transform() API with doesn't trigger that code path at all. 
But it *is* affected by exportToProj4() no longer returning +datum or +towgs84 
in cases where it used to be, which basically makes working with anything != 
WGS 84 fundamentaly broken. The only "fix" would be to backport the fully 
fledged PROJ 6 support of 3.10 which is obviously unreasonable to do in 3.4

> think we SHOULD drop
> Windows LTR support early rather than releasing a 3.4 build based on
> proj6/gdal3.

+1

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] LTR management [was Re: Delaying 3.10.1?]

[Even Rouault]

Jürgen,

> I've been "playing" with appveyor (doesn't work - hits the 1h timeout),
> azure-pipelines (which kind of works) and github workflows (which does not
> yet work) to implement the Windows CI many people are keen on.  Progress is
> very slow - every single step takes ages…   A successful run takes more
> than 2-3h on azure-pipelines…

Did you try to contact azure-pipeline support ? They might be receptive to a 
request to increase both the CPU count and disk space. Especially if you say 
that QGIS is a cool open source project where people can display... Bing Maps 
!

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] [gdal-dev] ERROR: Too many connections: max 64

[Even Rouault]

On jeudi 28 novembre 2019 20:19:10 CET Shane Carey wrote:
> Hi,
> 
> I'm doing some processing with Pyqgis and after processing, the file
> remains locked and eventually I get:
> ERROR: Too many connections: max 64

All I can say is that this error messages comes from Spatialite that can only 
handle 64 connections in a process at most. I did a fix (don't remember when, 
but must be included in QGIS 3.4) in the QGIS OGR provider to be able to open 
GeoPackage datasets (since the OGR GeoPackage driver enables Spatialite on 
them) that has many layers (so that they share the same connection). Not sure 
about the specificities of your use case.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS & PROJ 6 & SAGA

[Even Rouault]

William,

> I must then build and include both PROJ 6 and PROJ.4 in the QGIS package.

Caution...

> There is no option to build SAGA without PROJ, but I could hack the
> makefiles to skip it.  Or I could build it with the old PROJ and just not
> include those modules in the QGIS package.

From https://packages.debian.org/sid/saga , I see that SAGA has a dependency 
to both GDAL and PROJ. If your GDAL is a GDAL 3, it must be built against PROJ 
6. If you build SAGA against GDAL 3 and PROJ 4 or 5, then SAGA will 
effectively link against 2 different PROJ versions. This is a perfect recipee 
for potential runtime crashes in SAGA depending on how lucky/unlucky you are 
regarding the order into which libraries and symbols are loaded and resolved.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] BWTA2017 grid support in QGIS >= 3.10

[Even Rouault]

> @Jürgen Fischer: is there a change that you could "cherry-pick" the
> corresponding pull requests and patch them into the existing PROJ release
> used by osgeo4w for the upcoming pointrelease?

Just wants to note that a PROJ 6.3.0 release, with the
accompanying proj-datumgrid- backages, is scheduled for the beginning of 
January.

Even


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] osgeo4w's qgis-dev package broken, proj_7_0.dll missing

[Even Rouault]

> > Great work!!

+1 !

> 
> Thanks.  GDAL 2 and prøj 5 are now separate packages - prøj 5 being built
> statically and shipping with it's own files, although it's actually using
> the files from prøj 6 at runtime (as that's where also the grids are) - but
> those are identical except for ITRF2005 where +ds=-0.8 is instead of
> +ds=-0.08 - and I guess that's a fix).

Yes, a unfortunate Unicode minus sign that has been corrected to ASCII minus 
sign. Definitely a fix that also applies to PROJ 5. Anyway that only affects 
people playing manually with +init=ITRF2005, which I suspect are only happy 
fews.

Just want to warn in advance that a potential future PROJ 7 that would use 
grids in GeoTIFF format instead of GTX/NTv2 is something likely, and you will 
*not* want PROJ 5 to use such grids... But at that time, QGIS 3.4 will be 
retired hopefully.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] OGC API Features provider

[Even Rouault]

On mardi 10 décembre 2019 20:36:22 CET Stefan Keller wrote:
> Hi,
> 
> As I already talked about elsewhere I'm implementing a "minimal OGC-API
> Features Server".
> 
> Now I'd like to test this service with QGIS as client. And I also
> tested QGIS with the "kataster" service [1] the mentioned in the docs
> [2].
> Unfortunately I count not get both services to work (display) in QGIS.
> 
> * Our own service is being asked by QGIS with bbox and "limit=10&" any
> time a zoom or pan occurs. But QGIS displays just 10 which are not
> replaced by others whatever I do with QGIS.

Perhaps your service doesn't really return true unique ids ?

> * The kataster service asks about the CRS, then reports a large number
> of features (which means there must be a different "limit="), and
> finally does not answer requests from QGIS anymore.

Works for me.

> 
> Question 1: Any idea what QGIS makes it's OGC-API-Features reader to
> add limit=10?

Because that's the page size it detects from your service, or more presumably 
the default value if your service doesn't implement /api properly. You can 
tune it in the "WFS option" group of the "Modify WFS connection" dialog

> 
> Question 2: Does anybody know about any another OGC-API Features
> service we could use to test?

See list in comment of https://github.com/qgis/QGIS/pull/32262

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] OGC API Features provider

[Even Rouault]

> Finally, did I understood you correctly regarding the QGIS client options:
> 
> At 10. Dec. 2019 20:48 Even Rouault wrote:
> > > Question 1: Any idea what QGIS makes it's OGC-API-Features reader to
> > > add limit=10?
> > 
> > Because that's the page size it detects from your service, or more
> > presumably the default value if your service doesn't implement /api
> > properly. You can tune it in the "WFS option" group of the "Modify WFS
> > connection" dialog
> I've added the WFS3 (alias OGC API-F) service using QGIS menu item
> "Add Vector Layer..." in menu "Layer".

No, this way you'll use the OGR WFS3/OAPIF driver

I mean the "Add WFS Layer" entry (which should be renamed by the way to "Add 
WFS / OGC API - Features Layer") which will make you go to the "WFS / OGC API 
- Features" tab of Data Source Manager. There you will use the native QGIS 
provider (if using QGIS master)

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] What to do about GDAL and PROJ for 3.10.2?

[Even Rouault]

> Any news from GDAL team?

I was floating the idea of having a GDAL 3.0.3 release somewhere in January, 
but that's dependent on a volunteer taking on that. Someone was willing to 
give his first try at releasing GDAL, but I'm not sure if he's still 
available, and if that can be done before Jan 17th. I'll check back.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] What to do about GDAL and PROJ for 3.10.2?

[Even Rouault]

On jeudi 19 décembre 2019 13:09:38 CET Even Rouault wrote:
> > Any news from GDAL team?
> 
> I was floating the idea of having a GDAL 3.0.3 release somewhere in January,
> but that's dependent on a volunteer taking on that. Someone was willing to
> give his first try at releasing GDAL, but I'm not sure if he's still
> available, and if that can be done before Jan 17th. I'll check back.

OK so the volunteer is still on. We'll try to begin the release procedure 
earlier than usual, so he has time to warm up to the exercise. Hopefully we 
should have a 3.0.3 release officially approved for Jan 17th (but no guarantee 
as stated in the license text :-))

In the meantime would be good if people could confirm that
https://github.com/OSGeo/gdal/tree/release/3.0  is in good shape for QGIS 
needs (I believe I've backported all the needed fixes, but wouln't hurt to get 
some confirmation)

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] What to do about GDAL and PROJ for 3.10.2?

[Even Rouault]

GDAL 3.0.3 has just been released.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Environmental policy

[Even Rouault]

Regarding

> prohibit the use of QGIS to the fossil-fuels industry.

and

> ...may I suggest to add GLOBAL PEACE & WAR issues to this QGIS
> discussion on environmental questions ?

That's not a good idea IHMO. Don't mix licensing & ethics. Free software 
licenses explicitly don't discriminate against fields of endehavor (see clause 
6 of https://opensource.org/osd-annotated ), and adding discrimination is 
currently generally considered as being a bad idea and likely not enforcable 
in most juridictions, although they are debates around attempts at having 
ethical open source licensing
(https://medium.com/@gmcgath/the-futility-of-the-hippocratic-license-b6a9ad981dec).

Things can get really complex. For example, regarding prohibiting the use to 
the fossil-fuels industry, what about car makers: should they be considered as 
being part of the fossil-fuel industry, and thus prohibited from using the 
software and thus using proprietary mapping solution instead, etc etc. (Open 
Source) licensing is (or is perceived to be) a sufficiently complex topic 
already to not complicate it further.

Personal note: my contributions to FOSS4G related software all started because 
I was able to use them in the context of working in the defense industry in my 
past life: is it about peace or war ? both, depending from the point of view.

I should also remind, that for good or worse, the EPSG database that is 
critical to proper CRS management in our software is mostly maintained by 
IOGP.

The world is complex.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] FYI: Qt LTS Releases To Be Restricted To Commercial Customers

[Even Rouault]

Hi,

Just read this:
https://www.phoronix.com/scan.php?page=news_item&px=Qt-Going-More-Commercial

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] FYI: Qt LTS Releases To Be Restricted To Commercial Customers

[Even Rouault]

On lundi 27 janvier 2020 17:35:22 CET Paolo Cavallini wrote:
> Thanks Even. As I read it, this essentially means we are more or less
> forced to use the latest version, not the LTS (or to stick to an older
> one). This could mean more bugs.

QGIS sources aren't bound to a particular QT version. Currently the requirement 
is just >= 5.9
So pretty much depends on what binary distributions actually ship.

I've tried to follow the discussion on the thread announcing it at
https://lists.qt-project.org/pipermail/development/2020-January/thread.html#38316
but it is still unclear if QT Company will publish the source of their LTS 
branches or not.
At the very least, bugfixes that apply to the latest -dev branch, and older 
branches,
will be available, so people can *potentially* backport them (pending extra 
burden for
FOSS QT, typically Linux ones, distribution maintainers to scrutanize what must 
be
backported, which according to [1] would be very hard to do). If they don't 
publish the
source of the LTS branches, and a bug only affects a LTS branch and not the 
latest -dev branch, then
non-commercial customers will not benefit from it.

A lot of speculations in the above thoughts, and from a outsider of the working 
of the QT
project. If QT Company has the ultimate say to which branches are maintained or 
not,
or if other companies might jump in to bring back public LTS support...

Anyway, somewhat related to the above considerations, it looks like as far as 
OSGeo4W is
concerned, there's a QT 5.11.3 release with security fixes (cf [2]) whereas 
5.11.2 is shipped now.
But as the 5.11 branch is no maintained anymore, it is
well possible that 5.11.3 contains know issues, including security related ones,
fixed in later versions.
The 5.12 series is (should we say "was" now ?) announced as the LTS one with 
support until
5.12.2021 according to [3], so could be (have been?) a good candidate to track.
Perhaps some upgrade policy should be defined (only makes sense if there's 
manpower to
implement it of course)

Even

[1] https://lists.qt-project.org/pipermail/development/2020-January/038388.html
[2] 
https://www.qt.io/blog/2018/12/04/qt-5-11-3-released-important-security-updates
[3] https://en.wikipedia.org/wiki/Qt_version_history

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Potential vulnerabilities

[Even Rouault]

Nadia,

Thanks for investigating QGIS server security. However, I would expect a
vulnerability report to go a bit beyond than just using a generic security
scanner that can have false positives, especially here as all components
involved are open source so it is possible to look at the code, instrument it 
etc..
So a report should point to the exact line of code where the vulnerability
is triggered and/or provide an exploit.

For the long GET request, this is very very unlikely to be a buffer overflow.

Considering that the following is a valid request:
https://www.cybertest.it/gis/index.php/lizmap/service/?REQUEST=GetCapabilities&SERVICE=WMS&VERSION=1.3.0&project=demogis&repository=demogis

And the same but with just FOO instead of WMS for the value of SERVICE leads to 
the 500 error:
https://www.cybertest.it/gis/index.php/lizmap/service/?REQUEST=GetCapabilities&SERVICE=FOO&VERSION=1.3.0&project=demogis&repository=demogis

Looking at the error message, a bit of googling shows that it comes from LizMap
source code, not QGIS server:
https://github.com/3liz/lizmap-web-client/blob/master/lib/jelix/core/response/error.en_US.php

Furthermore Jelix is a PHP component, so not native code, hence buffer overflow
vulnerabilities leading to arbitrary code execution aren't relevant here 
(unless you'd
trigger a vulnerability of the PHP executable itself!)

I haven't look at the other things reported, but they are likely to be
LizMap specific rather than QGIS-server, unless otherwise proven.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Potential vulnerabilities

[Even Rouault]

> For example, is it possible to compromise QGIS Desktop via a
> opening/connecting to a compromised shapefile/Geopackage/web-service/CSV
> etc etc? I have no idea, but it'd definitely be a useful thing to
> investigate.

For file formats, part of the security/insecurity would fall on GDAL (and 
underlying libraries). GDAL has been integrated with oss-fuzz [1] since 3 
years and we have fixed a big number of issues raised by it (not all strictly 
security related).

That said, that doesn't test the pure QGIS side of things, since there could 
be issues in QGIS provider code, or in non-GDAL code paths (text delimited 
provider, web service providers). One technical issue with oss-fuzz is that it 
requires statically linked binaries, due to constraints how they deploy the 
binaries in their stress-testing cloud.

An alternative would be to use AFL ([2]) (which is one of the backends used by 
oss-fuzz), which doesn't have this statically linked binary requirement.

On the client side, testing the security of web service providers would be 
pretty challenging since you'd have to simulate potentially hostile servers 
and look at how the client reacts to those hostile responses.

For the server side, you could imagine having a ossfuzz/afl integration where 
what would be fuzzed would be the request sent to the server.

Even


[1] https://github.com/google/oss-fuzz
[2] https://github.com/google/AFL

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Proj 7 auto grid download

[Even Rouault]

On vendredi 20 mars 2020 19:26:03 CET Jeremy Palmer wrote:
> > On Fri, 20. Mar 2020 at 08:12:35 +0100, Andreas Neumann wrote:
> > > Isn't that potentially a quite large amount of data? Most QGIS users
> > 
> > that I
> > 
> > > know mainly work in one country or region and only need one gridshift
> > 
> > file.
> > 
> > > Of course there are exceptions.
> 
> Why not ship with the same grids that come with proj-datumgrid

I'd suggest to reavaluate that. proj-datumgrid grew organically, without a 
particular thinking of which grids were essential, and which were not. Maybe 
looking at the countries where most QGIS users live could be a way of taking 
an informed decision. I guess the UK grid could be relevant for example in a 
base package.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] [Qgis-psc] Position on Qt wrt The QT Company announcements

[Even Rouault]

On jeudi 9 avril 2020 21:30:02 CEST Martin Dobias wrote:
> Hi Vincent
> 
> On Thu, Apr 9, 2020, 18:32 Vincent Picavet (ml) 
> 
> wrote:
> > Hi all, PSC,
> > 
> > Olaf Schmidt-Wishhöfer from KDE project has made a statement yesterday
> > about a
> > really concerning situation regarding the OpenSource state of Qt.
> > 
> > https://mail.kde.org/pipermail/kde-community/2020q2/006098.html
> 
> The Qt Company has published a short blog post today saying that those
> discussions do not reflect their views and plans:
> https://www.qt.io/blog/qt-and-open-source
> 
> So maybe better to wait for some clarifications to make sure we are not
> rushing to spread claims that may not be correct...?

That's true, but it seems The QT Company has been lately testing the open 
source side of the 
QT community, with this recent event and the announcement in January to keep 
the LTS 
branches closed for 12 months, to apparently try getting more concessions in 
the contract 
with the KDE Free Qt foundation.
And even in their above correction post, they remain super vague and don't 
answer the 
points that would be wrong in Olaf Schmidt-Wishhöfer post

I doubt that the QT company would decide to go for the plan of delaying the 
open source 
version by 12 months, as the consequence (the last version of QT being 
potentially released 
as BSD) could actually quite harm their own business by allowing other 
commercial forks!

But whatever the outcome of the apparently cool discussions within the board of 
the KDE 
Free Qt foundation between the KDE e.v and QT Company representatives, I don't 
think a 
statement of support from QGIS.org to the open source side of the QT project 
would hurt.

As far as which body to officially support, this is a bit difficult. As the 
board of the KDE Free 
Qt foundation is made of 2 representatives from KDE e.V and 2 from The QT 
Company, it 
seems difficult to imagine that it would continue to exist as such, or be still 
relevant, in the 
event The QT company would execute their 12-month-delay plan. And before 
financially 
supporting the KDE Free Qt foundation or whatever other body would represent 
best the 
interests of a FOSS QT (I guess a new body gathering together KDE, KDAB and all 
other 
parties would be more relevant in the event a FOSS QT fork would be needed), we 
should 
probably have a look at its current finances/budget (from a quick search, 
couldn't find one 
regarding KDE Free Qt foundation, apart from the 200 000 KRO founding capital 
mentionned 
in their status [1])

In the discussion thread, I see a message from the KDE e.V vice president ([2]) 
opening 
discussion lines.

Even

[1] https://kde.org/community/whatiskde/KDEFreeQt_Statutes_09_final.pdf
[2] https://mail.kde.org/pipermail/kde-community/2020q2/006107.html

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] proj7 6 src db?

[Even Rouault]

Richard,

> 
> Probably not important, but I get the following output when running my
> build:
> 
> $ ninja install
> [0/1] Re-running CMake...
> -- QGIS version: 3.13.0 Master (31300)
> ...
> -- Found Proj: /usr/lib/x86_64-linux-gnu/libproj.so version 7 (7.0.0)
> ...
> -- Using PROJ 6 srs database.

To be understood as PROJ >= 6.

See
https://github.com/qgis/QGIS/blob/6a7719f9befbc68ed168c895a6d4e25a629feb0d/
resources/CMakeLists.txt#L47:

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] GeoPackage - where are we -where do we go

[Even Rouault]

> * The fid requirement
> 
>I sometimes want my features to be identified by uuids or others.
> They also tend to accumulate if derived datasets are created (through
> processing etc). If I need some pseudo stable primary key there is a
> rowid builtin into sqlite, we don't need a second one.
> 
>Possible mitigation: alter the ogr implementation. possibly alter the
> standard (required?)

I think the main issue here is that we expose the FID as a normal column in 
QGIS. This is a 
QGIS only behaviour, that could be easily removed.
Keeping a integer primary key column is indeed a requirement of the standard, 
and can be 
useful in circumstances where you run VACUUM on the DB, which would alter rowid 
in case of 
deleted features and possibly confuse QGIS identification by feature it if that 
happened after 
a layer has been loaded (but that's a bit of a edge scenario).

> 
> * The modification on r/o open
> 
>Has caused too much pain on git.
> 
>Possible mitigation: a) switch to journal mode=delete (not an easy
> option because of https://issues.qgis.org/issues/15351) 

> b) only switch
> to wal mode when layers are put into edit mode (I have strong doubts
> this is a safe thing to do)

That should be investigated/tested, but indeed it's not compleletely obvious 
that a 
connection that has been opened in the default journal mode=delete will "see" 
that it has 
been turned to WAL by another. I believe I looked in the SQLite doc about that 
scenario, but 
didn't find anything.

> 
> * The network share freeze
> 
>Our default file should play nicely with (windows) network shares.
> It's clear to everyone that we can't expect concurrent writes. But it
> should "just work" for concurrent read by many.
> 
>Possible mitigation: switch to journal mode=delete for network shares
> (we are looking into this)

I don't think journal mode=delete wil prevent SQLite3 from creating locks (or 
trying to do), to 
detect potential writes. I'm not sure if that can result in deadlock scenarios 
if they are 
network issues. One possibility to avoid all (deal)lock issues is to use the 
SQLITE_USE_OGR_VFS=YES env variable that will use GDAL I/O layer instead of 
SQLite3 built-
in one. The side effect of this is that as GDAL I/O layer doesn't implement 
locking, no locking 
attempt is done. So it has been obvserved in 
https://github.com/qgis/QGIS/issues/
27899#issuecomment-535413602 that it actually resulted in speedups.
Alternate implementation with identical effects&drawbacks: use the SQLite3 uri 
syntax with 
the immutable=1 or nolock=1. See https://www.sqlite.org/uri.html 

Of course, in such case, edits should be disabled, or enabled only with a big 
big red warning, 
since database corruption would occur for sure if 2 people tried to edit the DB 
simulatenously. I also thing that in a scenario 1 writer, other readers, the 
readers could 
possibly see inconsistant/broken state in a transient way. That could actually 
arise if there's a 
single machine editing & viewing the database, for example if a rendering 
thread reads 
during the time the DB is written.
(but the same could probably seen with some editing scenarios on shapefiles)

> 
> * The wal file appearing next to the file
> 
>It is confusing to newcomers and looks almost like a sidecar file. I
> would care less if it was put into some system cache folder instead of
> just into my data folder. Or at least if it was a hidden file.

By overriding sqlite3 I/O callback, I'm wondering if we couldn't move the .wal 
file somewhere 
else in the filesystem (of course that would only work for OGR enabled 
consumer, but 
probably good enough). That said, the modification of the first 16 bytes of the 
main .gpkg 
file, which cause issue for file synchronization, would remain, as it is a 
design constraint of 
WAL.

>Possible mitigation: switch to journal mode=delete (not an easy
> option because of https://issues.qgis.org/issues/15351)

I was wondering if we could consult the SQLite3 main author 
(http://www.hwaci.com/drh/) 
on all those locking and concurrency issues, as he has probably experience with 
similar 
scenarios.

> *The requirement for a single geometry column per table
> 
>I just don't see a good reason to forbid that
> 
>Possible mitigation: a) alter the standard. b) ignore the standard
> and patch the ogr implementation.

I don't think the Geopackage OGC SWG would be keen to change that. I believe I 
floated the 
idea around a few years ago, but they wanted GeoPackage core to remain simple 
and adding 
multiple geometry columns goes against that.
That said, GeoPackage has an extension mechanism, and it would be possible on 
the OGR 
side to define one to flag tables with multiple geometry columns as extended 
(I've just 
verified for exemple that the definition of the gpkg_geometry_columns system 
table has a 
unique constraint on the tuple (table_name, column_name), and not just 
table_name, so this 
is a provision for that 

Re: [QGIS-Developer] GeoPackage - where are we -where do we go

[Even Rouault]

> > *The requirement for a single geometry column per table
> > 
> >I just don't see a good reason to forbid that
> >
> >Possible mitigation: a) alter the standard. b) ignore the standard
> > 
> > and patch the ogr implementation.
> 
> I don't think the Geopackage OGC SWG would be keen to change that. I believe
> I floated the idea around a few years ago, but they wanted GeoPackage core
> to remain simple and adding multiple geometry columns goes against that.

Actually that might be just my past reading of the below blog post that Jeff 
Yutzler, the 
editor of the spec just reminded me about:

http://geopackage.blogspot.com/2015/10/one-geometry-per-table.html

The link to the modeling guidelines in this post is now dead. They are at:

http://www.geopackage.org/guidance/modeling.html

Even


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Flatgeobuf for processing

[Even Rouault]

On samedi 9 mai 2020 08:54:06 CEST Matthias Kuhn wrote:
> Hi Richard,
> 
> On 5/8/20 5:24 PM, Richard Duivenvoorde wrote:
> > About Tobias' flatgeobuf: instead of a shp/gpkg file alternative, would
> > this not be a very good candidate to store our intermediate processing
> > steps in (which was shp, not shure what it is now?)?
> 
> Good shout!
> 
> I would love to see someone systematically test it with processing.

A few insights of the GDAL implementation (CC'ing Björn):
- the default behaviour is to generate a flatgeobuf file with the spatial index
- to do so, a temporary file is created with the features serialized as soon as 
they are passed 
to GDAL
- the bounding box of each feature is added in a in-memory vector (40 bytes / 
feature)
- this vector is sorted with the Packed Hilbert R-Tree sorting order
- the final FlatGeobuf file is written with the features also sorted 
accordingly (doesn't require 
bring them all into memory, so not an issue)

One potential issue could be for huge files (> tens of millions of features), 
where the size of 
the vector storing the bounding box could become significant (10 millions 
features = 400 
MB), especially on 32 bit processes.

So probably not going to be a limitation for common usage (and someone really 
eager to 
limit the RAM usage could probably use a merge sort, with the array split into 
sub ones, 
stored onto several files on disk, and bringing into RAM only chunks for final 
sorting. That 
would be fun to write !!!)

If not writing a spatial index, the writing is done in a streaming way in the 
final file, without 
storing in RAM the bounding box of each feature.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS and spatialite

[Even Rouault]

Andreas,

> I am struggling with libspatialite. What libspatialite versions are you
> guys using?  I am trying to compile libspatialite-4.3.0a, but it
> complains about not finding "proj_api.h" (although this file is
> available at /usr/local/include). I tried both with proj-6.3.2 and
> proj-7.0.1. libspatialite-4.3.0a isn't happy with either proj version.

For 4.3.0a:

CFLAGS="-DACCEPT_USE_OF_DEPRECATED_PROJ_API_H -O2 should help:
https://github.com/OSGeo/gdal/blob/master/gdal/docker/alpine-normal/Dockerfile#L248

You can also build against the development repository of Spatialite:
https://github.com/OSGeo/gdal/blob/master/gdal/ci/travis/ubuntu_1804/install.sh#L27
(you could probably drop the "fossil update 90180e065d" which must have been a
workaround against a regression, and the other peculiarities), so mostly:

fossil clone https://www.gaia-gis.it/fossil/libspatialite libspatialite.fossil
mkdir sl
cd sl
fossil open ../libspatialite.fossil
./configure [...]

> I noticed that there 
> hadn't been a lot of activity with this library in the past months/years.

Yes, I've tried to maintain a communication channel with the developper, but 
with
moderate success only. The development is not abandoned, but only at a rather 
slow
pace.

> On a related question: could I build QGIS without libspatialite? I
> failed to do so, but maybe there are some tricks.

I don't think you can without some surgery in the code base.
FWIW, the WFS provider depends on it for caching (could potentially be replaced 
by
something else)

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Issues with CRS in QGIS master on Ubuntu 20.04

[Even Rouault]

> No transform is available between EPSG:3763 - ETRS89 / Portugal TM06 and
> Unknown CRS: GEOGCRS["unknown",DATUM["unknown",ELLIPSOID["GRS 1….
> proj_create_operations: cannot build geodeticCRS 4326: SQLite error on
> SELECT name, ellipsoid_auth_name, ellipsoid_code, prime_meridian_auth_name,
> prime_meridian_code, area_of_use_auth_name, area_of_use_code,
> publication_date, deprecated FROM geodetic_datum WHERE auth_name = ? AND
> code = ?: no such column: publication_date

You have definitely a mix of PROJ versions. Your PROJ library tries to access 
the proj.db of an 
older version.

> 
> Then I get several others, involving EPSG:4326, EPSG:3857 and Unknown CRS.
> 
> Another issue is that, apparently, QGIS does not recognize the prj file
> from shapefiles, even if it was created by himself. For instance, this
> sample shapefile
> https://drive.google.com/uc?export=download&id=1e7WxgPoSeTx7RiZHIaxdX0kvL8Uu
> Ok9Z

Works for me. Related to your above issue

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] GeoPackage - where are we -where do we go

[Even Rouault]

On mardi 2 juin 2020 15:58:38 CEST Olivier Dalang wrote:
> Hi !
> 
> About WAL related issues :
> 
> I just tried to reproduce the original issue that lead to using WAL on
> local files (https://github.com/qgis/QGIS/issues/23283) but could not
> reproduce it (recent master, Windows 10). Does the issue still exist
> on other platforms ? (this can be tested by changing the
> QgsSetting qgis/walForSqlite3 to false and following the steps in the
> issue).

Did you also try running the tests that were added per
https://github.com/qgis/QGIS/commit/b6b8759efbeb833d0d3dbf6df008087701361ad3
with WAL disabled ?

To reproduce the issue, you might need a sufficiently large GeoPackage file so 
that redraw 
time is not too instant and/or possible move quickly your map just before 
validating an edit 
so that a redraw is in progress while changes are committed to the database.
I have unfortunately no memories of if/how I reproduced the issue. Perhaps I 
just came with 
the unit tests first.

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] [Qgis-user] (how to) register gsb transformation in proj.db?

[Even Rouault]

On vendredi 5 juin 2020 11:13:09 CEST Thomas Schüttenberg wrote:
> Hi all!
> 
> Because I better should not roll out QGIS with the afore mentioned premanent
> ballpark-warning in place, I investigatetd further and stumbled across an
> "allowFallback"-setting:
> 
> By placing a default datum transformation in the settings/options.../CRS you
> get new entries in your QGIS3.ini [find it on win at
> %APPDATA%\QGIS\QGIS3\], which are the operation parameters themselve and a
> setting to allow the approximate “ballpark” transformation [...] between a
> source and destination CRS pair, in the case that the preferred coordinate
> operation fails".
> 
> [Projections]
> EPSG%3A31467\EPSG%3A25832_allowFallback=true
> EPSG%3A31467\EPSG%3A25832_coordinateOp="+proj=pipeline +step +inv
> +proj=tmerc +lat_0=0 +lon_0=9 +k=1 +x_0=350 +y_0=0 +ellps=bessel +step
> +proj=hgridshift +grids=Minden.gsb +step +proj=utm +zone=32 +ellps=GRS80"
> 
> This is also writen to any project file containing layers with the crs',
> covered by the default datum transformation. It appears within in the tag
>  and causes the warning
> to show up every time this project will be loaded, although the datum
> transformation is already working an no fallback seems necessary.
> 
> If the _allowFallback=-setting in QGIS3.ini is (manually) edited to 'false',
> newly saved project files also contain 
> and the warning will not be displayed again on project load.
> 
> Reference to my quite hacky findings can be found in the QGIS Python API
> Doc, search for 'allowFallback':
> https://qgis.org/pyqgis/3.10/search.html?q=allowFallback Some of the
> allowFallback stuff is labeled "New in version 3.12", however I currently
> investigate on 3.10.5-LTR. Might this indicate a backporting issue?
> (Otherwise, the implementation of my datum transformation very well might
> still be incomplete, don't know.)
> 
> Just wanted to inform you about my (current) workaround, any further hints
> are certainly appreciated. Greetings!

I'm not super familiar with the QGIS details about the selection and preference 
settings of 
transformations. I now that Nyall made a lot of efforts to make this as smooth 
as possible 
(difficult challenge to have user friendliness and accurate behaviour in that 
field!).
Looking at your record for minden.gsb, it uses a very large area of use. Once 
integrated into 
EPSG, it will probably use a much smaller one fitted to the grid extent, and 
I'm not sure how 
QGIS will react when zooming out sufficiently so that the grid doesn't cover 
the displayed 
extent.
You could simulate that by creating a new record into the area table which uses 
the real 
extent of your grid

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] [Qgis-user] (how to) register gsb transformation in proj.db?

[Even Rouault]

> Yes, I already thought of that, too. Can the extend be derrived 
from a
> gsb-file? Otherwise I have to estimate.

Command line with "gdalinfo Minden.gsb" and look at Corner 
Coordinates
Or just open the gsb file in QGIS as a raster and look at the 
properties.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] WFS cache invalidation / trigger repaint

[Even Rouault]

On vendredi 5 juin 2020 21:44:22 CEST Denis Rouzaud wrote:
> Hi all,
> 
> After digging a bit more, it seems that the cache invalidation is not
> working on the WFS layer.
> 
> If I call emitDataChanged on any of its layers dependencies, or call
> emitDataChanged on this layer or even call triggerRepaint, the layer stays
> the same.
> Only, re-opening the project refreshes the layer.
> 
> I am not very familiar with WFS layers, anyone has an idea about this?

F5 seems to do the job for me. At least looking at the traces, it does discard
the on-disk spatialite cache to recreate a new one. I didn't test with changing
contents on the server to confirm.

The provider implements reloadProviderData():
https://github.com/qgis/QGIS/blob/master/src/providers/wfs/qgswfsprovider.cpp#L775

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] [Qgis-user] More user-information?

[Even Rouault]

> After some Q&A, it appeared that he had opened the shapefile 
from a ZIP...

Not addressing your general concern, but upgrading to GDAL 3.1.0 
will fix that particular point: update of .shp.zip is now supported. 
(performance will probably not be rocketting if the .shp.zip is too 
big)

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] webp for qgis server ?

[Even Rouault]

Matthias,

thanks for the analysis. There are however a few unexpected results.

1) I'd expect gpkg pyramid_JPEG and COG_JPEG to have very similar sizes, even 
COG_JPEG 
being potentially slightly smaller.
And I'd also expect COG_JPEG to be slighly faster (but with less confidence 
that my 
statement about size)

Has by chance the source raster an alpha band ? In which case gpkg_pyramid_JPEG 
would 
have dropped it, whereas COG_JPEG will encode it as DEFLATE compressed mask, 
but still 
the difference is surprising

Another explanation might be the block size. GPKG defaults to 256x256 tiles, 
whereas 
COG_JPEG to 512x512. Perhaps that affect compression efficiency. And 
performance? 
(depends if your bench maintains the GDAL raster opened between requests or not)

If you didn't specify quality settings, both COG_JPEG and GPKG JPEG should use 
the same 
quality of 75%

2) For the same compression type, block sizes and number of overviews, MBTiles 
(the report 
doesn't specify the compression scheme for it) and GeoPacakge should also have 
similar sizes 
and performance. They are really close brothers, with just a few systems tables 
different.

Even

> Hi all,
> 
> At OPENGIS.ch we have recently looked into different raster formats. The
> results can be read here:
> https://www.opengis.ch/2020/06/09/offline-wms-benchmarking-raster-formats-fo
> r-qfield/
> 
> Not that surprising, but one of the interesting findings was that webp
> is very efficient. Low filesize, reasonable rendering performance,
> support for transparency. In short, it has all the potential for being
> used as default transport format for WM(T)S.
> 
> Looking at our server implementation, this format is not supported. Did
> someone ever think about or even look into that?
> 
> Regards


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Create plain table (without geometry) in gpkg?

[Even Rouault]

On mardi 23 juin 2020 15:34:58 CEST Raymond Nijssen wrote:
> My python plugin should create a table without geometry in a gpkg to
> store some additional (meta) info about the layers. I could use python
> bindings for sqlite of course, but the plugin users might not have that
> installed.
> 
> What are the options from within qgis? Is python-sqlite available, or
> can I run sql queries in antother way? Or can I create a table similar
> to creating a new maplayer?
> 
> My (ugly) solution would be creating a point layer and use that
> attribute table and storing POINT(0 0) as geometries.
> Or even worse, creating a .csv sidecar file for my gpkg.
> 
> Hope anyone here knows a better solution!

Raymond,

You should be able to just use the normal QGIS vector API for that. For example 
QgsVectorFileWriter.create() with geometryType = NoGeometry

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS for Apple: Intel vs. arm

[Even Rouault]

On mercredi 24 juin 2020 11:53:37 CEST Peter Petrik wrote:
> Hi,
> 
> As building on iOS already works (
> https://github.com/lutraconsulting/input-sdk/tree/master/ios), I imagine it
> would be possible to  run another architecture build for MacOS with a
> similar approach. It is in Apple's interest to facilitate transition to
> developers. To evaluate how hard it would be, that is more tricky at this
> point. First of all, Qt has to support this new architecture, I haven't
> found a statement from them yet if they plan it and when. So even Qt
> supports it (hopefully from Qt5 releases and not Qt6), we have to build all
> the deps with the new architecture, which takes some time to create
> recipes. And of course find a hardware where it all can be built (probably
> another machine , yes). Not saying that it is very frustrating to do
> development of such a new package if you do not own such hardware
> yourself...

One point I've in mind with my GDAL hat on is that GDAL has a number of x86 
specific speed 
optimizations using SIMD intrinsincs in warping/reprojection resampling, 
overview/rasterio 
resampling, statistics computation, hillshading, gdalgrid. Currently we have no 
accelaration 
for other architectures.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Create plain table (without geometry) in gpkg?

[Even Rouault]

Julien,

> I was looking to create an empty GPKG layer and try the solution Even
> provide, but it creates an invalid file.
> 
> fields=QgsFields()
> fields.append(QgsField("note", QVariant.Double))
> QgsVectorFileWriter.create("/tmp/test.gpkg", fields,
> QgsWkbTypes.MultiLineString, QgsProject.instance().crs(),
> QgsCoordinateTransformContext(), QgsVectorFileWriter.SaveVectorOptions())
> 
> I try also:
> lyr = "mylayer"
> QgsVectorFileWriter.create("/tmp/again.gpkg", fields,
> QgsWkbTypes.MultiLineString, QgsProject.instance().crs(),
> QgsCoordinateTransformContext(), QgsVectorFileWriter.SaveVectorOptions(),
> QgsFeatureSink.SinkFlags(), None, lyr)
> 
> It looks like there is no layer in the produced GPKG. Is it an issue, or am
> I misunderstanding the use of QgsVectorFileWriter?

The issue is likely QgsVectorFileWriter.SaveVectorOptions() with no driverName 
set. There's 
no extension based guessing. Probably that the SaveVectorOptions inner class 
could have a 
"GPKG" default value for driverName, as it is the default driver name in other 
methods of 
qgsvectorfilewriter.h

There are some usage examples of QgsVectorFileWriter in
https://github.com/qgis/QGIS/blob/master/tests/src/python/test_qgsvectorfilewriter.py
  
(not of create() though)

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS testing workflow with QgsMultiRenderChecker (on Windows and Travis)

[Even Rouault]

On mardi 30 juin 2020 14:20:50 CEST Rizky Maulana Nugraha wrote:
> Hi Olivier,
> 
> Maybe this helps. You can run build in debug mode in travis. I usually just
> make a private repo with a minimum script to run that said tests. Then 
you
> can ssh into the build.

Slightly more convenient is to use locally the Docker build method that is 
used by QGIS
See https://github.com/qgis/QGIS/tree/master/.docker#building

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] Plan for a next Spatialite release ? Should the community fork it ?

[Even Rouault]

Hi Sandro,

I hope this emails finds you, and you are doing well. We had a few exchanges in 
private over 
past months about a next Spatialite release, but I'm still unclear on when a 
next release will 
materialize, hence this public call, with the Spatialite list, and GDAL and 
QGIS communities 
CC'ed. The intend of this email is to be constructive hopefully.

Downstream projects of Spatialite will have soon a critical need for a new 
release of 
Spatialite, as there is no released version of Spatialite that uses the PROJ >= 
6 API. Spatialite 
4.3.0a, the latest stable release, dating from almost 5 years now, and uses the 
deprecated 
proj_api.h.
proj_api.h should normally have been removed in March this year, for PROJ 7.0, 
as announced 
2.5 years ago, but this was defered, mostly because Spatialite wasn't ready 
yet. PROJ 8, 
planned for March 2021, will certainly remove proj_api.h, so we will have an 
issue if no 
Spatialite release has been done in the meantime.

If no new Spatialite release is to happen soon, I can see 2 potential solutions:

a) drop Spatialite support from GDAL and QGIS. That would be a fair amount of 
work, and a 
functional regression for both projects. Besides support for the Spatialite 
format itself, both 
GDAL and QGIS benefit from its SQL spatial function to do analysis on 
GeoPackage files for 
example. There are also hidden dependencies to Spatialite, like the QGIS WFS 
client using 
Spatialite internally to do local caching of downloaded features.

b) or do a community fork of the current state of the Spatialite fossil 
repository, which 
implements the needed PROJ >= 6 support. This is certainly not an ideal 
solution, as that 
could create confusion on users on where the "official" Spatialite is. I would 
not call this a 
"hostile" fork, but more a "pragmatic" fork... If I were to do it, this would 
probably be a github 
repo, to make contributions from the community easier. But beyond the question 
of forking 
or not, I do think using github (or gitlab) for the official project would make 
it easier to foster 
collaboration.

Can you tell us about your plan and a realistic schedule for a Spatialite 
(would that be 5.0 or 
6.0 ?) release ? Can the community help somehow ? Is there anything preventing 
from 
releasing current fossil repo now ? If that may related to new features not yet 
implemented 
you've in mind and that you would like to land in a release, can't they wait 
after a release has 
been done with what exists already in Fossil ?

I'd say that if things don't seem to be moving, I'd probably go myself to 
implement solution b) 
this fall, hopefully with support of others devs reading this email as I'm 
already more than 
busy on other OSS projects.

Cheers,

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Raster layers: average resampling disappeared

[Even Rouault]

On mardi 28 juillet 2020 17:06:05 CEST Andreas Neumann wrote:
> Thanks for sharing this discussion.
> 
> So when you load an old project in QGIS master, the average method would
> be replaced with bilinear now?
> 
> Is "bilinear" the closest method to the previous "average"?

This is just a label change. "average" was actually implemented by bilinear.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS, DateTime and TimeZones

[Even Rouault]

> E.g. GDAL has a flag if a
> datetime is UTC or "not utc/unknown", and that's it.

Small precision: it also has support for explicit timezones 
(like UTC+0200)

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] (QGIS 2020 Grant Report) Use of FileGeodatabase spatial index in OpenFileGDB driver

[Even Rouault]

Hi,

I want to report about the outcome of
https://github.com/qgis/QGIS-Enhancement-Proposals/issues/172
"Use of FileGeodatabase spatial index in OpenFileGDB driver"

This work has been successfully completed into GDAL master (for upcoming GDAL 
3.2) per 
https://github.com/OSGeo/gdal/pull/2771 , and automatically benefits to QGIS 
when it uses 
the OpenFileGDB driver.

Some figures on a version from of few years ago of the dataset at
https://data.linz.govt.nz/layer/50772-nz-primary-parcels/, featuring 2 484 646 
polygons, 773 
MB:"ogrinfo -spat 174.949909 -41.143842 175.684529 -40.594600 -al -so 
-noextent" (thus 
counting the number of features intersecting the spatial filter), which returns 
81 046 
polygons, nows runs in 400 ms with GDAL 3.2dev and the OpenFileGDB driver, 
versus 6.7 s 
before (full scan), vs 890 ms with the FileGDB driver (with FileGDB SDK 1.5).
All figures given with debug builds, and on a hot run (so with most pages 
cached by the OS).

Interactive display of this dataset in QGIS with the OpenFileGDB driver is as 
fluid as with the 
FileGDB one. Comparing behaviour of OpenFileGDB and FileGDB drivers with strace 
shows 
that they read a similar amount of data in the .spx file, which confirms it is 
uses correctly.

The filegdb reverse engineered specification was also updated in
https://github.com/rouault/dump_gdbtable/wiki/FGDB-Spec#specification-of-spx-files

Thanks to qgis.org grant program for making this improvement possible.

Best regards,

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] cppcheck false positive

[Even Rouault]

On mardi 29 septembre 2020 11:55:40 CEST Alessandro Pasotti wrote:
> Looks like cppcheck is not smart enough...

Yeah, it hardly understands C++ (it doesn't need to be able to fully compile a 
file to analyze it, 
if I remember correctly)

> 
> const double factor { std::pow( 10, - mPrecisionSpinBox->value() ) };

I noticed this too. It is easily confused by curly initializer list, when used 
in a method body. It 
might also have sometimes issues with lambdas and variable scope.

> const double factor { std::pow( 10, - mPrecisionSpinBox->value() ) };

Using ( ) or = should fix it

const double factor (std::pow( 10, - mPrecisionSpinBox->value() ));
const double factor = std::pow( 10, - mPrecisionSpinBox->value() );

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] cppcheck false positive

[Even Rouault]

On mardi 29 septembre 2020 12:15:41 CEST Alessandro Pasotti wrote:
> Hi Even,
> 
> Thank you, I knew how to fix it but I was wondering if it is possible
> to reduce the false positives (see my other thread about spellchek on
> testdata), life is too short to be wasted in fighting with a dumb
> software :)
> Maybe a more recent version of cppcheck does nor have this issue?

I'm afraid not. my latest attempts with recent cppcheck HEAD lead to number of  
new false 
positives on GDAL that weren't reported before.
The cppcheck version of ubuntu18.04 is not so bad actually.

When things go bad you can add a suppression

// cppcheck-suppress {name_of_cppcheck_diagnostic}

Or we can disable a category in scripts/cppcheck.sh if it is too unreliable

Even

> 
> 
> On Tue, Sep 29, 2020 at 12:11 PM Even Rouault
> 
>  wrote:
> > On mardi 29 septembre 2020 11:55:40 CEST Alessandro Pasotti wrote:
> > > Looks like cppcheck is not smart enough...
> > 
> > Yeah, it hardly understands C++ (it doesn't need to be able to fully
> > compile a file to analyze it, if I remember correctly)> 
> > > const double factor { std::pow( 10, - mPrecisionSpinBox->value() ) };
> > 
> > I noticed this too. It is easily confused by curly initializer list, when
> > used in a method body. It might also have sometimes issues with lambdas
> > and variable scope.> 
> > > const double factor { std::pow( 10, - mPrecisionSpinBox->value() ) };
> > 
> > Using ( ) or = should fix it
> > 
> > 
> > 
> > const double factor (std::pow( 10, - mPrecisionSpinBox->value() ));
> > 
> > const double factor = std::pow( 10, - mPrecisionSpinBox->value() );
> > 
> > 
> > 
> > Even
> > 
> > 
> > 
> > --
> > 
> > Spatialys - Geospatial professional services
> > 
> > http://www.spatialys.com


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS 2.18 compiled against GDAL 3?

[Even Rouault]

Andreas,

> Can QGIS 2.18 be built against GDAL 3.x?

Hum, I guess it would build, although definitely not a vetted combination. I 
wouldn't expect 
datum transformations to work very well, since QGIS would be dependent on 
towgs84 
coming in PROJ.4 strings, and GDAL 3 / PROJ >= 6 will not populate that often 
now.

You might have to add
-DACCEPT_USE_OF_DEPRECATED_PROJ_API_H
in CMAKE_CXX_FLAGS to get QGIS to build against PROJ >= 6

You should check also which PROJ version your spatialite lib is linking to.

You might instead to choose the 'internal PROJ' trick 
(https://trac.osgeo.org/gdal/wiki/
BuildingOnUnixGDAL25dev) to have only GDAL built against PROJ >= 6 and have the 
rest of 
the stack (spatialite, QGIS) still using PROJ 5

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] GPKG and FID -- can we fix this mess?

[Even Rouault]

Hi Nyall,

> - The type constraint on the fid column makes it really hard to
> translate datasets with an existing, non-numeric "fid" column into
> geopackage. Eg. GML files often have a textual fid string, and
> attempting to convert these to gpkg results in a string of errors
> about string values not being usable as fid values, and an empty
> result layer. The only workaround here is to translate first to an
> alternative format (such as shp!), delete the fid column, and THEN
> save as gpkg.

What do you do exactly to get such issues ? If you open a GML file, you'll get 
a 'gml_id' string 
column, so when saving that to GPKG or whatever, you'll also get a regular 
'gml_id' column. 
This has nothing to do with the GPKG fid column. Or do you do something to 
inject the 
content of the 'gml_id' into the GPKG 'fid' column ? I can't reproduce a 
problem with a plain 
ogr2ogr or Export/Save features as in QGIS (with default settings at least)

> - The fid unique constraint, while understandable, results in a HUGE
> raft of issues while working with these. It's SO easy to get a
> situation where you have duplicate fids in an edit buffer, and no way
> to save these features back to the gpkg. You get a series of 1000s of
> errors about duplicate fid, and then an ambiguous state where you're
> completely unsure exactly what's been saved and what's about to be
> lost. This isn't just attributable to a single tool in QGIS -- it's
> possible to end up with duplicate fids through so many different
> operations, including really simple stuff like copying and pasting
> features!

Isn't the main issue here that we expose the fid column as a regular QGIS 
field, instead of 
keeping it as the fid specific property of a QgsFeature, as it should probably 
have remained ? 
That's really the main specificity on how the GPKG format is handled in the OGR 
provider. 

> I propose that we
> 
> 1. demote fids to being only a "semi-permanent" row identifier, with
> the message being that "sometimes these WILL change and you can't rely
> on them as a permanent id field for joins and row identification". If
> users require a permanent unique identifier (i.e. a primary key) on
> their table then THEY have to make and manage that themselves, just
> like shapefiles etc.

Why don't we just treat the fid as the regular FID returned by OGR for other 
drivers ?
I'm not familiar with the join fonctionnality in QGIS: but isn't there a way to 
use the 
QgsFeature.id() to do a join ? That could be a solution to have a permanent 
stable id. But if 
not, yes requiring users to create their own managed unique identifier would be 
understandable if they want to have control on the value of the identifier. 

> 
> 2. expose fids as a read-only field. Users can still see them if they
> want, but they cannot edit them.

Sounds reasonable. But perhaps not exposing them as a column at all (and thus 
content that 
can be duplicated by error), and keeping it as the QgsFeature.id(), would be 
even more safer. 

> 
> 3. make QGIS (or GDAL?) ALWAYS generate a completely new fid whenever
> a row is changed or added. Throwaway the old value, make a new one on
> EVERY edit/addition.

I'd be -1 on that, at least on the GDAL side. That would break an important and 
reasonable 
assumption of the format. That's how a row is identified... Why would we do 
that specifically 
on GPKG and not Postgres or other databases ?

> Yes, these changes will break existing workflows, and possibly break
> existing tools/scripts. But honestly, in my experience and the
> experience of my customers, there's a COMPLETE lack of faith and trust
> in GPKG at this stage. EVERYONE has their horror stories of lost data
> and mangled datasets. We've got to do something drastic, and we've got
> to do it sooner rather than later to salvage what little hope does
> remain for this format.

To sum up my understanding of the problem: it seems to me that all the issues 
originate from 
exposing the OGRFeature.GetFID() content as a QGIS 'fid' column instead of just 
putting it in 
QgsFeature.id(). Otherwise we'd have problems with many other OGR formats. 
Maybe I'm 
missing something.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] GPKG and FID -- can we fix this mess?

[Even Rouault]

> Well -- here's an example file:
> https://github.com/qgis/QGIS/blob/master/python/plugins/processing/tests/tes
> tdata/dissolve_polys.gml
> 
> Not sure how that file was created in the first place, but I've seen
> many like it!

Ah ok, this is a GML2 file, which reports indeed a 'fid' field (comimg from 
the XML 'id' attribute). I was testing with GML3, which reports instead a 
'gml_id' field (coming from XML 'gml:id' attribute)

That issue seems independent from the others. Nothing in the GeoPackage spec 
mandates that the unique identifier should be called 'fid'. This is just the 
default value proposed by the driver, and it is tunable. Maybe changing it to 
'ogr_fid' or 'ogc_fid' as in other database-based drivers would make such 
collision less likely ? There might be some subtle impacts if users of 
GeoPackage or scripts expect the fid column to be called 'fid' and not 
something else.

> Yes, ideally. But at this stage we can't completely hide the fid field
> without breaking existing QGIS projects. Breaking scripts is bad, but
> breaking projects is a complete no-go!

Good point...

> QgsFeature::id() isn't intended to be even semi-permanent. Just
> "mostly constant for the duration of a single data provider's
> lifetime" (i.e. a QGIS session).

Interesting. But yes indeed, I can think of the QGIS WFS provider that will 
also return QgsFeature::id() that aren't stable (reloading the layer is 
probably sufficient to cause them to change)

> Because, for fid at least, it's just an "internal detail" that we're
> showing. To use the postgres analogy we don't manage internal record
> identifiers with the returned features, just the actual exposed
> columns themselves and leave the rest to the backend. And here I think
> the backend (GDAL, or QGIS' OGR provider) should manage fids
> transparently from the client (the QgsVectorLayer).

Hum this is where things get really interesting. The PostgreSQL provider can 
expose a integer primary key as a regular column. So the sole fact of exposing 
a integer primary key as a regular column is not necessarily a recipee for 
disaster. There must be some important difference(s) that we must identify  
before jumping into conclusions.

One of the differences I can see is that the PostgreSQL provider allows to 
modify the value of the primary key column. Which isn't supported by 
OGR_L_SetFeature() (since OGR itself doesn't return the GPKG fid column as a 
OGR field, this is a QGIS only behaviour). So currently 
QgsOGRProvider::changeAttributeValues() will error out if you try to modify 
the content of the 'fid' column to a value different of QgsFeature.id())

I see the Postgres provider has some logic to have a feature id map in some 
cases (when the primary key cannot be mapped directly to a QGIS FID if I 
understand well), but for the GPKG case we aren't in that situation.

So, apart from the situation of changing the FID of an existing feature, at 
first sight, there doesn't seem to be that much difference between the GPKG 
situation and Postgres with a integer primary key that maps directly to a QGIS 
fid. Do I miss something (wouldn't be surprising. I skimmed quickly over the 
code), or is it just that difference that causes the mess?

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Fwd: GPKG and FID -- can we fix this mess?

[Even Rouault]

> I think this is a great idea -- we could retain any compliant fid
> values without change, but as soon as we hit a duplicate fid or
> non-integer fid value then we discard it and generate a new one.
> 
> What do you think Even?

Are you talking about the addFeatures() method of the provider ?

That would be a provider-level change I guess ? Why not, but I'm still puzzled 
why we need something in the GPKG case and not with PostgreSQL with a integer 
primary key.
One difference I see is that in the GPKG case, when creating features through 
the GUI, I can set a new feature to a fid value already taken, whereas if I 
try to do the same with a Postgres layer, the GUI prevents me from doing that. 
There must be some extra hint that the Postgres provider provides, and GPKG 
does not.
(I can even see that in the same edit session If I try to create 2 features 
with the same fid value with a Postgres layer, the GUI prevents me from doing 
that, so before trying any INSERT)

Hum, one difference I can see is that QgsOgrProvider::skipConstraintCheck() 
does not take into account providerProperty( EvaluateDefaultValues, false )  
whereas QgsPostgresProvider::skipConstraintCheck() does. And in the
QgsOgrProvider::skipConstraintCheck() case, for the fid column, the return of 
the method will be true ( mDefaultValues being equal to tr( "Autogenerate" ), 
so not empty )

Interestingly, putting the same implementation as the postgres provider makes 
GPKG behave as it for the scenarios presented above !
Let's see how CI likes that:
https://github.com/qgis/QGIS/pull/39388

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] GPKG and FID -- can we fix this mess?

[Even Rouault]

> The way I see it, fid can be seen very similarly to rowid or shapefile
> fids. A semi-stable unique identifier. Just - and that's the big difference
> - those are not part of the data hence the system can transparently deal
> with duplicates and can fill holes once in a while (shp -> repack, sqlite
> rowid -> vacuum).

Well, no, a fid is by definition a identifier. Something that must be stable 
ideally. In the shp or sqlite with rowid, this is not a genuine fid.
If you claim to have a fid, this is something you could use to do joins or 
create foreign keys upon. It shouldn't change after database recompaction.
> 
> If I could choose I would just make fids disappear (not only from the
> interface but from all the gpkg implementation).

Really, it looks to me "only" to be an implementation issue on our side. The 
concept of having a fid column is not something horrific per se. Many other 
databases have one !

> So just to have brought up the idea: Could we make fids optional for newly
> created gpkgs. Or is my fear that this will affect interoperability in a
> bad way correct?

You would create non-standard compliant GeoPackage. They would fail the OGC 
compliance testing and probably might be rejected by other implementations, 
which would defeat the purpose of being an exchange format.

Hum actually digging further, you would break even QGIS itself when using a 
spatial index. The triggers that update the RTree need a fid column. So if you 
used rowid instead and do a VACCUUM, you would break the spatial index (the 
RTree nodes would refer to other rows than the ones it has been built 
upon...).

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] GPKG and FID -- can we fix this mess?

[Even Rouault]

On jeudi 15 octobre 2020 17:48:30 CEST Matthias Kuhn wrote:
> If there is a managed database where there has been a conscious decision to
> have an integer id column (postgres case) and workflows are built around
> this: filling the id on insert, building foreign keys against it etc, that
> works just fine.
> 
> But if GeoPackage is used as a generic dataformat (typical Shapefile
> replacement), requirements are different and workflows less managed. E.g.
> The "simple" case of merging two datasets already poses the question what
> to do with the two colliding fid attributes. Delete the original fid
> columns and regenerate one. Or preserve both and generate a new one
> (leading to fid, fid_1 and fid_2).
> 
> If we look at the fid attribute as part of the data (where we have foreign
> keys pointing two etc) we want it to be preserved.
> If we look at it as a purely internal technical id (for RTree management
> etc) we don't care for it and can just regenerate and forget about it.
> 
> It looks as if we need to keep it as a technical id.
> I like the idea to transparently generate it in addFeature.
> I would also prefer to hide it away from the user and only leave it
> available through $id. This would make it clear to users that it's for
> internal use and no one would expect it to be preserved after a merge
> operation. For backwards compatibility there could be a provider flag to
> re-enable it.
> I hope that makes sense.

As I like playing devil's advocate :-), one issue I can think of is : how can 
you know when you open a GeoPackage, possibly created by another software, if 
the integer primary key column is a technical one (that could perhaps be 
hidden), or a "real" one part of the data and intended to be user visible ? So 
we would need some user feedback... Or maybe more simply, for pure QGIS 
workflows, we could decide for a new value for the fid column name when we 
create a GPKG table, like "hidden_fid", and when opening such GeoPackage, 
decide to not expose it as a field. That could solve issues for GPKG created 
from now by QGIS, while still providing some backward compatibility. And 
people wanting to still have a visible fid, and ready to deal with possible 
annoyances, could overload this "hidden_fid" value with another value.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Difference between QgsVectorFileWriter and QgsOgrProvider when creating gpkg?

[Even Rouault]

Hi Nyall,

> But I can't explain this. In both cases the resultant gpkg has a
> spatial index. Does anyone know why one of the methods is so much
> slower than the other? (And ultimately, can we fix
> QgsVectorLayerExporter so it uses the same fast approach!)

Candidate fix and explanations at:
https://github.com/qgis/QGIS/pull/39439

Note that I haven't given this more than unit testing, but given what you 
mentionned, I'm rather confident this should fix the issue.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Difference between QgsVectorFileWriter and QgsOgrProvider when creating gpkg?

[Even Rouault]

On dimanche 18 octobre 2020 15:35:11 CEST Nyall Dawson wrote:
> On Sat, 17 Oct 2020 at 20:02, Even Rouault  
wrote:
> > Hi Nyall,
> > 
> > > But I can't explain this. In both cases the resultant gpkg has a
> > > spatial index. Does anyone know why one of the methods is so much
> > > slower than the other? (And ultimately, can we fix
> > > QgsVectorLayerExporter so it uses the same fast approach!)
> > 
> > Candidate fix and explanations at:
> > https://github.com/qgis/QGIS/pull/39439
> 
> Thanks Even!
> 
> Just for my own curiosity, is there some logic in GDAL which defers
> the spatial index creation on the immediate first use of a newly
> created geopackage layer?

This isn't a GDAL core mechanism (since GDAL has no abstraction for a spatial 
index), but a possibly per-driver optimization. Which is implemented in the 
GPKG driver. On a freshly new created GeoPackage layer (if you don't close and 
reopen the dataset), the spatial index creation will be deferred until the 
first call of GetNextFeature() or SyncToDisk() (which will happen if you close 
the dataset or run ExecuteSQL()).

For the update/append scenario to an existing table, one thing that can help 
is to run "PRAGMA synchronous = OFF". For a freshly new created database, 
PRAGMA synchronous = OFF is executed, which avoids fsync() calls. This is OK 
for a new database as there's no prior existing data that could be corrupted 
if things go wrong. For an existing one, PRAGMA synchronous = OFF would be 
more risky (that said, according to https://www.sqlite.org/
pragma.html#pragma_synchronous, QGIS crashing would still be OK w.r.t data 
integrity, but not if the operating system crashes or computer is halted in a 
unclean way). From a test on a 2 GB file appended into another one of the same 
size, on a system with a rather slow rotational disk, PRAGMA synchronous = OFF 
makes ogr2ogr -append run 4 times faster.

Another improvement is to defer the update of the spatial index when appending 
features. I've implemented it in
https://github.com/OSGeo/gdal/pull/3079. It can make append time to be divided 
by 1.5  when inserting features in large batch within transactions (with 
synchronous = OFF enabled, otherwise I'm not patient enough :-))

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Difference between QgsVectorFileWriter and QgsOgrProvider when creating gpkg?

[Even Rouault]

> Wow, great! So now Tim's screencast has directgly lead to a bunch of
> improvements from GDAL up :D

For clarity I didn't turn on automatically PRAGMA synchronous = OFF for 
existing databases.

For current GDAL version, this can be done by setting the 
OGR_SQLITE_SYNCHRONOUS configuration option to OFF.

And starting with GDAL 3.2, also as an open option of the GPKG driver:
"PRELUDE_STATEMENTS=PRAGMA synchronous = OFF" (to reduce the scope to a given 
file)

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] [gdal-dev] QGIS from OSGeo4W crashes with latest update on NetCDF

[Even Rouault]

On mardi 20 octobre 2020 16:12:56 CEST Pedro Venâncio wrote:
> Hi Joaquim Luís,
> 
> This is an IPMA NetCDF, it was working before the update.
> 
> And I think it is in fact a HDF5 library version problem:
> > gdalinfo AROME_OPER_001_FC_SP_PT2_025_RH_2-HTGL_2020102000.nc
> 
> Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid
> argument".
> Warning! ***HDF5 library version mismatched error***

Jürgen told me that he has rebuilt the netCDF lib against the new HDF5 lib, so 
hopefully if you update your OSGeo4W install (to netcdf 4.6.2-2 I think), that 
should work now.

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] Fwd: [TC-Announce] OGC Geopackage Standards Working Group updates its Charter; public comment requested

[Even Rouault]

Of potential interest. "Develop a logical model for GeoPackage so that future 
implementations can be separated from the current requirement to use SQLite." 
is intriguing

--  Forwarded Message  --

Subject: [TC-Announce] OGC Geopackage Standards Working Group updates its 
Charter; public comment requested
Date: mercredi 28 octobre 2020, 10:07:46 CET
From: announce--- via TC-Announce 
To: tc-annou...@lists.opengeospatial.org

OGC Geopackage Standards Working Group updates its Charter; public comment
requested

New purpose of SWG is to revise the GeoPackage Encoding Standard and revise
existing, or produce new, extensions to improve and increase functionality



Contact: i...@ogc.org 

28 October 2020: The Open Geospatial Consortium (OGC) seeks public comment
on a revision of the Geopackage Standards Working Group (SWG) charter.
Comments are due by November 17, 2020.

The Geopackage Standards Working Group (SWG)

provides a consensus forum for revisions to the GeoPackage Encoding
Standard including approved extensions as well as the development of new
GeoPackage extensions. The SWG adopted a new charter outlining plans for
future development of the Standard.

GeoPackage was originally developed with three main goals in mind: 

 To be a convenient, efficient, and interoperable container for geospatial
information; 
 To enable operations in all computing environments, including those with
Disconnected, Degraded, Intermittent, or Limited (DDIL) network
connectivity; 
 For GeoPackage to be extensible, allowing it to evolve to meet future
operational needs.

For a good primer on Geopackage please read the post entitled
#GeoPackageDay 2020 - what is GeoPackage?

on the OGC Blog

.

The revised charter outlines the new objectives for the SWG:

 Develop a logical model for GeoPackage so that future implementations can
be separated from the current requirement to use SQLite.
 Revise existing GeoPackage standards and extensions, namely the GeoPackage
Encoding Standard 1.3.0, the GeoPackage Tiled Gridded Coverage Extension,
and the GeoPackage Related Tables Extension. In particular, the SWG will
review all outstanding Change Request Proposals submitted before the
adoption of this charter.
 Develop new GeoPackage extensions that have been requested by OGC Domain
Working Groups

.
 Perform outreach to promote GeoPackage understanding and use.

Revisions to the version 1.3.0 of the GeoPackage Encoding Standard will
improve the interoperability, capability, and clarity of the Standard. The
Tiled Gridded Coverage and Related Tables Extensions have not been revised
since their initial release and will likely benefit from additional review.
New extensions would enable new capabilities that have been requested by
the OGC community.

The Geopackage SWG plans to perform these activities by the end of calendar
year 2022.

For a good primer on Geopackage please read the post entitled
#GeoPackageDay 2020 - what is GeoPackage?

or any of the many Geopackage posts available on the OGC Blog

.

The draft revised GeoPackage SWG charter

is available for review and comment on the OGC Portal. Comments are due by
November 17, 2020, and should be submitted via the method outlined on the
draft revised GeoPackage SWG charter's public comment request page

.

About OGC

The Open Geospatial Consortium (OGC) is an international consortium of more
than 500 businesses, government agencies, research organizations, and
universities driven to make geospatial (location) information and services
FAIR - Findable, Accessible, Interoperable, and Reusable.

OGC's member-driven consensus process creates royalty free, publicly
available geospatial standards. Existing at the cutting edge, OGC actively
analyzes and anticipates emerging tech trends, and runs an agile,
collaborative Research and Development (R&D) lab that builds and tests
innovative prototype solutions to members' use cases.

OGC members together form a global f

[QGIS-Developer] Changes in Travis-CI billing

[Even Rouault]

Hi,

I was pointed to
https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

If my understanding is correct (which I'm not completely sure given all the 
subtelties), QGIS builds will now have a initial credit of 1000 minutes of 
build time (so only 17 hours), and then we'd have to beg for more, but I can't 
imagine we can ask every day a renewal of this credit...

Looks like Travis-CI won't be a realistic option any more unless paying.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Changes in Travis-CI billing

[Even Rouault]

> QGIS might not be affected by this since we don't have more than 5
> concurrent builds.

That's not my understanding. The 1, 2 or 5 concurrent builds plans are paying 
plans if you look at https://travis-ci.com/organizations/qgis/plan .
The free plan of QGIS is subject to the 10,000 credits system.

For reference, OSGeo has a premium plan for 10 concurrent builds that costs 
5379 USD/year, and we have a 30% discount on it due to OSGeo being a non-
profit, bringing it to 3765 USD. (I've just written to Travis support to 
confirm that the OSGeo plan isn't affected by the changes)

> Anyway, I guess we could slowly move everything to Github workflows as we
> have had much better experience lately than on Travis.

That's going to be a urgent topic if my understanding is correct. Probably 
asking Travis support for one month worth of credits could be done for the 
time to transition, but not sure they'll be enthusiastic to do so if it isn't 
to switch to a paying plan.

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Plugin licensing requirements

[Even Rouault]

On jeudi 7 janvier 2021 18:18:03 CET Vincent Picavet (ml) wrote:
> Hi,
> 
> On 07/01/2021 10:43, Nyall Dawson wrote:
> > On Thu, 7 Jan 2021 at 19:13, Topi Tjukanov  wrote:
> [..]
> 
> >> So if this really is a requirement, should this be enforced somehow and
> >> checked when plugins are accepted to the official repository?
> > 
> > Agreed, but keep in mind that it's a little tricky sometimes. A plugin
> > ONLY
> > has to make the modules which import QGIS classes GPL (and consequently
> > any
> > other modules which import these modules). It is entirely possible to
> > separate plugins into two isolated components, a non-GPL "core" layer
> > which
> > does NOT use any QGIS modules, and a GPL layer which imports both the QGIS
> > modules and the non-gpl core. (This is how the licensed SLYR plugin works,
> > for reference).
> 
> I would not bet any cent on these assertions. Would you have any serious
> references supporting this ?
> 
> The GPL licence makes the code "viral" through the notion of "link". For
> Python module, it is through import. A link is bidirectionnal, and
> therefore your whole package if distributed is considered GPL.
> 
> As far as I know, the only way to have a GPL/proprietary coexistence is to
> have distinct processes and inter-process communication ( through files,
> web interfaces... ). And it can be even trickier if there are strong
> dependencies between modules, which can be interpreted as "derived product"
> or "link", even if there is no link as in "library link" between the
> modules.

Those situations are tricky. Nyall's case looks to me a bit similar to the 
NVIDIA Linux kernel module:
- Linux kernel core : GPL
- shim layer for the NVIDIA driver: GPL
- NVIDIA blob that doesn't call GPL code: proprietary
NVIDIA claims that the blob isn't considered a derived work of the Linux 
kernel as it is OS-independent work.
Only a judge can ultimately decide what is a derived work or not.

Reference to a discussion regarding the NVIDIA case:
https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47ir1bZGAK/comp/
linux/gpl_modules.html

https://www.gnu.org/licenses/gpl-faq.en.html#GPLIncompatibleLibs might also be 
relevant.

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] undefined symbol: GDALAutoCreateWarpedVRTEx

[Even Rouault]

On mardi 19 janvier 2021 09:55:46 CET Jorge Gustavo Rocha wrote:
> Hi,
> 
> Just an additional comment. The GDALAutoCreateWarpedVRTEx is there, as
> far as I can see.
> 
> nm -g libqgis_core.so.3.16.3 | grep GDALAutoCreateWarpedVRTEx
>  U GDALAutoCreateWarpedVRTEx
> 
> But if the symbol is defined in the library, why do I get this error?

The U means undefined. Which is expected. libqgis_core imports the symbol from 
libgdal.
What I suspect here is that QGIS is built against a libgdal 3.2.x (QGIS only 
tries to use GDALAutoCreateWarpedVRTEx if GDAL headers are >= 3.2), but at 
runtime links against an older one.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] crssync issues

[Even Rouault]

Andreas,

if you've built yourself PROJ on a system that has older proj versions, I'd 
suppose that one of the QGIS dependency links against that older proj (might 
be spatialite for example), and at runtime you get a clash between it and 
7.2.x

Check the output of:
ldd /home/an/dev/QGIS/build/output/bin/crssync | grep proj

if you get 2 lines, then you're screwed

Then run the following to find which library links against the system libproj:

for i in `ldd /home/an/dev/QGIS/output/bin/crssync | awk '{print $3}'`; do \
  ldd $i | grep proj >/dev/null && (echo $i; ldd $i | grep proj); done

and you'll have to rebuild the library/ies that link against system proj 
against proj 7.2.x

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Date, Time and Time Zones

[Even Rouault]

 but the question I have is whether a vector layer were saved to 
something like a GPKG would the timezone even be preserved?


Short answer: not by default

Longer answer:

- see https://github.com/opengeospatial/geopackage/issues/530

- and https://github.com/OSGeo/gdal/issues/3423#issuecomment-772405674

- and https://gdal.org/drivers/vector/gpkg.html#dataset-creation-options

So basically:

- the GeoPackage spec only supports date/time encoded as ISO 8601 with 
Zulu timezone : "-MM-DDTHH:MM:SS.SSSZ"


- on reading side, the OGR Geopackage driver can understand non-Zulu 
timezones


- on writing side, by default the OGR Geopackage driver will convert 
datetimes to the Zulu timezone (applying the appropriate shift), unless 
the DATETIME_FORMAT=WITH_TZ dataset creation option is passed at 
creation time, in which case the original timezone is preserved. Such 
datasets may not be read correctly by third party implementations.


Even


http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Date, Time and Time Zones

[Even Rouault]

| Also I wonder if it would be possible to work with the GPKG developers 
to provide better timezone support there as well.


This is more about finding how to convince the GeoPackage spec 
maintainers that it is worth adding it than a technical one. I guess 
they wouldn't want to break compatibility with the core specification, 
so a clean way forward would be to deal with that as an extension, since 
GeoPackage has a mechanism to indicate that a given file uses extensions.


The gpkg_extensions table (https://www.geopackage.org/spec/#_extensions) 
could for example have one record to declare each (layer, field) that 
uses an extended date-time format


|table_name = |||{name_of_table_using_an_extented_datetime_column}
||

||column_name = {name_of_column_using_extended_datetime}||

||extension_name = ogr_extented_datetime (if the OGC SWG would accept to 
officialize it as a registered extension, that would become 
gpkg_extended_datetime)

||

||definition = ||https://gdal.org/geopackage_extended_datetime.html 
(or URL to updated GeoPackage spec if officialized)



scope = read-write

What would be to decide is if the extension consists in relaxing the 
constraints of the existing DATETIME data type (to accept any time zone, 
or lack of time zone), or adding a new DATETIME_EXTENDED data type.




Do you know what vector formats support timezones properly?


For general purpose formats: FlatGeobuf, GML, GeoJSON (well there's no 
datetime data type in JSON, so this is by guessing that a string looks 
like a ISO8601 formatted datetime...), CSV (same remark as GeoJSON)



--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Questions about QGIS plugin sales

[Even Rouault]

Jeferson,

yes, you can sell plugins, but you'll have to distribute the source code 
of it, or make a written offer to have access to it, to your customers 
under the GPL v2+ license of QGIS (which mean they could also 
potentially redistribute the source code freely, in all meaning of the 
word, to anyone else)


See https://www.gnu.org/licenses/gpl-faq.en.html#DoesTheGPLAllowMoney

And also https://lists.osgeo.org/pipermail/discuss/2021-March/039273.html

Even

Le 13/05/2021 à 21:08, jeferson da Silva a écrit :

Hello,

My name is Jeferson and I have a company in Brazil called Trindade 
Company, which map cities with Drones and recently created a plugin 
and interfaces for QGIS, for urban taxation, do tax calculations 
automatically and synchronize with a bank of data. My question is, can 
I sell these plugins? How can my company be a representative or 
partner of QGIS?

Attached images of the plugin and interface.
QGIS Version: 3.18.2

*/Jeferson da Silva de Jesus: Agricultural technician, majoring in 
Systems Development Analysis./*

*/CEO from Trindade Company. /*
*/WhatsApp: +55 71 99261-9642/*

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] problems debugging tests (and most other targets) with Qt-Creator

[Even Rouault]

Hi Alessandro,

Isn't it just the bump of version numbers after the release and stale 
.so.3.19.* files in output/lib ? (but that caused for me crashes for the 
qgis application)


Even

Le 23/06/2021 à 19:36, Alessandro Pasotti a écrit :

Hi,

I really don't know what I have changed but since yesterday I cannot 
debug any target except for the main `qgis` application with QT 
Creator (ubuntu 20.04, QT creator 4.15., clang-10 and gdb).


gdb hangs forever at 100% CPU and eats all RAM until it gets killed by 
OOM.


Does anyone have the same problem?

--
Alessandro Pasotti
QCooperative: www.qcooperative.net 
ItOpen: www.itopen.it 

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] GDAL-warp and gdal-translate strange behaviour regarding output NODATA vaue

[Even Rouault]

By design, only speaking about behaviour of the GDAL utilities themselves:

- gdal_translate -a_nodata just modifies the nodata metadata, and 
doesn't alter pixel values


- gdalwarp [-srcnodata xxx] -dstnodata yyy modifies the nodata metadata 
and pixel values


Le 09/07/2021 à 01:02, Alexandre Neto a écrit :

Hi,

I was quite surprised with the behavior of gdal_translate (in clip 
raster by extent) and gdal_warp (reproject) when the user sets up an 
output nodata value different from the original nodata value of the 
input files.


My intuition was telling me that by using that option, all the old 
nodata pixel values would be transferred to the new nodata values. So, 
if I have a raster with nodata = -3.40282e+38 and I use as output 
nodata value -, I would expect that all -3.40282e+38 pixels would 
be converted to - and considered to be NODATA. But I was wrong. By 
doing this, I ended up with rasters containing pixels with actual 
-3.40282e+38 value, no longer considered as nodata.


Is it just me that finds this unexpected?

On the other hand, if you use the same similar option in the DGAL 
raster calculator it works as (I) expected.


Best regards,

Alexandre Neto
User Support
www.cooperative.net 

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

[Even Rouault]

Le 15/07/2021 à 21:51, Andrea Giudiceandrea a écrit :

Il 15/07/2021 20:51, Andrea Giudiceandrea ha scritto:
Instead, QGIS imports correctly XLSX files as UTF-8 encoded, while 
XLS files are wrongly imported as "system" encoded, even selecting 
UTF-8 [3] encoding in the Data Source Manager vector import window.


Looking at GDA/OGR code, I see that the layer imported via XLSX driver 
advertise the UTF-8 strings (OLCStringsAsUTF8) in the capabilities, 
while an XLS layer does not.


So, this could be the underlying root cause of the issue.

Good catch! Could you issue a PR to fix this ?


Regards.

Andrea
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

[Even Rouault]

Le 15/07/2021 à 22:30, Nyall Dawson a écrit :



On Fri, 16 Jul 2021, 4:52 am Andrea Giudiceandrea, 
mailto:andreaer...@libero.it>> wrote:


> Isn't this limitation ultimately that GDAL isn't reading the
encoding
> correctly? (Or perhaps it's a limitation in the underlying freexl
> library...)
>
> Nyall


Hi Nyall and Andreas,
it seems to me GDAL/OGR [1] reads XLS and XLSX files with the
relative
proper encoding [2] and ogrinfo outputs the text in UTF-8 for both
the
formats.


I don't think that's completely correct -- looking at the freexl 
documentation it seems that only some xls file versions are utf8, and 
others have a codepage indicating the encoding which needs to be read 
from the xls metadata:


"Any BIFF version from BIFF2 to BIFF5 simply supports CodePage based 
character encoding, i.e. each character
simply requires 8 bits to be represented (single byte). Correct 
representation of characters requires knowing which
one CodePage table has to be applied. This can be determined from the 
workbook or worksheet metadata (it is the

CODEPAGE record).
BIFF8 is much more sophisticated, since any text string is usually 
encoded as Unicode in UTF-16 Little Endian
[UTF-16LE] format. This encoding is a multi-byte encoding (two bytes 
are required to represent a single character),

but being universal no character table is required."

Yes, but FreeXL does the recoding to UTF-8


Nyall




Instead, QGIS imports correctly XLSX files as UTF-8 encoded, while
XLS
files are wrongly imported as "system" encoded, even selecting
UTF-8 [3]
encoding in the Data Source Manager vector import window.

After importing a XLS file, changing the "Data source encoding" of
the
layer to "UTF-8" fixes the text codecs in my tests.

So, I think QGIS should automatically import also XLS files as UTF-8
encoded.

Best regards.

Andrea Giudiceandrea

[1] tested on Windows /OSGeo4W: GDAL 3.1.4 / FreeXL 1.0.2 / Expat
2.1.0
and GDAL 3.2.2 / FreeXL 1.0.6 / Expat 2.2.10
[2] text in XLS (BIFF8) files are internally encoded in UTF-16LE
[3] by the way, there are incorrectly two "UTF-8" codecs listed in
the
"Encoding" drop down menu list...
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org 
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Unsubscribe:
https://lists.osgeo.org/mailman/listinfo/qgis-developer



___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Bug with GeoPackage and Expressions?

[Even Rouault]

SQLite, which is the underneath database engine for GeoPackage, doesn't 
support ILIKE. The SQLite LIKE operator is actually case insensitive for 
ASCII characters. See paragraph 5 "The LIKE, GLOB, REGEXP, and MATCH 
operators" of https://www.sqlite.org/lang_expr.html


Le 28/07/2021 à 21:43, C Hamilton a écrit :
I think I found a bug and was wondering if someone could verify this. 
I am on Windows 10 and tried this with both QGIS 3.16.9 and 3.20.1.


I tried the following filter expression on the Natural Earth data set 
ne_10m_populated_places 
(https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_populated_places.zip 
). 
Once loaded into QGIS right mouse click on the layer and select 
"Properties...", then click on the "Source" tab, click on "Query 
Builder" and then use this query


"NAME" ILIKE '%Bom%'

This works fine. If I export ne_10m_populated_places as a GeoPackage 
and try the same thing with the GeoPackage version of the layer, I get 
the error:


"An error occurred when executing the query, please check the 
expression syntax."


If I use LIKE instead of ILIKE then it works. It is like ILIKE was not 
implemented for GeoPackage. Can anyone verify this? I think you can 
use any vector layer for this test with one that is a shapefile and 
the other a GeoPackage.


Thanks,

Calvin


___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Trouble compiling QGIS

[Even Rouault]

Andreas

Several things to check:

- Is your GDAL build a clean one ? That is is it from a fresh build 
directory, or are you rebuilding in a directory where a previous build 
was done. If the later, make sure to "make clean" before rebuilding


- Is your GDAL build using your custom GEOS one ? Otherwise QGIS might 
link directly to the custom one and indirectly to the system one through 
GDAL. Check the output of "ldd output/bin/qgis | grep geos"


- try "valgrind output/bin/crssync". Emitted stack traces can give 
interesting hints


Even

Le 03/09/2021 à 15:30, Andreas Neumann a écrit :


Hi,

I have troubles compiling QGIS on my Linux Ubuntu 20.04 machine.

I use a self-compiled GEOS and GDAL.

The build always hangs at "crssync" - here is the ninja build output I 
get. Do you have any idea what might be wrong or what I could do in 
order to fix this problem?


Thank you for your hints,

Andreas



[0/1] Re-running CMake...
-- QGIS version: 3.21.0 Master (32100)
-- Found OpenCL C++ headers: /usr/include
-- ccache found
-- Found Proj: /usr/local/lib/libproj.so version 8 (8.1.0)
-- Found GEOS: /usr/local/lib/libgeos_c.so (3.9.1)
-- Found GDAL: /usr/local/lib/libgdal.so (3.3.1)
-- Found Expat: /usr/lib/x86_64-linux-gnu/libexpat.so
-- Found Spatialindex: /usr/lib/x86_64-linux-gnu/libspatialindex.so
-- Found Qwt: /usr/lib/libqwt-qt5.so (6.1.4)
-- Found libzip: /usr/lib/x86_64-linux-gnu/libzip.so
-- Found Sqlite3: /usr/lib/x86_64-linux-gnu/libsqlite3.so
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-pthread
-- Found zlib: /usr/lib/x86_64-linux-gnu/libz.so
-- Found exiv2: /usr/lib/x86_64-linux-gnu/libexiv2.so
-- Found PostgreSQL: /usr/local/pgsql/lib/libpq.so
-- Found SpatiaLite: /usr/local/lib/libspatialite.so
-- Qt WebKit support enabled
-- Found ZSTD: /usr/lib/x86_64-linux-gnu/libzstd.so
-- Found laz-perf: /usr/local/include
-- Found PDAL: 
/usr/local/lib/libpdalcpp.so;/usr/local/lib/libpdal_util.so (2.0.1)

-- Found Qt version: 5.12.8
-- Found QScintilla2: /usr/lib/libqscintilla2_qt5.so (2.11.2)
-- Found QtKeychain: /usr/lib/x86_64-linux-gnu/libqt5keychain.so
-- Found QCA: /usr/lib/x86_64-linux-gnu/libqca-qt5.so (2.2.1)
-- Found QCA OpenSSL plugin
-- Pedantic compiler settings enabled
-- Found GSL: -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas -lm
-- Found Python executable: /usr/bin/python3.8 (version 3.8.10)
-- Python library: /usr/lib/x86_64-linux-gnu/libpython3.8.so
-- Python site-packages: /usr/lib/python3/dist-packages
-- Found PyQt5 version: 5.14.1
-- Found SIP version: 4.19.21
-- Found QScintilla2 PyQt module: 2.11.2
-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS) 
(found version "")

-- Could NOT find NetCDF (missing: NETCDF_LIBRARY NETCDF_INCLUDE_DIR)
-- Using PROJ >= 6 srs database.
-- Ctest Binary Directory set to: 
/home/bjsvwneu/dev/QGIS/build/output/bin

-- Configuring done
-- Generating done
-- Build files have been written to: /home/bjsvwneu/dev/QGIS/build
[1/6072] Generating qgsversion.h, qgsversion.inc
Files "/home/bjsvwneu/dev/QGIS/build/qgsversion.h.temp" to 
"/home/bjsvwneu/dev/QGIS/build/qgsversion.h" are different.

[3/6072] Generating qgsexpression_texts.cpp
Files 
"/home/bjsvwneu/dev/QGIS/build/src/core/qgsexpression_texts.cpp.temp" 
to "/home/bjsvwneu/dev/QGIS/build/src/core/qgsexpression_texts.cpp" 
are different.
[2905/6072] Generating gui/sip_guipart0.cpp, gui/sip_guipart1.cpp, 
gui/sip_guipart2.cpp, gui/sip_guipart3.cpp, gui/sip_guipart4.cpp, 
g...i/sip_guipart10.cpp, gui/sip_guipart11.cpp, gui/sip_guipart12.cpp, 
gui/sip_guipart13.cpp, gui/sip_guipart14.cpp, gui/sip_guipart15.cpp


[4061/6072] Building CXX object 
src/providers/mdal/CMakeFiles/provider_mdal.dir/__/__/__/external/mdal/mdal_utils.cpp.o 


../external/mdal/mdal_utils.cpp: In function ‘int MDAL::toInt(size_t)’:
../external/mdal/mdal_utils.cpp:143:14: warning: comparison of integer 
expressions of different signedness: ‘const size_t’ {aka ‘const long 
unsigned int’} and ‘int’ [-Wsign-compare]

 143 |   if ( value > std::numeric_limits::max() )
 |    ~~^
[4172/6072] Running crssync
FAILED: src/crssync/CMakeFiles/synccrsdb
cd /home/bjsvwneu/dev/QGIS/build/src/crssync && 
/home/bjsvwneu/dev/QGIS/build/output/bin/crssync

free(): invalid pointer
Aborted (core dumped)
[4173/6072] Building CXX object 
src/plugins/topology/CMakeFiles/topolplugin.dir/topolplugin_autogen/mocs_compilation.cpp.o 


ninja: build stopped: subcommand failed.


___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-develo

[QGIS-Developer] Leveraging Conda-Forge to create QGIS installers ?

[Even Rouault]

Hi,

Probably a topic that can raise passions and on which I'm moderately 
legitimate to speak, but shouldn't we seriously consider leveraging the 
Conda / Conda-Forge (https://conda-forge.org/) ecosystem for QGIS 
packaging, especially on the Windows and Mac platforms ? QGIS depends on 
a lot of external dependencies, and building them and updating them is 
really about maintaining a packaging system, and QGIS has two such 
separate and bespoke systems for Windows (OSGeo4W) and Mac 
(QGIS-Mac-Packager).  The ideal vision would be that the QGIS project 
mostly maintains the bits specific to QGIS, but not be the sole 
maintainer of its dependencies such as QT, GDAL (and its many 
dependencies), PDAL, GRASS etc, as it is today. Conda-Forge provides a 
truly collaborative environment and active community that already 
bundles a number of those dependencies, and QGIS is already there (not 
full capabilities yet, due to some dependencies missing. That would be 
one of the points to address). The Conda-Forge community is really 
vibrant (if you look at 
https://github.com/conda-forge/staged-recipes/pulls?q=is%3Apr+is%3Aclosed, 
you can see that 20 packages were added in the last 24 hours!). It is 
also a NumFocus sponsored project. It has support from a number of 
institutions. It is unlikely to disappear anytime soon.


There would certainly work needed to build installers from them. I found 
https://github.com/conda/constructor project where you can build 
standalone installers from Conda packages, but was told it is perhaps 
not super mature.  Even if QGIS needs require a dedicated installer with 
custom bits, leveraging already packaged dependencies would probably be 
a big enough win compared to the current situation where the whole stack 
needs to be built and rebuilt from scratch by only a few knowledgeable 
people, on non-shared infrastructure.


There would be the possibility to pin dependencies at certain known good 
points, for example to base LTR builds on top of them.


I guess also that Conda based installers could help for plugins that 
require installing native or Python dependencies, but that'd be already 
more a secondary advantage.


Another proof that Conda is to be taken seriously: 
https://developers.arcgis.com/python/guide/understanding-conda/


I'm not saying this is a magical solution: there would clearly be a 
significant amount of work and technical hurdles to solve to reach the 
same degree of maturity as our current installers, but it is probably an 
investment worth considering for the long term.


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS does not recognize CRS of specific layers generated by GeoServer

[Even Rouault]

Petro,

I've queued https://github.com/OSGeo/gdal/pull/4908 which will "fix" 
that particular case (more precisely anything based on ETRS89). I'm not 
super satisfied with this, and I believe there's no miracle solution for 
all potential use cases. What GDAL returns in both 2.4.0 and 3.4.0 is 
actually pretty much the same object, despite the different 
representation (WKT1 vs WKT2). The root cause of the issue is with the 
TOWGS84[] node. Those nodes are not part of CRS definitions. They date 
back to the time where the GDAL & PROJ stack wasn't capable of more 
powerful transformation, and so a particular datum transformation to WGS 
84 was picked up, and encoded in the CRS definition. When reading such 
definitions nowadays, we have a dilemna: did the user really meant to 
use the particular transformation that was in TOWGS84[], or was it just 
there because that was the default one. We ere on the side of caution 
(most of time, there are a few exceptions there and there in GDAL), and 
keep the transformation information, which makes that this CRS is 
formally not the official definition, and thus considered as ""custom""


However, for ETRS89, I believe we can safely remove it, since as far as 
I know, there's no better transformation between ETRS89 (which is known 
modeled as a "datum ensemble" by EPSG, meaning any of the 11 ETRF 
realizations) and WGS 84 (itself a datum ensemble) than the 
TOWGS84[0.0,0.0,0.0,0.0,0.0,0.0,0.0], and it is unlikely there will be 
an alternative one in the future, given the dynamic nature of the WGS 84 
datum (maybe I'm speaking too fast. When ETRS89 and WGS 84 coordinates 
will have drifted too far away in the future, maybe there will some 
non-null transformation added in the database, so that users who want to 
transform to a 'past' or 'future' coordinate epoch of WGS 84 can do that 
in a not too bad way.)


(I should not that this .prj file isn't really compliant with the Esri 
dialect of WKT, which is supposed to be used for shapefiles, since 
normally it shouldn't contain TOWGS84 or AUHTORITY code)


Even


Le 27/11/2021 à 19:13, Pedro Venâncio a écrit :

Hi all,

I'm seeing that QGIS 3.16 / 3.22 does not recognize the CRS of 
specific layers generated by GeoServer, but the old QGIS 2.18.28 
recognize.


For instance, this shapefile: https://si.icnf.pt/shp/rnap 



Checking ogrinfo in different versions:

ogrinfo --version
GDAL 2.4.0, released 2018/12/14 (Used by QGIS 2.18.28)

ogrinfo -al rnap.shp
INFO: Open of `rnap.shp'
      using driver `ESRI Shapefile' successful.

Layer name: rnap
Metadata:
  DBF_DATE_LAST_UPDATE=1921-11-27
Geometry: Polygon
Feature Count: 48
Extent: (-123561.201500, -300415.40) - (162054.118700, 268136.980100)
Layer SRS WKT:
PROJCS["ETRS89 / Portugal TM06",
    GEOGCS["ETRS89",
        DATUM["European Terrestrial Reference System 1989",
            SPHEROID["GRS 1980",6378137.0,298.257222101,
                AUTHORITY["EPSG","7019"]],
            TOWGS84[0.0,0.0,0.0,0.0,0.0,0.0,0.0],
            AUTHORITY["EPSG","6258"]],
        PRIMEM["Greenwich",0.0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.017453292519943295],
        AXIS["Geodetic longitude",EAST],
        AXIS["Geodetic latitude",NORTH],
        AUTHORITY["EPSG","4258"]],
    PROJECTION["Transverse_Mercator",
        AUTHORITY["EPSG","9807"]],
    PARAMETER["central_meridian",-8.1331084],
    PARAMETER["latitude_of_origin",39.6682583334],
    PARAMETER["scale_factor",1.0],
    PARAMETER["false_easting",0.0],
    PARAMETER["false_northing",0.0],
    UNIT["m",1.0],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","3763"]]
nome_ap: String (254.0)
classifica: String (254.0)
publica1: String (254.0)
publica2: String (254.0)
jurisdicao: String (254.0)
area_ha: Real (33.15)
sigla: String (254.0)



ogrinfo --version
GDAL 3.4.0, released 2021/11/04 (Used by QGIS 3.22)

ogrinfo -al rnap.shp
INFO: Open of `rnap.shp'
      using driver `ESRI Shapefile' successful.

Layer name: rnap
Metadata:
  DBF_DATE_LAST_UPDATE=1920-08-17
Geometry: Polygon
Feature Count: 48
Extent: (-123561.201500, -300415.40) - (162054.118700, 268136.980100)
Layer SRS WKT:
BOUNDCRS[
    SOURCECRS[
        PROJCRS["ETRS89 / Portugal TM06",
            BASEGEOGCRS["ETRS89",
                ENSEMBLE["European Terrestrial Reference System 1989 
ensemble",

                    MEMBER["European Terrestrial Reference Frame 1989"],
                    MEMBER["European Terrestrial Reference Frame 1990"],
                    MEMBER["European Terrestrial Reference Frame 1991"],
                    MEMBER["European Terrestrial Reference Frame 1992"],
                    MEMBER["European Terrestrial Reference Frame 1993"],
                    MEMBER["European Terrestrial Reference Frame 1994"],
                    MEMBER["European Terrestrial Reference Frame 1996"],
                    MEMBER["European Terrestrial Reference Frame 1997"],
  

Re: [QGIS-Developer] QGIS does not recognize CRS of specific layers generated by GeoServer

[Even Rouault]

I meant "Pedro", sorry for the typo.

Le 27/11/2021 à 20:38, Even Rouault a écrit :


Petro,

I've queued https://github.com/OSGeo/gdal/pull/4908 which will "fix" 
that particular case (more precisely anything based on ETRS89). I'm 
not super satisfied with this, and I believe there's no miracle 
solution for all potential use cases. What GDAL returns in both 2.4.0 
and 3.4.0 is actually pretty much the same object, despite the 
different representation (WKT1 vs WKT2). The root cause of the issue 
is with the TOWGS84[] node. Those nodes are not part of CRS 
definitions. They date back to the time where the GDAL & PROJ stack 
wasn't capable of more powerful transformation, and so a particular 
datum transformation to WGS 84 was picked up, and encoded in the CRS 
definition. When reading such definitions nowadays, we have a dilemna: 
did the user really meant to use the particular transformation that 
was in TOWGS84[], or was it just there because that was the default 
one. We ere on the side of caution (most of time, there are a few 
exceptions there and there in GDAL), and keep the transformation 
information, which makes that this CRS is formally not the official 
definition, and thus considered as ""custom""


However, for ETRS89, I believe we can safely remove it, since as far 
as I know, there's no better transformation between ETRS89 (which is 
known modeled as a "datum ensemble" by EPSG, meaning any of the 11 
ETRF realizations) and WGS 84 (itself a datum ensemble) than the 
TOWGS84[0.0,0.0,0.0,0.0,0.0,0.0,0.0], and it is unlikely there will be 
an alternative one in the future, given the dynamic nature of the WGS 
84 datum (maybe I'm speaking too fast. When ETRS89 and WGS 84 
coordinates will have drifted too far away in the future, maybe there 
will some non-null transformation added in the database, so that users 
who want to transform to a 'past' or 'future' coordinate epoch of WGS 
84 can do that in a not too bad way.)


(I should not that this .prj file isn't really compliant with the Esri 
dialect of WKT, which is supposed to be used for shapefiles, since 
normally it shouldn't contain TOWGS84 or AUHTORITY code)


Even


Le 27/11/2021 à 19:13, Pedro Venâncio a écrit :

Hi all,

I'm seeing that QGIS 3.16 / 3.22 does not recognize the CRS of 
specific layers generated by GeoServer, but the old QGIS 2.18.28 
recognize.


For instance, this shapefile: https://si.icnf.pt/shp/rnap 
<https://si.icnf.pt/shp/rnap>


Checking ogrinfo in different versions:

ogrinfo --version
GDAL 2.4.0, released 2018/12/14 (Used by QGIS 2.18.28)

ogrinfo -al rnap.shp
INFO: Open of `rnap.shp'
      using driver `ESRI Shapefile' successful.

Layer name: rnap
Metadata:
  DBF_DATE_LAST_UPDATE=1921-11-27
Geometry: Polygon
Feature Count: 48
Extent: (-123561.201500, -300415.40) - (162054.118700, 268136.980100)
Layer SRS WKT:
PROJCS["ETRS89 / Portugal TM06",
    GEOGCS["ETRS89",
        DATUM["European Terrestrial Reference System 1989",
            SPHEROID["GRS 1980",6378137.0,298.257222101,
                AUTHORITY["EPSG","7019"]],
            TOWGS84[0.0,0.0,0.0,0.0,0.0,0.0,0.0],
            AUTHORITY["EPSG","6258"]],
        PRIMEM["Greenwich",0.0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.017453292519943295],
        AXIS["Geodetic longitude",EAST],
        AXIS["Geodetic latitude",NORTH],
        AUTHORITY["EPSG","4258"]],
    PROJECTION["Transverse_Mercator",
        AUTHORITY["EPSG","9807"]],
    PARAMETER["central_meridian",-8.1331084],
    PARAMETER["latitude_of_origin",39.6682583334],
    PARAMETER["scale_factor",1.0],
    PARAMETER["false_easting",0.0],
    PARAMETER["false_northing",0.0],
    UNIT["m",1.0],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","3763"]]
nome_ap: String (254.0)
classifica: String (254.0)
publica1: String (254.0)
publica2: String (254.0)
jurisdicao: String (254.0)
area_ha: Real (33.15)
sigla: String (254.0)



ogrinfo --version
GDAL 3.4.0, released 2021/11/04 (Used by QGIS 3.22)

ogrinfo -al rnap.shp
INFO: Open of `rnap.shp'
      using driver `ESRI Shapefile' successful.

Layer name: rnap
Metadata:
  DBF_DATE_LAST_UPDATE=1920-08-17
Geometry: Polygon
Feature Count: 48
Extent: (-123561.201500, -300415.40) - (162054.118700, 268136.980100)
Layer SRS WKT:
BOUNDCRS[
    SOURCECRS[
        PROJCRS["ETRS89 / Portugal TM06",
            BASEGEOGCRS["ETRS89",
                ENSEMBLE["European Terrestrial Reference System 1989 
ensemble",

                    MEMBER[&quo

Re: [Qgis-developer] WCS and Raster improvements for QGIS

[Even Rouault]

Hi Tim,

> - Implement support for WCS as a native QGIS raster driver

Just curious : has levering and/or improving the GDAL WCS driver been considered
?
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] WCS and Raster improvements for QGIS

[Even Rouault]

> Yes - Radim already made a first pass implementation using GDAL WCS
> though he said there were some problems with it - I think the most
> critical being that it doesn't support proxy access (I am speaking
> under correction here as I haven't looked into it in that much detail
> myself).

This should be rather easy to add. It is just a matter of setting the right
option to CPLHTTPFetch() (and thus the curl library), namely :

PROXY=val, to make requests go through a proxy server, where val is of the
form proxy.server.com:port_number
PROXYUSERPWD=val, where val is of the form username:password

Note: They can also already be used by setting the GDAL_HTTP_PROXY and
GDAL_HTTP_PROXYUSERPWD configuration options/environmenet variable, which will
act on all GDAL/OGR drivers that go through the CPLHTTPFetch() function.

I see Radim is already a GDAL committer, so there shouldn't be too much overhead
if he wants to hack in the WCS driver.

> I will let Radim comment on any other issues. I think from
> the project perspective we don't mind too much which approach is used
> if the result is efficient and integrates nicely into QGIS.
>
> Regards
>
> Tim
>
>
>
> --
> Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
> ==
> Please do not email me off-list with technical
> support questions. Using the lists will gain
> more exposure for your issues and the knowledge
> surrounding your issue will be shared with all.
>
> Visit http://linfiniti.com to find out about:
>  * QGIS programming and support services
>  * Mapserver and PostGIS based hosting plans
>  * FOSS Consulting Services
> Skype: timlinux
> Irc: timlinux on #qgis at freenode.net
> ==
>


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] WCS and Raster improvements for QGIS

[Even Rouault]

Hi Radim,

> - (I know that OS people don't like to here it and  they will argue
> that is is not a way ahead.) WCS is not widespread yet and there are
> various issues with implementation in UMN Mapserver (and other
> servers) and GDAL. Combination of that issues made it for example
> impossible to support WCS 1.1 in QGIS via GDAL WCS provider. In QGIS
> we can use various hacks to get over various issues with servers, for
> example axis switching etc. which would not be accepted in GDAL.

Well, when developing the WFS driver in OGR, I have had to implement various
hacks to support the varying behaviour of the WFS servers in the wild, so I
don't think it is an absolute obstacle. There's probably still work to deal with
axis switching in GDAL. For geographic SRS, the support for axis switching is
already there (I didn't check if it is used in the WCS driver, but it is there
at the OGRSpatialReference level, and it is already used in the WMS and WFS
drivers). I have also a patch pending to deal with projected SRS that have axis
switched, if that's what is missing. If the issue is that some servers don't
behave accordingly to the spec, we could imagine to have an option in the
description XML to pass that information to the driver.

> Then
> there is a question of time, if you look at deployment on Linux, it
> takes months or years to get fixed package into a distribution. In
> last two days, I found for example new issues with both UMN Mapserver
> and GDAL which we can get around in QGIS, but we would need to wait
> years until fixed versions get to distributions.

Not false, but I'm wondering why a new version of QGIS would be packaged faster
than a GDAL one. QGIS could document that QGIS X needs at least GDAL >= Y if it
needs Y for a functionality.

> - question/suggestion, GDAL i using CPLHTTPFetch() to get data from
> server, would it be possible to add a possibility to set user function
> to be used instead of default CPLHTTPFetch()? Something like
> SetHTTPFetchFunction( myHTTPFetch ). That would make it possible to
> use GDAL providers with QgsNetworkAccessManager for example.

I was actually thinking to something like that when reading Marco's email.
Sounds perfectly doable. The only point that could be tricky is if you need a
user_data pointer that is specific to each WCS connexion.

My intention was not to force the use of GDAL, but to know if there were
technical reasons that would prevent from using it. This work could be an
opportunity to make its WCS driver better, for QGIS benefit, and all other
projects that rely on GDAL.

Best regards,

Even
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] WCS and Raster improvements for QGIS

[Even Rouault]

> Yes, requests must be somehow assigned to layers/providers so that
> progress may be emitted from the right place.  It could be maybe done
> using GDALDatasetH pointer? Just another hack.

I've no ideal solution for this. Your suggestion could be indeed a possibility :
adding 2 methods GDALDatasetSetUserData() & GDALDatasetGetUserData() and the
user data being passed to the custom http fetcher if it is defined. Except that
I think that in the Open() method of the WCS driver, there are already HTTP
requests done to determine some characteristics of the dataset, so that is not
practical since the dataset handle isn't yet known at that time. Would need more
thinking.

>
> In theory, we are going to use threads, which could complicate it even
> more. It is not yet clear to me if QgsRasterDataProvider::readBlock of
> an instance is also going to be called from multiple threads
> simultaneously?

That can be a problem. And it isn't specific to the WCS driver. It is generally
not safe to access the *same* dataset handle concurrently from multiple threads
with most GDAL drivers, because the XXXDataset or XXXRasterBand classes may have
state variables that aren't protected against concurrent access. Looking a bit
at the WCS driver, I can see that the osResultFilename and pabySavedDataBuffer
member variables would certainly not appreciate concurrent access on the same
dataset handle. But looking at how they are used, they is no need to make them
persist between different calls to the IReadBlock() method. So they could be
easily made local variables passed to the GDALOpenResult() and
FlushMemoryResult() methods.

But reading your later email where you mention that the native QGIS WCS driver
is almost there, there's probably no need to dig further into this.

Thanks for having shared your thoughts,

Even
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Memory data provider persistence

[Even Rouault]

Le mercredi 25 juillet 2012 20:03:16, Chris Crook a écrit :
> Hi .. yes I don't like masking the FID field and I'm not thinking this is a
> long term solution, even before the problem in my implementation with the
> field index issue (which I think I will be able to fix).  I'm looking at a
> couple of options over the next few days .. and open to any contributions,
> suggestions, code, etc :-)

Chris,

did you try setting GML_EXPOSE_FID to NO before opening the GML as we 
discussed on the GDAL trac system ? This should fix your issue without much 
code, basically calling CPLSetConfigOption("GML_EXPOSE_FID", "NO") (or 
gdal.SetConfigOption("GML_EXPOSE_FID", "NO") if you're coding in Python) before 
doing the OGROpen() call with the GML file.

Actually, to avoid "polluting" other portions of code, you should save the 
original value and restore it afterwards, so :

in C/C++ :

char* pszOldVal = CPLGetConfigOption("GML_EXPOSE_FID") ? 
strdup(CPLGetConfigOption("GML_EXPOSE_FID")) : NULL;
CPLSetConfigOption("GML_EXPOSE_FID", "NO");
OGRDataSource hDS  = OGROpen();
CPLSetConfigOption("GML_EXPOSE_FID", pszOldVal);
free(pszOldVal);

in Python :

old_val = gdal.GetConfigOption("GML_EXPOSE_FID")
gdal.SetConfigOption("GML_EXPOSE_FID", "NO")
ds = ogr.Open()
gdal.SetConfigOption("GML_EXPOSE_FID",old_val)

And to avoid confusion, yes, you should also avoid that a user creates a 'FID' 
field.

> 
> Cheers
> Chris
> 
> From: haubourg [regis.haubo...@eau-adour-garonne.fr]
> Sent: 23 July 2012 20:58
> To: qgis-developer@lists.osgeo.org
> Subject: Re: [Qgis-developer] Memory data provider persistence
> 
> Hi Chris, thanks for your feedback. I start thinking using a spatialite is
> a great idea. I prefer this than having rules to mask FID field. This can
> generate very hard-to-find bugs..
> 
> Having a complex Qgs zipfile containing either a spatialite DB or xml +
> data would be great but is a big evolution. IMHO, such a break should be
> kept for a major version change in QGIS (2.0?).
> 
> Is that a work you are on ? Need contributors?
> 
> 
> 
> 
> 
> 
> --
> View this message in context:
> http://osgeo-org.1560.n6.nabble.com/Memory-data-provider-persistence-tp410
> 8012p4989995.html Sent from the Quantum GIS - Developer mailing list
> archive at Nabble.com.
> 
> This message contains information, which is confidential and may be subject
> to legal privilege. If you are not the intended recipient, you must not
> peruse, use, disseminate, distribute or copy this message. If you have
> received this message in error, please notify us immediately (Phone 0800
> 665 463 or i...@linz.govt.nz) and destroy the original message. LINZ
> accepts no responsibility for changes to this email, or for any
> attachments, after its transmission from LINZ. Thank You.
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Memory data provider persistence

[Even Rouault]

Le jeudi 26 juillet 2012 21:41:57, Chris Crook a écrit :
> Hi Even
> 
> Thanks for the suggestion and code examples - very helpful. I haven't
> followed up on this yet partly because this has prompted me to look at
> other storage formats for the data (something I'd been putting off).
> 
> If I understand the GML driver documentation properly then I would have to
> manage the possibility of the user having a field called FID, as after
> I've saved the data, then reloaded with GMS_EXPOSE_FID set to NO, that
> field will be lost?

If a user creates a 'fid' field, then that one will be used to set the fid 
attribute of the element. It will not be "lost" strictly speaking, but indeed 
if GML_EXPOSE_FID is set to NO, then it will not be visible.

> As you suggest, renaming the fields before saving and
> then after reloading would fix handle this.

Not sure to understand you here : are you speaking about the mechanism for 
saving the old value of GML_EXPOSE_FID, setting GML_EXPOSE_FID=OLD, calling 
OGROpen() and restoring the original value of GML_EXPOSE_FID ? If so, this 
will not change anything about the behaviour with a user field called 'fid'.

The intention here was not to interfere with other parts of QGIS that might 
manipulate GML files. For example, if a user opens a GML file through "Add 
Vector Layer", he might want to see the fid reported.

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Memory data provider persistence

[Even Rouault]

Le jeudi 26 juillet 2012 22:52:47, Chris Crook a écrit :
> Hi Even
> 
> Sorry .. lost wasn't quite correct.  The point is that my code for saving
> some users data and then reloading needs to take account of whether they
> have a field called fid or not if I want the list of fields unchanged
> after saving and reloading the data.
> 
> Either I have to record somewhere that the fid field is real (ie was in the
> original data set), and set the GML_EXPOSE_FID accordingly so that it
> appears in the reloaded data, or alternatively rename the users fields an
> a reversible way before I save the data so that none of the saved fields
> can be called fid (eg prefix all fields with "u_" for example), and set
> GML_EXPOSE_FID to NO.
> 
> At least that is my understanding of the way the driver works now?

Well, you could do it, but I think you're infliciting yourself too much pain, 
but it's up to you as you're the coder ;-) Personnaly, I'd just prevent the 
user from creating a field called "fid", saying it is a reserved name. But your 
solutions would certainly work.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Large raster (ecw) identify very long

[Even Rouault]

Selon Radim Blazek :

> On Wed, Aug 22, 2012 at 12:29 PM, haubourg
>  wrote:
> >
> > Radim Blazek-2 wrote
> >>
> >>
> >> QGIS is using GDALRasterIO() which reads a single pixel on original
> >> resolution. AFAIK, ECW is using tiles internally so it should be all
> >> very fast. I can imagine 2 problems:
> >>
> >> - the tiles in ECW file are too big - can you verify somehow how big
> >> are the tiles?
> >>
> > Hi Radim, I'm not ecw specialist, but I understand it is a wavelet
> > compression algorithm, not  a tiled one.
> > There is one big image with pyramids inside.
>
> I am not sure, but I thought that it is using wavelet, pyramids and
> tiles. Without tiles it could not be fast enough on higher resolution.
>
> > Radim Blazek-2 wrote
> >>
> >> - GDAL is reading bigger portion of data than necessary (cc to GDAL list)
> >>
> >> Is it drawing of the raster also so slow (on raster resolution zoom)?
> >>
> >> Radim
> >>
> >>
> > No everything is fine when drawing data, whatever resolution I use, this is
> > the big thing with ecw.
> > I must say I find that it was a bit faster with 1.7.4 than 1.8 and later.
> > But this could be related to other factors like internal network
> > capacities.. I haven't benched on it yet.
>
> I found in GDAL ecwdataset.cpp that it is treating  single row
> requests in IRasterIO in a special way:
> if( nYSize == 1 )
> {
> return GDALRasterBand::IRasterIO(eRWFlag, nXOff, nYOff, nXSize,
> nYSize,
>  pData, nBufXSize, nBufYSize,
>  eBufType, nPixelSpace, nLineSpace );
> }


Regis,

what are the dimensions (in pixels) of your big ECW ?

Radim,

I tried the following Python script that must be representative of how QGIS must
do picking (I guess it does a RasterIO(,  x, y, 1, 1, ... 1, 1) )

from osgeo import gdal
import random
import sys

ds = gdal.Open(sys.argv[1])
xsize = ds.RasterXSize
ysize = ds.RasterYSize

while True:
x = random.randint(0, xsize)
y = random.randint(0, ysize)
data = ds.ReadRaster(x,y,1,1)

After some time, the python process occupies the 1/4 of the total RAM. This is
the default behaviour of the ECW SDK documented in
http://gdal.org/frmt_ecw.html. I then set ECW_CACHE_MAXMEM=100 (1 million
bytes) and the memory usage was very small as expected. So I believe there is no
memory leak in the driver. Note: in the old ECW SDK 3.3 (I don't know for the
newer ones), there was a bug in some cases : if the RAM was > 8 GB, RAM / 4 > 2
GB which overflowed a 32 bit, resulting in the SDK allocating memory without
limit.

The ECW driver is quite complex in its reading strategy to establish "views",
but from what I've captured and you noticed, when you ask a window with a 1
pixel height, it goes to IReadBlock() which will fetch one entire line and put
it in the GDAL cache. This is to avoid issuing a SetView() each time a line is
read. The intent here is to be clever for the common line-by-line pattern
access. The drawback of this is that if the raster has a big width and you only
want to read one single pixel, the cost for reading the whole line might be much
greater than the cost of establishing SetView() for your single pixel.

I suppose your workaround in QGIS will be to read 1x2 pixel or something like
that.

Ultimately, we would probably need the following GDAL patch. I can't really say
if it will improve performance a lot because my biggest ECW is "just"
1x1.

Index: ecwdataset.cpp
===
--- ecwdataset.cpp  (revision 24824)
+++ ecwdataset.cpp  (working copy)
@@ -246,8 +246,10 @@
 /*  We will drop down to the block oriented API if only a single*/
 /*  scanline was requested. This is based on the assumption that*/
 /*  doing lots of single scanline windows is expensive. */
+/*  But for single pixel reading (picking use case), this is not a  */
+/*  good strategy for big rasters.  */
 /*  */
-if( nYSize == 1 )
+if( nYSize == 1 && nXSize != 1 )
 {
 #ifdef NOISY_DEBUG
 CPLDebug( "ECWRasterBand",
@@ -1038,7 +1040,7 @@
 /*  If we are requesting a single line at 1:1, we do a multi-band   */
 /*  AdviseRead() and then TryWinRasterIO() again.   */
 /*  */
-if( nYSize == 1 && nBufYSize == 1 && nBandCount > 1 )
+if( nYSize == 1 && nXSize != 1 && nBufYSize == 1 && nBandCount > 1 )
 {
 CPLErr eErr;

@@ -1064,8 +1066,8 @@
 /*  band case where we post a big window for the view, and allow*/
 /*  sequential reads.   */
 /*  */
-if( nXSize < nBufXSize || nYSize < nBufYSize || nYSize == 1
-|| nBan

Re: [Qgis-developer] Large raster (ecw) identify very long

[Even Rouault]

Le mercredi 22 août 2012 20:13:39, Radim Blazek a écrit :
> Even,
> thanks for exhaustive explanation and testing.
> 
> On Wed, Aug 22, 2012 at 2:37 PM, Even Rouault
> 
>  wrote:
> >> I found in GDAL ecwdataset.cpp that it is treating  single row
> > 
> >> requests in IRasterIO in a special way:
> > I tried the following Python script that must be representative of how
> > QGIS must do picking (I guess it does a RasterIO(,  x, y, 1, 1, ...
> > 1, 1) )
> 
> Yes.
> 
> > I suppose your workaround in QGIS will be to read 1x2 pixel or something
> > like that.
> 
> Yes, I have used 2x2.

Hum, I humbly suggest that the fix should be done in the ECW driver rather than 
in QGIS. The issue with the 2x2 workaround is that if other GDAL drivers have 
optimizations for the 1x1 pixel use case (which would be reasonable), they 
will be unused now.

> 
> Radim
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Large raster (ecw) identify very long

[Even Rouault]

> Until it get fixed in GDAL we can use
>
>   if ( GDALGetDriverShortName() == "ECW")
>
> and once you fix that
>
>   #if defined(GDAL_VERSION_NUM) && GDAL_VERSION_NUM < x.x
>   if ( GDALGetDriverShortName() == "ECW")

Seems reasonnable. Except I suggest using a string comparison function and not
== ;-)

Would you mind creating a ticket in GDAL trac about that ?

>
> Radim
>


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Large raster (ecw) identify very long

[Even Rouault]

Le mercredi 22 août 2012 20:13:39, Radim Blazek a écrit :
> Even,
> thanks for exhaustive explanation and testing.
> 
> On Wed, Aug 22, 2012 at 2:37 PM, Even Rouault
> 
>  wrote:
> >> I found in GDAL ecwdataset.cpp that it is treating  single row
> > 
> >> requests in IRasterIO in a special way:
> > I tried the following Python script that must be representative of how
> > QGIS must do picking (I guess it does a RasterIO(,  x, y, 1, 1, ...
> > 1, 1) )
> 
> Yes.
> 
> > I suppose your workaround in QGIS will be to read 1x2 pixel or something
> > like that.
> 
> Yes, I have used 2x2.

Radim,

I've tested the big ECW (1.8 GB, 141970 x 141970) provided by Jukka and could 
indeed reproduce a noticable performance issue when simulating picking with my 
tiny GDAL python script. In QGIS, the performance was still OK however, a few 
hundreds of milliseconds for each picking. It is just when you do a lot of 
iterations that you begin to notice the slowness, or if the dataset is really 
huge.

I've fixed that in http://trac.osgeo.org/gdal/ticket/4790 in trunk and 1.9 
branch. So it will be in GDAL 1.9.2. Actually, as I think that the ECW driver 
is generally available through a plugin (and not built-in in GDAL DLL), this 
could be available sooner if the ECW plugin is rebuilt with the fixed sources.

Best regards,

Even
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] invalid conversion from 'void*' to 'OGRSpatialReferenceHS*'

[Even Rouault]

Selon Marco Bernasocchi :

> hi every body, updating to th letest android qt version is being tougher
> than planned ... I got all the dependencies ok now but building qgis gives
> me this errors below. any Ideas? this is using android ndk 8b with gcc 4.4.3
>
> thanks a lot
>
> /home/marco/dev/Quantum-GIS/src/core/qgscoordinatereferencesystem.cpp:
> In destructor
> 'QgsCoordinateReferenceSystem::~QgsCoordinateReferenceSystem()':
> /home/marco/dev/Quantum-GIS/src/core/qgscoordinatereferencesystem.cpp:78:
> error:
> invalid conversion from 'void*' to 'OGRSpatialReferenceHS*'

There's an error in
https://github.com/qgis/Quantum-GIS/blob/master/src/core/qgscoordinatereferencesystem.h#L475

The pointer should be of type OGRSpatialReferenceH instead of void*

The compilation error must be unnoticed for people compiling QGIS without
-DDEBUG defined, but when -DDEBUG is actually defined, OGRSpatialReferenceH is
no longer aliased to void*. See
http://trac.osgeo.org/gdal/browser/trunk/gdal/ogr/ogr_api.h#L59
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

[Qgis-developer] Inconsistant use of VSIMalloc() / malloc() and VSIFree() / free()

[Even Rouault]

Hi,

I'm running QGIS master with a GDAL trunk built with a mechanism to detect 
unbalanced use of VSIMalloc() and VSIFree().

To enable it, edit gdal/port/cpl_vsisimple.cpp and uncomment the //#define 
DEBUG_VSIMALLOC line at the beginning of the file. When this is enabled, the 
pointer returned by VSIMalloc() shouldn't be passed to free(). And if 
VSIFree() is provided a pointer allocated by malloc(), it will abort.

This is mainly usefull to detect issues on Windows : if QGIS and GDAL are 
built with different MSVC versions, the C runtimes might not be compatible, and 
a free() done in DLL A of a pointer allocated by malloc() in a DLL B might 
crash. For example, when called from QGIS, free(VSIMalloc()) or 
VSIFree(malloc()) can result in crashes.

And indeed I get crashes in QGIS when it is enabled, for example on the free() 
in qgsrasterdrawer.cpp:73

I've searched in QGIS code and I can see that several classes implement void * 
X::readBlock( int bandNo, QgsRectangle  const & extent, int width, int 
height ), but they return a buffer sometimes allocated by malloc(), or 
sometimes by VSIMalloc().

Some choice must be made ... My intuition would be that using regular malloc() 
/ free() would be better.

Basically, you would restrict the use of VSIFree() to only strings/buffers 
returned by GDAL. To avoid unnecessary changes in existing code, balanced use 
of VSIMalloc() / VSIFree() in the context of a method is OK. The real issue is 
when a method returns a void* pointer and you don't know which free function 
must be called on it.

Best regards,

Even

PS: The same holds for CPLMalloc() / CPLFree(). In the context of the above 
analysis, they can be considered as strictly equivalent to VSIMalloc() / 
VSIFree().
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Inconsistant use of VSIMalloc() / malloc() and VSIFree() / free()

[Even Rouault]

Le samedi 29 septembre 2012 12:58:09, Paolo Cavallini a écrit :
> Il 28/09/2012 22:45, Even Rouault ha scritto:
> > Hi,
> > 
> > I'm running QGIS master with a GDAL trunk built with a mechanism to
> > detect unbalanced use of VSIMalloc() and VSIFree().
> 
> Hi Even,
> thanks for pointing this ou. Perhaps better opening a ticket, not to forget
> about this? And/or adding it to coding guidelines?

I think some discussion is needed first to agree on the technical solution. 
>From the headers, I see that Marco and Radim are the most active contributors 
in that area.

> All the best.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Inconsistant use of VSIMalloc() / malloc() and VSIFree() / free()

[Even Rouault]

Le samedi 29 septembre 2012 15:24:59, Etienne Tourigny a écrit :
> Hi,
> 
> my opinion is that VSIMalloc/VSIFree should be used mainly by the
> gdal/ogr providers, and if needed elsewhere (e.g. when calling
> gdal/ogr code)

This is even not needed for code that interact with GDAL/OGR. GDAL/OGR 
functions that take memory buffers just works fine with whatever memory 
allocator you want (malloc, new[], etc...)

The only requirement is that VSIFree( ) *must* be used to free a pointer 
allocated by GDAL, to make sure that the CRT used for free() matches the CRT 
used for malloc().

> , but alloc/free only internally to a function/class.
> 
> Basically, following Even's recommendations.
> 
> for example, in QgsRasterDataProvider::readBlock :
>   // TODO: replace VSIMalloc, it is GDAL function
>   void * data = VSIMalloc( dataTypeSize( bandNo ) * width * height );
> ...
> return data;

You didn't finish your sentence here. So I suppose you meant that the TODO 
should be implemented by replacing VSIMalloc() by malloc() ?

> 
> 
> QgsRasterFileWriter makes much use of VSIMalloc/VSIFree but that's
> probably ok, because its use seems to be limited to internal use and
> passed to QgsRasterIterator, but freed locally
> 
> However, QgsRasterFileWriter::writeImageRaster uses VSIMalloc()
> followed by CPLFree() - is that ok?

Yes, CPLFree(VSIMalloc()) is OK

The difference between VSIMalloc() and CPLMalloc() is that CPLMalloc() calls 
internally VSIMalloc() and aborts if the allocation fails. Thus CPLMalloc() 
never returns a NULL pointer and is only meant for very small allocations that 
shouldn't fail. My personnal opinion on this is that it should be completely 
avoided and replaced by VSIMalloc() with a proper check of the return value.

VSIFree() and CPLFree() are the same: in port/cpl_conv.h : "#define CPLFree 
VSIFree"

> 
> Even - did you try to save a raster (using new raster save as dialog)
> with your modifications?

I didn't do any modification yet. With the memory allocation debugging enabled, 
QGIS just crashes when opening a raster, so I couldn't test saving ;-)

> 
> 
> On the other hand, it could go the other way around - make sure that
> all frees are using VSIFree() instead of free().

That's one possibility, but I was wondering if there are other raster 
providers than GDAL. In that case, they are unlikely to allocate with 
VSIMalloc().

> 
> Even - what are the fundamental advantages of using VSIMalloc()
> instead of malloc()?

Basically none, from the application's point of view when GDAL is compiled 
with default options.

For the purpose of GDAL developement, VSIMalloc() / VSIFree() are interesting, 
because they can be instrumented to provide debugging and statistics 
facilities. For example, if you enable both DEBUG_VSIMALLOC and 
DEBUG_VSIMALLOC_STATS, you get the following report at the end of each process 
using GDAL :

Current VSI memory usage: 0 bytes
Maximum VSI memory usage: 23781 bytes
Number of calls to VSIMalloc()  : 770
Number of calls to VSICalloc()  : 0
Number of calls to VSIRealloc() : 235
Number of calls to VSIFree(): 770
VSIMalloc + VSICalloc - VSIFree : 0

This is a cheap way of tracking most memory leaks with quite unnoticeable 
slowdown. When DEBUG_VSIMALLOC is enabled, there's also a check in VSIFree() 
to detect that the pointer passed is valid (i.e. allocated by VSIMalloc()) and 
to detect double-frees. A kind of poor man's "valgrind --leak-check=full" to 
sum it up ;-) For my developement activities in GDAL, I enable that all the 
time.

Of course, other applications can benefit from that by using 
VSIMalloc()/VSIFree(). But you must be very carefull to correctly pair 
VSIMalloc() with VSIFree(), and malloc() with free() : any other combination 
will lead to chaos.


Even
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer