[gdal-dev] WFS: How to use feature ordering?

[Craig de Stigter via gdal-dev]

Hey folks

We're having a lot of trouble getting data out of a WFS server that doesn't
have primary keys. (previous work on this)



The URL that's causing us grief is this one:
https://opendata.maps.vic.gov.au/geoserver/wfs?REQUEST=GetFeature=WFS=2.0.0=open-data-platform:v_s_parcel_proposed=0=2

* we can't disable pagination, because the server only returns 5000 records
even if we ask for more.
* we can't paginate without ordering, because there's no PK.

So, adding a `=parcel_pfi` to the querystring fixes the issue. But I
can't seem to get GDAL to do that for us.

There does seem to be code for applying an "ORDER BY" clause from
ExecuteSQL to the WFS 1.1+ SORTBY field. However, I can't get it to work.

I made a test script

to demonstrate the problem. It uses ExecuteSQL to apply an ORDER BY clause,
but the resulting curl queries don't have a SORTBY parameter. Then it
crashes after the server returns a 400 error due to the missing SORTBY.

I assume there's something I need to do to cause GDAL to use the SORTBY
parameter but I'm not sure what it is. Any tips would be appreciated.

-- 
Regards,
Craig

Platform Engineer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Shapefile: Write alternative PRJ WKT format

[Craig de Stigter]

Hi there

We export data in a variety of coordinate systems and formats. Most formats
work with most coordinate systems.

However, the SHP driver always calls MorphToESRI() on its SRS objects
before writing them as WKT to the PRJ files.

I haven't been able to find information on why it does this. It means
certain coordinate systems (geocentric, for instance) don't work with SHP
and raise an exception.

Could we add a creation option to customise this behaviour?

-- 
Regards,
Craig

Platform Engineer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Deciding whether to use WFS paging after opening the datasource

[Craig de Stigter]

Hi folks

We're having trouble with OGR trying to fetch a response from this WFS2
URL:
https://geo.irceline.be/wfs?SERVICE=WFS=2.0.0=GetFeature=realtime:pm25_24hmean_station=100=0

The error from the server is

Cannot do natural order without a primary key, please add it or specify a
> manual sort over existing attributes
>

This seems fair enough on the surface - we specified *STARTINDEX=0* but the
server is unable to support paging for this dataset due to no natural sort
order.

We'd happily disable paging for this dataset - it has a fairly low number
of features - but we would prefer to implement a general purpose solution
rather than special-case this particular dataset.

It would make sense to open the datasource, check the feature count by
doing a 'hits' query (we're doing this anyway), and then use paging if
there are more than 100 features - otherwise don't use paging.

However I can't find a way to accomplish this with GDAL. The
*OGR_WFS_PAGING_ALLOWED* config var is only checked when the datasource is
opened - so you can't open the datasource, check the feature count and then
change the paging behaviour.

I can imagine either of these changes in GDAL itself might fix this:

1. GDAL could check *OGR_WFS_PAGING_ALLOWED* when actually first issuing
GetFeature requests, rather than during datasource creation
2. GDAL could omit the pagination parameters if the feature count is
already known and is less than the page size

How would you suggest we proceed?

-- 
Regards,
Craig

Platform Engineer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] VSI CIFS driver

[Craig de Stigter]

It was a good idea. We've been trying to make the FUSE thing work, but
there are a lot of hoops to jump through.

   - We did find `smbnetfs`, which is a *slightly* more updated package
   with seemingly more sensible usage information
   - Docker doesn't support it unless you have CAP_SYS_ADMIN:
   https://github.com/docker/for-linux/issues/321 - which defeats the
   purpose; we might as well just use `mount.cifs` at that point.

We're going to switch back to looking into implementing a VSI driver for
wrapping libsmb.

On Mon, 16 Dec 2019 at 15:41, Craig de Stigter <
craig.destig...@koordinates.com> wrote:

> Sorry, that's `fusesmb`, not `smbfuse`
>
> On Mon, 16 Dec 2019 at 15:40, Craig de Stigter <
> craig.destig...@koordinates.com> wrote:
>
>> I haven't quite exhausted my frustration budget for this yet but here's
>> what I've found so far:
>>
>>
>>- `smbfuse` appears to be abandoned since 2007, though Ubuntu is
>>still packaging it through 19.10.
>>- I can't really find anything else in this space.
>>- To use FUSE at all you need the `fuse` module, which isn't enabled
>>in any of the default images I tried.
>>- Some systems don't seem to support loading kernel modules, e.g. the
>>xhyve VM on Docker for Desktop (well, the Mac version anyway) doesn't seem
>>to allow `modprobe`
>>
>> I think it's probably more straightforward to implement a VSI CIFS than a
>> FUSE CIFS.
>>
>>
>>
>> On Sat, 14 Dec 2019 at 20:42, Craig de Stigter <
>> craig.destig...@koordinates.com> wrote:
>>
>>> Thanks Even
>>>
>>> That sounds like a possible path forward, although it works for FUSE
>>> filesystems only, so we'd have to find a good FUSE implementation of CIFS
>>> (currently we're using the kernel provided cifs)
>>>
>>> If anyone can recommend a well-maintained FUSE CIFS implementation I'd
>>> gladly look into it. After some googling I don't see much that fills me
>>> with confidence, but I will continue to search at work next week.
>>>
>>> I'll keep you updated  on what I discover. Thanks
>>> Craig
>>>
>>> On Fri, 13 Dec 2019 at 23:38, Even Rouault 
>>> wrote:
>>>
>>>> Craig,
>>>>
>>>> > We have to access some customer data via their CIFS shares on a
>>>> regular
>>>> > basis.
>>>> >
>>>> > To date we have been doing it via a setuid script which mounts and
>>>> unmounts
>>>> > the CIFS share locally, then treats the files as local. We're
>>>> > containerising our workers, and using kernel mounts is no longer
>>>> possible
>>>> > (at least, not even slightly securely.)
>>>>
>>>> Is upgrading to Kernel 4.18 an option ?
>>>> See https://www.phoronix.com/scan.php?page=news_item=Linux-4.18-FUSE
>>>>
>>>> >
>>>> > This means we need a userspace CIFS implementation for a lot of our
>>>> GDAL
>>>> > code, and a VSI CIFS driver would seem to be the right thing to do.
>>>> > Presumably it will be a fairly thin wrapper of libsmb.
>>>> >
>>>> > 2. Would it be something GDAL might accept as a core contribution, or
>>>> > should we expect to maintain it outside of core?
>>>>
>>>> Hard to know where to draw the line. If recent kernels would make this
>>>> no
>>>> longer needed, then its value in core might be low.
>>>>
>>>> > 3. We noticed https://github.com/OSGeo/gdal/pull/1289 should allow
>>>> VSI
>>>> > plugins to be registered from Python,
>>>>
>>>> It allows VSI plugins to be registered with a C interface. So if you'd
>>>> want ot
>>>> use that from Python, you'd likely have to play with ctypes or similar
>>>> technology.
>>>>
>>>> > Are there any docs on how to actually do this?
>>>>
>>>> Well, the Doxygen docs of VSIAllocFilesystemPluginCallbacksStruct and
>>>> VSIInstallPluginHandler in cpl_vsi.h . And the tutorial you'll add if
>>>> you
>>>> proceed that way ;-)
>>>>
>>>>
>>>> Even
>>>>
>>>> --
>>>> Spatialys - Geospatial professional services
>>>> http://www.spatialys.com
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Craig
>>>
>>> Developer
>>> Koordinates
>>>
>>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
>>> <https://twitter.com/koordinates>
>>>
>>
>>
>> --
>> Regards,
>> Craig
>>
>> Developer
>> Koordinates
>>
>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
>> <https://twitter.com/koordinates>
>>
>
>
> --
> Regards,
> Craig
>
> Developer
> Koordinates
>
> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
> <https://twitter.com/koordinates>
>


-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
<https://twitter.com/koordinates>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] VSI CIFS driver

[Craig de Stigter]

Sorry, that's `fusesmb`, not `smbfuse`

On Mon, 16 Dec 2019 at 15:40, Craig de Stigter <
craig.destig...@koordinates.com> wrote:

> I haven't quite exhausted my frustration budget for this yet but here's
> what I've found so far:
>
>
>- `smbfuse` appears to be abandoned since 2007, though Ubuntu is still
>packaging it through 19.10.
>- I can't really find anything else in this space.
>- To use FUSE at all you need the `fuse` module, which isn't enabled
>in any of the default images I tried.
>- Some systems don't seem to support loading kernel modules, e.g. the
>xhyve VM on Docker for Desktop (well, the Mac version anyway) doesn't seem
>to allow `modprobe`
>
> I think it's probably more straightforward to implement a VSI CIFS than a
> FUSE CIFS.
>
>
>
> On Sat, 14 Dec 2019 at 20:42, Craig de Stigter <
> craig.destig...@koordinates.com> wrote:
>
>> Thanks Even
>>
>> That sounds like a possible path forward, although it works for FUSE
>> filesystems only, so we'd have to find a good FUSE implementation of CIFS
>> (currently we're using the kernel provided cifs)
>>
>> If anyone can recommend a well-maintained FUSE CIFS implementation I'd
>> gladly look into it. After some googling I don't see much that fills me
>> with confidence, but I will continue to search at work next week.
>>
>> I'll keep you updated  on what I discover. Thanks
>> Craig
>>
>> On Fri, 13 Dec 2019 at 23:38, Even Rouault 
>> wrote:
>>
>>> Craig,
>>>
>>> > We have to access some customer data via their CIFS shares on a regular
>>> > basis.
>>> >
>>> > To date we have been doing it via a setuid script which mounts and
>>> unmounts
>>> > the CIFS share locally, then treats the files as local. We're
>>> > containerising our workers, and using kernel mounts is no longer
>>> possible
>>> > (at least, not even slightly securely.)
>>>
>>> Is upgrading to Kernel 4.18 an option ?
>>> See https://www.phoronix.com/scan.php?page=news_item=Linux-4.18-FUSE
>>>
>>> >
>>> > This means we need a userspace CIFS implementation for a lot of our
>>> GDAL
>>> > code, and a VSI CIFS driver would seem to be the right thing to do.
>>> > Presumably it will be a fairly thin wrapper of libsmb.
>>> >
>>> > 2. Would it be something GDAL might accept as a core contribution, or
>>> > should we expect to maintain it outside of core?
>>>
>>> Hard to know where to draw the line. If recent kernels would make this
>>> no
>>> longer needed, then its value in core might be low.
>>>
>>> > 3. We noticed https://github.com/OSGeo/gdal/pull/1289 should allow VSI
>>> > plugins to be registered from Python,
>>>
>>> It allows VSI plugins to be registered with a C interface. So if you'd
>>> want ot
>>> use that from Python, you'd likely have to play with ctypes or similar
>>> technology.
>>>
>>> > Are there any docs on how to actually do this?
>>>
>>> Well, the Doxygen docs of VSIAllocFilesystemPluginCallbacksStruct and
>>> VSIInstallPluginHandler in cpl_vsi.h . And the tutorial you'll add if
>>> you
>>> proceed that way ;-)
>>>
>>>
>>> Even
>>>
>>> --
>>> Spatialys - Geospatial professional services
>>> http://www.spatialys.com
>>>
>>
>>
>> --
>> Regards,
>> Craig
>>
>> Developer
>> Koordinates
>>
>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
>> <https://twitter.com/koordinates>
>>
>
>
> --
> Regards,
> Craig
>
> Developer
> Koordinates
>
> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
> <https://twitter.com/koordinates>
>


-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
<https://twitter.com/koordinates>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] VSI CIFS driver

[Craig de Stigter]

I haven't quite exhausted my frustration budget for this yet but here's
what I've found so far:


   - `smbfuse` appears to be abandoned since 2007, though Ubuntu is still
   packaging it through 19.10.
   - I can't really find anything else in this space.
   - To use FUSE at all you need the `fuse` module, which isn't enabled in
   any of the default images I tried.
   - Some systems don't seem to support loading kernel modules, e.g. the
   xhyve VM on Docker for Desktop (well, the Mac version anyway) doesn't seem
   to allow `modprobe`

I think it's probably more straightforward to implement a VSI CIFS than a
FUSE CIFS.



On Sat, 14 Dec 2019 at 20:42, Craig de Stigter <
craig.destig...@koordinates.com> wrote:

> Thanks Even
>
> That sounds like a possible path forward, although it works for FUSE
> filesystems only, so we'd have to find a good FUSE implementation of CIFS
> (currently we're using the kernel provided cifs)
>
> If anyone can recommend a well-maintained FUSE CIFS implementation I'd
> gladly look into it. After some googling I don't see much that fills me
> with confidence, but I will continue to search at work next week.
>
> I'll keep you updated  on what I discover. Thanks
> Craig
>
> On Fri, 13 Dec 2019 at 23:38, Even Rouault 
> wrote:
>
>> Craig,
>>
>> > We have to access some customer data via their CIFS shares on a regular
>> > basis.
>> >
>> > To date we have been doing it via a setuid script which mounts and
>> unmounts
>> > the CIFS share locally, then treats the files as local. We're
>> > containerising our workers, and using kernel mounts is no longer
>> possible
>> > (at least, not even slightly securely.)
>>
>> Is upgrading to Kernel 4.18 an option ?
>> See https://www.phoronix.com/scan.php?page=news_item=Linux-4.18-FUSE
>>
>> >
>> > This means we need a userspace CIFS implementation for a lot of our GDAL
>> > code, and a VSI CIFS driver would seem to be the right thing to do.
>> > Presumably it will be a fairly thin wrapper of libsmb.
>> >
>> > 2. Would it be something GDAL might accept as a core contribution, or
>> > should we expect to maintain it outside of core?
>>
>> Hard to know where to draw the line. If recent kernels would make this no
>> longer needed, then its value in core might be low.
>>
>> > 3. We noticed https://github.com/OSGeo/gdal/pull/1289 should allow VSI
>> > plugins to be registered from Python,
>>
>> It allows VSI plugins to be registered with a C interface. So if you'd
>> want ot
>> use that from Python, you'd likely have to play with ctypes or similar
>> technology.
>>
>> > Are there any docs on how to actually do this?
>>
>> Well, the Doxygen docs of VSIAllocFilesystemPluginCallbacksStruct and
>> VSIInstallPluginHandler in cpl_vsi.h . And the tutorial you'll add if you
>> proceed that way ;-)
>>
>>
>> Even
>>
>> --
>> Spatialys - Geospatial professional services
>> http://www.spatialys.com
>>
>
>
> --
> Regards,
> Craig
>
> Developer
> Koordinates
>
> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
> <https://twitter.com/koordinates>
>


-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
<https://twitter.com/koordinates>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] VSI CIFS driver

[Craig de Stigter]

Thanks Even

That sounds like a possible path forward, although it works for FUSE
filesystems only, so we'd have to find a good FUSE implementation of CIFS
(currently we're using the kernel provided cifs)

If anyone can recommend a well-maintained FUSE CIFS implementation I'd
gladly look into it. After some googling I don't see much that fills me
with confidence, but I will continue to search at work next week.

I'll keep you updated  on what I discover. Thanks
Craig

On Fri, 13 Dec 2019 at 23:38, Even Rouault 
wrote:

> Craig,
>
> > We have to access some customer data via their CIFS shares on a regular
> > basis.
> >
> > To date we have been doing it via a setuid script which mounts and
> unmounts
> > the CIFS share locally, then treats the files as local. We're
> > containerising our workers, and using kernel mounts is no longer possible
> > (at least, not even slightly securely.)
>
> Is upgrading to Kernel 4.18 an option ?
> See https://www.phoronix.com/scan.php?page=news_item=Linux-4.18-FUSE
>
> >
> > This means we need a userspace CIFS implementation for a lot of our GDAL
> > code, and a VSI CIFS driver would seem to be the right thing to do.
> > Presumably it will be a fairly thin wrapper of libsmb.
> >
> > 2. Would it be something GDAL might accept as a core contribution, or
> > should we expect to maintain it outside of core?
>
> Hard to know where to draw the line. If recent kernels would make this no
> longer needed, then its value in core might be low.
>
> > 3. We noticed https://github.com/OSGeo/gdal/pull/1289 should allow VSI
> > plugins to be registered from Python,
>
> It allows VSI plugins to be registered with a C interface. So if you'd
> want ot
> use that from Python, you'd likely have to play with ctypes or similar
> technology.
>
> > Are there any docs on how to actually do this?
>
> Well, the Doxygen docs of VSIAllocFilesystemPluginCallbacksStruct and
> VSIInstallPluginHandler in cpl_vsi.h . And the tutorial you'll add if you
> proceed that way ;-)
>
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>


-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] VSI CIFS driver

[Craig de Stigter]

Hey folks!

We have to access some customer data via their CIFS shares on a regular
basis.

To date we have been doing it via a setuid script which mounts and unmounts
the CIFS share locally, then treats the files as local. We're
containerising our workers, and using kernel mounts is no longer possible
(at least, not even slightly securely.)

This means we need a userspace CIFS implementation for a lot of our GDAL
code, and a VSI CIFS driver would seem to be the right thing to do.
Presumably it will be a fairly thin wrapper of libsmb.

1. Has anyone tried that? Any comments?

2. Would it be something GDAL might accept as a core contribution, or
should we expect to maintain it outside of core?

3. We noticed https://github.com/OSGeo/gdal/pull/1289 should allow VSI
plugins to be registered from Python, which means we could possibly
implement this in Python if we decided that was the best approach. Because
we're Python experts and not C++ experts, we're likely to choose this path
if the answer to (2) is "no". Are there any docs on how to actually do
this?


Thanks :)

-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Python IOError during file object destructor

[Craig de Stigter]

Hey there

We're using the python bindings to do all sorts of translates and warps and
things, and I don't have a proper traceback but we got this in some of our
processes lately:

  Exit code: 0
  stderr:
...
close failed in file object destructor:
IOError: [Errno 5] Input/output error


I *suspect strongly* that this IOError is happening during process
shutdown, and also that it is coming from GDAL rather than our code (I
could be wrong!)

The IOError itself must have come from a disk problem, but instead of being
raised, this IOError was silently ignored, leading to problems with a
corrupted file later.

As far as I can tell all GDAL datasets have gone out of scope long before
the process exits. However I wondered if we should be calling some kind of
shutdown function to force GDAL to close all its file handles, or
something. I recall various things like that existing (e.g.
`GDALDestroyDriverManager()`) from my pytest-related work, but at present
we're not using anything like that anywhere.

TIA

-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Vagrant build environment

[Craig de Stigter]

Hi folks

I sat down to finish off a pull request I started months ago, and realised
that GDAL's vagrant build no longer works. It fails quite early in
provisioning.

I think this is probably because Ubuntu 14.04 is no longer in upstream
support, and the ubuntugis PPA has stopped supplying packages for it:

> default: Unable to locate package postgresql-9.3-postgis-2.2
> default: Unable to locate package libopenjp2-7-dev
> default: Unable to locate package libopenjpip7
> default: Unable to locate package libopenjp3d7
>

Is there a newer recommended way to easily get a dev build of GDAL running,
with a reasonably complete set of drivers? An updated vagrantfile for
Ubuntu 18.04 would be pretty ideal...

-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ubuntu_1804

[Craig de Stigter]

I didnt write it but I had to fiddle with the build scripts quite a bit for
the  pytest refactor. Travis only supports Trusty (recently added Xenial,
though I don't think GDAL is using that yet either.)

The ubuntu_1804 job builds an entire Ubuntu 18.04 install in a chroot
(using debootstrap) inside the travis Trusty environment. It's not terribly
elegant but gets the job done. So the kernel is the travis trusty kernel
but the userland is ubuntu 18.04.

Hope that clarifies things.

Craig de Stigter

On Fri, 25 Jan 2019, 12:34 Alan Stewart,  wrote:

> The raw log for the ubuntu_1804 Travis job (specifically #15454.1) seems
> to indicate that the OS is Ubuntu 14.04 . From the build name I assumed it
> would be Ubuntu 18.04?
>
>
>
> Alan Stewart
>
> Senior Software Engineer
>
> *Terra**Go **Technologies*
>
> 3200 Windy Hill Road, Suite 1550W
>
> Atlanta, GA 30339 USA
>
> O.  +1 678.391.9615
>
>
>
> www.terragotech.com
>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 72: Run tests with pytest

[Craig de Stigter]

(Just saw Jonathan's email)

That approach to `ids` makes sense to me. Let's try and get the current PR
merged and I'll submit that as a followup PR (I have a few other minor
changes I want to make too, but have been holding off since the current one
is too enormous as-is)

On Tue, 11 Dec 2018 at 13:07 Craig de Stigter <
craig.destig...@koordinates.com> wrote:

> Thanks for the votes and kind words :)
>
> I'd applied Kurt's super-minor wording suggestions but haven't added any
> new sections (on fiona, extra examples, etc) yet. I'll try and incorporate
> those changes in the next couple days.
>
> Cheers
> Craig
>
> On Tue, 11 Dec 2018 at 11:07 Kurt Schwehr  wrote:
>
>> To follow on to Howard's comment about testing in python...
>>
>> I'm looking forward to the day that GDAL can drop all python 2.x support
>> and testing!
>>
>> And anyone working on C++ testing is welcome to the code here.  I'll
>> happily donate it with a license change to GDAL (it's apache 2.0 right
>> now).  You are free refactor however.  Or just use any bits that you think
>> are helpful if going to a different testing library.  I can also donate
>> some of the 80+k files in my fuzzer corpus as test inputs that cover weird
>> corners of the code.
>>
>> https://github.com/schwehr/gdal-autotest2/tree/master/cpp
>>
>> On Mon, Dec 10, 2018 at 11:39 AM Howard Butler  wrote:
>>
>>> +1
>>>
>>> I watched the ticket traffic and shuddered :) Thank you. GDAL's testing
>>> probably predates five or six Python testing regimes/eras.  I also have
>>> the concern about GDAL's testing going through Python, but this RFC will
>>> make it much easier for people to contribute and improve the story.
>>>
>>>
>>> Congratulations on a huge lift!
>>>
>>>
>>> Howard
>>>
>>>
>>> On 12/9/18 9:48 PM, Kurt Schwehr wrote:
>>> > I haven't had a chance to read the RFC yet, so I can't yet vote.
>>> However,
>>> > a huge thank you to Craig and everyone else who put in effort to make
>>> this
>>> > happen!
>>> >
>>> > On Sun, Dec 9, 2018 at 7:05 PM Craig de Stigter <
>>> > craig.destig...@koordinates.com> wrote:
>>> >
>>> >> Jonathan
>>> >>
>>> >>> it's worth spending a little thought on coming up with a scheme for
>>> >> test-ids.
>>> >>
>>> >> I've been through the list of parametrized tests and tweaked the `ids`
>>> >> kwargs to make them a little more helpful at first glance:
>>> >>
>>> https://github.com/OSGeo/gdal/pull/963/commits/8db599e7bc08b7dc73d81591898ed0f5f4243a58
>>> >>
>>> >> I didn't see any way to use `pytest_make_parametrize_id` really; IDs
>>> >> rightly vary enough between tests that I can't see that hook being
>>> very
>>> >> useful here.
>>> >>
>>> >> Cheers
>>> >> Craig de Stigter
>>> >>
>>> >> On Mon, 10 Dec 2018 at 09:59 jratike80 <
>>> >> jukka.rahko...@maanmittauslaitos.fi> wrote:
>>> >>
>>> >>> +0
>>> >>>
>>> >>> -Jukka Rahkonen-
>>> >>>
>>> >>>
>>> >>> Even Rouault-2 wrote
>>> >>>> PSC members,
>>> >>>>
>>> >>>> gentle reminder to cast your vote on this.
>>> >>>>
>>> >>>> Thanks,
>>> >>>>
>>> >>>> Even
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Sent from:
>>> http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
>>> >>> ___
>>> >>> gdal-dev mailing list
>>> >>> gdal-dev@lists.osgeo.org
>>> >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>> >> ___
>>> >> gdal-dev mailing list
>>> >> gdal-dev@lists.osgeo.org
>>> >> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>> >
>>> >
>>> >
>>> > ___
>>> > gdal-dev mailing list
>>> > gdal-dev@lists.osgeo.org
>>> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>>> ___
>>> gdal-dev mailing list
>>> gdal-dev@lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>>
>>
>> --
>> --
>> http://schwehr.org
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 72: Run tests with pytest

[Craig de Stigter]

Thanks for the votes and kind words :)

I'd applied Kurt's super-minor wording suggestions but haven't added any
new sections (on fiona, extra examples, etc) yet. I'll try and incorporate
those changes in the next couple days.

Cheers
Craig

On Tue, 11 Dec 2018 at 11:07 Kurt Schwehr  wrote:

> To follow on to Howard's comment about testing in python...
>
> I'm looking forward to the day that GDAL can drop all python 2.x support
> and testing!
>
> And anyone working on C++ testing is welcome to the code here.  I'll
> happily donate it with a license change to GDAL (it's apache 2.0 right
> now).  You are free refactor however.  Or just use any bits that you think
> are helpful if going to a different testing library.  I can also donate
> some of the 80+k files in my fuzzer corpus as test inputs that cover weird
> corners of the code.
>
> https://github.com/schwehr/gdal-autotest2/tree/master/cpp
>
> On Mon, Dec 10, 2018 at 11:39 AM Howard Butler  wrote:
>
>> +1
>>
>> I watched the ticket traffic and shuddered :) Thank you. GDAL's testing
>> probably predates five or six Python testing regimes/eras.  I also have
>> the concern about GDAL's testing going through Python, but this RFC will
>> make it much easier for people to contribute and improve the story.
>>
>>
>> Congratulations on a huge lift!
>>
>>
>> Howard
>>
>>
>> On 12/9/18 9:48 PM, Kurt Schwehr wrote:
>> > I haven't had a chance to read the RFC yet, so I can't yet vote.
>> However,
>> > a huge thank you to Craig and everyone else who put in effort to make
>> this
>> > happen!
>> >
>> > On Sun, Dec 9, 2018 at 7:05 PM Craig de Stigter <
>> > craig.destig...@koordinates.com> wrote:
>> >
>> >> Jonathan
>> >>
>> >>> it's worth spending a little thought on coming up with a scheme for
>> >> test-ids.
>> >>
>> >> I've been through the list of parametrized tests and tweaked the `ids`
>> >> kwargs to make them a little more helpful at first glance:
>> >>
>> https://github.com/OSGeo/gdal/pull/963/commits/8db599e7bc08b7dc73d81591898ed0f5f4243a58
>> >>
>> >> I didn't see any way to use `pytest_make_parametrize_id` really; IDs
>> >> rightly vary enough between tests that I can't see that hook being very
>> >> useful here.
>> >>
>> >> Cheers
>> >> Craig de Stigter
>> >>
>> >> On Mon, 10 Dec 2018 at 09:59 jratike80 <
>> >> jukka.rahko...@maanmittauslaitos.fi> wrote:
>> >>
>> >>> +0
>> >>>
>> >>> -Jukka Rahkonen-
>> >>>
>> >>>
>> >>> Even Rouault-2 wrote
>> >>>> PSC members,
>> >>>>
>> >>>> gentle reminder to cast your vote on this.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Even
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
>> >>> ___
>> >>> gdal-dev mailing list
>> >>> gdal-dev@lists.osgeo.org
>> >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>> >> ___
>> >> gdal-dev mailing list
>> >> gdal-dev@lists.osgeo.org
>> >> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>> >
>> >
>> >
>> > ___
>> > gdal-dev mailing list
>> > gdal-dev@lists.osgeo.org
>> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
> --
> --
> http://schwehr.org
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 72: Run tests with pytest

[Craig de Stigter]

Jonathan

> it's worth spending a little thought on coming up with a scheme for
test-ids.

I've been through the list of parametrized tests and tweaked the `ids`
kwargs to make them a little more helpful at first glance:
https://github.com/OSGeo/gdal/pull/963/commits/8db599e7bc08b7dc73d81591898ed0f5f4243a58

I didn't see any way to use `pytest_make_parametrize_id` really; IDs
rightly vary enough between tests that I can't see that hook being very
useful here.

Cheers
Craig de Stigter

On Mon, 10 Dec 2018 at 09:59 jratike80 
wrote:

> +0
>
> -Jukka Rahkonen-
>
>
> Even Rouault-2 wrote
> > PSC members,
> >
> > gentle reminder to cast your vote on this.
> >
> > Thanks,
> >
> > Even
>
>
>
>
>
> --
> Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Motion: adopt RFC 72: Run tests with pytest

[Craig de Stigter]

Hi

I appreciate your comments on the pytest proposal and all the support to
help get it this far. Given no actionable improvements have been suggested,
and the feedback thus far seems encouraging...

I move to adopt RFC 72: Run tests with pytest.

https://trac.osgeo.org/gdal/wiki/rfc72_pytest


Cheers
Craig de Stigter
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for discussion on RFC 72: Update autotest suite to use pytest

[Craig de Stigter]

Thanks Sean & Even

> I'm still concerned about using the SWIG bindings to test the C API

Fair enough, though as you noted, this is a bit tangential; RFC72 doesn't
make any changes to this situation, it's merely a refactor of the Python
tests.

> It uses a lot of magic, which some Python developers don't like, and so
it isn't necessarily *the way* to test in Python

That's true. Some people don't like Django for the same reason; a lot of
its internals are full of metaclasses and descriptors. SQLAlchemy is in the
same boat. Magic can be difficult to grok, and difficult to maintain.
However I think in all three cases they've acquired a significant user base
for the simple reason that the internal magic enables a really clean,
simple and effective interface.

I don't know if I'd want to *develop* pytest; I've seen some of the code
and I think it would take me a long time to understand it well enough to
work on it. But I do think the interface it presents is really excellent.

> The example shown in the RFC, for example: is making a fixture for every
gdal-bin program or adding checker fixtures that make multiple asserts
patterns that will help or hurt long term

Note that the example given in the RFC is just one possible future. I'd
hope those questions would be answered in pull requests as they happen. For
instance, the `require_files` fixture in the example doesn't exist yet. I'm
not committed to making those particular changes if there's opposition to
them.

That said, the advantages of a `require_files` fixture at least seem
reasonably clear to me: it would make the requirements of each test more
explicit and reduce boilerplate. I think this would also allow us to run
tests in parallel without them interfering with each other. The fixture
would put just the required files in a temp directory for each test, and
remove it afterwards. Being able to parallelise the tests would be a huge
win.

For what's actually implemented in the phase 1 pull request, see the
conftest.py
<https://github.com/craigds/gdal/blob/pytestify/autotest/conftest.py> file.
There's not much there beyond what's required for compatibility with the
old test runner. Notably, there's a
`@pytest.mark.require_driver(drivername)` marker, which I've applied to
some of the tests I had to change manually.

Cheers
Craig

On Wed, 5 Dec 2018 at 06:52 Sean Gillies  wrote:

> Craig,
>
> Thanks for doing this huge lift. Not all heroes wear capes!
>
> I'm still concerned about using the SWIG bindings to test the C API
> because subtle defects in the wrappers could hide deeper bugs and wish
> there were more direct tests, but RFC 72 is a big improvement.
>
> I use pytest in most of my projects and like how it reduces the friction
> for writing tests. It uses a lot of magic, which some Python developers
> don't like, and so it isn't necessarily *the way* to test in Python. I
> think there is a small risk that the hard to understand patterns of the
> home-grown approach could be replaced by hard to understand fixtures in a
> pytest approach. The example shown in the RFC, for example: is making a
> fixture for every gdal-bin program or adding checker fixtures that make
> multiple asserts patterns that will help or hurt long term? I'm not super
> concerned, just want to point out that pytest isn't a panacea and doesn't
> default to the golden path of testing.
>
> Thanks again!
>
> On Mon, Dec 3, 2018 at 4:49 PM Craig de Stigter <
> craig.destig...@koordinates.com> wrote:
>
>> Hi,
>>
>> This is a call for discussion on RFC 72: Update autotest suite to use
>> pytest
>>
>> https://trac.osgeo.org/gdal/wiki/rfc72_pytest
>>
>> Summary:
>>
>> The document proposes and describes conversion of the existing Python
>> autotest suite to use the ​pytest framework.
>>
>> Using pytest provides significant productivity gains for writing, reading
>> and debugging python tests, compared with the current home-grown approach.
>>
>> --
>>
>> Please submit feedback soon, as we'd like to have this merged in time for
>> the imminent GDAL 2.4.0 RC if possible.
>>
>> Thanks
>> Craig de Stigter
>>
>>
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
> --
> Sean Gillies
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Call for discussion on RFC 72: Update autotest suite to use pytest

[Craig de Stigter]

Hi,

This is a call for discussion on RFC 72: Update autotest suite to use pytest

https://trac.osgeo.org/gdal/wiki/rfc72_pytest

Summary:

The document proposes and describes conversion of the existing Python
autotest suite to use the ​pytest framework.

Using pytest provides significant productivity gains for writing, reading
and debugging python tests, compared with the current home-grown approach.

--

Please submit feedback soon, as we'd like to have this merged in time for
the imminent GDAL 2.4.0 RC if possible.

Thanks
Craig de Stigter
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Refactoring the test suite with pytest

[Craig de Stigter]

I posted the link earlier in the thread, but to clarify:
https://github.com/OSGeo/gdal/pull/963

On Fri, 26 Oct 2018 at 14:54 Craig de Stigter <
craig.destig...@koordinates.com> wrote:

> My work on this branch is starting to wind down. There's still a few test
> failures to contend with, and I need to work through some inconsistencies
> in the different CI environments.
>
> Could I request some review of the PR, especially around scope and overall
> approach? I believe the PR description is reasonably useful to describe
> what I've been up to, but please ask questions if you have them, or
> comment/review on the PR itself.
>
> Thanks
> Craig de Stigter
>
> On Fri, 5 Oct 2018 at 22:50 Kurt Schwehr  wrote:
>
>> The python tests in autotest2 are pretty much a subset of GDAL's
>> autotest.  The C++ and Java tests are mostly outside of what autotest has.
>>
>> On Tue, Oct 2, 2018 at 2:05 AM Craig de Stigter <
>> craig.destig...@koordinates.com> wrote:
>>
>>> Thanks. Are those tests entirely based on the osgeo git tests, or are
>>> there some extra ones? It might be worth getting the extra ones in at least.
>>>
>>>
>>> On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr  wrote:
>>>
>>>> I am definitely interested in improvements to the test infrastructure,
>>>> but always fear slowing down Even or getting bogged down in a massive
>>>> effort.
>>>>
>>>> In addition to working in the normal osgeo github world...
>>>>
>>>> My current state: I started "autotest2
>>>> <https://github.com/schwehr/gdal-autotest2/tree/master/python>" back
>>>> in 2013-14 after being unable to make autotest work for me and released
>>>> much of it as opensource.  It's unittest based for python and
>>>> GoogleTest/gmock for c++.  The Java isn't very far along, but it's Truth +
>>>> JUnit4.  It's in use inside google typically running every 10 to 60 minutes
>>>> depending on what's going on and a couple times a day in ASAN, MSAN, and
>>>> TSAN modes.  On the python side, it doesn't cover that much compared to the
>>>> full python test suite in autotest.  But in doing rewrites, I did have to
>>>> make each test independent.  I also have to allow control of where tests
>>>> write temporary data as I work in a system where the source tree is read
>>>> only.  I am probably close to making the tests python 3.7+ only (but likely
>>>> to work on 3.5+).  Whenever I start using a driver for the first time, I
>>>> try to rewrite the python test and create at least basic C++ tests, but I
>>>> never have enough time to do as complete of a job as I would like (err...
>>>> like the partially done pull request I have in to Proj).
>>>>
>>>> You are welcome to use anything in autotest2 for the core gdal tree if
>>>> it fits.  I (meaning Google) are happy to contribute any or all of the code
>>>> to the GDAL tree under the GDAL license.
>>>>
>>>> On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
>>>> craig.destig...@koordinates.com> wrote:
>>>>
>>>>> Hi folks
>>>>>
>>>>> I've been looking at a potential large change to the GDAL test suite,
>>>>> replacing some of the worst parts. I thought I'd bring it up for 
>>>>> discussion
>>>>> here, in case anyone has any wisdom or strong feelings about it.
>>>>>
>>>>> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>>>>>
>>>>> Essentially my plan is to use pytest
>>>>> <https://docs.pytest.org/en/latest/> as both a test runner and a
>>>>> helper library. We would remove or replace large parts of the `gdaltest`
>>>>> utilities with pytest equivalents.
>>>>>
>>>>> To give an idea of the main changes, this code:
>>>>>
>>>>> if ds.GetLayer(0).GetSpatialRef() is not None:
>>>>>> gdaltest.post_reason('did not get expected spatial ref')
>>>>>> return 'fail'
>>>>>>
>>>>> would change to:
>>>>>
>>>>>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>>>>>> spatial ref'
>>>>>>
>>>>>
>>>>> The pytest runner gives contextual tracebacks from assert statements,
>>>>> as well as relevant variables and line numbers.
>>>>>
>&

Re: [gdal-dev] Refactoring the test suite with pytest

[Craig de Stigter]

My work on this branch is starting to wind down. There's still a few test
failures to contend with, and I need to work through some inconsistencies
in the different CI environments.

Could I request some review of the PR, especially around scope and overall
approach? I believe the PR description is reasonably useful to describe
what I've been up to, but please ask questions if you have them, or
comment/review on the PR itself.

Thanks
Craig de Stigter

On Fri, 5 Oct 2018 at 22:50 Kurt Schwehr  wrote:

> The python tests in autotest2 are pretty much a subset of GDAL's
> autotest.  The C++ and Java tests are mostly outside of what autotest has.
>
> On Tue, Oct 2, 2018 at 2:05 AM Craig de Stigter <
> craig.destig...@koordinates.com> wrote:
>
>> Thanks. Are those tests entirely based on the osgeo git tests, or are
>> there some extra ones? It might be worth getting the extra ones in at least.
>>
>>
>> On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr  wrote:
>>
>>> I am definitely interested in improvements to the test infrastructure,
>>> but always fear slowing down Even or getting bogged down in a massive
>>> effort.
>>>
>>> In addition to working in the normal osgeo github world...
>>>
>>> My current state: I started "autotest2
>>> <https://github.com/schwehr/gdal-autotest2/tree/master/python>" back in
>>> 2013-14 after being unable to make autotest work for me and released much
>>> of it as opensource.  It's unittest based for python and GoogleTest/gmock
>>> for c++.  The Java isn't very far along, but it's Truth + JUnit4.  It's in
>>> use inside google typically running every 10 to 60 minutes depending on
>>> what's going on and a couple times a day in ASAN, MSAN, and TSAN modes.  On
>>> the python side, it doesn't cover that much compared to the full python
>>> test suite in autotest.  But in doing rewrites, I did have to make each
>>> test independent.  I also have to allow control of where tests write
>>> temporary data as I work in a system where the source tree is read only.  I
>>> am probably close to making the tests python 3.7+ only (but likely to work
>>> on 3.5+).  Whenever I start using a driver for the first time, I try to
>>> rewrite the python test and create at least basic C++ tests, but I never
>>> have enough time to do as complete of a job as I would like (err... like
>>> the partially done pull request I have in to Proj).
>>>
>>> You are welcome to use anything in autotest2 for the core gdal tree if
>>> it fits.  I (meaning Google) are happy to contribute any or all of the code
>>> to the GDAL tree under the GDAL license.
>>>
>>> On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
>>> craig.destig...@koordinates.com> wrote:
>>>
>>>> Hi folks
>>>>
>>>> I've been looking at a potential large change to the GDAL test suite,
>>>> replacing some of the worst parts. I thought I'd bring it up for discussion
>>>> here, in case anyone has any wisdom or strong feelings about it.
>>>>
>>>> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>>>>
>>>> Essentially my plan is to use pytest
>>>> <https://docs.pytest.org/en/latest/> as both a test runner and a
>>>> helper library. We would remove or replace large parts of the `gdaltest`
>>>> utilities with pytest equivalents.
>>>>
>>>> To give an idea of the main changes, this code:
>>>>
>>>> if ds.GetLayer(0).GetSpatialRef() is not None:
>>>>> gdaltest.post_reason('did not get expected spatial ref')
>>>>> return 'fail'
>>>>>
>>>> would change to:
>>>>
>>>>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>>>>> spatial ref'
>>>>>
>>>>
>>>> The pytest runner gives contextual tracebacks from assert statements,
>>>> as well as relevant variables and line numbers.
>>>>
>>>> Other changes:
>>>>  * The `gdaltest_list` and the `__main__` entry point at the bottom of
>>>> each file will be removed. Some tests will be renamed if they don't already
>>>> start with `test_`.
>>>>  * The `_cleanup` fixture will be turned into a module-scoped
>>>> fixture
>>>> <https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
>>>> This allows users to run any number of tests in 

Re: [gdal-dev] Refactoring the test suite with pytest

[Craig de Stigter]

I've got some initial steps toward this work at
https://github.com/OSGeo/gdal/pull/963

It doesn't have any of the automated conversion yet but I've added a todo
list and am hoping to have something semi-useful running in CI soon.

On Tue, 2 Oct 2018 at 22:04 Craig de Stigter <
craig.destig...@koordinates.com> wrote:

> Thanks. Are those tests entirely based on the osgeo git tests, or are
> there some extra ones? It might be worth getting the extra ones in at least.
>
>
> On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr  wrote:
>
>> I am definitely interested in improvements to the test infrastructure,
>> but always fear slowing down Even or getting bogged down in a massive
>> effort.
>>
>> In addition to working in the normal osgeo github world...
>>
>> My current state: I started "autotest2
>> <https://github.com/schwehr/gdal-autotest2/tree/master/python>" back in
>> 2013-14 after being unable to make autotest work for me and released much
>> of it as opensource.  It's unittest based for python and GoogleTest/gmock
>> for c++.  The Java isn't very far along, but it's Truth + JUnit4.  It's in
>> use inside google typically running every 10 to 60 minutes depending on
>> what's going on and a couple times a day in ASAN, MSAN, and TSAN modes.  On
>> the python side, it doesn't cover that much compared to the full python
>> test suite in autotest.  But in doing rewrites, I did have to make each
>> test independent.  I also have to allow control of where tests write
>> temporary data as I work in a system where the source tree is read only.  I
>> am probably close to making the tests python 3.7+ only (but likely to work
>> on 3.5+).  Whenever I start using a driver for the first time, I try to
>> rewrite the python test and create at least basic C++ tests, but I never
>> have enough time to do as complete of a job as I would like (err... like
>> the partially done pull request I have in to Proj).
>>
>> You are welcome to use anything in autotest2 for the core gdal tree if it
>> fits.  I (meaning Google) are happy to contribute any or all of the code to
>> the GDAL tree under the GDAL license.
>>
>> On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
>> craig.destig...@koordinates.com> wrote:
>>
>>> Hi folks
>>>
>>> I've been looking at a potential large change to the GDAL test suite,
>>> replacing some of the worst parts. I thought I'd bring it up for discussion
>>> here, in case anyone has any wisdom or strong feelings about it.
>>>
>>> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>>>
>>> Essentially my plan is to use pytest
>>> <https://docs.pytest.org/en/latest/> as both a test runner and a helper
>>> library. We would remove or replace large parts of the `gdaltest` utilities
>>> with pytest equivalents.
>>>
>>> To give an idea of the main changes, this code:
>>>
>>> if ds.GetLayer(0).GetSpatialRef() is not None:
>>>> gdaltest.post_reason('did not get expected spatial ref')
>>>> return 'fail'
>>>>
>>> would change to:
>>>
>>>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>>>> spatial ref'
>>>>
>>>
>>> The pytest runner gives contextual tracebacks from assert statements, as
>>> well as relevant variables and line numbers.
>>>
>>> Other changes:
>>>  * The `gdaltest_list` and the `__main__` entry point at the bottom of
>>> each file will be removed. Some tests will be renamed if they don't already
>>> start with `test_`.
>>>  * The `_cleanup` fixture will be turned into a module-scoped
>>> fixture
>>> <https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
>>> This allows users to run any number of tests in the module, and the cleanup
>>> fixture will be invoked once after all of them are finished. If I find
>>> corresponding setup functions I will make them into fixtures too, but I
>>> haven't seen those so far in my digging.
>>>
>>> Other, more intrusive changes are possible but this is what I'm
>>> considering targeting as a first step. It is fairly straightforward to
>>> automate this change; I have a script in progress already to do that.
>>>
>>> I intend to make a compatibility shim for the old-style tests that can't
>>> easily be automatically updated. The shim would be a decorator, perhaps
>>> called `@old_test`, which wou

Re: [gdal-dev] Refactoring the test suite with pytest

[Craig de Stigter]

Thanks. Are those tests entirely based on the osgeo git tests, or are there
some extra ones? It might be worth getting the extra ones in at least.


On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr  wrote:

> I am definitely interested in improvements to the test infrastructure, but
> always fear slowing down Even or getting bogged down in a massive effort.
>
> In addition to working in the normal osgeo github world...
>
> My current state: I started "autotest2
> <https://github.com/schwehr/gdal-autotest2/tree/master/python>" back in
> 2013-14 after being unable to make autotest work for me and released much
> of it as opensource.  It's unittest based for python and GoogleTest/gmock
> for c++.  The Java isn't very far along, but it's Truth + JUnit4.  It's in
> use inside google typically running every 10 to 60 minutes depending on
> what's going on and a couple times a day in ASAN, MSAN, and TSAN modes.  On
> the python side, it doesn't cover that much compared to the full python
> test suite in autotest.  But in doing rewrites, I did have to make each
> test independent.  I also have to allow control of where tests write
> temporary data as I work in a system where the source tree is read only.  I
> am probably close to making the tests python 3.7+ only (but likely to work
> on 3.5+).  Whenever I start using a driver for the first time, I try to
> rewrite the python test and create at least basic C++ tests, but I never
> have enough time to do as complete of a job as I would like (err... like
> the partially done pull request I have in to Proj).
>
> You are welcome to use anything in autotest2 for the core gdal tree if it
> fits.  I (meaning Google) are happy to contribute any or all of the code to
> the GDAL tree under the GDAL license.
>
> On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
> craig.destig...@koordinates.com> wrote:
>
>> Hi folks
>>
>> I've been looking at a potential large change to the GDAL test suite,
>> replacing some of the worst parts. I thought I'd bring it up for discussion
>> here, in case anyone has any wisdom or strong feelings about it.
>>
>> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>>
>> Essentially my plan is to use pytest <https://docs.pytest.org/en/latest/>
>> as both a test runner and a helper library. We would remove or replace
>> large parts of the `gdaltest` utilities with pytest equivalents.
>>
>> To give an idea of the main changes, this code:
>>
>> if ds.GetLayer(0).GetSpatialRef() is not None:
>>> gdaltest.post_reason('did not get expected spatial ref')
>>> return 'fail'
>>>
>> would change to:
>>
>>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>>> spatial ref'
>>>
>>
>> The pytest runner gives contextual tracebacks from assert statements, as
>> well as relevant variables and line numbers.
>>
>> Other changes:
>>  * The `gdaltest_list` and the `__main__` entry point at the bottom of
>> each file will be removed. Some tests will be renamed if they don't already
>> start with `test_`.
>>  * The `_cleanup` fixture will be turned into a module-scoped
>> fixture
>> <https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
>> This allows users to run any number of tests in the module, and the cleanup
>> fixture will be invoked once after all of them are finished. If I find
>> corresponding setup functions I will make them into fixtures too, but I
>> haven't seen those so far in my digging.
>>
>> Other, more intrusive changes are possible but this is what I'm
>> considering targeting as a first step. It is fairly straightforward to
>> automate this change; I have a script in progress already to do that.
>>
>> I intend to make a compatibility shim for the old-style tests that can't
>> easily be automatically updated. The shim would be a decorator, perhaps
>> called `@old_test`, which would take 'skip' or 'fail' return values from
>> tests and turn them into the appropriate `assert False` or `pytest.skip()`
>> calls.
>>
>> I don't intend to make any changes to the C++ tests.
>>
>> A couple of complicating factors to be aware of:
>>
>>- pytest only supports python 2.7+. The tests already only run in
>>2.7+ in Travis, so no big deal there. I think it's perfectly reasonable to
>>only support 2.7+ these days, given that 2.6 has been unsupported since
>>2013, but I thought I'd mention it.
>>- pytest has no builtin way of *ordering* tests. It is assumed that
>> 

[gdal-dev] Refactoring the test suite with pytest

[Craig de Stigter]

Hi folks

I've been looking at a potential large change to the GDAL test suite,
replacing some of the worst parts. I thought I'd bring it up for discussion
here, in case anyone has any wisdom or strong feelings about it.

I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .

Essentially my plan is to use pytest <https://docs.pytest.org/en/latest/>
as both a test runner and a helper library. We would remove or replace
large parts of the `gdaltest` utilities with pytest equivalents.

To give an idea of the main changes, this code:

if ds.GetLayer(0).GetSpatialRef() is not None:
> gdaltest.post_reason('did not get expected spatial ref')
> return 'fail'
>
would change to:

> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
> spatial ref'
>

The pytest runner gives contextual tracebacks from assert statements, as
well as relevant variables and line numbers.

Other changes:
 * The `gdaltest_list` and the `__main__` entry point at the bottom of each
file will be removed. Some tests will be renamed if they don't already
start with `test_`.
 * The `_cleanup` fixture will be turned into a module-scoped fixture
<https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
This allows users to run any number of tests in the module, and the cleanup
fixture will be invoked once after all of them are finished. If I find
corresponding setup functions I will make them into fixtures too, but I
haven't seen those so far in my digging.

Other, more intrusive changes are possible but this is what I'm considering
targeting as a first step. It is fairly straightforward to automate this
change; I have a script in progress already to do that.

I intend to make a compatibility shim for the old-style tests that can't
easily be automatically updated. The shim would be a decorator, perhaps
called `@old_test`, which would take 'skip' or 'fail' return values from
tests and turn them into the appropriate `assert False` or `pytest.skip()`
calls.

I don't intend to make any changes to the C++ tests.

A couple of complicating factors to be aware of:

   - pytest only supports python 2.7+. The tests already only run in 2.7+
   in Travis, so no big deal there. I think it's perfectly reasonable to only
   support 2.7+ these days, given that 2.6 has been unsupported since 2013,
   but I thought I'd mention it.
   - pytest has no builtin way of *ordering* tests. It is assumed that
   tests are independent of each other. In practice, all tests are run in
   alphabetical name order within a given module. It appears to me that some
   of GDAL's tests are *not* independent of each other within the same
   module. However, most are already in alphabetical name order, so I don't
   believe this is a problem; we are not worse off with pytest than previously.


Please let me know your thoughts.

-- 
Regards,
Craig de Stigter

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
<https://twitter.com/koordinates>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] vsicurl configuration design decisions

[Craig de Stigter]

Hi folks

We're slightly invested in this because we use VSI paths reasonably
heavily, though not so much for cloud services yet.


> One downside is that you need to URLEncode the URL, which can make it
> painful when composing it at hand.


True, but that does eliminate ambiguity in the URL, and does so in a
well-known way.

Does the current scheme use any encoding? How would you escape text in
option values that might use `=` and `,` etc? Or are there guaranteed to be
no freeform-text options in these paths?



Tangential, but related: I've also just discovered the 2.2+ curly-brace
syntax for vsizip/vsitar paths, which allows nested archives:

/vsizip/{/vsizip/{/path/to/outer.zip}/path/to/inner.zip}/file.shp


The curly braces are a definite improvement on the ambiguous older syntax
for these paths. However, we noted the nesting order looks inside-out, and
thought it would have been more intuitive to put the path *inside* the
archive in the braces. i.e. nesting would look like:

> /vsizip//path/to/outer.zip/{/vsizip//path/to/inner.zip/{file.shp}}


Of course, this latter syntax was added in 2.2, so perhaps that ship has
already sailed.

>From our experiences with vsicurl and vsizip urls, it feels like
eliminating ambiguity in these paths is pretty important, more so than
trivial composability. Just my 2¢ :)


On 13 October 2017 at 07:42, Even Rouault 
wrote:

> Hi Sean,
>
>
>
> > Is the future of open and creation options?
>
>
>
> I don't understand your above sentence.
>
>
>
> > Do you imagine this extended
>
> > to, say, block size, compression, number of threads? An RFC that
> discussed
>
> > the scope of this and at what level of abstraction it is implemented at
>
> > might be warranted? I'd be happy to participate.
>
>
>
> Not clear what you've in mind. Are you thinking about some formalism to
> define and specify options for VSI filenames ?
>
>
>
> > On the other hand, https://example.com/foo.tif identifies only a single
>
> > resource, whereas /viscurl/url=https://example.com/foo.tif can identify
> a
>
> > GeoTIFF along with all of its sidecars. I presume that the new GDAL cloud
>
> > utilities like gdal_cp.py take care of the auxiliary files, yes?
>
>
>
> No. They should perhaps be named cpl_xxx since they really operate at the
> VSI/file level. Auxiliary/sidecar files are concepts that exist only at the
> driver level/
>
>
>
> Copy of datasets + side car files can be done with "gdalmanage copy"
>
>
>
> >
>
> > My final concern about the virtual file opening options is the syntax.
>
> > These /vsicurl/option1=val1[,optionN=valN]*,url=http://
> example.com/foo.tif
>
> > identifiers (or filenames or whatever we call them) may spread from GDAL
>
> > into the wider geospatial programming domain. Speaking from my experience
>
> > with Rasterio, open source Python GIS developers expect the /vsi*
> filenames
>
> > to "just work" in all software. Can we consider using a more standard
>
> > syntax? One that has parsers already deployed everywhere?
>
>
>
> I don't really see a use of parsing those /vsi names by user code. User
> code has to compose them, not parse them. But I can see your point for
> something more standardized.
>
>
>
> > That syntax gives the /vsi* filenames the form of a "reflector" URL such
> as
>
> > we see in Google searches (for example:
>
> > https://www.google.com/url?sa=t=j==s=web;
> cd=1=0ahUKEwj
>
> > C6e7hvevWAhXmjFQKHWsHDyMQFggmMAA=http%3A%2F%2Fwww.gdal.org
> %2F=AOvVaw
>
> > 3fbRv5TusYwkXgz2Acf2kt) and there are abundant tools and a body of
> knowledge
>
> > about how to parse and work with these.
>
>
>
> One downside is that you need to URLEncode the URL, which can make it
> painful when composing it at hand.
>
>
>
>
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] cubic / bilinear resampling with gdalwarp looks similar to nearest neighbour

[Craig de Stigter]

Apologies for the long delay. I've finally had opportunity to look into this
again.

I'm still unsure why exactly the bilinear and bicubic resampling produces
such second-rate output in GDAL. I have a few questions:

1. Is there a fundamental reason why the bilinear resampling couldn't use
all the pixels from the source region, rather than just the 4 corner pixels?
A destination pixel would then be the weighted average of all the source
pixels for each band.

2. Suppose a 1/6 sampling ratio, i.e. a 6x6 region (0, 0)--(6, 6) in the
source image corresponds to a 1x1 region in the output.
What source pixels are used by the bilinear resampling in GDAL?

Wikipedia http://en.wikipedia.org/wiki/Bilinear_interpolation suggests it
should be the 'nearest' four pixels to the desired destination pixel, i.e.
(2, 2), (2, 3), (3, 2), (3, 3). Is that what GDAL is doing, or is it using
(0, 0), (0, 5), (5, 0), (5, 5) ?

3. Assuming a 'no' answer to (1), if I were to contribute a patch to make
bilinear/bicubic resampling take all the source pixels into account, would
it meet much opposition? It would probably make large-scale downsampling
using these methods much slower, though its questionable whether anyone
would complain given that the quality of current output is so poor...

Thanks for your help
Craig de Stigter

On Thu, Sep 30, 2010 at 12:30 PM, Seth Price s...@pricepages.org wrote:

 The only way to 'force' it is to use cubicspline or lanczos. Everything
 else has a hardcoded filter size. If those are too slow, I'd suggest using
 the OpenCL code I wrote over the summer. It's many times faster. (It
 should be in trunk soon if it isn't already.)

 http://github.com/mailseth/OpenCL-integration-for-GRASS---GDAL

 ~Seth



 On Wed, September 29, 2010 4:12 pm, Craig de Stigter wrote:
  Thanks Even
 
  However if I do the same thing with reprojection I get the same result.
  gdalwarp output looks sharp and ugly, imagemagick output looks nice.
 
  Is there any way to force the bilinear/cubic interpolation to notice more
  pixels? The other interpolation methods are much slower, I imagine even
  forcing cubic to notice all of the 5x5 pixels would still be faster.
 
  Regards
  Craig de Stigter
 
  On Thu, Sep 30, 2010 at 10:34 AM, Even Rouault
  even.roua...@mines-paris.org
  wrote:
 
  Craig,
 
  The main reason is that you are using gdalwarp to reduce an image by a
  large
  factor and not reproject it, which generally only involves image
  resizing
  by
  small factors. The implementation of the bilinear and cubic resampling
  algorithms is currently not designed for the pure resizing use case and
  takes
  only into account a few source pixels (basically if to compute a
  destination
  pixel you would need to take into account a 5x5 source window, they will
  only
  used the 4 corners of that square, which fine usually since those are
  just
  the
  immediate neighbours of the source pixel), whereas cubicspline and
  lancsoz
  take into account more pixels (all the pixels in the 5x5 square).
 
  Best regards,
 
  Even
 
  Le mercredi 29 septembre 2010 22:27:01, Craig de Stigter a écrit :
   Hi folks
  
   I filed this bug http://trac.osgeo.org/gdal/ticket/3740 on gdal a
  few
   weeks back and haven't heard anything since.
  
   Would someone mind taking a look at it?
  
   Thanks
   Craig de Stigter
 
 
 
 
  --
  Koordinates Ltd
  PO Box 1604, Shortland St, Auckland, New Zealand
  Phone +64-9-966 0433 Fax +64-9-969 0045
  Web http://www.koordinates.com
  ___
  gdal-dev mailing list
  gdal-dev@lists.osgeo.org
  http://lists.osgeo.org/mailman/listinfo/gdal-dev





-- 
Koordinates Ltd
PO Box 1604, Shortland St, Auckland, New Zealand
Phone +64-9-966 0433 Fax +64-9-969 0045
Web http://www.koordinates.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] cubic / bilinear resampling with gdalwarp looks similar to nearest neighbour

[Craig de Stigter]

Hi folks

I filed this bug http://trac.osgeo.org/gdal/ticket/3740 on gdal a few
weeks back and haven't heard anything since.

Would someone mind taking a look at it?

Thanks
Craig de Stigter

-- 
Koordinates Ltd
PO Box 1604, Shortland St, Auckland, New Zealand
Phone +64-9-966 0433 Fax +64-9-969 0045
Web http://www.koordinates.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] cubic / bilinear resampling with gdalwarp looks similar to nearest neighbour

[Craig de Stigter]

Thanks Even

However if I do the same thing with reprojection I get the same result.
gdalwarp output looks sharp and ugly, imagemagick output looks nice.

Is there any way to force the bilinear/cubic interpolation to notice more
pixels? The other interpolation methods are much slower, I imagine even
forcing cubic to notice all of the 5x5 pixels would still be faster.

Regards
Craig de Stigter

On Thu, Sep 30, 2010 at 10:34 AM, Even Rouault even.roua...@mines-paris.org
 wrote:

 Craig,

 The main reason is that you are using gdalwarp to reduce an image by a
 large
 factor and not reproject it, which generally only involves image resizing
 by
 small factors. The implementation of the bilinear and cubic resampling
 algorithms is currently not designed for the pure resizing use case and
 takes
 only into account a few source pixels (basically if to compute a
 destination
 pixel you would need to take into account a 5x5 source window, they will
 only
 used the 4 corners of that square, which fine usually since those are just
 the
 immediate neighbours of the source pixel), whereas cubicspline and lancsoz
 take into account more pixels (all the pixels in the 5x5 square).

 Best regards,

 Even

 Le mercredi 29 septembre 2010 22:27:01, Craig de Stigter a écrit :
  Hi folks
 
  I filed this bug http://trac.osgeo.org/gdal/ticket/3740 on gdal a few
  weeks back and haven't heard anything since.
 
  Would someone mind taking a look at it?
 
  Thanks
  Craig de Stigter




-- 
Koordinates Ltd
PO Box 1604, Shortland St, Auckland, New Zealand
Phone +64-9-966 0433 Fax +64-9-969 0045
Web http://www.koordinates.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev