Re: [gdal-dev] ogr2ogr and WFS 2

2021-11-30 Thread Hernán De Angelis 
Sorry, in the message below I meant one needs to explicitly specify 
OGR_WFS_PAGING_ALLOWED YES, with a suitable OGR_WFS_PAGE_SIZE.




Den 2021-11-30 kl. 09:34, skrev Hernán De Angelis:
Thank you, Jukka. I did that and worked. I also had another case where 
it is important to get this right. It is when the total number of 
features exposed by the service exceeds the maximum number of features 
allowed to be fetched in one request (CountDefault). In this case it 
is also important to set the paging to NO, in addition with an 
adequate OGR_WFS_PAGE_SIZE so that the layer can be downloaded 
entirely, in several pages, without hitting the limit.


Best

Hernán


Den 2021-11-30 kl. 08:59, skrev Rahkonen Jukka (MML):


Hi,

There is something about WFS 2 and paging in 
https://gdal.org/drivers/vector/wfs.html


“Paging with WFS 2.0

The WFS driver will automatically detect if server supports paging, 
when requesting a WFS 2.0 server. The page size (number of features 
fetched in a single request) is limited to 100 by default when not 
declared by the server. It can be changed by setting the 
OGR_WFS_PAGE_SIZE configuration option, or by specifying COUNT as a 
query parameter in the URL of the connection string.


If only the N first features must be downloaded and paging through 
the whole layer is not desirable, the OGR_WFS_PAGING_ALLOWED 
configuration option should be set to OFF.”


So by default normally GDAL goes to paging mode that makes it 
possible to read the whole featuretype, possibly restricted with BBOX 
or other filters but it should be possible to disable paging with 
--config OGR_WFS_PAGING_ALLOWED OFF. I would try that.


-Jukka Rahkonen-

*Lähettäjä:* gdal-dev  *Puolesta 
*Hernán De Angelis

*Lähetetty:* maanantai 29. marraskuuta 2021 12.43
*Vastaanottaja:* gdal-dev@lists.osgeo.org
*Aihe:* [gdal-dev] ogr2ogr and WFS 2

Dear all

I seem to have problems understanding how ogr2ogr interacts with WFS 
and hope someone would be able to clarify.


I have this WFS:

https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?

The capabilities document states that it supports WFS 1.0.0, 1.1.0 
and 2.0.0.


If I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=1.0.0=GetFeature=stationsregistret:active_site=100 
<https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=1.0.0=GetFeature=stationsregistret:active_site=100>"


I get a small and nice GPKG file very fast.

But if I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100 
<https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100>"


It downloads the whole layer.

This behaviour can be seriously impractical for services exposing a 
large number of features.



Pay attention that if I do:

curl 
"https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100; 
<https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100>


it retrieves a short document fast with just 100 features. The same 
of course if I invoke the service using WFS 1.0.0 or 1.1.0.



What is going on here? Is it me that I misunderstood something about 
how ogr2ogr is supposed to interact with WFS services?


Using GDAL 3.4.0, released 2021/11/04. But the same behaviour was 
observed in 3.3.2.



Thanks in advance for any clarification


Hernán


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr and WFS 2

2021-11-30 Thread Hernán De Angelis 
Thank you, Jukka. I did that and worked. I also had another case where 
it is important to get this right. It is when the total number of 
features exposed by the service exceeds the maximum number of features 
allowed to be fetched in one request (CountDefault). In this case it is 
also important to set the paging to NO, in addition with an adequate 
OGR_WFS_PAGE_SIZE so that the layer can be downloaded entirely, in 
several pages, without hitting the limit.


Best

Hernán


Den 2021-11-30 kl. 08:59, skrev Rahkonen Jukka (MML):


Hi,

There is something about WFS 2 and paging in 
https://gdal.org/drivers/vector/wfs.html


“Paging with WFS 2.0

The WFS driver will automatically detect if server supports paging, 
when requesting a WFS 2.0 server. The page size (number of features 
fetched in a single request) is limited to 100 by default when not 
declared by the server. It can be changed by setting the 
OGR_WFS_PAGE_SIZE configuration option, or by specifying COUNT as a 
query parameter in the URL of the connection string.


If only the N first features must be downloaded and paging through the 
whole layer is not desirable, the OGR_WFS_PAGING_ALLOWED configuration 
option should be set to OFF.”


So by default normally GDAL goes to paging mode that makes it possible 
to read the whole featuretype, possibly restricted with BBOX or other 
filters but it should be possible to disable paging with --config 
OGR_WFS_PAGING_ALLOWED OFF. I would try that.


-Jukka Rahkonen-

*Lähettäjä:* gdal-dev  *Puolesta 
*Hernán De Angelis

*Lähetetty:* maanantai 29. marraskuuta 2021 12.43
*Vastaanottaja:* gdal-dev@lists.osgeo.org
*Aihe:* [gdal-dev] ogr2ogr and WFS 2

Dear all

I seem to have problems understanding how ogr2ogr interacts with WFS 
and hope someone would be able to clarify.


I have this WFS:

https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?

The capabilities document states that it supports WFS 1.0.0, 1.1.0 and 
2.0.0.


If I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=1.0.0=GetFeature=stationsregistret:active_site=100 
<https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=1.0.0=GetFeature=stationsregistret:active_site=100>"


I get a small and nice GPKG file very fast.

But if I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100 
<https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100>"


It downloads the whole layer.

This behaviour can be seriously impractical for services exposing a 
large number of features.



Pay attention that if I do:

curl 
"https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100; 
<https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100>


it retrieves a short document fast with just 100 features. The same of 
course if I invoke the service using WFS 1.0.0 or 1.1.0.



What is going on here? Is it me that I misunderstood something about 
how ogr2ogr is supposed to interact with WFS services?


Using GDAL 3.4.0, released 2021/11/04. But the same behaviour was 
observed in 3.3.2.



Thanks in advance for any clarification


Hernán


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr and WFS 2

2021-11-29 Thread Rahkonen Jukka (MML) 
Hi,

There is something about WFS 2 and paging in 
https://gdal.org/drivers/vector/wfs.html

“Paging with WFS 2.0
The WFS driver will automatically detect if server supports paging, when 
requesting a WFS 2.0 server. The page size (number of features fetched in a 
single request) is limited to 100 by default when not declared by the server. 
It can be changed by setting the OGR_WFS_PAGE_SIZE configuration option, or by 
specifying COUNT as a query parameter in the URL of the connection string.

If only the N first features must be downloaded and paging through the whole 
layer is not desirable, the OGR_WFS_PAGING_ALLOWED configuration option should 
be set to OFF.”

So by default normally GDAL goes to paging mode that makes it possible to read 
the whole featuretype, possibly restricted with BBOX or other filters but it 
should be possible to disable paging with --config OGR_WFS_PAGING_ALLOWED OFF. 
I would try that.

-Jukka Rahkonen-

Lähettäjä: gdal-dev  Puolesta Hernán De 
Angelis
Lähetetty: maanantai 29. marraskuuta 2021 12.43
Vastaanottaja: gdal-dev@lists.osgeo.org
Aihe: [gdal-dev] ogr2ogr and WFS 2

Dear all

I seem to have problems understanding how ogr2ogr interacts with WFS and hope 
someone would be able to clarify.

I have this WFS:

https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?

The capabilities document states that it supports WFS 1.0.0, 1.1.0 and 2.0.0.

If I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=1.0.0=GetFeature=stationsregistret:active_site=100;

I get a small and nice GPKG file very fast.

But if I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100;

It downloads the whole layer.

This behaviour can be seriously impractical for services exposing a large 
number of features.


Pay attention that if I do:

curl 
"https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100;<https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100>

it retrieves a short document fast with just 100 features. The same of course 
if I invoke the service using WFS 1.0.0 or 1.1.0.


What is going on here? Is it me that I misunderstood something about how 
ogr2ogr is supposed to interact with WFS services?

Using GDAL 3.4.0, released 2021/11/04. But the same behaviour was observed in 
3.3.2.


Thanks in advance for any clarification


Hernán


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr and WFS 2

2021-11-29 Thread Hernán De Angelis 
Thank you for the clarification, Even. Now I understand how this works. 
I also think the pull request will help to make ogr2ogr behave more as 
is intuitively expected.


Have a nice day!

Hernán


Den 2021-11-29 kl. 13:11, skrev Even Rouault:


Hernán,

The URL you specify in the WFS: connection string isn't in general the 
one of a GetFeature request, but only the endpoint of the service. 
When you specify a GetFeature request, some heuristics trigger, but in 
WFS 2, the COUNT parameter is interpreted as the page size, but as 
this server supports paging, it is actually overridden by the page 
size returned by the server (this part is a bug, addressed per 
https://github.com/OSGeo/gdal/pull/4914), and anyway whatever the page 
size used, it scrolls by default through the results.


If you add  --config OGR_WFS_PAGING_ALLOWED NO you'll get the behavior 
you expect. The above mentioned pull request also clarifies this 
hopefully.


Even

Le 29/11/2021 à 11:43, Hernán De Angelis a écrit :

Dear all

I seem to have problems understanding how ogr2ogr interacts with WFS 
and hope someone would be able to clarify.


I have this WFS:

https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?

The capabilities document states that it supports WFS 1.0.0, 1.1.0 
and 2.0.0.


If I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=1.0.0=GetFeature=stationsregistret:active_site=100;


I get a small and nice GPKG file very fast.

But if I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100;


It downloads the whole layer.

This behaviour can be seriously impractical for services exposing a 
large number of features.



Pay attention that if I do:

curl 
"https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100;


it retrieves a short document fast with just 100 features. The same 
of course if I invoke the service using WFS 1.0.0 or 1.1.0.



What is going on here? Is it me that I misunderstood something about 
how ogr2ogr is supposed to interact with WFS services?


Using GDAL 3.4.0, released 2021/11/04. But the same behaviour was 
observed in 3.3.2.



Thanks in advance for any clarification


Hernán




___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr and WFS 2

2021-11-29 Thread Even Rouault 

Hernán,

The URL you specify in the WFS: connection string isn't in general the 
one of a GetFeature request, but only the endpoint of the service. When 
you specify a GetFeature request, some heuristics trigger, but in WFS 2, 
the COUNT parameter is interpreted as the page size, but as this server 
supports paging, it is actually overridden by the page size returned by 
the server (this part is a bug, addressed per 
https://github.com/OSGeo/gdal/pull/4914), and anyway whatever the page 
size used, it scrolls by default through the results.


If you add  --config OGR_WFS_PAGING_ALLOWED NO you'll get the behavior 
you expect. The above mentioned pull request also clarifies this hopefully.


Even

Le 29/11/2021 à 11:43, Hernán De Angelis a écrit :

Dear all

I seem to have problems understanding how ogr2ogr interacts with WFS 
and hope someone would be able to clarify.


I have this WFS:

https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?

The capabilities document states that it supports WFS 1.0.0, 1.1.0 and 
2.0.0.


If I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=1.0.0=GetFeature=stationsregistret:active_site=100;


I get a small and nice GPKG file very fast.

But if I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100;


It downloads the whole layer.

This behaviour can be seriously impractical for services exposing a 
large number of features.



Pay attention that if I do:

curl 
"https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100;


it retrieves a short document fast with just 100 features. The same of 
course if I invoke the service using WFS 1.0.0 or 1.1.0.



What is going on here? Is it me that I misunderstood something about 
how ogr2ogr is supposed to interact with WFS services?


Using GDAL 3.4.0, released 2021/11/04. But the same behaviour was 
observed in 3.3.2.



Thanks in advance for any clarification


Hernán




___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] ogr2ogr and WFS 2

2021-11-29 Thread Hernán De Angelis 

Dear all

I seem to have problems understanding how ogr2ogr interacts with WFS and 
hope someone would be able to clarify.


I have this WFS:

https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?

The capabilities document states that it supports WFS 1.0.0, 1.1.0 and 
2.0.0.


If I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=1.0.0=GetFeature=stationsregistret:active_site=100;


I get a small and nice GPKG file very fast.

But if I do:

ogr2ogr -f GPKG test.gpkg 
"WFS:https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100;


It downloads the whole layer.

This behaviour can be seriously impractical for services exposing a 
large number of features.



Pay attention that if I do:

curl 
"https://stationsregister.miljodatasamverkan.se/geoserver/stationsregistret/wfs?SERVICE=WFS=2.0.0=GetFeature=stationsregistret:active_site=100;


it retrieves a short document fast with just 100 features. The same of 
course if I invoke the service using WFS 1.0.0 or 1.1.0.



What is going on here? Is it me that I misunderstood something about how 
ogr2ogr is supposed to interact with WFS services?


Using GDAL 3.4.0, released 2021/11/04. But the same behaviour was 
observed in 3.3.2.



Thanks in advance for any clarification


Hernán


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev