Re: [gdal-dev] WFS requests failing with /vsicurl/ against QGIS Server: missing Content-Length header

Mon, 29 May 2023 16:14:28 -0700 

Hi Evan,

Thank you for the prompted answer.
I'm discussing this because I testing ogr2ogr node module and it adds (always) the /vsicurl/ prefix for any resource starting with http [1]. 
The point is /vsicurl/ can only be used with some http resources.
My question was (but you already answered it) if every WFS request should be callable with /vsicurl/.
Geoserver supports both /vsicurl/ and /vsicurl_streaming/ which seems more flexible. That's why I thought that QGIS Server should add Content-Length headers too.
If you consider that this is not a limitation of QGIS Server, let's keep as it is. I'll use /vsicurl_streaming/ for both Geoserver and QGIS Server calls. 

Thanks for the feedback on this.

Best regards,

Jorge
[1] https://github.com/wavded/ogr2ogr/blob/729f321dd96414e256aa67be1bc76f0b0a9b21dc/index.ts#L105 


On 29/05/23 23:54, Even Rouault wrote:

Jorge,
Which error for which invocation of the WFS driver do you get ? Your below tries are already low level attempts.
e.g the below invocation works for me without any error with GDAL master (in "ghcr.io/osgeo/gdal:alpine-small-3.7.0" Docker image too)
ogr2ogr tmp.gpkg "WFS:https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS";
The WFS driver doesn't use /vsicurl/ but /vsicurl_streaming/ . The former requires HTTP GET Range request support, while the later doesn't.
and when using "ogrinfo http://something";, you are *not* using /vsicurl or /vsicurl_streaming. You are using the hidden HTTP driver that downloads locally the file before passing it to a driver.
I don't believe there is an issue with QGIS Server. It probably streams its output (didn't check though), so it cannot know in advance the Content-Length. The headers of the HEAD requests aren't particularly helpful too. Looking at the ones of the GET request, I see "Transfer-Encoding: chunked" for QGIS server, which is in line with a streamed output and OK 

Even

Le 30/05/2023 à 00:23, Jorge Gustavo Rocha a écrit :

Hi,
I'm getting errors with WFS calls to QGIS Server using /vsicurl/. Against Geoserver I didn't notice this issue.
I can work on a solution, if the problem is on the QGIS Server side. I would love to have a second opinion on this. 

What I have found:

1) The is no Content-Length: header in QGIS Server answer:
curl -L -I "https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016"; 
HTTP/1.1 200 OK
Date: Mon, 29 May 2023 22:12:32 GMT
Server: Apache/2.4.41 (Ubuntu)
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT
Access-Control-Max-Age: 1000
Access-Control-Allow-Headers: x-requested-with, Content-Type, origin, authorization, accept, client-security-token 
Content-Type: text/xml; subtype=gml/2.1.2; charset=utf-8

2) The is a Content-Length: header in Geoserver answer:
curl -L -I "https://wssig3.azores.gov.pt/geoserver/ows?service=WFS&request=GetFeature&typename=ef:EnvironmentalMonitoringFacility&version=2.0.0&srsname=EPSG:5015"; 
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
Content-Type: application/gml+xml; version=3.2
Content-Disposition: inline; filename=features.xml
Content-Length: 494599
Server: Jetty(9.4.18.v20190429)

Example of testing calls:

This works (without vsicurl):
ogrinfo "https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016"; INFO: Open of `https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016' 
      using driver `GML' successful.
1: ponto_cotado

This does not work:
ogrinfo "/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016"; --debug on HTTP: libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.14 
VSICURL: HEAD did not provide file size. Retrying with GET
VSICURL: GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016): response_code=200, curl error msg=Failure writing output to destination 
VSICURL: Request at offset 0, after end of file
VSICURL: Request at offset 0, after end of file
VSICURL: GetFileList(/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin) 
VSICURL: HEAD did not provide file size. Retrying with GET
VSICURL: GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5011): response_code=200, curl error msg=Failure writing output to destination 
VSICURL: Request at offset 0, after end of file
VSICURL: Request at offset 0, after end of file
VSICURL: Request at offset 0, after end of file
VSICURL: Request at offset 0, after end of file
ogrinfo failed - unable to open '/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016'. 
DGNv8: Driver cleanup

Using docker, I can not use /vsicurl/ with WFS too:
docker run -ti --rm osgeo/gdal ogrinfo "/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016"; --debug on HTTP: libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13 
VSICURL: HEAD did not provide file size. Retrying with GET
VSICURL: GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016): response_code=200, curl error msg=Failure writing output to destination 
VSICURL: Request at offset 0, after end of file
VSICURL: Request at offset 0, after end of file
VSICURL: GetFileList(/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin) 
VSICURL: HEAD did not provide file size. Retrying with GET
VSICURL: GetFileSize(https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5011): response_code=200, curl error msg=Failure writing output to destination 
VSICURL: Request at offset 0, after end of file
VSICURL: Request at offset 0, after end of file
VSICURL: Request at offset 0, after end of file
VSICURL: Request at offset 0, after end of file
ogrinfo failed - unable to open '/vsicurl/https://irig-madeira.pt/postgresql/drote/public/baram_2020/cgi-bin/qgis_mapserv.fcgi?service=WFS&version=2.0.0&request=GetFeature&typename=ponto_cotado&srsname=EPSG:5016'. 

For a Geoserver layer, vsicurl is working:
ogrinfo "https://wssig3.azores.gov.pt/geoserver/ows?service=WFS&request=GetFeature&typename=ef:EnvironmentalMonitoringFacility&version=2.0.0&srsname=EPSG:5015"; INFO: Open of `https://wssig3.azores.gov.pt/geoserver/ows?service=WFS&request=GetFeature&typename=ef:EnvironmentalMonitoringFacility&version=2.0.0&srsname=EPSG:5015' 
      using driver `GML' successful.
1: EnvironmentalMonitoringFacility
Can I assume that the problem is the missing Content-Length header in QGIS Server WFS answer? Should I file a bug about this in QGIS? 

Regards,

Jorge



--
OSGeo Portugal
NIF: 509588190
email: [email protected]
https://www.osgeopt.pt/

Próximo evento: SASIG 2017, 20 a 22 de novembro, Porto
Mais informações: http://osgeopt.pt/sasig2017/
_______________________________________________
gdal-dev mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Reply via email to