Re: [QGIS-Developer] [Qgis-developer] [Server] performance questions

[René-Luc Dhont]

Hi Régis,

Is this work, about "Speed up project loading by skipping data checks. 
Useful in qgis server context or project with huge database views or 
materialized view.", still relevant ?


https://github.com/pblottiere/QGIS/tree/trust

Regards,
René-Luc

Le 15/02/2017 à 18:48, Régis Haubourg a écrit :

Hi all,
reviving the thread - Oslandia had the opportunity to work on speed up
improvements.
So Paul  introduced an option to avoid unecessary checks on startup called
"Trust project when datasource has no metadata (id, extent, geometry type).
His branch is here https://github.com/pblottiere/QGIS/tree/trust

we plan to change the associated tooltip to: "Speed up project loading by
skipping data checks. Useful in qgis server context or project with huge
database views or materialized view."

If anyone has very big databases using views, it would be cool to test to
estimate how faster a project loads (and a getCapabilities is generated then
fort QGIS server)

Cheers
Régis



-
--
Régis Haubourg
GIS administrator and project manager
Administrateur de données géographiques et chef de projet SIG
Agence de l'eau Adour-Garonne

--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5308088.html
Sent from the QGIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[kimaidou]

Hi,

Thanks for this work.
I heard in a recent PR that Richard Duivenvoorde is struggling with big
PostgreSQL datasets (see [1] )

[1]
https://github.com/qgis/QGIS-Enhancement-Proposals/issues/79#issuecomment-279628851

Michaël

2017-02-16 11:04 GMT+01:00 Luca Manganelli :

>
> On Wed, Feb 15, 2017 at 6:48 PM, Régis Haubourg 
> wrote:
>
>> So Paul  introduced an option to avoid unecessary checks on startup called
>> "Trust project when datasource has no metadata (id, extent, geometry
>> type).
>> His branch is here https://github.com/pblottiere/QGIS/tree/trust
>
>
> ​This is awesome! Looking forward in merging into QGIS.
>
> But now I see that the Travis build gives an error...
>
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Luca Manganelli]

On Wed, Feb 15, 2017 at 6:48 PM, Régis Haubourg 
wrote:

> So Paul  introduced an option to avoid unecessary checks on startup called
> "Trust project when datasource has no metadata (id, extent, geometry type).
> His branch is here https://github.com/pblottiere/QGIS/tree/trust


​This is awesome! Looking forward in merging into QGIS.

But now I see that the Travis build gives an error...
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Régis Haubourg]

Hi all,
reviving the thread - Oslandia had the opportunity to work on speed up
improvements. 
So Paul  introduced an option to avoid unecessary checks on startup called
"Trust project when datasource has no metadata (id, extent, geometry type). 
His branch is here https://github.com/pblottiere/QGIS/tree/trust 

we plan to change the associated tooltip to: "Speed up project loading by
skipping data checks. Useful in qgis server context or project with huge
database views or materialized view."

If anyone has very big databases using views, it would be cool to test to
estimate how faster a project loads (and a getCapabilities is generated then
fort QGIS server)

Cheers
Régis 



-
--
Régis Haubourg
GIS administrator and project manager
Administrateur de données géographiques et chef de projet SIG
Agence de l'eau Adour-Garonne 

--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5308088.html
Sent from the QGIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Jörg Habenicht]

Hello Andreas,

I did forward the question to our internal development, returned some
answers for clarity:

Am 22.04.2016 um 16:45 schrieb Neumann, Andreas:
> Hi Jörg,
> 
> Unfortunately I am not the suitable person to ask. In only occasionally
> develop Python and don't know C/C++. However, I am a long-time QGIS
> server user. I have big projects as well (with 80/90 layers). They
> usually render in 1-2 seconds, not 20 seconds like you cite. So maybe
> there are other issues/bottlenecks?

It seems so.

> Do you have an exceptional number of features in your layers?

Yes.

> Do you use scale thresholds to filter features.

No.

> Typically it doesn't make any sense to render too many features at small
> scales, because you can't depict them anyway if there are too many.
> 
> I assume you use Postgis as a data source - since you talk about DB
> queries.

Ah yes, correct.

> Do you have the proper spatial index and other indexes?

We do have indices installed along the tables, sometimes more than one.
We did not analyse each and every table, but the handling should perform
sufficiently.

> Do you have a recent version of QGIS server?

Fairly recent, version 9.4

> I think version 2.10 and later
> introduced filtering inside the DB (instead of filtering in QGIS), which
> should come with a performance improvement.

No, we dont use DB server filtering.

> 20 seconds is definitely too long a wait for a map rendering, but I
> guess you can bring this down to 1-2 seconds if properly optimized (both
> in DB and in QGIS).
> 
> It may well be that there is room for improvements in rendering speed of
> QGIS server, but multi-threading is probaby the most complicated way of
> improving performance in QGIS server. When I asked Marco Hugentobler,
> the main QGIS Server developer, he told me it would be almost impossible
> to make QGIS server multi-threaded. 3LIZ is also into QGIS server
> development since some time. So these are the guys to ask.
> 
> And then - the other issue is that you want to reserve your other CPU
> cores for parallel requests as well. You usually don't want to dedicate
> all your CPU power to a single http request.

Oh yes certainly I want to.
If an eight core server uses only one core to calculate the result, we
could have a speed improvement factor 2-6 (my guess) for the single
answer. With negligible slowdown of the application because of the
additional handling of multi threading. So in this case the server
delivers the resulting picture faster than before.
And if the same server already has a multi threaded request running on
eight cores, then the linux kernel divides the cpu calculation. So that
a single request still gains an estimated improvement of 1-3.
So in the result we have no significant speed loss in case of a fully
loaded server. But much gain in case of an idle server.


> 
> But if there is a way to improve speed in QGIS server I am all for it ...
> 
> Greetings,
> 
> Andreas
> 
[snip]

cu
Jörg

-- 

mWerk GmbH
Dipl.-Ing. Jörg Habenicht
Landwehrstr. 76
30519 Hannover
(T) +49 511  8033
(F) +49 511  8041
(E) j...@mwerk.net
Amtsgericht Hannover HRB 206522
Geschäftsführer
Reiner Brachvogel
Dennis Kornehl
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Jörg Habenicht]

Am 23.04.2016 um 11:47 schrieb Matthias Kuhn:
> Hi Jörg,
> 
> On 22/04/16 18:24, Jörg Habenicht wrote:
>> Hi Matthias,
>>
[snip]

>> It seems to me I have to extract the code inside the loop (in
>> QgsMapRenderer::render(), "while ( li.hasPrevious() )") into a separate
>> method and call QtConcurrent on that.
> You shouldn't have to deal directly with QtConcurrent, this code is
> alredy present.
> 
> https://qgis.org/api/classQgsMapRendererJob.html#details
> 
> Have a look at QgsMapCanvas::refreshMap() which sets up a job and
> connects the signal finished() to a local slot and continues execution.
> On the server instead you probably want to use a QFutureWatcher with
> waitForFinished() to wait for the painting job (with its threads) finish
> before responding with the freshly painted image.

Oh yes, good hint.
thank you

> 
> Matthias

cu
Jörg

-- 

mWerk GmbH
Dipl.-Ing. Jörg Habenicht
Landwehrstr. 76
30519 Hannover
(T) +49 511  8033
(F) +49 511  8041
(E) j...@mwerk.net
Amtsgericht Hannover HRB 206522
Geschäftsführer
Reiner Brachvogel
Dennis Kornehl
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Matthias Kuhn]

Hi Jörg,

On 22/04/16 18:24, Jörg Habenicht wrote:

Hi Matthias,

Am 22.04.2016 um 17:20 schrieb Matthias Kuhn:

Hi,

I don't know the server code very well, so I may be wrong, but I think
that making the server multithreaded rendering (MTR) capable should be
possible. And while it is certainly often possible to increase the
performance with filtering, rules and caching there are certainly also
situations where MTR can help a server to generate quick responses (many
layers, few requests).

As far as I can see, the main task would be porting the code from
QgsMapRenderer to QgsMapRendererJob and QgsMapSettings. A first step
into this direction can be found in a recently closed pull request [1],
in particular this commit [2].

I'll have a look.
It seems to me I have to extract the code inside the loop (in
QgsMapRenderer::render(), "while ( li.hasPrevious() )") into a separate
method and call QtConcurrent on that.
You shouldn't have to deal directly with QtConcurrent, this code is 
alredy present.


https://qgis.org/api/classQgsMapRendererJob.html#details

Have a look at QgsMapCanvas::refreshMap() which sets up a job and 
connects the signal finished() to a local slot and continues execution.
On the server instead you probably want to use a QFutureWatcher with 
waitForFinished() to wait for the painting job (with its threads) finish 
before responding with the freshly painted image.


Matthias
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Jörg Habenicht]

Hello Marco.

Am 22.04.2016 um 17:44 schrieb Marco Hugentobler:
> Hi
> 
> Yes, rendering multithreaded in the server is possible (but needs some
> work of course).
> 
> What I said was that the effect is probably not that big, because the
> fcgi module already distributes concurrent requests to different
> processes.

we already got this technology, but in those cases described below the
work distribution does not help. Currently we got around 10
clicks/second at max, but the user has to wait an awful long time for
the map to update in his/her session.

We have measured the average response time which is around 1 second. But
using FCGI and parallel QGis-server processes does not help in this case.


> In the special case of very rare requests to the server, it
> might have a considerable effect. But even then it is very unlikely that
> a 20sec request will render in 1-2 sec (what you need for interactive
> web maps).

I'd estimate it to be nearly linear for painting the pictures (and maybe
also for calling the DB). On your server we can see one core to be fully
utilized calculating with low IO and all others idling around.

I'd like to give it a try in the below mentioned method. Do I have to
expect serious side effects? Do I have to consider something I forgot in
my design?


> 
> Regards,
> Marco

cu
Jörg

> 
[snip]
>>> Il 22 apr 2016 3:27 PM, "Jörg Habenicht" >> > ha scritto:
>>>
>>> Hi Andreas, dear list,
>>>
[snip]
>>>
>>> I have scanned the git-Qgis sources (commit
>>> 81744ecf90a8f6c1e2e94fdacb07e3dca6987dcc) for a suitable
>>> multithreading
>>> entry. Found QgsMapRenderer::render() because of the line "while (
>>> li.hasPrevious() )". It seems in this loop the layers are painted in
>>> serial onto the resulting picture.
>>> If I can build an auto parallel computing with QtConcurrent in this
>>> place there seems to be the most gain. I mean to parallelize the loop
>>> with concurrent DB requests, concurrent painting in separate pictures
>>> and in serial map the single pictures onto the result picture.
>>>
>>> Right now I don't know if I can do DB requests in parallel, or
>>> where the
>>> code for database requests is buried.
>>> And how about the calls inside the loop, are there side effects
>>> preventing multi threading?
>>>
>>>
>>> Do you got hints, flames or suggestions?
>>>
>>>
>>> cu
>>> Jörg
>>>
>>>
[snip]

-- 

mWerk GmbH
Dipl.-Ing. Jörg Habenicht
Landwehrstr. 76
30519 Hannover
(T) +49 511  8033
(F) +49 511  8041
(E) j...@mwerk.net
Amtsgericht Hannover HRB 206522
Geschäftsführer
Reiner Brachvogel
Dennis Kornehl
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Jörg Habenicht]

Hi Matthias,

Am 22.04.2016 um 17:20 schrieb Matthias Kuhn:
> Hi,
> 
> I don't know the server code very well, so I may be wrong, but I think
> that making the server multithreaded rendering (MTR) capable should be
> possible. And while it is certainly often possible to increase the
> performance with filtering, rules and caching there are certainly also
> situations where MTR can help a server to generate quick responses (many
> layers, few requests).
> 
> As far as I can see, the main task would be porting the code from
> QgsMapRenderer to QgsMapRendererJob and QgsMapSettings. A first step
> into this direction can be found in a recently closed pull request [1],
> in particular this commit [2].

I'll have a look.
It seems to me I have to extract the code inside the loop (in
QgsMapRenderer::render(), "while ( li.hasPrevious() )") into a separate
method and call QtConcurrent on that.


> 
> Concerning parallel DB connection, there's no difference between server
> and desktop use. This code is already in production use and should be
> fairly robust already.
> 
> I hope I didn't base this message on wrong assumptions and hope someone
> of the server developers can shed more light onto this.

thanks

> 
> Regards
> Matthias
> 
> [1] https://github.com/qgis/QGIS/pull/2989/files
> [2]
> https://github.com/qgis/QGIS/pull/2989/commits/4cba6393544aeb23569d6c15f19e67f4ff8bf19c
> 
> 
[snip]


-- 

mWerk GmbH
Dipl.-Ing. Jörg Habenicht
Landwehrstr. 76
30519 Hannover
(T) +49 511  8033
(F) +49 511  8041
(E) j...@mwerk.net
Amtsgericht Hannover HRB 206522
Geschäftsführer
Reiner Brachvogel
Dennis Kornehl
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Marco Hugentobler]

Hi

Yes, rendering multithreaded in the server is possible (but needs some 
work of course).


What I said was that the effect is probably not that big, because the 
fcgi module already distributes concurrent requests to different 
processes. In the special case of very rare requests to the server, it 
might have a considerable effect. But even then it is very unlikely that 
a 20sec request will render in 1-2 sec (what you need for interactive 
web maps).


Regards,
Marco

On 22.04.2016 17:20, Matthias Kuhn wrote:

Hi,

I don't know the server code very well, so I may be wrong, but I think 
that making the server multithreaded rendering (MTR) capable should be 
possible. And while it is certainly often possible to increase the 
performance with filtering, rules and caching there are certainly also 
situations where MTR can help a server to generate quick responses 
(many layers, few requests).


As far as I can see, the main task would be porting the code from 
QgsMapRenderer to QgsMapRendererJob and QgsMapSettings. A first step 
into this direction can be found in a recently closed pull request 
[1], in particular this commit [2].


Concerning parallel DB connection, there's no difference between 
server and desktop use. This code is already in production use and 
should be fairly robust already.


I hope I didn't base this message on wrong assumptions and hope 
someone of the server developers can shed more light onto this.


Regards
Matthias

[1] https://github.com/qgis/QGIS/pull/2989/files
[2] 
https://github.com/qgis/QGIS/pull/2989/commits/4cba6393544aeb23569d6c15f19e67f4ff8bf19c



On 22/04/16 17:00, Andrea Peri wrote:


I guess a more time consuming parameter is the number of vertex in 
the feature involved in the map rather than the layer numbers.


A.

Il 22 apr 2016 3:27 PM, "Jörg Habenicht" > ha scritto:


Hi Andreas, dear list,

sorry for the warm up of this old topic.

There is in deed a need to enable multi threading in Qgis-Server.

In our company we use around 60 different layer to lay some topics on
the map. This may be an extreme example to read, but quite common
in our
environment, i.e. we can not lower this number significantly.

A warmed up QGis-Server process which already read and cached the
Configuration file qgis-xy.conf still needs around 20 seconds to
calculate the complete resulting map picture.

During this calculation there is only one thread working, serializing
the DB-requests, calculation the layer picture and mapping the
picture
on the resulting picture.
The process is running on a multi core server. If we can utilize the
idle cores to do useful work, this can be an easy example to speed up
the process (significant).

I'd like to share my developing experience to do this change, but I
would need further information and some support to do some useful
patches.

I have scanned the git-Qgis sources (commit
81744ecf90a8f6c1e2e94fdacb07e3dca6987dcc) for a suitable
multithreading
entry. Found QgsMapRenderer::render() because of the line "while (
li.hasPrevious() )". It seems in this loop the layers are painted in
serial onto the resulting picture.
If I can build an auto parallel computing with QtConcurrent in this
place there seems to be the most gain. I mean to parallelize the loop
with concurrent DB requests, concurrent painting in separate pictures
and in serial map the single pictures onto the result picture.

Right now I don't know if I can do DB requests in parallel, or
where the
code for database requests is buried.
And how about the calls inside the loop, are there side effects
preventing multi threading?


Do you got hints, flames or suggestions?


cu
Jörg


Am 23.02.2016 um 16:39 schrieb Neumann, Andreas:
> Hi Régis,
>
> QGIS server is single-threaded. You can't simply turn on
multi-threaded
> rendering in QGIS server. The other issue is that each Apache FCGI
> process has it's own set of cache. So if you have 5-10 parallel
Apache
> threads or processes each thread/process has it's own cache. If
your
> client hits a thread that has the cache unititialized you will
have wait
> longer until this cache is fillled.
>
> I once asked Marco whether one could
>
> a) have a shared cache for all Apache threads or processes

There is no shared cache, because Apache spawns QGis-Processes
with its
own memory layout. these Processes know nothing about its
neighbor, and
you have a hard time to exchange data between all processes.
How about a scheduler, which assigns requests to warmed up
Qgis-processes?

>
> b) support multi-threaded rendering
>
> But he said it would be very complicated to implement a) and
b), if not
> impossible. Maybe other devs have other views. Note that 

Re: [Qgis-developer] [Server] performance questions

[Neumann, Andreas]

Hi Jörg, 

Unfortunately I am not the suitable person to ask. In only occasionally
develop Python and don't know C/C++. However, I am a long-time QGIS
server user. I have big projects as well (with 80/90 layers). They
usually render in 1-2 seconds, not 20 seconds like you cite. So maybe
there are other issues/bottlenecks? Do you have an exceptional number of
features in your layers? Do you use scale thresholds to filter features.
Typically it doesn't make any sense to render too many features at small
scales, because you can't depict them anyway if there are too many. 

I assume you use Postgis as a data source - since you talk about DB
queries. Do you have the proper spatial index and other indexes? Do you
have a recent version of QGIS server? I think version 2.10 and later
introduced filtering inside the DB (instead of filtering in QGIS), which
should come with a performance improvement. 

20 seconds is definitely too long a wait for a map rendering, but I
guess you can bring this down to 1-2 seconds if properly optimized (both
in DB and in QGIS). 

It may well be that there is room for improvements in rendering speed of
QGIS server, but multi-threading is probaby the most complicated way of
improving performance in QGIS server. When I asked Marco Hugentobler,
the main QGIS Server developer, he told me it would be almost impossible
to make QGIS server multi-threaded. 3LIZ is also into QGIS server
development since some time. So these are the guys to ask. 

And then - the other issue is that you want to reserve your other CPU
cores for parallel requests as well. You usually don't want to dedicate
all your CPU power to a single http request. 

But if there is a way to improve speed in QGIS server I am all for it
... 

Greetings, 

Andreas 

On 2016-04-22 15:20, Jörg Habenicht wrote:

> Hi Andreas, dear list,
> 
> sorry for the warm up of this old topic.
> 
> There is in deed a need to enable multi threading in Qgis-Server.
> 
> In our company we use around 60 different layer to lay some topics on
> the map. This may be an extreme example to read, but quite common in our
> environment, i.e. we can not lower this number significantly.
> 
> A warmed up QGis-Server process which already read and cached the
> Configuration file qgis-xy.conf still needs around 20 seconds to
> calculate the complete resulting map picture.
> 
> During this calculation there is only one thread working, serializing
> the DB-requests, calculation the layer picture and mapping the picture
> on the resulting picture.
> The process is running on a multi core server. If we can utilize the
> idle cores to do useful work, this can be an easy example to speed up
> the process (significant).
> 
> I'd like to share my developing experience to do this change, but I
> would need further information and some support to do some useful patches.
> 
> I have scanned the git-Qgis sources (commit
> 81744ecf90a8f6c1e2e94fdacb07e3dca6987dcc) for a suitable multithreading
> entry. Found QgsMapRenderer::render() because of the line "while (
> li.hasPrevious() )". It seems in this loop the layers are painted in
> serial onto the resulting picture.
> If I can build an auto parallel computing with QtConcurrent in this
> place there seems to be the most gain. I mean to parallelize the loop
> with concurrent DB requests, concurrent painting in separate pictures
> and in serial map the single pictures onto the result picture.
> 
> Right now I don't know if I can do DB requests in parallel, or where the
> code for database requests is buried.
> And how about the calls inside the loop, are there side effects
> preventing multi threading?
> 
> Do you got hints, flames or suggestions?
> 
> cu
> Jörg
> 
> Am 23.02.2016 um 16:39 schrieb Neumann, Andreas: 
> 
>> Hi Régis,
>> 
>> QGIS server is single-threaded. You can't simply turn on multi-threaded
>> rendering in QGIS server. The other issue is that each Apache FCGI
>> process has it's own set of cache. So if you have 5-10 parallel Apache
>> threads or processes each thread/process has it's own cache. If your
>> client hits a thread that has the cache unititialized you will have wait
>> longer until this cache is fillled.
>> 
>> I once asked Marco whether one could
>> 
>> a) have a shared cache for all Apache threads or processes
> 
> There is no shared cache, because Apache spawns QGis-Processes with its
> own memory layout. these Processes know nothing about its neighbor, and
> you have a hard time to exchange data between all processes.
> How about a scheduler, which assigns requests to warmed up Qgis-processes?
> 
>> b) support multi-threaded rendering
>> 
>> But he said it would be very complicated to implement a) and b), if not
>> impossible. Maybe other devs have other views. Note that was about 2
>> years ago, when I last discussed this with Marco.
>> 
>> -
>> 
>> Nevertheless I believe there would be room for performance improvements
>> in QGIS server - if interested parties share their 

Re: [Qgis-developer] [Server] performance questions

[Jörg Habenicht]

Hi Andreas, dear list,

sorry for the warm up of this old topic.

There is in deed a need to enable multi threading in Qgis-Server.

In our company we use around 60 different layer to lay some topics on
the map. This may be an extreme example to read, but quite common in our
environment, i.e. we can not lower this number significantly.

A warmed up QGis-Server process which already read and cached the
Configuration file qgis-xy.conf still needs around 20 seconds to
calculate the complete resulting map picture.

During this calculation there is only one thread working, serializing
the DB-requests, calculation the layer picture and mapping the picture
on the resulting picture.
The process is running on a multi core server. If we can utilize the
idle cores to do useful work, this can be an easy example to speed up
the process (significant).

I'd like to share my developing experience to do this change, but I
would need further information and some support to do some useful patches.

I have scanned the git-Qgis sources (commit
81744ecf90a8f6c1e2e94fdacb07e3dca6987dcc) for a suitable multithreading
entry. Found QgsMapRenderer::render() because of the line "while (
li.hasPrevious() )". It seems in this loop the layers are painted in
serial onto the resulting picture.
If I can build an auto parallel computing with QtConcurrent in this
place there seems to be the most gain. I mean to parallelize the loop
with concurrent DB requests, concurrent painting in separate pictures
and in serial map the single pictures onto the result picture.

Right now I don't know if I can do DB requests in parallel, or where the
code for database requests is buried.
And how about the calls inside the loop, are there side effects
preventing multi threading?


Do you got hints, flames or suggestions?


cu
Jörg


Am 23.02.2016 um 16:39 schrieb Neumann, Andreas:
> Hi Régis,
> 
> QGIS server is single-threaded. You can't simply turn on multi-threaded
> rendering in QGIS server. The other issue is that each Apache FCGI
> process has it's own set of cache. So if you have 5-10 parallel Apache
> threads or processes each thread/process has it's own cache. If your
> client hits a thread that has the cache unititialized you will have wait
> longer until this cache is fillled.
> 
> I once asked Marco whether one could
> 
> a) have a shared cache for all Apache threads or processes

There is no shared cache, because Apache spawns QGis-Processes with its
own memory layout. these Processes know nothing about its neighbor, and
you have a hard time to exchange data between all processes.
How about a scheduler, which assigns requests to warmed up Qgis-processes?

> 
> b) support multi-threaded rendering
> 
> But he said it would be very complicated to implement a) and b), if not
> impossible. Maybe other devs have other views. Note that was about 2
> years ago, when I last discussed this with Marco.
> 
> -
> 
> Nevertheless I believe there would be room for performance improvements
> in QGIS server - if interested parties share their efforts (be it dev
> time or financial resources).
> 
[snip]
> 
> Andreas
> 
[snip]
> 
>  
> 
> 
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> 

-- 

mWerk GmbH
Dipl.-Ing. Jörg Habenicht
Landwehrstr. 76
30519 Hannover
(T) +49 511  8033
(F) +49 511  8041
(E) j...@mwerk.net
Amtsgericht Hannover HRB 206522
Geschäftsführer
Reiner Brachvogel
Dennis Kornehl
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[kimaidou]

Hi Marco,


2016-02-29 8:40 GMT+01:00 Marco Hugentobler :

>
> Hi all
>
> This is not a server specific problem. It is that QGIS may need a long
> time to get metadata for some layers. So the best solution is (as it has
> been proposed already in this thread) to enhance QGIS to store the metadata
> (in the project file) and not read it from the db the first time.
>


I agree. We also need some behaviour to let the user decide how these
metadata are refreshed when needed : manually, or if it is older than X
seconds/minutes/hours/days.



> This will also speed up loading of large projects, it is not only usefull
> for the server. We should avoid to make any server specific things in core
> for that.
>


I agree with you, this "server specific thing" was not necessary, as
improvements in this area will be great for both desktop and server.



>
> >- there was a bug in QGIS Server which did not use cache whenever a
> GetMap request was called with an empty STYLE parameter
>
> No, the opposite. It did not use cache with a non-empty style parameter.
>


You are right, I wrote my email too fast.


Cheers
Michaël
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Marco Hugentobler]

Hi all

This is not a server specific problem. It is that QGIS may need a long 
time to get metadata for some layers. So the best solution is (as it has 
been proposed already in this thread) to enhance QGIS to store the 
metadata (in the project file) and not read it from the db the first 
time. This will also speed up loading of large projects, it is not only 
usefull for the server. We should avoid to make any server specific 
things in core for that.


>- there was a bug in QGIS Server which did not use cache whenever a 
GetMap request was called with an empty STYLE parameter


No, the opposite. It did not use cache with a non-empty style parameter.

Regards,
Marco

On 27.02.2016 10:32, kimaidou wrote:

Hi all,

I have made some comparison on requests done when QGIS Server is 
called for a simple GetMap, on the first initialization and aftewards 
when the cache is found.


Here are the logs

When QGIS builds the maplayer object, ie when there is no cache for 
this layer yet (first call, for example with a getcapabilities). This 
is only done if no cache is found for this layer in QGIS Server


1/ Log when estimated metadata is used (and stats are available in 
PostGIS)

https://paste.sh/yguqyRWL#EPcHHVH3uuijGgEtzH0pAlHb

2/ Log when QGIS does not use the metadata for a layer . Which means 
this is done for any view at present, since no estimated metadata is 
available, or also for a table if the user forgot to check the option 
"use metadata" when he first created the Postgis connection (which can 
be often the case because this option is not checked by default in 
PostgreSQL connection dialog )

https://paste.sh/frCglZ9A#A1nQ6BXtdBeEiZWQmTJP8v9l

3/ Log when a layer is already in QGIS Server cache, for example for a 
GetMap request. This means for table and views, since it does not 
depend on metadata existence or not, but only on cache

https://paste.sh/Rkvqf757#K1MCRitovIxMblBVils1VF7r

My conclusions here :

In the first example 1/, we see all the queries QGIS does to get all 
the needed information for building the maplayer object

* postgis information : postgis_version() , postgis_geos_version()
* search for topology features -> not really usefull server side I 
think ( ? )

* feature count . In 1, estimated metadata is used, so this query is fast
* layer extent calculation . In 1/ QGIS uses st_estimatedextent, so it 
also should be fast



In the second example 2/, without available stats (no use of estimated 
metadata for views or when the option is not checked in layer ), we 
can see the performance killing queries (for complex or big datasets) 
done by QGIS to get the needed information :


* check if the field used as primary key is unique :
SELECT count(distinct ""nb"")=count(""nb"") FROM (SELECT row_number() 
OVER() AS nb, * FROM ""2013"".v_geo_parcelle) AS ""subQuery_0"""


* Get the extent :
SELECT st_extent(""geom"") FROM (SELECT row_number() OVER() AS nb, * 
FROM ""2013"".v_geo_parcelle) AS ""subQuery_0""



In 3/, QGIS has already built the cache for the maplayer, and only 
fetches the features, which is very fast.



My thoughts on how to improve QGIS server performance with PostgreSQL 
provider (and possibly for other providers as well )


* check that layer cache in QGIS server is well used.
- I do not know for example if the 100 default number is for the whole 
server (which is then reached very fast and will become useless if 
there are many projects served by QGIS), or for each projects.
- there was a bug in QGIS Server which did not use cache whenever a 
GetMap request was called with an empty STYLE parameter. It has been 
fixed but check your version is not older to this commit : 
https://github.com/qgis/QGIS/commit/a5450a78ffcbba9a75afa8be43460ad741fea050 



* Add an option in QGIS ( in OWS server properties ) to tell QGIS 
Server to not get some information if the user who publish the project 
knows what he does, meaning

  - use the extent stored in  and do not ask for it server side
  - do NOT check that the primary key is unique -> the person who 
built the project and publish it does know that his layer has a unique 
primary key.
Note that there is a limit in the number of layers QGIS Server will 
cache ( 100 by default).


* Add a even more hard way of telling QGIS Server "Never delete the 
cache for this layer, as the extent and the other properties wont 
change (even if I allow editing inside my known layer extent )" . This 
means the cache will be store permanently for this layer and would be 
lost only if the apache / nginx server is reloaded . I do not know if 
it is feasible


* Store layers metadata "the QGIS way" in the database, as we do for 
styles. This means the users building the project has enough rights on 
the postgreSQL database, but could be a handy QGIS solution for 
production ready projects. This way we could store the metadata for 
tables AND views, in a format close to what QGIS needs to build the 
maplayer object. The user would define some 

Re: [Qgis-developer] [Server] performance questions

[kimaidou]

Hi all,

I have made some comparison on requests done when QGIS Server is called for
a simple GetMap, on the first initialization and aftewards when the cache
is found.

Here are the logs

When QGIS builds the maplayer object, ie when there is no cache for this
layer yet (first call, for example with a getcapabilities). This is only
done if no cache is found for this layer in QGIS Server

1/ Log when estimated metadata is used (and stats are available in PostGIS)
https://paste.sh/yguqyRWL#EPcHHVH3uuijGgEtzH0pAlHb

2/ Log when QGIS does not use the metadata for a layer . Which means this
is done for any view at present, since no estimated metadata is available,
or also for a table if the user forgot to check the option "use metadata"
when he first created the Postgis connection (which can be often the case
because this option is not checked by default in PostgreSQL connection
dialog )
https://paste.sh/frCglZ9A#A1nQ6BXtdBeEiZWQmTJP8v9l

3/ Log when a layer is already in QGIS Server cache, for example for a
GetMap request. This means for table and views, since it does not depend on
metadata existence or not, but only on cache
https://paste.sh/Rkvqf757#K1MCRitovIxMblBVils1VF7r

My conclusions here :

In the first example 1/, we see all the queries QGIS does to get all the
needed information for building the maplayer object
* postgis information : postgis_version() ,  postgis_geos_version()
* search for topology features -> not really usefull server side I think (
? )
* feature count . In 1, estimated metadata is used, so this query is fast
* layer extent calculation . In 1/ QGIS uses st_estimatedextent, so it also
should be fast


In the second example 2/, without available stats (no use of estimated
metadata for views or when the option is not checked in layer ), we can see
the performance killing queries (for complex or big datasets) done by QGIS
to get the needed information :

* check if the field used as primary key is unique :
SELECT count(distinct ""nb"")=count(""nb"") FROM (SELECT row_number()
OVER() AS nb, * FROM ""2013"".v_geo_parcelle) AS ""subQuery_0"""

* Get the extent :
SELECT st_extent(""geom"") FROM (SELECT row_number() OVER() AS nb, * FROM
""2013"".v_geo_parcelle) AS ""subQuery_0""


In 3/, QGIS has already built the cache for the maplayer, and only fetches
the features, which is very fast.


My thoughts on how to improve QGIS server performance with PostgreSQL
provider (and possibly for other providers as well )

* check that layer cache in QGIS server is well used.
- I do not know for example if the 100 default number is for the whole
server (which is then reached very fast and will become useless if there
are many projects served by QGIS), or for each projects.
- there was a bug in QGIS Server which did not use cache whenever a GetMap
request was called with an empty STYLE parameter. It has been fixed but
check your version is not older to this commit :
https://github.com/qgis/QGIS/commit/a5450a78ffcbba9a75afa8be43460ad741fea050

* Add an option in QGIS ( in OWS server properties ) to tell QGIS Server to
not get some information if the user who publish the project knows what he
does, meaning
  - use the extent stored in  and do not ask for it server side
  - do NOT check that the primary key is unique -> the person who built the
project and publish it does know that his layer has a unique primary key.
Note that there is a limit in the number of layers QGIS Server will cache (
100 by default).

* Add a even more hard way of telling QGIS Server "Never delete the cache
for this layer, as the extent and the other properties wont change (even if
I allow editing inside my known layer extent )" . This means the cache will
be store permanently for this layer and would be lost only if the apache /
nginx server is reloaded . I do not know if it is feasible

* Store layers metadata "the QGIS way" in the database, as we do for
styles. This means the users building the project has enough rights on the
postgreSQL database, but could be a handy QGIS solution for production
ready projects. This way we could store the metadata for tables AND views,
in a format close to what QGIS needs to build the maplayer object. The user
would define some options, such as cache duration in seconds ( for example,
overwrite the cache if the stored information has more than 1 week, or keep
it as long as I do not manually remove it, via a -1 value for cache
duration ).

As a conclusion, I think we really need a "QGIS Server context true/false"
when using some of the core classes, such as maplayer, to be able to do
some performance improvements.

Régis, René-Luc and myself can have a look at it, if you need some work
done toward this goal.

Cheers,
Michaël

2016-02-23 21:52 GMT+01:00 Andreas Neumann :

> Hi Alessandro,
>
> On 23.02.2016 17:22, Alessandro Pasotti wrote:
>
>>
>>
>> Yes, but Andreas probably meant that plugin support is still compiled
>> into the server.
>>
>
> yes - if you compile 

Re: [Qgis-developer] [Server] performance questions

[Andreas Neumann]

Hi Alessandro,

On 23.02.2016 17:22, Alessandro Pasotti wrote:



Yes, but Andreas probably meant that plugin support is still compiled 
into the server.


yes - if you compile yourself, you can disable it in the CMAKE settings.



Still, we never measured any noticeable performance issue with plugins 
enabled (as long as thre aren't any of course, otherwise it will 
obviously depend on what the plugins do).


yes - only master versions between 2.12 and 2.14 were affected. It has 
been fixed meanwhile.




There was indeed a performance issue with the new access control 
implementation (now hopefullt fixed) but IIRC it's still only in 
master, to be release soon.


Also, we aren't aware of any security issue, if you (Andreas) know 
about any security issue I would definitely like to know more about that.
I am not aware of any security issues around QGIS server Python plugins. 
It is just good to disable stuff if you don't need it and opt-in if you 
need it.


Andreas
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Yves Jacolin]

On Tuesday, February 23, 2016 17:11:33 Alessandro Pasotti wrote:
> Mapserver takes extent from mapfile, not from
> data.
that's not really true, it depends if extent exist in mapfile or not and data 
format in input. If it doesn't exist in mapfile, mapserver will look at the 
data but such things are really to avoid.

Y.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Yves Jacolin]

On Tuesday, February 23, 2016 19:18:37 Alessandro Pasotti wrote:
> 2016-02-23 19:06 GMT+01:00 Yves Jacolin :
> > On Tuesday, February 23, 2016 17:11:33 Alessandro Pasotti wrote:
> > > Mapserver takes extent from mapfile, not from
> > > data.
> > 
> > that's not really true, it depends if extent exist in mapfile or not and
> > data
> > format in input. If it doesn't exist in mapfile, mapserver will look at
> > the
> > data but such things are really to avoid.
> > 
> > Y.
> 
> I never wrote that, Règis did it (wrong quoting?)
Not really, this is my mail application which get quoting wrong! Really sorry 
about that!

Y.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Alessandro Pasotti]

2016-02-23 19:06 GMT+01:00 Yves Jacolin :

> On Tuesday, February 23, 2016 17:11:33 Alessandro Pasotti wrote:
> > Mapserver takes extent from mapfile, not from
> > data.
> that's not really true, it depends if extent exist in mapfile or not and
> data
> format in input. If it doesn't exist in mapfile, mapserver will look at the
> data but such things are really to avoid.
>
> Y.
>


I never wrote that, Règis did it (wrong quoting?)


-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Alessandro Pasotti]

2016-02-23 17:23 GMT+01:00 Régis Haubourg <
regis.haubo...@eau-adour-garonne.fr>:

> Alessandro Pasotti-2 wrote
> > Yes, but Andreas probably meant that plugin support is still compiled
> into
> > the server.
> >
> > Still, we never measured any noticeable performance issue with plugins
> > enabled (as long as thre aren't any of course, otherwise it will
> obviously
> > depend on what the plugins do).
>
> Ok then, I will try compiling server qgis somewhere else.
> Cheers
> Régis
>
>
>
I was saying that unless you are using master  (and you don't since you're
on 2.12), you probably don't need to, but if you find any measurable
performance difference, please let me know.



-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Régis Haubourg]

Alessandro Pasotti-2 wrote
> Yes, but Andreas probably meant that plugin support is still compiled into
> the server.
> 
> Still, we never measured any noticeable performance issue with plugins
> enabled (as long as thre aren't any of course, otherwise it will obviously
> depend on what the plugins do).

Ok then, I will try compiling server qgis somewhere else. 
Cheers
Régis



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5252286.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Régis Haubourg]

Alessandro Pasotti-2 wrote
> You should definitely add a feature request.

Done here:  http://hub.qgis.org/issues/14364
   
Cheers

If someone interested in working on that, please contact me. 
Cheers
Régis



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5252275.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Régis Haubourg]

Neumann, Andreas wrote
> did you switch of QGIS server python plugins? If you don't need them, I
> recommend you switch them off - both for security reasons, but also
> there was a huge performance problem with QGIS server Python plugins
> just recently. See http://hub.qgis.org/issues/13919 - luckily this was
> fixed just recently. Depending on the version you use, this may have a
> negative impact on your installation. 

We did not activate explicitly server plugins yet. we have no
QGIS_PLUGINPATH variable set, is that enough to deactivate server plugins?
Régis



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5252272.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Alessandro Pasotti]

2016-02-23 16:55 GMT+01:00 Régis Haubourg <
regis.haubo...@eau-adour-garonne.fr>:

> Alessandro Pasotti-2 wrote
> > The answer from strk  makes sense to me, I might be wrong but calculating
> > estimated metadata on a query doesn't make much sense to me.
>
> Oh, I read too fast Strk's comment. Sorry for that. I won't explore that
> dead end no more.
> Another idea, maybe autodiscovering extent on data is a good thing for
> desktop, less for server. Mapserver takes extent from mapfile, not from
> data. Could we imagine adding options to force extents from project
> properties and prevent qgis server from asking that to postgis? Such
> queries
> are BTW  very consuming thing on big datasets and I can't imagine big
> servers with many sessions keeping on querying postgis geometries just to
> get such extent.
> Régis
>
>

Yes, that sounds a nice idea.

We could create a new config option for that and cache the extents.
I'm not sure about how deep into the providers this would lead us, or if it
can be better done into the server itself, but worth exploring.

You should definitely add a feature request.


-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Régis Haubourg]

Andrea Peri wrote
> To avoid also all these questions we use the spatialite db that is fast 
> like a shapefile toopen and have no the problem of connection and break 
> of connection like the postgres.

Good to know, thanks. unfortunately, my current use case is not compatible
with data replication in spatialite. For some others, I will test that. 
Régis



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5252270.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Régis Haubourg]

Alessandro Pasotti-2 wrote
> The answer from strk  makes sense to me, I might be wrong but calculating
> estimated metadata on a query doesn't make much sense to me.

Oh, I read too fast Strk's comment. Sorry for that. I won't explore that
dead end no more.
Another idea, maybe autodiscovering extent on data is a good thing for
desktop, less for server. Mapserver takes extent from mapfile, not from
data. Could we imagine adding options to force extents from project
properties and prevent qgis server from asking that to postgis? Such queries
are BTW  very consuming thing on big datasets and I can't imagine big
servers with many sessions keeping on querying postgis geometries just to
get such extent. 
Régis



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5252268.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[aperi2007]

Hi Regis,

The shapefile is obviously more fast in opne than a postgres connection.
Is due to the different need of resources.
The postgres need to assign a lot of resources for every connection that 
it is opening.


I do't know quite well the pgpool solution, but I guess it cannot 
helpyou because AFAIK it is only for dirct connection.
You should set is with a fastcgis but you have heowever the problem that 
postgres interrupt brutally the connection after some ten of minutes 
without any byte exchanging with the clinet.


These mean that alsoif you use a fastcgi protocol but have a service not 
used for more than 30 minutes it will break connection from postgres 
side and so it need to restart the connection (and so take some time to 
do it) when the user ask for a new map.


To avoid also all these questions we use the spatialite db that is fast 
like a shapefile toopen and have no the problem of connection and break 
of connection like the postgres.


A.


Il 23/02/2016 16:11, Régis Haubourg ha scritto:

Andrea Peri wrote

I Guess the time difference between the qgis desktop and qgis-server
when using a postgres,

is due to the protocol.
Infact the http is a connection-less session-less protocoll,
so after every map the qgis-server disconnect from postgres.

Instead when you work on a qgis desktop as a user on its own pc you
are connectin with a postgres and sray connected for all the time.

So the postgres don't waste time to open and staryup your connection.

Also don't forget that if you stay about 18-20 minutes without do
nothing on qgis in a postgres connection.
The postgres kill your connection and your qgis go in crazy state.

A.

Hi Andrea,
You are right for the overhead part. This would not big a big issue if we
did not have the overhead on postgres side for extent querying on SQL views.
The same dataset in shape is open fast even at first connection.
We use pgpool here so we do not have such trouble with idle connections
being killed.
Cheers
Régis



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5252247.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Alessandro Pasotti]

2016-02-23 16:29 GMT+01:00 Régis Haubourg <
regis.haubo...@eau-adour-garonne.fr>:

> Alessandro Pasotti-2 wrote
> >
> > Hi,
> >
> > For your point 2, is this bug related to your issue
> > http://hub.qgis.org/issues/13232 ?
> >
> > --
> > Alessandro Pasotti
> > w3:   www.itopen.it
>
> Hi Alessandro,
> probably not, we do not have inherited tables and we have autovacuum
> activated. St_estimated_extent is correctly called when loading postgis
> tables. The matter is that we use very few table, but mainly views and
> materialized views. Postgres does not collect metadatas on views:
>  See https://trac.osgeo.org/postgis/ticket/510
> Having that would be a great improvement. I would be avalaible for funding.
> I already asked to postgres community and had no feedback.. maybe I asked
> the wrong place.
>

The answer from strk  makes sense to me, I might be wrong but calculating
estimated metadata on a query doesn't make much sense to me.



> Any idea concerning multithreading configuration?
>
>
>
Sorry, don't, but I'll have a look.

-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Régis Haubourg]

Alessandro Pasotti-2 wrote
> 
> Hi,
> 
> For your point 2, is this bug related to your issue
> http://hub.qgis.org/issues/13232 ?
> 
> -- 
> Alessandro Pasotti
> w3:   www.itopen.it

Hi Alessandro, 
probably not, we do not have inherited tables and we have autovacuum
activated. St_estimated_extent is correctly called when loading postgis
tables. The matter is that we use very few table, but mainly views and
materialized views. Postgres does not collect metadatas on views: 
 See https://trac.osgeo.org/postgis/ticket/510 
Having that would be a great improvement. I would be avalaible for funding.
I already asked to postgres community and had no feedback.. maybe I asked
the wrong place. 

Any idea concerning multithreading configuration?




--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5252256.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Neumann, Andreas]

Hi Régis, 

QGIS server is single-threaded. You can't simply turn on multi-threaded
rendering in QGIS server. The other issue is that each Apache FCGI
process has it's own set of cache. So if you have 5-10 parallel Apache
threads or processes each thread/process has it's own cache. If your
client hits a thread that has the cache unititialized you will have wait
longer until this cache is fillled. 

I once asked Marco whether one could 

a) have a shared cache for all Apache threads or processes 

b) support multi-threaded rendering 

But he said it would be very complicated to implement a) and b), if not
impossible. Maybe other devs have other views. Note that was about 2
years ago, when I last discussed this with Marco. 

- 

Nevertheless I believe there would be room for performance improvements
in QGIS server - if interested parties share their efforts (be it dev
time or financial resources). 

BTW: 

did you switch of QGIS server python plugins? If you don't need them, I
recommend you switch them off - both for security reasons, but also
there was a huge performance problem with QGIS server Python plugins
just recently. See http://hub.qgis.org/issues/13919 - luckily this was
fixed just recently. Depending on the version you use, this may have a
negative impact on your installation. 

Andreas 

On 2016-02-23 15:20, Régis Haubourg wrote:

> Hi all, 
> we are working on using qgis server here and we face some performance issues
> on big data sets. Cache is not an option for those layers that will be
> edited by users. Currently performances are not acceptable for production
> uses.  Here is a couple of questions that I could not answer by myself on
> the net:
> 
> 1- Clarifying multithread on server side vs multithread on desktop:
> 
> We use qgis 2.12 server from qgis repository for debian on a 4 core Xeon
> (2.1Ghz) + 16Go RAM. 
> 
> We configured apache2 with MPM worker and we have a pool of 10
> qgis_mapserv.fcgi waiting for client request. 
> The behaviour here is that one client will only consume one fcgid process,
> when desktop can use several processes to make some parallel rendering of
> project layers. 
> When having QGIS desktop  using only one thread, I have similar rendering
> time. The difference is purely proportionnal to CPU speed difference. 
> Did we miss something in configuration process or is that the intended
> behavior? I did try to use QGIS_OPTION_PATH on a QGIS2.ini but I really
> can't be sure it is really being used.
> Do QGIS packages miss the multithread option on compile cmake options? 
> If this is expected, does the web architecture would allow to have fcgid do
> parallel rendering like desktop? Would that be a big work (I could partly
> fund that)?
> 
> 2- Overhead on first request with postgres complex view compared to
> shapefiles
> 
> We have almost only postgis views being used as QGIS layers. Qgis server
> builds get capabilities and requests views to get extent and primary keys.
> There is a severe overhead for the first wms request by a client session.
> When generating the image takes about 7s, the first request can take from 24
> to 30s. Desktop multihreaded will get the image in about 3s (same postgres
> server, same network). 
> That overhead starts to become annoying on desktop side too, but it's only
> qgs loading time, once in a work session. Does someone have any idea that
> one day postgis could collect such metadata on views so that QGIS do not
> have to scan all rows when data are in views? Direct access to tables, or
> same project with shp datasources do not show that overhead on server nor in
> desktop.
> 
> Thanks for your hints, 
> Cheers
> 
> Régis
> 
> --
> View this message in context: 
> http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233.html
> Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

  ___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Régis Haubourg]

Andrea Peri wrote
> I Guess the time difference between the qgis desktop and qgis-server
> when using a postgres,
> 
> is due to the protocol.
> Infact the http is a connection-less session-less protocoll,
> so after every map the qgis-server disconnect from postgres.
> 
> Instead when you work on a qgis desktop as a user on its own pc you
> are connectin with a postgres and sray connected for all the time.
> 
> So the postgres don't waste time to open and staryup your connection.
> 
> Also don't forget that if you stay about 18-20 minutes without do
> nothing on qgis in a postgres connection.
> The postgres kill your connection and your qgis go in crazy state.
> 
> A.

Hi Andrea, 
You are right for the overhead part. This would not big a big issue if we
did not have the overhead on postgres side for extent querying on SQL views.
The same dataset in shape is open fast even at first connection. 
We use pgpool here so we do not have such trouble with idle connections
being killed. 
Cheers
Régis



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233p5252247.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Alessandro Pasotti]

2016-02-23 15:20 GMT+01:00 Régis Haubourg <
regis.haubo...@eau-adour-garonne.fr>:

> Hi all,
> we are working on using qgis server here and we face some performance
> issues
> on big data sets. Cache is not an option for those layers that will be
> edited by users. Currently performances are not acceptable for production
> uses.  Here is a couple of questions that I could not answer by myself on
> the net:
>
> 1- Clarifying multithread on server side vs multithread on desktop:
>
> We use qgis 2.12 server from qgis repository for debian on a 4 core Xeon
> (2.1Ghz) + 16Go RAM.
>
> We configured apache2 with MPM worker and we have a pool of 10
> qgis_mapserv.fcgi waiting for client request.
> The behaviour here is that one client will only consume one fcgid process,
> when desktop can use several processes to make some parallel rendering of
> project layers.
> When having QGIS desktop  using only one thread, I have similar rendering
> time. The difference is purely proportionnal to CPU speed difference.
> Did we miss something in configuration process or is that the intended
> behavior? I did try to use QGIS_OPTION_PATH on a QGIS2.ini but I really
> can't be sure it is really being used.
> Do QGIS packages miss the multithread option on compile cmake options?
> If this is expected, does the web architecture would allow to have fcgid do
> parallel rendering like desktop? Would that be a big work (I could partly
> fund that)?
>
>
> 2- Overhead on first request with postgres complex view compared to
> shapefiles
>
> We have almost only postgis views being used as QGIS layers. Qgis server
> builds get capabilities and requests views to get extent and primary keys.
> There is a severe overhead for the first wms request by a client session.
> When generating the image takes about 7s, the first request can take from
> 24
> to 30s. Desktop multihreaded will get the image in about 3s (same postgres
> server, same network).
> That overhead starts to become annoying on desktop side too, but it's only
> qgs loading time, once in a work session. Does someone have any idea that
> one day postgis could collect such metadata on views so that QGIS do not
> have to scan all rows when data are in views? Direct access to tables, or
> same project with shp datasources do not show that overhead on server nor
> in
> desktop.
>


Hi,

For your point 2, is this bug related to your issue
http://hub.qgis.org/issues/13232 ?


-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] [Server] performance questions

[Andrea Peri]

I Guess the time difference between the qgis desktop and qgis-server
when using a postgres,

is due to the protocol.
Infact the http is a connection-less session-less protocoll,
so after every map the qgis-server disconnect from postgres.

Instead when you work on a qgis desktop as a user on its own pc you
are connectin with a postgres and sray connected for all the time.

So the postgres don't waste time to open and staryup your connection.

Also don't forget that if you stay about 18-20 minutes without do
nothing on qgis in a postgres connection.
The postgres kill your connection and your qgis go in crazy state.

A.


2016-02-23 15:20 GMT+01:00 Régis Haubourg :
> Hi all,
> we are working on using qgis server here and we face some performance issues
> on big data sets. Cache is not an option for those layers that will be
> edited by users. Currently performances are not acceptable for production
> uses.  Here is a couple of questions that I could not answer by myself on
> the net:
>
> 1- Clarifying multithread on server side vs multithread on desktop:
>
> We use qgis 2.12 server from qgis repository for debian on a 4 core Xeon
> (2.1Ghz) + 16Go RAM.
>
> We configured apache2 with MPM worker and we have a pool of 10
> qgis_mapserv.fcgi waiting for client request.
> The behaviour here is that one client will only consume one fcgid process,
> when desktop can use several processes to make some parallel rendering of
> project layers.
> When having QGIS desktop  using only one thread, I have similar rendering
> time. The difference is purely proportionnal to CPU speed difference.
> Did we miss something in configuration process or is that the intended
> behavior? I did try to use QGIS_OPTION_PATH on a QGIS2.ini but I really
> can't be sure it is really being used.
> Do QGIS packages miss the multithread option on compile cmake options?
> If this is expected, does the web architecture would allow to have fcgid do
> parallel rendering like desktop? Would that be a big work (I could partly
> fund that)?
>
>
> 2- Overhead on first request with postgres complex view compared to
> shapefiles
>
> We have almost only postgis views being used as QGIS layers. Qgis server
> builds get capabilities and requests views to get extent and primary keys.
> There is a severe overhead for the first wms request by a client session.
> When generating the image takes about 7s, the first request can take from 24
> to 30s. Desktop multihreaded will get the image in about 3s (same postgres
> server, same network).
> That overhead starts to become annoying on desktop side too, but it's only
> qgs loading time, once in a work session. Does someone have any idea that
> one day postgis could collect such metadata on views so that QGIS do not
> have to scan all rows when data are in views? Direct access to tables, or
> same project with shp datasources do not show that overhead on server nor in
> desktop.
>
>
> Thanks for your hints,
> Cheers
>
> Régis
>
>
>
> --
> View this message in context: 
> http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233.html
> Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer



-- 
-
Andrea Peri
. . . . . . . . .
qwerty àèìòù
-
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

[Qgis-developer] [Server] performance questions

[Régis Haubourg]

Hi all, 
we are working on using qgis server here and we face some performance issues
on big data sets. Cache is not an option for those layers that will be
edited by users. Currently performances are not acceptable for production
uses.  Here is a couple of questions that I could not answer by myself on
the net:

1- Clarifying multithread on server side vs multithread on desktop:

We use qgis 2.12 server from qgis repository for debian on a 4 core Xeon
(2.1Ghz) + 16Go RAM. 

We configured apache2 with MPM worker and we have a pool of 10
qgis_mapserv.fcgi waiting for client request. 
The behaviour here is that one client will only consume one fcgid process,
when desktop can use several processes to make some parallel rendering of
project layers. 
When having QGIS desktop  using only one thread, I have similar rendering
time. The difference is purely proportionnal to CPU speed difference. 
Did we miss something in configuration process or is that the intended
behavior? I did try to use QGIS_OPTION_PATH on a QGIS2.ini but I really
can't be sure it is really being used.
Do QGIS packages miss the multithread option on compile cmake options? 
If this is expected, does the web architecture would allow to have fcgid do
parallel rendering like desktop? Would that be a big work (I could partly
fund that)?


2- Overhead on first request with postgres complex view compared to
shapefiles

We have almost only postgis views being used as QGIS layers. Qgis server
builds get capabilities and requests views to get extent and primary keys.
There is a severe overhead for the first wms request by a client session.
When generating the image takes about 7s, the first request can take from 24
to 30s. Desktop multihreaded will get the image in about 3s (same postgres
server, same network). 
That overhead starts to become annoying on desktop side too, but it's only
qgs loading time, once in a work session. Does someone have any idea that
one day postgis could collect such metadata on views so that QGIS do not
have to scan all rows when data are in views? Direct access to tables, or
same project with shp datasources do not show that overhead on server nor in
desktop.


Thanks for your hints, 
Cheers

Régis



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Server-performance-questions-tp5252233.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer