Re: [QGIS-Developer] QGIS Server plugins and thread safety

[Nyall Dawson]

On Fri, 26 Oct 2018 at 00:28, G. Allegri  wrote:
>
> Moreover The C API offers the opportunity to release the GIL (see for example 
> numpy [1]).

For reference -- we do this also since QGIS 3.0. Every call to a QGIS
(c++) API releases the GIL and reacquires after the call. This means
it's possible to do multithreaded processes in PyQGIS using expensive
QGIS api calls, e.g. you could do a bunch of geometry intersections on
different threads without the GIL blocking other threads.

Nyall
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Server plugins and thread safety

[Radim Blazek]

OK, single thread only for now.

On Sun, Nov 19, 2017 at 5:39 PM Martin Dobias  wrote:

> Dealing with objects like map layers or project in non-main thread is likely
> to end up badly.

Just as a theoretical question: Why, if there are no static variables
and everything (except layer rendering, which works in threads
already) will happen on the same thread?

Radim

On Thu, Oct 25, 2018 at 4:28 PM G. Allegri  wrote:
>
> Python GIL  doess't guarantee thread safety, and it only applies to python 
> code. It only protects the internal state of the interpreter.
> Moreover The C API offers the opportunity to release the GIL (see for example 
> numpy [1]).
>
> Giovanni
>
> [1] https://stackoverflow.com/a/36480941
>
> Il giorno gio 25 ott 2018 alle ore 13:10 Alessandro Pasotti 
>  ha scritto:
>>
>>
>> I would go for multiprocess and stick to a single thread per-process.
>>
>> But David may have a better advice.
>>
>> Btw, it's nice to see a use case for Python Vector data providers :)
>>
>>
>>
>>
>> On Thu, Oct 25, 2018 at 1:02 PM Radim Blazek  wrote:
>>>
>>> If we narrow down the environment to uWSGI (with multiple threads
>>> enabled) running a Python application, which means that more instances
>>> of the same application may run in threads but never at the same time
>>> due to Python GIL (if I understand it well), is it still a problem? If
>>> we ensure that there are no static variables and singletons used by
>>> QGIS server? Is it then a problem to create/delete QObject on a non
>>> main thread?
>>>
>>> Radim
>>> On Sun, Nov 19, 2017 at 5:39 PM Martin Dobias  wrote:
>>> >
>>> > Hi Alessandro
>>> >
>>> > On Sun, Nov 19, 2017 at 10:34 AM, Alessandro Pasotti  
>>> > wrote:
>>> > > Hi,
>>> > >
>>> > > mi recent experiments with multi-threaded python wrappers for QGIS 
>>> > > server
>>> > > showed a critical flaw in my original implementation of the server 
>>> > > plugins.
>>> > >
>>> > > First, I want to stress that this is not a problem in FCGI server
>>> > > implementation but only if the server is used directly from python in a
>>> > > multi threaded server implementation.
>>> > >
>>> > > The problem is that the server interface that exposes the request 
>>> > > handler is
>>> > > a static property of the interface, that is changed on every request.
>>> > > This means that there is a race condition in setting/accessing the 
>>> > > handler
>>> > > from a plugin filter, and that a plugin filter might access to the 
>>> > > handler
>>> > > for a wrong request.
>>> >
>>> > I think this is probably just the tip of the iceberg: most of the
>>> > classes in qgis_core are not thread-safe, and I believe the
>>> > implementation of server's services is not expected to be run in
>>> > multiple threads at once either. So even though things in theory could
>>> > work if requests are handled in multiple threads, most likely there
>>> > will be crashes. Even though map rendering was made to run safe in
>>> > multiple worker threads, there is still the assumption that the map
>>> > rendering is started from the main thread where all the objects live
>>> > (I'm talking about the QObject-based classes). Dealing with objects
>>> > like map layers or project in non-main thread is likely to end up
>>> > badly.
>>> >
>>> > I would say for now we should simply stick with handling only one
>>> > request per server process at a time - just like how it is done with
>>> > the FastCGI implementation.
>>> >
>>> > Cheers
>>> > Martin
>>> > ___
>>> > QGIS-Developer mailing list
>>> > QGIS-Developer@lists.osgeo.org
>>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>>
>>
>> --
>> Alessandro Pasotti
>> w3:   www.itopen.it
>> ___
>> QGIS-Developer mailing list
>> QGIS-Developer@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Server plugins and thread safety

[G. Allegri]

Python GIL  doess't guarantee thread safety, and it only applies to python
code. It only protects the internal state of the interpreter.
Moreover The C API offers the opportunity to release the GIL (see for
example numpy [1]).

Giovanni

[1] https://stackoverflow.com/a/36480941

Il giorno gio 25 ott 2018 alle ore 13:10 Alessandro Pasotti <
apaso...@gmail.com> ha scritto:

>
> I would go for multiprocess and stick to a single thread per-process.
>
> But David may have a better advice.
>
> Btw, it's nice to see a use case for Python Vector data providers :)
>
>
>
>
> On Thu, Oct 25, 2018 at 1:02 PM Radim Blazek 
> wrote:
>
>> If we narrow down the environment to uWSGI (with multiple threads
>> enabled) running a Python application, which means that more instances
>> of the same application may run in threads but never at the same time
>> due to Python GIL (if I understand it well), is it still a problem? If
>> we ensure that there are no static variables and singletons used by
>> QGIS server? Is it then a problem to create/delete QObject on a non
>> main thread?
>>
>> Radim
>> On Sun, Nov 19, 2017 at 5:39 PM Martin Dobias 
>> wrote:
>> >
>> > Hi Alessandro
>> >
>> > On Sun, Nov 19, 2017 at 10:34 AM, Alessandro Pasotti <
>> apaso...@gmail.com> wrote:
>> > > Hi,
>> > >
>> > > mi recent experiments with multi-threaded python wrappers for QGIS
>> server
>> > > showed a critical flaw in my original implementation of the server
>> plugins.
>> > >
>> > > First, I want to stress that this is not a problem in FCGI server
>> > > implementation but only if the server is used directly from python in
>> a
>> > > multi threaded server implementation.
>> > >
>> > > The problem is that the server interface that exposes the request
>> handler is
>> > > a static property of the interface, that is changed on every request.
>> > > This means that there is a race condition in setting/accessing the
>> handler
>> > > from a plugin filter, and that a plugin filter might access to the
>> handler
>> > > for a wrong request.
>> >
>> > I think this is probably just the tip of the iceberg: most of the
>> > classes in qgis_core are not thread-safe, and I believe the
>> > implementation of server's services is not expected to be run in
>> > multiple threads at once either. So even though things in theory could
>> > work if requests are handled in multiple threads, most likely there
>> > will be crashes. Even though map rendering was made to run safe in
>> > multiple worker threads, there is still the assumption that the map
>> > rendering is started from the main thread where all the objects live
>> > (I'm talking about the QObject-based classes). Dealing with objects
>> > like map layers or project in non-main thread is likely to end up
>> > badly.
>> >
>> > I would say for now we should simply stick with handling only one
>> > request per server process at a time - just like how it is done with
>> > the FastCGI implementation.
>> >
>> > Cheers
>> > Martin
>> > ___
>> > QGIS-Developer mailing list
>> > QGIS-Developer@lists.osgeo.org
>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Server plugins and thread safety

[Alessandro Pasotti]

I would go for multiprocess and stick to a single thread per-process.

But David may have a better advice.

Btw, it's nice to see a use case for Python Vector data providers :)




On Thu, Oct 25, 2018 at 1:02 PM Radim Blazek  wrote:

> If we narrow down the environment to uWSGI (with multiple threads
> enabled) running a Python application, which means that more instances
> of the same application may run in threads but never at the same time
> due to Python GIL (if I understand it well), is it still a problem? If
> we ensure that there are no static variables and singletons used by
> QGIS server? Is it then a problem to create/delete QObject on a non
> main thread?
>
> Radim
> On Sun, Nov 19, 2017 at 5:39 PM Martin Dobias  wrote:
> >
> > Hi Alessandro
> >
> > On Sun, Nov 19, 2017 at 10:34 AM, Alessandro Pasotti 
> wrote:
> > > Hi,
> > >
> > > mi recent experiments with multi-threaded python wrappers for QGIS
> server
> > > showed a critical flaw in my original implementation of the server
> plugins.
> > >
> > > First, I want to stress that this is not a problem in FCGI server
> > > implementation but only if the server is used directly from python in a
> > > multi threaded server implementation.
> > >
> > > The problem is that the server interface that exposes the request
> handler is
> > > a static property of the interface, that is changed on every request.
> > > This means that there is a race condition in setting/accessing the
> handler
> > > from a plugin filter, and that a plugin filter might access to the
> handler
> > > for a wrong request.
> >
> > I think this is probably just the tip of the iceberg: most of the
> > classes in qgis_core are not thread-safe, and I believe the
> > implementation of server's services is not expected to be run in
> > multiple threads at once either. So even though things in theory could
> > work if requests are handled in multiple threads, most likely there
> > will be crashes. Even though map rendering was made to run safe in
> > multiple worker threads, there is still the assumption that the map
> > rendering is started from the main thread where all the objects live
> > (I'm talking about the QObject-based classes). Dealing with objects
> > like map layers or project in non-main thread is likely to end up
> > badly.
> >
> > I would say for now we should simply stick with handling only one
> > request per server process at a time - just like how it is done with
> > the FastCGI implementation.
> >
> > Cheers
> > Martin
> > ___
> > QGIS-Developer mailing list
> > QGIS-Developer@lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>


-- 
Alessandro Pasotti
w3:   www.itopen.it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Server plugins and thread safety

[Radim Blazek]

If we narrow down the environment to uWSGI (with multiple threads
enabled) running a Python application, which means that more instances
of the same application may run in threads but never at the same time
due to Python GIL (if I understand it well), is it still a problem? If
we ensure that there are no static variables and singletons used by
QGIS server? Is it then a problem to create/delete QObject on a non
main thread?

Radim
On Sun, Nov 19, 2017 at 5:39 PM Martin Dobias  wrote:
>
> Hi Alessandro
>
> On Sun, Nov 19, 2017 at 10:34 AM, Alessandro Pasotti  
> wrote:
> > Hi,
> >
> > mi recent experiments with multi-threaded python wrappers for QGIS server
> > showed a critical flaw in my original implementation of the server plugins.
> >
> > First, I want to stress that this is not a problem in FCGI server
> > implementation but only if the server is used directly from python in a
> > multi threaded server implementation.
> >
> > The problem is that the server interface that exposes the request handler is
> > a static property of the interface, that is changed on every request.
> > This means that there is a race condition in setting/accessing the handler
> > from a plugin filter, and that a plugin filter might access to the handler
> > for a wrong request.
>
> I think this is probably just the tip of the iceberg: most of the
> classes in qgis_core are not thread-safe, and I believe the
> implementation of server's services is not expected to be run in
> multiple threads at once either. So even though things in theory could
> work if requests are handled in multiple threads, most likely there
> will be crashes. Even though map rendering was made to run safe in
> multiple worker threads, there is still the assumption that the map
> rendering is started from the main thread where all the objects live
> (I'm talking about the QObject-based classes). Dealing with objects
> like map layers or project in non-main thread is likely to end up
> badly.
>
> I would say for now we should simply stick with handling only one
> request per server process at a time - just like how it is done with
> the FastCGI implementation.
>
> Cheers
> Martin
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Server plugins and thread safety

[Martin Dobias]

Hi Alessandro

On Sun, Nov 19, 2017 at 10:34 AM, Alessandro Pasotti  wrote:
> Hi,
>
> mi recent experiments with multi-threaded python wrappers for QGIS server
> showed a critical flaw in my original implementation of the server plugins.
>
> First, I want to stress that this is not a problem in FCGI server
> implementation but only if the server is used directly from python in a
> multi threaded server implementation.
>
> The problem is that the server interface that exposes the request handler is
> a static property of the interface, that is changed on every request.
> This means that there is a race condition in setting/accessing the handler
> from a plugin filter, and that a plugin filter might access to the handler
> for a wrong request.

I think this is probably just the tip of the iceberg: most of the
classes in qgis_core are not thread-safe, and I believe the
implementation of server's services is not expected to be run in
multiple threads at once either. So even though things in theory could
work if requests are handled in multiple threads, most likely there
will be crashes. Even though map rendering was made to run safe in
multiple worker threads, there is still the assumption that the map
rendering is started from the main thread where all the objects live
(I'm talking about the QObject-based classes). Dealing with objects
like map layers or project in non-main thread is likely to end up
badly.

I would say for now we should simply stick with handling only one
request per server process at a time - just like how it is done with
the FastCGI implementation.

Cheers
Martin
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Server plugins and thread safety

[David Marteau]

Hi,

IMHO, The multi process approach has to be preferred over  multi-threading

Some thought about it:

Forking sever instance for each request is imho a bad idea as it is not 
scalable and defeat resource  control.

I think that it is not a problem  by itself to consider the server engine as 
single worker process:  there is numerous cases where it is much better handled 
using appropriates tools:

- Use supervisor with fcgi prefork to handle a pool of server (not sure is is 
appropriate here)
- Embed the server in a asynchronous framework (like tornado, or python 3 
asyncio) and use multiprocessing to handle your server workers.
- Embed as wsgi  application and use gunicorn mumérous options to handle 
requests queues and the number workers.
…


David. 


IMHO this can be handled in much better way with other tools:

> Le 19 nov. 2017 à 10:34, Alessandro Pasotti  a écrit :
> 
> Hi,
> 
> mi recent experiments with multi-threaded python wrappers for QGIS server 
> showed a critical flaw in my original implementation of the server plugins.
> 
> First, I want to stress that this is not a problem in FCGI server 
> implementation but only if the server is used directly from python in a multi 
> threaded server implementation.
> 
> The problem is that the server interface that exposes the request handler is 
> a static property of the interface, that is changed on every request.
> This means that there is a race condition in setting/accessing the handler 
> from a plugin filter, and that a plugin filter might access to the handler 
> for a wrong request.
> 
> The solution is to pass the request handler (or the request and/or response 
> objects depending on the filter) to the plugin filters and leave the 
> interface for static properties only.
> 
> This would be a big API change for the server interface and filters and I'm 
> not sure we have time/resources to do that now.
> 
> As an alternative we could simply document the issue and change the testing 
> code to create a new server instance to server every request (to be verified, 
> and with consitent performance degradation).
> 
> What's your opinion about this issue?
> 
> BTW,  tomorrow I'll file a ticket for this problem.
> 
> -- 
> Alessandro Pasotti
> w3:   www.itopen.it 
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Server plugins and thread safety

[Alessandro Pasotti]

Maybe I've found a simple and effective no-API-break solution: I'm working
on a QThreadStorage based implementation of the server iface request
handler that might be the simplest solution to this issue.

Btw this is only one of the most obvious issues in the server thread
safety, there are certainly more areas that need to be checked before we
can say that QgsServer class it thread-safe.

Happy sunday :)



On Sun, Nov 19, 2017 at 10:34 AM, Alessandro Pasotti 
wrote:

> Hi,
>
> mi recent experiments with multi-threaded python wrappers for QGIS server
> showed a critical flaw in my original implementation of the server plugins.
>
> First, I want to stress that this is not a problem in FCGI server
> implementation but only if the server is used directly from python in a
> multi threaded server implementation.
>
> The problem is that the server interface that exposes the request handler
> is a static property of the interface, that is changed on every request.
> This means that there is a race condition in setting/accessing the handler
> from a plugin filter, and that a plugin filter might access to the handler
> for a wrong request.
>
> The solution is to pass the request handler (or the request and/or
> response objects depending on the filter) to the plugin filters and leave
> the interface for static properties only.
>
> This would be a big API change for the server interface and filters and
> I'm not sure we have time/resources to do that now.
>
> As an alternative we could simply document the issue and change the
> testing code to create a new server instance to server every request (to be
> verified, and with consitent performance degradation).
>
> What's your opinion about this issue?
>
> BTW,  tomorrow I'll file a ticket for this problem.
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>



-- 
Alessandro Pasotti
w3:   www.itopen.it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Server plugins and thread safety

[Luigi Pirelli]

I didn't follow, shared cache is a solved problem for the server?
Luigi Pirelli

**
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* 
https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
**


On 19 November 2017 at 10:34, Alessandro Pasotti  wrote:
> Hi,
>
> mi recent experiments with multi-threaded python wrappers for QGIS server
> showed a critical flaw in my original implementation of the server plugins.
>
> First, I want to stress that this is not a problem in FCGI server
> implementation but only if the server is used directly from python in a
> multi threaded server implementation.
>
> The problem is that the server interface that exposes the request handler is
> a static property of the interface, that is changed on every request.
> This means that there is a race condition in setting/accessing the handler
> from a plugin filter, and that a plugin filter might access to the handler
> for a wrong request.
>
> The solution is to pass the request handler (or the request and/or response
> objects depending on the filter) to the plugin filters and leave the
> interface for static properties only.
>
> This would be a big API change for the server interface and filters and I'm
> not sure we have time/resources to do that now.
>
> As an alternative we could simply document the issue and change the testing
> code to create a new server instance to server every request (to be
> verified, and with consitent performance degradation).
>
> What's your opinion about this issue?
>
> BTW,  tomorrow I'll file a ticket for this problem.
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer