Re: [openstack-dev] [all] [designate] [heat] [python3] deadlock with eventlet and ThreadPoolExecutor in py3.7

[Corey Bryant]

Ok thanks again for the input.

Corey

On Wed, Jul 25, 2018 at 2:15 PM, Joshua Harlow 
wrote:

> So the only diff is that GreenThreadPoolExecutor was customized to work
> for eventlet (with a similar/same api as ThreadPoolExecutor); as for
> performance I would expect (under eventlet) that GreenThreadPoolExecutor
> would have better performance because it can use the native eventlet green
> objects (which it does try to use) instead of having to go threw the layers
> that ThreadPoolExecutor would have to use to achieve the same (and in this
> case as you found out it looks like those layers are not patched correctly
> in the newest ThreadPoolExecutor).
>
> Otherwise yes, under eventlet imho swap out the executor (assuming you can
> do this) and under threading swap in threadpool executor (ideally if done
> correctly the same stuff should 'just work').
>
> Corey Bryant wrote:
>
>> Josh,
>>
>> Thanks for the input. GreenThreadPoolExecutor does not have the deadlock
>> issue, so that is promising (at least with futurist 1.6.0).
>>
>> Does ThreadPoolExecutor have better performance than
>> GreenThreadPoolExecutor? Curious if we could just swap out
>> ThreadPoolExecutor for GreenThreadPoolExecutor.
>>
>> Thanks,
>> Corey
>>
>> On Wed, Jul 25, 2018 at 12:54 PM, Joshua Harlow > > wrote:
>>
>> Have you tried the following instead of threadpoolexecutor (which
>> honestly should work as well, even under eventlet + eventlet
>> patching).
>>
>> https://docs.openstack.org/futurist/latest/reference/index.
>> html#futurist.GreenThreadPoolExecutor
>> > html#futurist.GreenThreadPoolExecutor>
>>
>> If you have the ability to specify which executor your code is
>> using, and you are running under eventlet I'd give preference to the
>> green thread pool executor under that situation (and if not running
>> under eventlet then prefer the threadpool executor variant).
>>
>> As for @tomoto question; honestly openstack was created before
>> asyncio was a thing so that was a reason and assuming eventlet
>> patching is actually working then all the existing stdlib stuff
>> should keep on working under eventlet (including
>> concurrent.futures); otherwise eventlet.monkey_patch isn't working
>> and that's breaking the eventlet api. If their contract is that only
>> certain things work when monkey patched, that's fair, but that needs
>> to be documented somewhere (honestly it's time imho to get the hell
>> off eventlet everywhere but that likely requires rewrites of a lot
>> of things, oops...).
>>
>> -Josh
>>
>> Corey Bryant wrote:
>>
>> Hi All,
>>
>> I'm trying to add Py3 packaging support for Ubuntu Rocky and
>> while there
>> are a lot of issues involved with supporting Py3.7, this is one
>> of the
>> big ones that I could use a hand with.
>>
>> With py3.7, there's a deadlock when eventlet monkeypatch of stdlib
>> thread modules is combined with use of ThreadPoolExecutor. I
>> know this
>> affects at least designate. The same or similar also affects heat
>> (though I've not dug into the code the traceback after canceling
>> tests
>> matches that seen with designate). And it may affect other
>> projects that
>> I haven't touched yet.
>>
>> How to recreate [1]:
>> * designate: Add a tox.ini py37 target and run
>> designate.tests.test_workers.test_processing.TestProcessingE
>> xecutor.test_execute_multiple_tasks
>> * heat: Add a tox.ini py37 target and run tests
>> * general: Run bpo34173-recreate.py
>> > > from
>> issue
>> 34173 (see below).
>> [1] ubuntu cosmic has py3.7
>>
>> In issue 508 (see below) @tomoto asks "Eventlet and asyncio
>> solve same
>> problem. Why would you want concurrent.futures and eventlet in
>> same
>> application?"
>>
>> I told @tomoto that I'd seek input to that question from
>> upstream. I
>> know there've been efforts to move away from eventlet but I just
>> don't
>> have the knowledge to  provide a good answer to him.
>>
>> Here are the bugs/issues I currently have open for this:
>> https://github.com/eventlet/eventlet/issues/508
>> 
>> > >
>> https://bugs.launchpad.net/designate/+bug/1782647
>> 
>> > >
>>  

Re: [openstack-dev] [all] [designate] [heat] [python3] deadlock with eventlet and ThreadPoolExecutor in py3.7

[Joshua Harlow]

So the only diff is that GreenThreadPoolExecutor was customized to work 
for eventlet (with a similar/same api as ThreadPoolExecutor); as for 
performance I would expect (under eventlet) that GreenThreadPoolExecutor 
would have better performance because it can use the native eventlet 
green objects (which it does try to use) instead of having to go threw 
the layers that ThreadPoolExecutor would have to use to achieve the same 
(and in this case as you found out it looks like those layers are not 
patched correctly in the newest ThreadPoolExecutor).


Otherwise yes, under eventlet imho swap out the executor (assuming you 
can do this) and under threading swap in threadpool executor (ideally if 
done correctly the same stuff should 'just work').


Corey Bryant wrote:

Josh,

Thanks for the input. GreenThreadPoolExecutor does not have the deadlock
issue, so that is promising (at least with futurist 1.6.0).

Does ThreadPoolExecutor have better performance than
GreenThreadPoolExecutor? Curious if we could just swap out
ThreadPoolExecutor for GreenThreadPoolExecutor.

Thanks,
Corey

On Wed, Jul 25, 2018 at 12:54 PM, Joshua Harlow mailto:harlo...@fastmail.com>> wrote:

Have you tried the following instead of threadpoolexecutor (which
honestly should work as well, even under eventlet + eventlet patching).


https://docs.openstack.org/futurist/latest/reference/index.html#futurist.GreenThreadPoolExecutor



If you have the ability to specify which executor your code is
using, and you are running under eventlet I'd give preference to the
green thread pool executor under that situation (and if not running
under eventlet then prefer the threadpool executor variant).

As for @tomoto question; honestly openstack was created before
asyncio was a thing so that was a reason and assuming eventlet
patching is actually working then all the existing stdlib stuff
should keep on working under eventlet (including
concurrent.futures); otherwise eventlet.monkey_patch isn't working
and that's breaking the eventlet api. If their contract is that only
certain things work when monkey patched, that's fair, but that needs
to be documented somewhere (honestly it's time imho to get the hell
off eventlet everywhere but that likely requires rewrites of a lot
of things, oops...).

-Josh

Corey Bryant wrote:

Hi All,

I'm trying to add Py3 packaging support for Ubuntu Rocky and
while there
are a lot of issues involved with supporting Py3.7, this is one
of the
big ones that I could use a hand with.

With py3.7, there's a deadlock when eventlet monkeypatch of stdlib
thread modules is combined with use of ThreadPoolExecutor. I
know this
affects at least designate. The same or similar also affects heat
(though I've not dug into the code the traceback after canceling
tests
matches that seen with designate). And it may affect other
projects that
I haven't touched yet.

How to recreate [1]:
* designate: Add a tox.ini py37 target and run

designate.tests.test_workers.test_processing.TestProcessingExecutor.test_execute_multiple_tasks
* heat: Add a tox.ini py37 target and run tests
* general: Run bpo34173-recreate.py
> from issue
34173 (see below).
[1] ubuntu cosmic has py3.7

In issue 508 (see below) @tomoto asks "Eventlet and asyncio
solve same
problem. Why would you want concurrent.futures and eventlet in same
application?"

I told @tomoto that I'd seek input to that question from upstream. I
know there've been efforts to move away from eventlet but I just
don't
have the knowledge to  provide a good answer to him.

Here are the bugs/issues I currently have open for this:
https://github.com/eventlet/eventlet/issues/508

>
https://bugs.launchpad.net/designate/+bug/1782647

>
https://bugs.python.org/issue34173

>

Any help with this would be greatly appreciated!

Thanks,
Corey


__
OpenStack Development 

Re: [openstack-dev] [all] [designate] [heat] [python3] deadlock with eventlet and ThreadPoolExecutor in py3.7

[Corey Bryant]

Josh,

Thanks for the input. GreenThreadPoolExecutor does not have the deadlock
issue, so that is promising (at least with futurist 1.6.0).

Does ThreadPoolExecutor have better performance than
GreenThreadPoolExecutor? Curious if we could just swap out
ThreadPoolExecutor for GreenThreadPoolExecutor.

Thanks,
Corey

On Wed, Jul 25, 2018 at 12:54 PM, Joshua Harlow 
wrote:

> Have you tried the following instead of threadpoolexecutor (which honestly
> should work as well, even under eventlet + eventlet patching).
>
> https://docs.openstack.org/futurist/latest/reference/index.
> html#futurist.GreenThreadPoolExecutor
>
> If you have the ability to specify which executor your code is using, and
> you are running under eventlet I'd give preference to the green thread pool
> executor under that situation (and if not running under eventlet then
> prefer the threadpool executor variant).
>
> As for @tomoto question; honestly openstack was created before asyncio was
> a thing so that was a reason and assuming eventlet patching is actually
> working then all the existing stdlib stuff should keep on working under
> eventlet (including concurrent.futures); otherwise eventlet.monkey_patch
> isn't working and that's breaking the eventlet api. If their contract is
> that only certain things work when monkey patched, that's fair, but that
> needs to be documented somewhere (honestly it's time imho to get the hell
> off eventlet everywhere but that likely requires rewrites of a lot of
> things, oops...).
>
> -Josh
>
> Corey Bryant wrote:
>
>> Hi All,
>>
>> I'm trying to add Py3 packaging support for Ubuntu Rocky and while there
>> are a lot of issues involved with supporting Py3.7, this is one of the
>> big ones that I could use a hand with.
>>
>> With py3.7, there's a deadlock when eventlet monkeypatch of stdlib
>> thread modules is combined with use of ThreadPoolExecutor. I know this
>> affects at least designate. The same or similar also affects heat
>> (though I've not dug into the code the traceback after canceling tests
>> matches that seen with designate). And it may affect other projects that
>> I haven't touched yet.
>>
>> How to recreate [1]:
>> * designate: Add a tox.ini py37 target and run
>> designate.tests.test_workers.test_processing.TestProcessingE
>> xecutor.test_execute_multiple_tasks
>> * heat: Add a tox.ini py37 target and run tests
>> * general: Run bpo34173-recreate.py
>>  from issue
>> 34173 (see below).
>> [1] ubuntu cosmic has py3.7
>>
>> In issue 508 (see below) @tomoto asks "Eventlet and asyncio solve same
>> problem. Why would you want concurrent.futures and eventlet in same
>> application?"
>>
>> I told @tomoto that I'd seek input to that question from upstream. I
>> know there've been efforts to move away from eventlet but I just don't
>> have the knowledge to  provide a good answer to him.
>>
>> Here are the bugs/issues I currently have open for this:
>> https://github.com/eventlet/eventlet/issues/508
>> 
>> https://bugs.launchpad.net/designate/+bug/1782647
>> 
>> https://bugs.python.org/issue34173 
>>
>> Any help with this would be greatly appreciated!
>>
>> Thanks,
>> Corey
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [designate] [heat] [python3] deadlock with eventlet and ThreadPoolExecutor in py3.7

[Joshua Harlow]

Have you tried the following instead of threadpoolexecutor (which 
honestly should work as well, even under eventlet + eventlet patching).


https://docs.openstack.org/futurist/latest/reference/index.html#futurist.GreenThreadPoolExecutor

If you have the ability to specify which executor your code is using, 
and you are running under eventlet I'd give preference to the green 
thread pool executor under that situation (and if not running under 
eventlet then prefer the threadpool executor variant).


As for @tomoto question; honestly openstack was created before asyncio 
was a thing so that was a reason and assuming eventlet patching is 
actually working then all the existing stdlib stuff should keep on 
working under eventlet (including concurrent.futures); otherwise 
eventlet.monkey_patch isn't working and that's breaking the eventlet 
api. If their contract is that only certain things work when monkey 
patched, that's fair, but that needs to be documented somewhere 
(honestly it's time imho to get the hell off eventlet everywhere but 
that likely requires rewrites of a lot of things, oops...).


-Josh

Corey Bryant wrote:

Hi All,

I'm trying to add Py3 packaging support for Ubuntu Rocky and while there
are a lot of issues involved with supporting Py3.7, this is one of the
big ones that I could use a hand with.

With py3.7, there's a deadlock when eventlet monkeypatch of stdlib
thread modules is combined with use of ThreadPoolExecutor. I know this
affects at least designate. The same or similar also affects heat
(though I've not dug into the code the traceback after canceling tests
matches that seen with designate). And it may affect other projects that
I haven't touched yet.

How to recreate [1]:
* designate: Add a tox.ini py37 target and run
designate.tests.test_workers.test_processing.TestProcessingExecutor.test_execute_multiple_tasks
* heat: Add a tox.ini py37 target and run tests
* general: Run bpo34173-recreate.py
 from issue
34173 (see below).
[1] ubuntu cosmic has py3.7

In issue 508 (see below) @tomoto asks "Eventlet and asyncio solve same
problem. Why would you want concurrent.futures and eventlet in same
application?"

I told @tomoto that I'd seek input to that question from upstream. I
know there've been efforts to move away from eventlet but I just don't
have the knowledge to  provide a good answer to him.

Here are the bugs/issues I currently have open for this:
https://github.com/eventlet/eventlet/issues/508

https://bugs.launchpad.net/designate/+bug/1782647

https://bugs.python.org/issue34173 

Any help with this would be greatly appreciated!

Thanks,
Corey

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] [designate] [heat] [python3] deadlock with eventlet and ThreadPoolExecutor in py3.7

[Corey Bryant]

Hi All,

I'm trying to add Py3 packaging support for Ubuntu Rocky and while there
are a lot of issues involved with supporting Py3.7, this is one of the big
ones that I could use a hand with.

With py3.7, there's a deadlock when eventlet monkeypatch of stdlib thread
modules is combined with use of ThreadPoolExecutor. I know this affects at
least designate. The same or similar also affects heat (though I've not dug
into the code the traceback after canceling tests matches that seen with
designate). And it may affect other projects that I haven't touched yet.

How to recreate [1]:
* designate: Add a tox.ini py37 target and run designate.tests.test_workers.
test_processing.TestProcessingExecutor.test_execute_multiple_tasks
* heat: Add a tox.ini py37 target and run tests
* general: Run bpo34173-recreate.py
 from issue 34173
(see below).
[1] ubuntu cosmic has py3.7

In issue 508 (see below) @tomoto asks "Eventlet and asyncio solve same
problem. Why would you want concurrent.futures and eventlet in same
application?"

I told @tomoto that I'd seek input to that question from upstream. I know
there've been efforts to move away from eventlet but I just don't have the
knowledge to  provide a good answer to him.

Here are the bugs/issues I currently have open for this:
https://github.com/eventlet/eventlet/issues/508
https://bugs.launchpad.net/designate/+bug/1782647
https://bugs.python.org/issue34173

Any help with this would be greatly appreciated!

Thanks,
Corey
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev