Re: [Python-Dev] cffi in stdlib

2013-12-19 Thread Maciej Fijalkowski 
On Thu, Dec 19, 2013 at 3:17 AM, Gregory P. Smith  wrote:
>
>
>
> On Tue, Dec 17, 2013 at 8:43 AM, Stefan Krah  wrote:
>>
>> Maciej Fijalkowski  wrote:
>> > I would like to discuss on the language summit a potential inclusion
>> > of cffi[1] into stdlib. This is a project Armin Rigo has been working
>> > for a while, with some input from other developers.
>>
>> I've tried cffi (admittedly only in a toy script) and find it very nice
>> to use.
>>
>> Here's a comparison (pi benchmark) between wrapping libmpdec using a
>> C-extension (_decimal), cffi and ctypes:
>>
>>
>> +---+--+--+-+
>> |   | _decimal |  ctypes  |   cffi  |
>> +===+==+==+=+
>> | cpython-tip (with-system-ffi) |   0.19s  |   5.40s  |  5.14s  |
>> +---+--+--+-+
>> | cpython-2.7 (with-system-ffi) |n/a   |   4.46s  |  5.18s  |
>> +---+--+--+-+
>> |  Ubuntu-cpython-2.7   |n/a   |   3.63s  |-|
>> +---+--+--+-+
>> |  pypy-2.2.1-linux64   |n/a   |  125.9s  |  0.94s  |
>> +---+--+--+-+
>> | pypy3-2.1-beta1-linux64   |n/a   |  264.9s  |  2.93s  |
>> +---+--+--+-+
>>
>>
>> I guess the key points are that C-extensions are hard to beat and that
>> cffi performance on pypy-2 is outstanding. Additionally it's worth noting
>> that Ubuntu does something in their Python build that we should do, too.
>
>
> Ubuntu compiles their Python with FDO (feedback directed optimization /
> profile guided optimization) enabled. All distros should do this if they
> don't already. It's generally 20% interpreter speedup. Our makefile already
> supports it but it isn't the default build as it takes a long time given
> that it needs to compile everything twice and do a profiled benchmark run
> between compilations.
>
> -gps

Hey Greg.

We found out that this only speedups benchmarks that you tried during
profiling and not others, so we disabled it for the default pypy
build. Can you provide me with some more detailed study on how it
speeds up interpreters in general and CPython in particular?

Cheers,
fijal
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] thread issues when embedding Python

2013-12-19 Thread Nick Coghlan 
On 19 December 2013 07:58, Daniel Pocock  wrote:
>
>
> On 18/12/13 16:29, Victor Stinner wrote:
>> 2013/12/18 Antoine Pitrou :
>>> You only need to call PyEval_InitThreads() once in the main Python
>>> thread.
>>
>> This is not well documented. For your information, PyGILState_Ensure()
>> now calls PyEval_InitThreads() in Python 3.4, see:
>> http://bugs.python.org/issue19576
>
>
> I did see that - but from my own experience, I do not believe it is
> calling PyThreadState_New(..) and it is not even checking if
> PyThreadState_New(..) has ever been called for the active thread
>
> Consequently, the thread is blocked or there is a seg fault
>
> I've now written up a much more thorough overview of my experience on my
> blog:
>
> http://danielpocock.com/embedding-python-multi-threaded-cpp

You absolutely should *NOT* have to call PyThreadState_New before
calling PyGILState_Ensure, as it is designed to call it for you (see
http://hg.python.org/cpython/file/2.7/Python/pystate.c#l598). If
calling PyThreadState_New works, but calling PyGILState_Ensure does
not, then something else is broken (such as not initialising the
thread local storage for the GIL state APIs).

I don't see anything in your article about how you ensure that the
main thread of the application *before anything else related to the
embedded Python happens* calls both Py_Initialize() and
PyEval_InitThreads(). If you don't do that, then all bets are off in
terms of multithreading support.

Regards,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] thread issues when embedding Python

2013-12-19 Thread Daniel Pocock 
On 19/12/13 12:22, Nick Coghlan wrote:
> On 19 December 2013 07:58, Daniel Pocock  wrote:
>>
>> On 18/12/13 16:29, Victor Stinner wrote:
>>> 2013/12/18 Antoine Pitrou :
 You only need to call PyEval_InitThreads() once in the main Python
 thread.
>>> This is not well documented. For your information, PyGILState_Ensure()
>>> now calls PyEval_InitThreads() in Python 3.4, see:
>>> http://bugs.python.org/issue19576
>>
>> I did see that - but from my own experience, I do not believe it is
>> calling PyThreadState_New(..) and it is not even checking if
>> PyThreadState_New(..) has ever been called for the active thread
>>
>> Consequently, the thread is blocked or there is a seg fault
>>
>> I've now written up a much more thorough overview of my experience on my
>> blog:
>>
>> http://danielpocock.com/embedding-python-multi-threaded-cpp
> You absolutely should *NOT* have to call PyThreadState_New before
> calling PyGILState_Ensure, as it is designed to call it for you (see
> http://hg.python.org/cpython/file/2.7/Python/pystate.c#l598). If
> calling PyThreadState_New works, but calling PyGILState_Ensure does
> not, then something else is broken (such as not initialising the
> thread local storage for the GIL state APIs).
>
> I don't see anything in your article about how you ensure that the
> main thread of the application *before anything else related to the
> embedded Python happens* calls both Py_Initialize() and
> PyEval_InitThreads(). If you don't do that, then all bets are off in
> terms of multithreading support.

I definitely do both of those things in the method PyRoutePlugin::init(..)

It is in PyRoutePlugin.cxx:

http://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/PyRoutePlugin.cxx?view=markup#l88


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] thread issues when embedding Python

2013-12-19 Thread Nick Coghlan 
On 19 December 2013 21:28, Daniel Pocock  wrote:
> On 19/12/13 12:22, Nick Coghlan wrote:
>> I don't see anything in your article about how you ensure that the
>> main thread of the application *before anything else related to the
>> embedded Python happens* calls both Py_Initialize() and
>> PyEval_InitThreads(). If you don't do that, then all bets are off in
>> terms of multithreading support.
>
> I definitely do both of those things in the method PyRoutePlugin::init(..)
>
> It is in PyRoutePlugin.cxx:
>
> http://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/PyRoutePlugin.cxx?view=markup#l88

I can't see an immediately obvious explanation for why your current
approach based on PyExternalUser::Use gets things working, while the
PyThreadSupport approach fails immediately. However, you'd need to be
able to reproduce the problem with a much simpler embedding
application and without PyCXX involved anywhere to confirm it as a
possible CPython bug, or to identify exactly what is missing from the
current embedding initialisation instructions.

The reason for that is the fact that the GIL state API is unit tested
on a wide variety of platforms inside a fully initialised interpreter
and that means we know this *does* work in the absence of external
interference: 
http://hg.python.org/cpython/file/16bfddf5a091/Modules/_testcapimodule.c#l1335

I also asked Graham Dumpleton (author of mod_wsgi, one of the more
complex CPython embedding scenarios currently in existence) to take a
look, and he didn't see any obvious explanation for the discrepancy
either, so you may want to try a cut down implementation without PyCXX
to see if that's the culprit.

Regards,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-12-19 Thread Stefan Krah 
Gregory P. Smith  wrote:
> Ubuntu compiles their Python with FDO (feedback directed optimization / 
> profile
> guided optimization) enabled. All distros should do this if they don't 
> already.
> It's generally 20% interpreter speedup. Our makefile already supports it but 
> it
> isn't the default build as it takes a long time given that it needs to compile
> everything twice and do a profiled benchmark run between compilations.

Yes, I didn't know we already had `make profile-opt`.  With that option
the self-compiled results are nearly the same as with the Ubuntu version,
the remaining difference might be due to Ubuntu's use of -flto, as
Matthias suggests in http://bugs.python.org/issue17781 .


Stefan Krah


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

2013-12-19 Thread Gregory P. Smith 
On Thu, Dec 19, 2013 at 2:07 AM, Maciej Fijalkowski wrote:

> On Thu, Dec 19, 2013 at 3:17 AM, Gregory P. Smith  wrote:
> >
> >
> >
> > On Tue, Dec 17, 2013 at 8:43 AM, Stefan Krah 
> wrote:
> >>
> >> Maciej Fijalkowski  wrote:
> >> > I would like to discuss on the language summit a potential inclusion
> >> > of cffi[1] into stdlib. This is a project Armin Rigo has been working
> >> > for a while, with some input from other developers.
> >>
> >> I've tried cffi (admittedly only in a toy script) and find it very nice
> >> to use.
> >>
> >> Here's a comparison (pi benchmark) between wrapping libmpdec using a
> >> C-extension (_decimal), cffi and ctypes:
> >>
> >>
> >> +---+--+--+-+
> >> |   | _decimal |  ctypes  |   cffi  |
> >> +===+==+==+=+
> >> | cpython-tip (with-system-ffi) |   0.19s  |   5.40s  |  5.14s  |
> >> +---+--+--+-+
> >> | cpython-2.7 (with-system-ffi) |n/a   |   4.46s  |  5.18s  |
> >> +---+--+--+-+
> >> |  Ubuntu-cpython-2.7   |n/a   |   3.63s  |-|
> >> +---+--+--+-+
> >> |  pypy-2.2.1-linux64   |n/a   |  125.9s  |  0.94s  |
> >> +---+--+--+-+
> >> | pypy3-2.1-beta1-linux64   |n/a   |  264.9s  |  2.93s  |
> >> +---+--+--+-+
> >>
> >>
> >> I guess the key points are that C-extensions are hard to beat and that
> >> cffi performance on pypy-2 is outstanding. Additionally it's worth
> noting
> >> that Ubuntu does something in their Python build that we should do, too.
> >
> >
> > Ubuntu compiles their Python with FDO (feedback directed optimization /
> > profile guided optimization) enabled. All distros should do this if they
> > don't already. It's generally 20% interpreter speedup. Our makefile
> already
> > supports it but it isn't the default build as it takes a long time given
> > that it needs to compile everything twice and do a profiled benchmark run
> > between compilations.
> >
> > -gps
>
> Hey Greg.
>
> We found out that this only speedups benchmarks that you tried during
> profiling and not others, so we disabled it for the default pypy
> build. Can you provide me with some more detailed study on how it
> speeds up interpreters in general and CPython in particular?
>

That's a common concern for profile based builds but it turns out not to
matter a whole lot which workloads you choose for the CPython interpreter
to collect profiles for a FDO build. I believe ubuntu's packages just use
the test suite. In our own tests at work this produced good results.
Interpreter loops and other common code paths in the interpreter have a
*lot* of low hanging fruit in terms of more optimal code generation.

Link time optimization adds additional benefits IF you can get it working
(not always easy or reliable right now as Matthias mentions in issue17781).

-gps
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How long the wrong type of argument should we limit (or not) in the error message (C-api)?

2013-12-19 Thread Eric V. Smith 
On 12/16/2013 03:49 PM, Nick Coghlan wrote:
> 
> On 17 Dec 2013 02:23, "Eric V. Smith"  > wrote:
>>
>> On 12/16/2013 10:29 AM, Walter Dörwald wrote:
>> > I'd vote for including the module name in the string and using
>> > __qualname__ instead of __name__, i.e. make "{:T}".format(obj)
>> > equivalent to
>> > "{0.__class__.__module__}.{0.__class__.qualname__}".format(obj).
>>
>> That's not possible in general. The format specifier interpretation is
>> done by each type. So, you could add this to str.__format__ and
>> int.__format__, but you can't add it to an arbitrary type's __format__.
>> For example, types not in the stdlib would never know about it.
> 
> That just suggests it would need to be a type coercion code, like !a,
> !r, and !s. However, that observation also suggests that starting with a
> "classname" or "typename" builtin would be more appropriate than jumping
> directly to a formatting code.

That's an excellent observation, Nick, including that it should be based
on a builtin. But I'd suggest something like classof(), and have it's
__format__ "do the right thing". But it all seems like overkill for this
problem.

> We've definitely drifted well into python-ideas territory at this point,
> though :)


True enough!

Eric.

> Cheers,
> Nick.
> 
>>
>> There's no logic for calling through to object.__format__ for unknown
>> specifiers. Look at datetime, for example. It uses strftime, so "T"
>> currently just prints a literal "T".
>>
>> And for object.__format__, we recently made it an error to specify any
>> format string. This is to prevent you from calling
>> format(an_object, ".30")
>> and "knowning" that it's interpreted by str.__format__ (because that's
>> the default conversion for object.__format__). If in the future
>> an_object's class added its own __format__, this code would break (or at
>> least do the wrong thing).
>>
>> But I really do like the idea! Maybe there's a way to just make
>> obj.__class__ recognize "T", so you could at least do:
>> format(obj.__class__, "T")
>> or equivalently:
>> "{:T}".format(obj.__class__)
>>
>> I realize that having to use .__class__ defeats some of the beauty of
>> this scheme.
>>
>> Eric.
>>
>>
>> ___
>> Python-Dev mailing list
>> [email protected] 
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
> 
> 
> 
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com
> 

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com