Re: [Python-Dev] Use of Cython

[Yury Selivanov]

On Tue, Sep 4, 2018 at 2:58 PM Stefan Behnel  wrote:
[..]
> Cython has four ways to provide type declarations: cdef statements in
> Cython code, external .pxd files for Python or Cython files, special
> decorators and declaration functions, and PEP-484/526 type annotations.

Great to hear that PEP 484 type annotations are supported.  Here's a
link to the docs:
https://cython.readthedocs.io/en/latest/src/tutorial/pure.html#static-typing

[..]
> > I know that Cython has a mode to use decorators in
> > pure Python code to annotate types, but they are less intuitive than
> > using typing annotations in 3.6+.
>
> You can use PEP-484/526 type annotations to declare Cython types in Python
> code that you intend to compile. It's entirely up to you, and it's an
> entirely subjective measure which "is better". Many people prefer Cython's
> non-Python syntax because it allows them to apply their existing C
> knowledge. For them, PEP-484 annotations may easily be non-intuitive in
> comparison.

Yeah, but if we decide to use Cython in CPython we probably need to
come up with something like PEP 7 to recommend one particular style
and have an overall guideline.  Using PEP 484 annotations means that
we have pure Python code that PyPy and other interpreters can still
run.

[..]
> > I'd be +0.5 on using Cython (optionally?) to compile some pure Python
> > code to make it 30-50% faster.  asyncio, for instance, would certainly
> > benefit from that.
>
> Since most of this (stdlib) Python code doesn't need to stay syntax
> compatible with Python < 3.6 (actually 3.8) anymore, you can probably get
> much higher speedups than that by statically typing some variables and
> functions here and there. I recently tried that with difflib, makes a big
> difference.

I'd be willing to try this in asyncio if we start using Cython.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Ethan Furman]

On 09/04/2018 11:55 AM, Stefan Behnel wrote:


Adding the right language would lower the bar, IMHO. Cython is Python. It
allows users with a Python background to implement C things without having
to thoroughly learn C/and/  the CPython C-API first. So, the way I see it,
rather than/adding/  a "third" language to the mix, it substantially lowers
the entry level from the current two and a half languages (Python + C +
C-API) to one and a half (Python + Cython).


As somebody who only has light exposure to C, I would very much like to have 
Cython be an option.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Stefan Behnel]

Yury Selivanov schrieb am 04.09.2018 um 18:19:
> On Sat, Sep 1, 2018 at 6:12 PM Stefan Behnel wrote:
>> Yury Selivanov schrieb am 07.08.2018 um 19:34:
>>> The first goal is to compile mypy with it to make it faster, so I hope
>>> that the project will be completed.
>>
>> That's not "the first goal". It's the /only/ goal. The only intention of
>> mypyc is to be able to compile and optimise enough of Python to speed up
>> the kind or style of code that mypy uses.
>>
>>> Essentially, mypyc will be similar
>>> to Cython, but mypyc is a *subset of Python*, not a superset.
>>
>> Which is bad, right? It means that there will be many things that simply
>> don't work, and that you need to change your code in order to make it
>> compile at all. Cython is way beyond that point by now. Even RPython will
>> probably continue to be way better than mypyc for quite a while, maybe
>> forever, who knows.
> 
> To be clear I'm not involved with mypyc, but my understanding is that
> the entire Python syntax will be supported, except some dynamic
> features like patching `globals()`, `locals()`, or classes, or
> __class__.

No, that's not the goal, at least from what I understood from my
discussions with Jukka. The goal is to make it compile mypy, be it by
supporting Python features in mypyc or by avoiding Python features in mypy.
I'm sure they will take any shortcut they can in order to avoid having to
make mypyc too capable, because mypyc is not more than a means to an end.
For example, they may easily get away without supporting generators and
closures, which are quite difficult to implement in C. But finding a
non-trivial piece of Python code out there that uses neither of the two is
probably not easy.

I'm also sure they will avoid Python semantics wherever they can, because
implementing them in the same way as CPython and Cython would mean that
certain constructs cannot safely be statically reasoned about, and thus
cannot be optimised. Avoiding (full) Python semantics relieves you from
these restrictions, and if you control both sides, the compiler and the
code that it compiles, then it becomes much easier to apply arbitrary
optimisations at will.

IMHO, what they are implementing is much closer to ShedSkin than to Cython.


>>> Interfacing with C libraries can be easily achieved with cffi.
>>
>> Except that it will be fairly slow. cffi is not designed for static
>> analysis but for runtime operations.
> 
> Could you please clarify this point?  My current understanding is that
> you can build a static compiler with a knowledge about cffi so that it
> can compile calls like `ffi.new("something_t[]", 80)` to pure C.

I'm sure there is a relatively large subset of cffi's API that could be
compiled statically, as long as the declartions and their usage are kept
simple and fully visible to the compiler. What that subset is remains to be
seen once someone actually tries to do it.


> Yeah, statically compiling cffi-enabled code is probably the way to go
> for mypyc and Cython.

I doubt it, given the expected restrictions and verbosity. But debating
this is useless as long as no-one attempts to actually write a static
compiler for cffi(-like) code.


> Using Cython/C types usually means
> that you need to use pxd/pyx files which means that the code isn't
> Python anymore.

I'm aware that this is a very common misconception that is difficult to get
out of people's heads. You probably got this idea from wrapping a native
library, in which case the only choice you have in order to declare an
external C-API is really to use Cython's special syntax. However, this
would not apply to most use cases in the CPython project context, and it
also does not necessarily apply to most of the code in a Cython module even
if it uses external libraries.

Cython has four ways to provide type declarations: cdef statements in
Cython code, external .pxd files for Python or Cython files, special
decorators and declaration functions, and PEP-484/526 type annotations.

All four have their use cases (e.g. syntax support in different Python
versions, efficiency of expression, readability for people with different
backgrounds, etc.), and all but the first allow users to keep their module
code in Python syntax. As long as you do not call into external native
code, it's your choice which of these you prefer for your code base,
project context and developer background. You can even mix them at will, if
you feel like it.


> I know that Cython has a mode to use decorators in
> pure Python code to annotate types, but they are less intuitive than
> using typing annotations in 3.6+.

You can use PEP-484/526 type annotations to declare Cython types in Python
code that you intend to compile. It's entirely up to you, and it's an
entirely subjective measure which "is better". Many people prefer Cython's
non-Python syntax because it allows them to apply their existing C
knowledge. For them, PEP-484 annotations may easily be non-intuitive in
comparison.


> For CPython 

Re: [Python-Dev] Use of Cython

[Yury Selivanov]

Hi Stefan,

On Sat, Sep 1, 2018 at 6:12 PM Stefan Behnel  wrote:
>
> Yury,
>
> given that people are starting to quote enthusiastically the comments you
> made below, let me set a couple of things straight.

To everyone reading this thread please keep in mind that I'm not in
position to "defend" mypyc or to "promote" it, and I'm not affiliated
with the project at all.  I am just excited about yet another tool to
statically compile Python and I'm discussing it only from a
theoretical standpoint.

>
> Yury Selivanov schrieb am 07.08.2018 um 19:34:
> > On Mon, Aug 6, 2018 at 11:49 AM Ronald Oussoren via Python-Dev wrote:
> >
> >> I have no strong opinion on using Cython for tests or in the stdlib, other 
> >> than that it is a fairly large dependency.  I do think that adding a 
> >> “Cython-lite” tool the CPython distribution would be less ideal, creating 
> >> and maintaining that tool would be a lot of work without clear benefits 
> >> over just using Cython.
> >
> > Speaking of which, Dropbox is working on a new compiler they call "mypyc".
> >
> > mypyc will compile type-annotated Python code to an optimized C.
>
> That's their plan. Saying that "it will" is a bit premature at this point.
> The list of failed attempts at writing static Python compilers is rather
> long, even if you only count those that compile the usual "easy subset" of
> Python.
>
> I wish them the best of luck and endurance, but they have a long way to go.

I fully agree with you here.

>
>
> > The
> > first goal is to compile mypy with it to make it faster, so I hope
> > that the project will be completed.
>
> That's not "the first goal". It's the /only/ goal. The only intention of
> mypyc is to be able to compile and optimise enough of Python to speed up
> the kind or style of code that mypy uses.
>
>
> > Essentially, mypyc will be similar
> > to Cython, but mypyc is a *subset of Python*, not a superset.
>
> Which is bad, right? It means that there will be many things that simply
> don't work, and that you need to change your code in order to make it
> compile at all. Cython is way beyond that point by now. Even RPython will
> probably continue to be way better than mypyc for quite a while, maybe
> forever, who knows.

To be clear I'm not involved with mypyc, but my understanding is that
the entire Python syntax will be supported, except some dynamic
features like patching `globals()`, `locals()`, or classes, or
__class__.  IMO this is *good* and in general Python programs don't do
that anyways.

>
>
> > Interfacing with C libraries can be easily achieved with cffi.
>
> Except that it will be fairly slow. cffi is not designed for static
> analysis but for runtime operations.

Could you please clarify this point?  My current understanding is that
you can build a static compiler with a knowledge about cffi so that it
can compile calls like `ffi.new("something_t[]", 80)` to pure C.

> You can obviously also use cffi from
> Cython – but then, why would you, if you can get much faster code much more
> easily without using cffi?

The "much more easily" part is debatable here and is highly
subjective.  For me using Cython is also easier *at this point*
because I've spent so much time working with it. Although getting
there wasn't easy for me :(

>
> That being said, if someone wants to write a static cffi optimiser for
> Cython, why not, I'd be happy to help with my advice. The cool thing is
> that this can be improved gradually, because compiling the cffi code
> probably already works out of the box. It's just not (much) faster than
> when interpreted.

Yeah, statically compiling cffi-enabled code is probably the way to go
for mypyc and Cython.

>
>
> > Being a
> > strict subset of Python means that mypyc code will execute just fine
> > in PyPy.
>
> So does normal (non-subset) Python code. You can run it in PyPy, have
> CPython interpret it, or compile it with Cython if you want it to run
> faster in CPython, all without having to limit yourself to a subset of
> Python. Seriously, you make this sound like requiring users to rewrite
> their code to make it compilable with mypyc was a good thing.

But that's the point: unless you add Cython types to your Python code
it gets only moderate speedups.  Using Cython/C types usually means
that you need to use pxd/pyx files which means that the code isn't
Python anymore.  I know that Cython has a mode to use decorators in
pure Python code to annotate types, but they are less intuitive than
using typing annotations in 3.6+.

[..]
> > I'd be more willing to start using mypyc+cffi in CPython stdlib
> > *eventually*, than Cython now.  Cython is a relatively complex and
> > still poorly documented language.
>
> You are free to improve the documentation or otherwise help us find and
> discuss concrete problems with it.

Fair point.

> Calling Cython a "poorly documented
> language" could easily feel offensive towards those who have put a lot of
> work into the documentation, wiki, tutorials, trainings and

Re: [Python-Dev] Use of Cython

[Stefan Behnel]

Yury,

given that people are starting to quote enthusiastically the comments you
made below, let me set a couple of things straight.

Yury Selivanov schrieb am 07.08.2018 um 19:34:
> On Mon, Aug 6, 2018 at 11:49 AM Ronald Oussoren via Python-Dev wrote:
> 
>> I have no strong opinion on using Cython for tests or in the stdlib, other 
>> than that it is a fairly large dependency.  I do think that adding a 
>> “Cython-lite” tool the CPython distribution would be less ideal, creating 
>> and maintaining that tool would be a lot of work without clear benefits over 
>> just using Cython.
> 
> Speaking of which, Dropbox is working on a new compiler they call "mypyc".
> 
> mypyc will compile type-annotated Python code to an optimized C.

That's their plan. Saying that "it will" is a bit premature at this point.
The list of failed attempts at writing static Python compilers is rather
long, even if you only count those that compile the usual "easy subset" of
Python.

I wish them the best of luck and endurance, but they have a long way to go.


> The
> first goal is to compile mypy with it to make it faster, so I hope
> that the project will be completed.

That's not "the first goal". It's the /only/ goal. The only intention of
mypyc is to be able to compile and optimise enough of Python to speed up
the kind or style of code that mypy uses.


> Essentially, mypyc will be similar
> to Cython, but mypyc is a *subset of Python*, not a superset.

Which is bad, right? It means that there will be many things that simply
don't work, and that you need to change your code in order to make it
compile at all. Cython is way beyond that point by now. Even RPython will
probably continue to be way better than mypyc for quite a while, maybe
forever, who knows.


> Interfacing with C libraries can be easily achieved with cffi.

Except that it will be fairly slow. cffi is not designed for static
analysis but for runtime operations. You can obviously also use cffi from
Cython – but then, why would you, if you can get much faster code much more
easily without using cffi?

That being said, if someone wants to write a static cffi optimiser for
Cython, why not, I'd be happy to help with my advice. The cool thing is
that this can be improved gradually, because compiling the cffi code
probably already works out of the box. It's just not (much) faster than
when interpreted.


> Being a
> strict subset of Python means that mypyc code will execute just fine
> in PyPy.

So does normal (non-subset) Python code. You can run it in PyPy, have
CPython interpret it, or compile it with Cython if you want it to run
faster in CPython, all without having to limit yourself to a subset of
Python. Seriously, you make this sound like requiring users to rewrite
their code to make it compilable with mypyc was a good thing.


> They can even apply some optimizations to it eventually, as
> it has a strict and static type system.

In case "they" refers to PyPy here, then I remember the PyPy project
stating very clearly that they are not interested in PEP-484 typing because
it is completely irrelevant for their JIT. It's really best for them to
ignore it.

That's similar for Cython, simply because PEP-484 typing isn't designed for
optimisation at all, definitely not for C-level optimisation. Still, Cython
can make some use of PEP-484 typing, if you use it to define specific C
types. That allows normal execution in CPython, static analysis with
PEP-484 analyser tools (e.g. PyCharm or mypy), and efficient optimisation
by Cython. The best of all worlds. See the docs on how to do that, it's
been supported for about a year now (and has been around in a similar,
non-PEP-484 form for years before that PEP even existed).


> I'd be more willing to start using mypyc+cffi in CPython stdlib
> *eventually*, than Cython now.  Cython is a relatively complex and
> still poorly documented language.

You are free to improve the documentation or otherwise help us find and
discuss concrete problems with it. Calling Cython a "poorly documented
language" could easily feel offensive towards those who have put a lot of
work into the documentation, wiki, tutorials, trainings and what not that
help people use the language. Even stack overflow is getting better and
better in documenting Cython these days, even though responses over there
that describe work-arounds tend to get outdated fairly quickly.

Besides, don't forget that it's Python, so consider reading the Python
documentation first if something is unclear. And maybe some documentation
of C data types as well. (.5 wink)


> I'm speaking from experience after
> writing thousands of lines of Cython in uvloop & asyncpg.  In skillful
> hands Cython is amazing, but I'd be cautious to advertise and use it
> in CPython.

Why not? You didn't actually give any reasons for that.


> I'm also -1 on using Cython to test C API. While writing C tests is
> annoying (I wrote a fair share myself), their very purpose is to make
> third-party tools/ex

Re: [Python-Dev] Use of Cython

[Abdur-Rahmaan Janhangeer]

since when they started working on it?(mypyc)

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ
Mauritius
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Stefan Behnel]

Hi,

this is getting a bit out of scope for this list. I propose to move further
questions about general Cython usage to he cython-users mailing list.

Matěj Cepl schrieb am 08.08.2018 um 12:44:
> On 2018-08-06, 15:13 GMT, Stefan Behnel wrote:
>> Not sure I understand this correctly, but I think we're on the 
>> same page here: writing test code in C is cumbersome, writing 
>> test code in a mix of C and Python across different files is 
>> aweful. And making it difficult to write or even just review 
>> test code simply means that people will either waste their 
>> precious contribution time on it, or try to get around it.
> 
> I was thinking about the same when porting M2Crypto to py3k 
> (M2Crypto is currently swig-based mix of C-code and Python). Is 
> it even possible to make a mix of Cython, swig-based C, and 
> Python?

As long as you take the decision at a per-module basis, sure. If you want
to mix them inside of a single module, then it's either Cython+C or Swig+C,
not all three. But as Antoine suggested, unless you really want an
identical mapper for whole range of different languages, Swig is likely not
what you should use these days.


> In the end I rather stayed with plain C, because the 
> combination seems unimaginably awful.

Probably worth expanding your imagination. :)

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Matěj Cepl]

On 2018-08-08, 11:30 GMT, Antoine Pitrou wrote:
> I'm not sure why anyone would want to use swig nowadays.

Legacy reasons, 47k lines of Python code (and 7k lines of swig 
*.i files).

Matěj
-- 
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
 
Do not long for the night, when people vanish in their place.
Be careful, do not turn to evil; for you have preferred this to
affliction.
  -- Job 36:20f (NASB)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Antoine Pitrou]

On Wed, 08 Aug 2018 12:44:50 +0200
Matěj Cepl  wrote:
> On 2018-08-06, 15:13 GMT, Stefan Behnel wrote:
> > Not sure I understand this correctly, but I think we're on the 
> > same page here: writing test code in C is cumbersome, writing 
> > test code in a mix of C and Python across different files is 
> > aweful. And making it difficult to write or even just review 
> > test code simply means that people will either waste their 
> > precious contribution time on it, or try to get around it.  
> 
> I was thinking about the same when porting M2Crypto to py3k 
> (M2Crypto is currently swig-based mix of C-code and Python). Is 
> it even possible to make a mix of Cython, swig-based C, and 
> Python? In the end I rather stayed with plain C, because the 
> combination seems unimaginably awful.

I'm not sure why anyone would want to use swig nowadays.

> (Also, is Cython the best of all of them? What about cffi or 
> Nuitka?)

This sounds like comparing apples to oranges.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Matěj Cepl]

On 2018-08-06, 15:13 GMT, Stefan Behnel wrote:
> Not sure I understand this correctly, but I think we're on the 
> same page here: writing test code in C is cumbersome, writing 
> test code in a mix of C and Python across different files is 
> aweful. And making it difficult to write or even just review 
> test code simply means that people will either waste their 
> precious contribution time on it, or try to get around it.

I was thinking about the same when porting M2Crypto to py3k 
(M2Crypto is currently swig-based mix of C-code and Python). Is 
it even possible to make a mix of Cython, swig-based C, and 
Python? In the end I rather stayed with plain C, because the 
combination seems unimaginably awful.

(Also, is Cython the best of all of them? What about cffi or 
Nuitka?)

Best,

Matěj
-- 
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
 
There is no reason to suppose that most human beings are engaged
in maximizing anything unless it be unhappiness, and even this
with incomplete success.
-- Ronald Coase
   Introduction to “The Firm, the Market, and the Law”

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Matěj Cepl]

On 2018-08-07, 17:34 GMT, Yury Selivanov wrote:
> Speaking of which, Dropbox is working on a new compiler they 
> call "mypyc".

How does it compare with Nuitka?

Matěj
-- 
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
 
If in desperation, read the documentation!
-- Brian D. Ripley, on R-help list

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Yury Selivanov]

On Mon, Aug 6, 2018 at 11:49 AM Ronald Oussoren via Python-Dev
 wrote:

> I have no strong opinion on using Cython for tests or in the stdlib, other 
> than that it is a fairly large dependency.  I do think that adding a 
> “Cython-lite” tool the CPython distribution would be less ideal, creating and 
> maintaining that tool would be a lot of work without clear benefits over just 
> using Cython.

Speaking of which, Dropbox is working on a new compiler they call "mypyc".

mypyc will compile type-annotated Python code to an optimized C. The
first goal is to compile mypy with it to make it faster, so I hope
that the project will be completed. Essentially, mypyc will be similar
to Cython, but mypyc is a *subset of Python*, not a superset.
Interfacing with C libraries can be easily achieved with cffi. Being a
strict subset of Python means that mypyc code will execute just fine
in PyPy. They can even apply some optimizations to it eventually, as
it has a strict and static type system.

I'd be more willing to start using mypyc+cffi in CPython stdlib
*eventually*, than Cython now.  Cython is a relatively complex and
still poorly documented language.  I'm speaking from experience after
writing thousands of lines of Cython in uvloop & asyncpg.  In skillful
hands Cython is amazing, but I'd be cautious to advertise and use it
in CPython.

I'm also -1 on using Cython to test C API. While writing C tests is
annoying (I wrote a fair share myself), their very purpose is to make
third-party tools/extensions more stable. Using a third-party tool to
test C API to track regressions that break third-party tools feels
wrong.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Ronald Oussoren via Python-Dev]

> On 6 Aug 2018, at 17:13, Stefan Behnel  wrote:
> 
> Ronald Oussoren via Python-Dev schrieb am 06.08.2018 um 15:25:
>>> On 5 Aug 2018, at 18:14, Nick Coghlan wrote:
>>> On 5 August 2018 at 18:06, Ronald Oussoren wrote:
 I’m not sure if I understand this, ctypes and cffi are used to access C 
 APIs
 without writing C code including the CPython API (see for example
 ).
 
 The code code below should be mostly equivalent to the Cython example 
 posted
 earlier:
 
 import unittest
 import ctypes
 from ctypes import pythonapi
 
 class PyObject(ctypes.Structure):
   _fields_ = (
   ('ob_refcnt', ctypes.c_ssize_t),
   )
 
 pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
 
 def refcount(v):
   return PyObject.from_address(id(v)).ob_refcnt
>>> 
>>> The quoted code is what I was referring to in:
>>> 
>>> ctypes & cffi likely wouldn't help as much in the case, since they
>>> don't eliminate the need to come up with custom code for parts 3 & 4,
>>> they just let you write that logic in Python rather than C.
>>> 
>> 
>> And earlier Nick wrote:
>>> 1. The test case itself (what action to take, which assertions to make 
>>> about it)
>>> 2. The C code to make the API call you want to test
>>> 3. The Python->C interface for the test case from 1 to pass test
>>> values in to the code from 2
>>> 4. The C->Python interface to get state of interest from 2 back to the
>>> test case from 1
>> 
>> For all of Cython, ctypes and cffi you almost never have to write (2), and 
>> hence (3) and (4), but can write that code in Python.
> 
> Which then means that you have a mix of Python and C in many cases.

Not really, for many cases the only C code required is the actual CPython API. 

> I guess
> that's what you meant with your next sentence:
> 
>> This is at the code of making it harder to know which bits of the CPython 
>> API are used in step (2), which makes it harder to review a testcase. 

Not really. What I was trying to say is that when you use C code to make the 
API call and collect information you have complete control of which APIs are 
used (in the PyList_Append test case you can be 100% sure that the test code 
only calls that API), with ctypes/cffi/Cython you have less control over the 
exact API calls made which could affect the test. Making assertions about 
refcounts is one example, my test case with ctypes works but you have to think 
about what the code does before you understand why the test case works and 
tests what you want to test.  In general that’s not a good quality for test 
code, that should be as obviously correct as possible. 

Cython is probably the best option in this regard (from what I’ve seen so far) 
because you can in the end basically write C code with Python syntax where 
needed.

> 
> Not sure I understand this correctly, but I think we're on the same page
> here: writing test code in C is cumbersome, writing test code in a mix of C
> and Python across different files is aweful. And making it difficult to
> write or even just review test code simply means that people will either
> waste their precious contribution time on it, or try to get around it.

Awful is a strong word. The separation of the test case in two different 
languages in two different source files is definitely less than ideal. 

> 
> 
>> BTW. In other projects I use tests there almost all of the test code is in 
>> C, the unittest runner only calls a C function and uses the result of that 
>> function to deduce if the test passed or failed. This only works nicely for 
>> fairly simple tests (such as the example test in this thread), not for more 
>> complicated and interesting tests due to having to write more C code.
> 
> I totally agree with that. For less trivial tests, people will often want
> to stear the test case at the C level, because some things are really
> difficult to do from Python. Good luck making assertions about reference
> counts when you're orchestrating the C-API through ctypes.

I agree, see above.


> And this is
> where Cython shines – your code *always* ends up running in C, regardless
> of how much of it is plain Python. But at any point, you can do pretty
> arbitrary C things, all in the same function. And unittest can execute that
> function directly for you, without having to write a Python wrapper or
> separate test runner.
> 
> And for the really hard cases, you can resort to writing a literal C code
> snippet in your Cython source file (as a string) and let Cython drop it
> into the file it generates, e.g. to quickly define a macro, a little C
> function, or an interface wrapper around a C macro that would otherwise be
> difficult to test. That little feature removes the last reason for
> resorting to a separate C file.

Another reason for liking the translation to C is that you can use and test th

Re: [Python-Dev] Use of Cython

[Stefan Behnel]

Ronald Oussoren via Python-Dev schrieb am 06.08.2018 um 15:25:
>> On 5 Aug 2018, at 18:14, Nick Coghlan wrote:
>> On 5 August 2018 at 18:06, Ronald Oussoren wrote:
>>> I’m not sure if I understand this, ctypes and cffi are used to access C APIs
>>> without writing C code including the CPython API (see for example
>>> ).
>>>
>>> The code code below should be mostly equivalent to the Cython example posted
>>> earlier:
>>>
>>> import unittest
>>> import ctypes
>>> from ctypes import pythonapi
>>>
>>> class PyObject(ctypes.Structure):
>>>_fields_ = (
>>>('ob_refcnt', ctypes.c_ssize_t),
>>>)
>>>
>>> pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
>>>
>>> def refcount(v):
>>>return PyObject.from_address(id(v)).ob_refcnt
>>
>> The quoted code is what I was referring to in:
>> 
>> ctypes & cffi likely wouldn't help as much in the case, since they
>> don't eliminate the need to come up with custom code for parts 3 & 4,
>> they just let you write that logic in Python rather than C.
>> 
> 
> And earlier Nick wrote:
>> 1. The test case itself (what action to take, which assertions to make about 
>> it)
>> 2. The C code to make the API call you want to test
>> 3. The Python->C interface for the test case from 1 to pass test
>> values in to the code from 2
>> 4. The C->Python interface to get state of interest from 2 back to the
>> test case from 1
> 
> For all of Cython, ctypes and cffi you almost never have to write (2), and 
> hence (3) and (4), but can write that code in Python.

Which then means that you have a mix of Python and C in many cases. I guess
that's what you meant with your next sentence:

> This is at the code of making it harder to know which bits of the CPython API 
> are used in step (2), which makes it harder to review a testcase. 

Not sure I understand this correctly, but I think we're on the same page
here: writing test code in C is cumbersome, writing test code in a mix of C
and Python across different files is aweful. And making it difficult to
write or even just review test code simply means that people will either
waste their precious contribution time on it, or try to get around it.


> BTW. In other projects I use tests there almost all of the test code is in C, 
> the unittest runner only calls a C function and uses the result of that 
> function to deduce if the test passed or failed. This only works nicely for 
> fairly simple tests (such as the example test in this thread), not for more 
> complicated and interesting tests due to having to write more C code.

I totally agree with that. For less trivial tests, people will often want
to stear the test case at the C level, because some things are really
difficult to do from Python. Good luck making assertions about reference
counts when you're orchestrating the C-API through ctypes. And this is
where Cython shines – your code *always* ends up running in C, regardless
of how much of it is plain Python. But at any point, you can do pretty
arbitrary C things, all in the same function. And unittest can execute that
function directly for you, without having to write a Python wrapper or
separate test runner.

And for the really hard cases, you can resort to writing a literal C code
snippet in your Cython source file (as a string) and let Cython drop it
into the file it generates, e.g. to quickly define a macro, a little C
function, or an interface wrapper around a C macro that would otherwise be
difficult to test. That little feature removes the last reason for
resorting to a separate C file.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Ronald Oussoren via Python-Dev]

> On 5 Aug 2018, at 18:14, Nick Coghlan  wrote:
> 
> On 5 August 2018 at 18:06, Ronald Oussoren  wrote:
>> I’m not sure if I understand this, ctypes and cffi are used to access C APIs
>> without writing C code including the CPython API (see for example
>> ).
>> 
>> The code code below should be mostly equivalent to the Cython example posted
>> earlier:
>> 
>> import unittest
>> import ctypes
>> from ctypes import pythonapi
>> 
>> class PyObject(ctypes.Structure):
>>_fields_ = (
>>('ob_refcnt', ctypes.c_ssize_t),
>>)
>> 
>> pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
>> 
>> def refcount(v):
>>return PyObject.from_address(id(v)).ob_refcnt
> 
> The quoted code is what I was referring to in:
> 
> ctypes & cffi likely wouldn't help as much in the case, since they
> don't eliminate the need to come up with custom code for parts 3 & 4,
> they just let you write that logic in Python rather than C.
> 

And earlier Nick wrote:
> 1. The test case itself (what action to take, which assertions to make about 
> it)
> 2. The C code to make the API call you want to test
> 3. The Python->C interface for the test case from 1 to pass test
> values in to the code from 2
> 4. The C->Python interface to get state of interest from 2 back to the
> test case from 1

For all of Cython, ctypes and cffi you almost never have to write (2), and 
hence (3) and (4), but can write that code in Python. This is at the code of 
making it harder to know which bits of the CPython API are used in step (2), 
which makes it harder to review a testcase. 

BTW. In other projects I use tests there almost all of the test code is in C, 
the unittest runner only calls a C function and uses the result of that 
function to deduce if the test passed or failed. This only works nicely for 
fairly simple tests (such as the example test in this thread), not for more 
complicated and interesting tests due to having to write more C code.

Ronald

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Nick Coghlan]

On 5 August 2018 at 18:06, Ronald Oussoren  wrote:
> I’m not sure if I understand this, ctypes and cffi are used to access C APIs
> without writing C code including the CPython API (see for example
> ).
>
> The code code below should be mostly equivalent to the Cython example posted
> earlier:
>
> import unittest
> import ctypes
> from ctypes import pythonapi
>
> class PyObject(ctypes.Structure):
> _fields_ = (
> ('ob_refcnt', ctypes.c_ssize_t),
> )
>
> pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
>
> def refcount(v):
> return PyObject.from_address(id(v)).ob_refcnt

The quoted code is what I was referring to in:

ctypes & cffi likely wouldn't help as much in the case, since they
don't eliminate the need to come up with custom code for parts 3 & 4,
they just let you write that logic in Python rather than C.


Cython has more machinery for accessing the CPython C API correctly
already built in to it, whereas ctypes has no type safety at all,
while cffi doesn't special case CPython's C API in particular.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Ronald Oussoren via Python-Dev]

> On 5 Aug 2018, at 03:15, Nick Coghlan  wrote:
> 
> On 5 August 2018 at 00:46, Stefan Behnel  wrote:
>> Antoine Pitrou schrieb am 04.08.2018 um 15:57:
>>> Actually, I think testing the C API is precisely the kind of area where
>>> you don't want to involve a third-party, especially not a moving target
>>> (Cython is actively maintained and generated code will vary after each
>>> new Cython release).  Besides, Cython itself calls the C API, which
>>> means you might end up involuntarily testing the C API against itself.
>>> 
>>> If anything, testing the C API using ctypes or cffi would probably be
>>> more reasonable... assuming we get ctypes / cffi to compile everywhere,
>>> which currently isn't the case.
>> 
>> I agree that you would rather not want to let Cython (or another tool)
>> generate the specific code that tests a specific C-API call, but you could
>> still use Cython to get around writing the setup, validation and unittest
>> boilerplate code in C. Basically, a test could then look something like
>> this (probably works, although I didn't test it):
>> 
>>from cpython.object cimport PyObject
>>from cpython.list cimport PyList_Append
>> 
>>def test_PyList_Append_on_empty_list():
>># setup code
>>l = []
>>assert len(l) == 0
>>value = "abc"
>>pyobj_value =  value
>>refcount_before = pyobj_value.ob_refcnt
>> 
>># conservative test call, translates to the expected C code,
>># although with exception propagation if it returns -1:
>>errcode = PyList_Append(l, value)
>> 
>># validation
>>assert errcode == 0
>>assert len(l) == 1
>>assert l[0] is value
>>assert pyobj_value.ob_refcnt == refcount_before + 1
>> 
>> 
>> If you don't want the exception handling, you can define your own
>> declaration of PyList_Append() that does not have it. But personally, I'd
>> rather use try-except in my test code than manually taking care of cleaning
>> up (unexpected) exceptions.
> 
> Exactly, that's the kind of thing I had in mind. At the moment,
> writing a new dedicated C API test requires designing 4 things:
> 
> 1. The test case itself (what action to take, which assertions to make about 
> it)
> 2. The C code to make the API call you want to test
> 3. The Python->C interface for the test case from 1 to pass test
> values in to the code from 2
> 4. The C->Python interface to get state of interest from 2 back to the
> test case from 1
> 
> If we were able to use Cython to handle 3 & 4 rather than having to
> hand craft it for every test, then I believe it would significantly
> lower the barrier to testing the C API directly rather than only
> testing it indirectly through the CPython implementation.
> 
> Having such a test suite available would then hopefully make it easier
> for other implementations to provide robust emulations of the public C
> API.
> 
> ctypes & cffi likely wouldn't help as much in the case, since they
> don't eliminate the need to come up with custom code for parts 3 & 4,
> they just let you write that logic in Python rather than C.

I’m not sure if I understand this, ctypes and cffi are used to access C APIs 
without writing C code including the CPython API (see for example 
>). 

The code code below should be mostly equivalent to the Cython example posted 
earlier:

import unittest
import ctypes
from ctypes import pythonapi

class PyObject(ctypes.Structure):
_fields_ = (
('ob_refcnt', ctypes.c_ssize_t),
)

pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]

def refcount(v):
return PyObject.from_address(id(v)).ob_refcnt


def test_PyList_Append_on_empty_list():
   # setup code
   l = []
   assert len(l) == 0
   value = "abc"


   refcount_before = refcount(value)

   errcode = pythonapi.PyList_Append(l, value)

   assert errcode == 0
   assert len(l) == 1
   assert l[0] is value
   assert refcount(value) == refcount_before + 1

I write “mostly” because I rarely use ctypes and am not 100% sure that I use 
the API correctly.

A problem with using ctypes is that this tests the ABI and to the API, which 
for example means you cannot test C macros this way.

Ronald

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Nick Coghlan]

On 5 August 2018 at 00:46, Stefan Behnel  wrote:
> Antoine Pitrou schrieb am 04.08.2018 um 15:57:
>> Actually, I think testing the C API is precisely the kind of area where
>> you don't want to involve a third-party, especially not a moving target
>> (Cython is actively maintained and generated code will vary after each
>> new Cython release).  Besides, Cython itself calls the C API, which
>> means you might end up involuntarily testing the C API against itself.
>>
>> If anything, testing the C API using ctypes or cffi would probably be
>> more reasonable... assuming we get ctypes / cffi to compile everywhere,
>> which currently isn't the case.
>
> I agree that you would rather not want to let Cython (or another tool)
> generate the specific code that tests a specific C-API call, but you could
> still use Cython to get around writing the setup, validation and unittest
> boilerplate code in C. Basically, a test could then look something like
> this (probably works, although I didn't test it):
>
> from cpython.object cimport PyObject
> from cpython.list cimport PyList_Append
>
> def test_PyList_Append_on_empty_list():
> # setup code
> l = []
> assert len(l) == 0
> value = "abc"
> pyobj_value =  value
> refcount_before = pyobj_value.ob_refcnt
>
> # conservative test call, translates to the expected C code,
> # although with exception propagation if it returns -1:
> errcode = PyList_Append(l, value)
>
> # validation
> assert errcode == 0
> assert len(l) == 1
> assert l[0] is value
> assert pyobj_value.ob_refcnt == refcount_before + 1
>
>
> If you don't want the exception handling, you can define your own
> declaration of PyList_Append() that does not have it. But personally, I'd
> rather use try-except in my test code than manually taking care of cleaning
> up (unexpected) exceptions.

Exactly, that's the kind of thing I had in mind. At the moment,
writing a new dedicated C API test requires designing 4 things:

1. The test case itself (what action to take, which assertions to make about it)
2. The C code to make the API call you want to test
3. The Python->C interface for the test case from 1 to pass test
values in to the code from 2
4. The C->Python interface to get state of interest from 2 back to the
test case from 1

If we were able to use Cython to handle 3 & 4 rather than having to
hand craft it for every test, then I believe it would significantly
lower the barrier to testing the C API directly rather than only
testing it indirectly through the CPython implementation.

Having such a test suite available would then hopefully make it easier
for other implementations to provide robust emulations of the public C
API.

ctypes & cffi likely wouldn't help as much in the case, since they
don't eliminate the need to come up with custom code for parts 3 & 4,
they just let you write that logic in Python rather than C.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Stefan Behnel]

Antoine Pitrou schrieb am 04.08.2018 um 15:57:
> Le 04/08/2018 à 15:13, Nick Coghlan a écrit :
>>
>> It'd be *really* nice to at least be able to write some of the C API
>> tests directly in Cython rather than having to fiddle about with
>> splitting the test between the regrtest parts that actually define the
>> test case and the extension module parts that expose the interfaces
>> that we want to test.
> 
> Actually, I think testing the C API is precisely the kind of area where
> you don't want to involve a third-party, especially not a moving target
> (Cython is actively maintained and generated code will vary after each
> new Cython release).  Besides, Cython itself calls the C API, which
> means you might end up involuntarily testing the C API against itself.
> 
> If anything, testing the C API using ctypes or cffi would probably be
> more reasonable... assuming we get ctypes / cffi to compile everywhere,
> which currently isn't the case.

I agree that you would rather not want to let Cython (or another tool)
generate the specific code that tests a specific C-API call, but you could
still use Cython to get around writing the setup, validation and unittest
boilerplate code in C. Basically, a test could then look something like
this (probably works, although I didn't test it):

from cpython.object cimport PyObject
from cpython.list cimport PyList_Append

def test_PyList_Append_on_empty_list():
# setup code
l = []
assert len(l) == 0
value = "abc"
pyobj_value =  value
refcount_before = pyobj_value.ob_refcnt

# conservative test call, translates to the expected C code,
# although with exception propagation if it returns -1:
errcode = PyList_Append(l, value)

# validation
assert errcode == 0
assert len(l) == 1
assert l[0] is value
assert pyobj_value.ob_refcnt == refcount_before + 1


If you don't want the exception handling, you can define your own
declaration of PyList_Append() that does not have it. But personally, I'd
rather use try-except in my test code than manually taking care of cleaning
up (unexpected) exceptions.


What we do in Cython, BTW, is write doctests in compiled ".pyx" files. That
allows us to execute certain parts of a test in Python (the doctest code)
and other parts in Cython (the compiled functions/classes that have the
doctests), and thus to do a direct comparison between Python and Cython. An
example that you could find in a test ".pyx" file:

def test_times2(x):
"""
doctest that gets executed by Python:

>>> test_times2(3) == 3 * 2
True
"""
# Cython compiled code in a compiled function that gets tested:
return x * 2


Given that CPython's current "_testcapimodule.c" is only a good 5000 lines
long (divide that by the number of public C-API functions and macros!), I'm
sure the above could help in improving the unit test coverage of the C-API
quite quickly.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Antoine Pitrou]

Hi Nick,

Le 04/08/2018 à 15:13, Nick Coghlan a écrit :
> 
> It'd be *really* nice to at least be able to write some of the C API
> tests directly in Cython rather than having to fiddle about with
> splitting the test between the regrtest parts that actually define the
> test case and the extension module parts that expose the interfaces
> that we want to test.

Actually, I think testing the C API is precisely the kind of area where
you don't want to involve a third-party, especially not a moving target
(Cython is actively maintained and generated code will vary after each
new Cython release).  Besides, Cython itself calls the C API, which
means you might end up involuntarily testing the C API against itself.

If anything, testing the C API using ctypes or cffi would probably be
more reasonable... assuming we get ctypes / cffi to compile everywhere,
which currently isn't the case.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com