[Python-Dev] Re: Object deallocation during the finalization of Python program

[Armin Rigo]

Hi Pau,

Also, the Cython documentation warns against doing this kind of things
(here, accessing the Python object stored in ``foo``).  From
https://cython.readthedocs.io/en/latest/src/userguide/special_methods.html:

You need to be careful what you do in a __dealloc__() method.
By the time your __dealloc__() method is called, the object
may already have been partially destroyed and may not be
in a valid state as far as Python is concerned, so you should
avoid invoking any Python operations which might touch the
object. In particular, don’t call any other methods of the object
or do anything which might cause the object to be resurrected.
It’s best if you stick to just deallocating C data.


Armin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JFMEQM6XPB6OLFNFMO4Q4PFM63XXXJDR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Adding a scarier warning to object.__del__?

[Armin Rigo]

Hi,

On Thu, 2 Jan 2020 at 03:59, Yonatan Zunger  wrote:
> It is possible (though not recommended!) for the __del__() method to postpone 
> destruction of the instance by creating a new reference to it. This is called 
> object resurrection. It is implementation-dependent whether __del__() is 
> called a second time when a resurrected object is about to be destroyed; the 
> current CPython implementation only calls it once.

"...in most cases."


Armin Rigo
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LHY2EM7PMSSUAMORBOHIUZAYKHILAMJO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Helpers for dynamic bytecode generation

[Armin Rigo]

Hi,

On Fri, 25 Oct 2019 at 04:13, Jonathan Goble  wrote:
>> Has anyone already done this that people know of? (Searching the Internetz 
>> didn't turn anything up) Failing that, to what extent is it reasonable to 
>> either consider assemble() as some kind of sane API point into compile.c

PyPy contains a complete rewrite of compile.c in Python, which should
be relatively easy to extract.  Here's the py3.6 version:
https://bitbucket.org/pypy/pypy/src/py3.6/pypy/interpreter/astcompiler/

Independently, you may also want to benchmark your code on PyPy
(*without* using any bytecode generation, just plain Python loops).


A bientôt,

Armin.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NUPFLTIO7GB2DNBFHKVBMKGYYZE4PPO4/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Armin Rigo]

Hi,

On Wed, 29 May 2019 at 08:07, Greg Ewing  wrote:
> Nick Coghlan wrote:
> > Having a single locals() call de-optimize an entire function would be
> > far from ideal.
>
> I don't see what would be so bad about that. The vast majority
> of functions have no need for locals().

You have the occasional big function that benefits a lot from being
JIT-compiled but which contains ``.format(**locals())``.  That occurs
in practice, and that's why PyPy is happy that there is a difference
between ``locals()`` and ``sys._getframe().f_locals``.  PyPy could be
made to support the full mutable view, but that's extra work that
isn't done so far and is a bit unlikely to occur at this point.  It
also raises the significantly the efforts for other JIT
implementations of Python if they have to support a full-featured
``locals()``; supporting ``_getframe().f_locals`` is to some extent
optional, but supporting ``locals()`` is not.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 558: Defined semantics for locals()

[Armin Rigo]

Hi,

On Thu, 23 May 2019 at 17:28, Steve Dower  wrote:
> On 23May2019 0636, Nick Coghlan wrote:
> > However, I think the PR does show that the proposed technique can be
> > implemented without too much additional code complexity, and will
> > hopefully be adaptable for all implementations that emulate the frame
> > API at all.
>
> Much excitement!
>
> One of the things I like best about this change is that it actually
> makes it *easier* for alternative implementations to use a simpler frame
> object without having to emulate CPython semantics (I'd love to get to a
> place where bug-for-bug compatibility wasn't required, but this is where
> we are right now so *shrug*).

Thanks Nick for getting this through!

A bientôt,
Armin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Armin Rigo]

Hi Neil,

On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer  wrote:
> Regarding the Py_TRACE_REFS fields, I think we can't do them without
> breaking the ABI because of the following.  For GC objects, they are
> always allocated by _PyObject_GC_New/_PyObject_GC_NewVar.  So, we
> can allocate the extra space needed for the GC linked list.  For
> non-GC objects, that's not the case.  Extensions can allocate using
> malloc() directly or their own allocator and then pass that memory
> to be initialized as a PyObject.
>
> I think that's a poor design and I think we should try to make slow
> progress in fixing it.

Such progress needs to start with the global static PyTypeObjects that
all extensions define.  This is going to be impossible to fix without
requiring a big fix in of *all* of them.  (Unless of course you mean
to still allow them, but then Py_TRACE_REF can't be implemented in a
way that doesn't break the ABI.)


A bientôt,

Armin.

On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer  wrote:
>
> On 2019-04-24, Victor Stinner wrote:
> > The current blocker issue is that the Py_DEBUG define imply the
> > Py_TRACE_REFS define
>
> I think your change to make Py_TRACE_REFS as separate configure flag
> is fine.  I've used the trace fields to debug occasionally but I
> don't use it often enough to need it enabled by Py_DEBUG.
>
> > Being able to switch between Python in release mode and Python in
> > debug mode is a first step. My long term plan would be to better
> > separate "Python" from its "runtime".
>
> Regarding the Py_TRACE_REFS fields, I think we can't do them without
> breaking the ABI because of the following.  For GC objects, they are
> always allocated by _PyObject_GC_New/_PyObject_GC_NewVar.  So, we
> can allocate the extra space needed for the GC linked list.  For
> non-GC objects, that's not the case.  Extensions can allocate using
> malloc() directly or their own allocator and then pass that memory
> to be initialized as a PyObject.
>
> I think that's a poor design and I think we should try to make slow
> progress in fixing it.  I think non-GC objects should also get
> allocated by a Python API.  In that case, the Py_TRACE_REFS
> functionality could be implemented in a way that doesn't break the
> ABI.  It also makes the CPython API more friendly for alternative
> Python runtimes like PyPy, etc.
>
> Note that this change would not prevent an extension from allocating
> memory with it's own allocator.  It just means that memory can't
> hold a PyObject.  The extension PyObject would need to have a
> pointer that points to this externally allocated memory.
>
> I can imagine there could be some situations when people really
> want a PyObject to reside in a certain memory location.  E.g. maybe
> you have some kind of special shared memory area.  In that case, I
> think we could have specialized APIs to create PyObjects using a
> specialized allocator.  Those APIs would not be supported by
> some runtimes (e.g. tracing/moving GC for PyObjects) and the APIs
> would not be used by most extensions.
>
> Regards,
>
>   Neil
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/armin.rigo%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Armin Rigo]

Hi,

On Tue, 19 Feb 2019 at 13:12, Victor Stinner  wrote:
> Please don't use _GET_ITEM() or _PyTuple_ITEMS(). It prevents
> to use a more efficient storage for tuple. Something like:
> https://pythoncapi.readthedocs.io/optimization_ideas.html#specialized-list-for-small-integers
>
> PyPy already has the issue right now.

Just to clarify PyPy's point of view (or at least mine):

1. No, it no longer has this issue.  You can misuse
``_GET_ITEM()`` freely with PyPy too.

2. This whole discussion is nice but is of little help to PyPy at this
point.  The performance hit comes mostly from emulating reference
counting and non-movable objects.  If the API was half the size and
did not contain anything with irregular behavior, it would have made
our job easier in the past, but now it's done---and it wouldn't have
improved the performance of the result.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C API changes

[Armin Rigo]

Hi Steve,

On 30/11/2018, Steve Dower  wrote:
> On 29Nov2018 2206, Armin Rigo wrote:
>> On Thu, 29 Nov 2018 at 18:19, Steve Dower  wrote:
>>> quo. We continue to not be able to change CPython internals at all,
>>> since that will break people using option B.
>>
>> No?  That will only break users if they only have an option-B
>> ``foo.cpython-318m-x86_64-linux-gnu.so``, no option-A .so and no
>> source code, and want to run it elsewhere than CPython 3.18.  That's
>> the same as today.  If you want option-B .so for N versions of
>> CPython, recompile the source code N times.
>>
>> Just to be clear, if done correctly there should be no need for
>> #ifdefs in the source code of the extension module.
>
> The problem is that if option B remains as compatible as it is today, we
> can't make option A faster enough to be attractive. The marketing pitch
> for this looks like: "rewrite all your existing code to be slower but
> works with PyPy, or don't rewrite your existing code and it'll be
> fastest with CPython and won't break in the future". This is status quo
> (where option A today is something like CFFI or Cython), and we can
> already see how many people have made the switch (FWIW, I totally prefer
> Cython over pure C for my own projects :) ).
>
> My proposed marketing pitch is: "rewrite your existing code to be
> forward-compatible today and faster in the future without more work, or
> be prepared to rewrite/update your source code for each CPython release
> to remain compatible with the low level API".
> (...)

Discussing marketing pitches on python-dev is not one of my favorite
past-times, so I'll excuse myself out of this conversation.  Instead,
I might try to implement the basics, check out the performance on
CPython and on PyPy, and seek out interest---I'm thinking about
Cython, for example, which might relatively easily be adapted to
generate that kind of code.  This might be a solution for the poor
performance of Cython on PyPy...  If everything works out, maybe I'll
come back here at some point, with the argument "the CPython C API is
blocking CPython from evolution more and more?  Here's one possible
path forward."


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C API changes

[Armin Rigo]

Hi,

On Thu, 29 Nov 2018 at 18:19, Steve Dower  wrote:
> quo. We continue to not be able to change CPython internals at all,
> since that will break people using option B.

No?  That will only break users if they only have an option-B
``foo.cpython-318m-x86_64-linux-gnu.so``, no option-A .so and no
source code, and want to run it elsewhere than CPython 3.18.  That's
the same as today.  If you want option-B .so for N versions of
CPython, recompile the source code N times.

Just to be clear, if done correctly there should be no need for
#ifdefs in the source code of the extension module.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C API changes

[Armin Rigo]

Hi Steve,

On Tue, 27 Nov 2018 at 19:14, Steve Dower  wrote:
> On 27Nov2018 0609, Victor Stinner wrote:
> > Note: Again, in my plan, the new C API would be an opt-in API. The old
> > C API would remain unchanged and fully supported. So there is no
> > impact on performance if you consider to use the old C API.
>
> This is one of the things that makes me think your plan is not feasible.

I can easily imagine the new API having two different implementations
even for CPython:

A) you can use the generic implementation, which produces a
cross-python-compatible .so.  All function calls go through the API at
runtime.  The same .so works on any version of CPython or PyPy.

B) you can use a different set of headers or a #define or something,
and you get a higher-performance version of your unmodified
code---with the issue that the .so only runs on the exact version of
CPython.  This is done by defining some of the functions as macros.  I
would expect this version to be of similar speed than the current C
API in most cases.

This might give a way forward: people would initially port their
extensions hoping to use the option B; once that is done, they can
easily measure---not guess--- the extra performance costs of the
option A, and decide based on actual data if the difference is really
worth the additional troubles of distributing many versions.  Even if
it is, they can distribute an A version for PyPy and for unsupported
CPython versions, and add a few B versions on top of that.


...Also, although I'm discussing it here, I think the whole approach
would be better if done as a third-party extension for now, without
requiring changes to CPython---just use the existing C API to
implement the CPython version.  The B option discussed above can even
be mostly *just* a set of macros, with a bit of runtime that we might
as well include in the produced .so in order to make it a standalone,
regular CPython C extension module.


A bientôt,

Armin.

PS: on CPython could use ``typedef struct { PyObject *_obj; }
PyHandle;``.  This works like a pointer, but you can't use ``==`` to
compare them.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C API changes

[Armin Rigo]

Hi,

On Sun, 25 Nov 2018 at 10:15, Stefan Behnel  wrote:
> Overall, this seems like something that PyPy could try out as an
> experiment, by just taking a simple extension module and replacing all
> increfs with newref assignments. And obviously implementing the whole thing
> for the C-API

Just to be clear, I suggested making a new API, not just tweaking
Py_INCREF() and hoping that all the rest works as it is.  I'm
skeptical about that.

To start with, a ``Py_NEWREF()`` like you describe *will* lead people
just renaming all ``Py_INCREF()`` to ``Py_NEWREF()`` ignoring the
return value, because that's the easiest change and it would work fine
on CPython.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C API changes

[Armin Rigo]

Hi Stefan,

On Sat, 24 Nov 2018 at 22:17, Stefan Behnel  wrote:
> Couldn't this also be achieved via reference counting? Count only in C
> space, and delete the "open object" when the refcount goes to 0?

The point is to remove the need to return the same handle to C code if
the object is the same one.  This saves one of the largest costs of
the C API emulation, which is looking up the object in a big
dictionary to know if there is already a ``PyObject *`` that
corresponds to it or not---for *all* objects that go from Python to C.

Once we do that, then there is no need for a refcount any more.  Yes,
you could add your custom refcount code in C, but in practice it is
rarely done.  For example, with POSIX file descriptors, when you would
need to "incref" a file descriptor, you instead use dup().  This gives
you a different file descriptor which can be closed independently of
the original one, but they both refer to the same file.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] C API changes

[Armin Rigo]

Hi Hugo, hi all,

On Sun, 18 Nov 2018 at 22:53, Hugh Fisher  wrote:
> I suggest that for the language reference, use the license plate
> or registration analogy to introduce "handle" and after that use
> handle throughout. It's short, distinctive, and either will match
> up with what the programmer already knows or won't clash if
> or when they encounter handles elsewhere.

FWIW, a "handle" is typically something that users of an API store and
pass around, and which can be used to do all operations on some
object.  It is whatever a specific implementation needs to describe
references to an object.  In the CPython C API, this is ``PyObject*``.
I think that using "handle" for something more abstract is just going
to create confusion.

Also FWIW, my own 2 cents on the topic of changing the C API: let's
entirely drop ``PyObject *`` and instead use more opaque
handles---like a ``PyHandle`` that is defined as a pointer-sized C
type but is not actually directly a pointer.  The main difference this
would make is that the user of the API cannot dereference anything
from the opaque handle, nor directly compare handles with each other
to learn about object identity.  They would work exactly like Windows
handles or POSIX file descriptors.  These handles would be returned by
C API calls, and would need to be closed when no longer used.  Several
different handles may refer to the same object, which stays alive for
at least as long as there are open handles to it.  Doing it this way
would untangle the notion of objects from their actual implementation.
In CPython objects would internally use reference counting, a handle
is really just a PyObject pointer in disguise, and closing a handle
decreases the reference counter.  In PyPy we'd have a global table of
"open objects", and a handle would be an index in that table; closing
a handle means writing NULL into that table entry.  No emulated
reference counting needed: we simply use the existing GC to keep alive
objects that are referenced from one or more table entries.  The cost
is limited to a single indirection.

The C API would change a lot, so it's not reasonable to do that in the
CPython repo.  But it could be a third-party project, attempting to
define an API like this and implement it well on top of both CPython
and PyPy.  IMHO this might be a better idea than just changing the API
of functions defined long ago to make them more regular (e.g. stop
returning borrowed references); by now this would mostly mean creating
more work for the PyPy team to track and adapt to the changes, with no
real benefits.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] short-circuiting runtime errors/exceptions in python debugger.

[Armin Rigo]

Hi,

On Sat, 27 Oct 2018 at 01:50, Steven D'Aprano  wrote:
> [...]
> > So I was wondering if it would be possible to keep that context around
> > if you are in the debugger and rewind the execution point to before
> > the statement was triggered.
>
> I think what you are looking for is a reverse debugger[1] also known as
> a time-travel debugger.

I think it's a bit different.  A reverse debugger is here to debug
complex conditions in a (static) program.  What Ed is looking for is a
way to catch easy failures, fix the obviously faulty line, and
continue running the program.

Of course I can't help but mention to Ed that this is precisely the
kind of easy failures that are found by *testing* your code,
particularly if that's code that only runs after hours of other code
has executed.  *Never* trust yourself to write correct code if you
don't know that it is correct after waiting for hours.

But assuming that you really, really are allergic to tests, then what
you're looking for reminds me of long-ago Python experiments with
resumable exceptions and patching code at runtime.  Both topics are
abandoned now.  Resumable exceptions was a cool hack of the
interpreter that nobody really found a use for (AFAIR); patching code
at runtime comes with a pile of messes---it only works in the simple
cases, but there is no general solution for that.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's change to C API!

[Armin Rigo]

Hi Antoine,

On 11 August 2018 at 15:19, Antoine Pitrou  wrote:
> Jython and IronPython never got significant manpower AFAIK, so even
> without being hindered by the C API, chances are they would never have
> gotten very far.  Both do not even seem to have stable releases
> implementing the Python 3 language...

I included IronPython and Jython because they are also rather complete
implementations of (some version of) Python that are actively used in
some contexts.  During the past 20 years, these two and PyPy are the
only generally-useful rather-complete alternate Python
implementations, and they each improve on some of the pain points (2)
(3) (4) hitting CPython.  Neither of them supports the C API
efficiently.  Whatever you argue, my opinion is that they got where
they are by first completely ignoring the C API.  Even Pyston did
that.

About its C API, CPython can continue to prefer the status quo.  I
tend to think that it's exactly what will occur, so I'm staying away
from capi-sig.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's change to C API!

[Armin Rigo]

Hi,

On 31 July 2018 at 13:55, Antoine Pitrou  wrote:
> It's just that I disagree that removing the C API will make CPython 2x
> faster.
>
> Actually, important modern optimizations for dynamic languages (such as
> inlining, type specialization, inline caches, object unboxing) don't
> seem to depend on the C API at all.

These are optimizations typically talked about in papers about dynamic
languages in general.  In my opinion, in the specific case of CPython,
they are all secondary to the following: (1) JIT, (2) GC, (3) object
model, (4) multithreading.

Currently, the C API only allows Psyco-style JITting (much slower than
PyPy).  All three other points might not be possible at all without a
seriously modified C API.  Why?  I have no proof, but only
circumstantial evidence.  Each of (2), (3), (4) has been done in at
least one other implementation: PyPy, Jython and IronPython.  Each of
these implementation has also got its share of troubles with emulating
the CPython C API.  You can continue to think that the C API has got
nothing to do with it.  I tend to think the opposite.  The continued
absence of major performance improvements for either CPython itself or
for any alternative Python implementation that *does* support the C
API natively is probably proof enough---I think that enough time has
passed, by now, to make this argument.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Error message for wrong number of arguments

[Armin Rigo]

Hi,

On 30 July 2018 at 22:19, Chris Barker via Python-Dev
 wrote:
> Oh well. This is a serious usability issue -- but what can you do?

I think that argument clinic knows if the built-in function is
supposed to be a method or a function.  It doesn't look too hard to
add a new flag METH_IS_METHOD or something, which would be taken in
consideration in the common cases, and which can be added manually OR
used automatically by argument clinic.  This would not be a 100%
solution out of the box, but if the new wording is right, it shouldn't
be a problem.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About [].append == [].append

[Armin Rigo]

Hi,

On 23 June 2018 at 10:54, Serhiy Storchaka  wrote:
> +1 too. But I think the right solution should be opposite: reverting
> issue1350060 changes and making all methods equality be based on the
> identity of __self__.

The arguments in this thread are the kind of discussion I was looking
for when I asked for one in https://bugs.python.org/issue1350060 :-)
Better 12 years late than never.  Fwiw I also tend to think nowadays
that the right solution should be the opposite, like you say.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add __reversed__ methods for dict

[Armin Rigo]

Hi Inada,

On 27 May 2018 at 09:12, INADA Naoki  wrote:
> When focusing to CPython, PyPy and MicroPython, no problem for adding
> __reverse__ in 3.8 seems OK.

Fwiw, the functionality that is present in OrderedDict but still
absent from 'dict' is: ``__reverse__``, discussed above, and
``move_to_end(last=False)``.  In PyPy3, OrderedDict is implemented not
like in the CPython stdlib but as just a thin dict subclass without
any extra data, using two custom built-ins from the ``__pypy__``
module for the two methods above (plus some pure Python code for other
methods like __eq__(), where it is possible to do so with the correct
complexity).


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (name := expression) doesn't fit the narrative of PEP 20

[Armin Rigo]

Hi,

On 26 April 2018 at 07:50, Raymond Hettinger
 wrote:
>> [Raymond Hettinger ]
>>> After re-reading all the proposed code samples, I believe that
>>> adopting the PEP will make the language harder to teach to people
>>> who are not already software engineers.
>
> (...)
>
> Python is special, in part, because it is not one of those languages.
> It has virtues that make it suitable even for elementary school children.
> We can show well-written Python code to non-computer folks and walk
> them through what it does without their brains melting (something I can't
> do with many of the other languages I've used).  There is a virtue
> in encouraging simple statements that read like English sentences
> organized into English-like paragraphs, presenting itself like
> "executable pseudocode".

I must admit that when I heard about this PEP I thought "this April
1st joke was already done long ago".  I'm sorry to discover that, this
time, it is not actually one.  Thank you, Raymond, for an unlikely
attempt at reminding people what made Python so special---in your
opinion, and mine.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backward incompatible change about docstring AST

[Armin Rigo]

Hi,

On 27 February 2018 at 15:32, Serhiy Storchaka  wrote:
> 1. CPython and PyPy set different position for multiline strings. PyPy sets
> the position of the start of string, but CPython sets the position of the
> end of the string. A program that utilizes the docstring position needs to
> handle both of these cases.

If that's true it's a PyPy bug.
https://bitbucket.org/pypy/pypy/issues/2767/docstring-position-in-the-ast


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python possible vulnerabilities in concurrency

[Armin Rigo]

Hi,

On 14 November 2017 at 14:55, Jan Claeys  wrote:
> Sounds like https://www.iso.org/standard/71094.html
> which is updating https://www.iso.org/standard/61457.html
> (which you can download from there if you search a bit; clearly either
> ISO doesn't have a UI/UX "standard" or they aren't following it...)

Just for completeness, I think that what you can download for free
from that second page only contains the first few sections ("Terms and
definitions").  It doesn't even go to "Purpose of this technical
report"---we need to pay $200 just to learn what the purpose is...

*Shrug*

Armin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Armin Rigo]

Hi Antoine,

On 8 November 2017 at 10:28, Antoine Pitrou  wrote:
> Yet, PyPy has no reference counting, and it doesn't seem to be a cause
> of concern.  Broken code is fixed along the way, when people notice.

It is a major cause of concern.  This is the main blocker for
pure-Python code compatibility between CPython and all other
implementations of Python, but most of the Python community plays nice
and says since long ago "don't do that".   As a result,
nowadays, people generally know better than rely on deterministic
__del__, and the language evolution usually cares not to add new
dependencies on that.  The problem is mostly confined to some
pre-existing large code bases (like OpenStack), where no good solution
exists.

It's roughly OK to have one big blocker that people need to know
about.  I don't think it's anywhere close to OK to have a miriad of
small ones.  PyPy has worked very hard to get where it is now, and
continues to regularly "fix" very obscure compatibility issues where
CPython's behaviour differs from its own documentation---by copying
the CPython actual behaviour, of course.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __del__ is not called after creating a new reference

[Armin Rigo]

Hi all,

On 20 March 2017 at 22:28, Nathaniel Smith  wrote:
> Modern CPython, and all extant versions of PyPy and Jython, guarantee that
> __del__ is called at most once.

Just a note, if someone actually depends on this: it is not true in
all cases.  For example, in CPython 3.5.3:

>>> class X:
...   __slots__=()#   <= note this!
...   def __del__(self):
... print("DEL")
... global resurrect
... resurrect = self
...
>>> print(X())
<__main__.X object at 0x7f5d1ad600d0>
DEL
>>> resurrect=None
DEL
>>> resurrect=None
DEL
>>> resurrect=None
DEL


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CALL_FUNCTION_EX arg and stack_effect

[Armin Rigo]

Hi Matthieu,

On 20 February 2017 at 19:44, Matthieu Dartiailh  wrote:
> What do you think ? Should I open an issue on https://bugs.python.org/ ?

Possibly related: http://bugs.python.org/issue24340


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deterministic builds of the interpreter

[Armin Rigo]

Hi Freddy,

On 16 February 2017 at 18:03, Freddy Rietdijk  wrote:
> As I mentioned, it seems only sets cause unreproducible
> bytecode. Sets have no order. But when generating the bytecode, I would
> expect there would still be an order since the code isn't actually executed,
> right?

No, the sets are built as real sets and then marshalled to .pyc files
in a separate step.  So on CPython an essentially random order will
end up in the .pyc file.  Even CPython 3.6 gives a deterministic order
to dictionaries but not sets.  You could ensure sets are marshalled in
a known order by changing the marshalling code, e.g. to emit them in
sorted order (on Python 2.x; on 3.x it is more messy because different
types are more often non-comparable).


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] re performance

[Armin Rigo]

Hi Sven,

On 26 January 2017 at 22:13, Sven R. Kunze  wrote:
> I recently refreshed regular expressions theoretical basics *indulging in
> reminiscences* So, I read https://swtch.com/~rsc/regexp/regexp1.html

Theoretical regular expressions and what Python/Perl/etc. call regular
expressions are a bit different.  You can read more about it at
https://en.wikipedia.org/wiki/Regular_expression#Implementations_and_running_times
.

Discussions about why they are different often focus on
backreferences, which is a rare feature.  Let me add two other points.

The theoretical kind of regexp is about giving a "yes/no" answer,
whereas the concrete "re" or "regexp" modules gives a match object,
which lets you ask for the subgroups' location, for example.  Strange
at it may seem, I am not aware of a way to do that using the
linear-time approach of the theory---if it answers "yes", then you
have no way of knowing *where* the subgroups matched.

Another issue is that the theoretical engine has no notion of
greedy/non-greedy matching.  Basically, you walk over the source
character and it answers "yes" or "no" after each of them.  This is
different from a typical backtracking implementation.  In Python:

>>> re.match(r'a*', 'aaa')
>>> re.match(r'a*?', 'aaa')

This matches either three or zero characters in Python.  The two
versions are however indistinguishable for the theoretical engine.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is there any remaining reason why weakref callbacks shouldn't be able to access the referenced object?

[Armin Rigo]

Hi,

Sorry to reply in this old thread.  We just noticed this on #pypy:

On 22 October 2016 at 05:32, Nick Coghlan  wrote:
> The weakref-before-__del__ ordering change in
> https://www.python.org/dev/peps/pep-0442/#disposal-of-cyclic-isolates
> only applies to cyclic garbage collection,so for normal refcount
> driven object cleanup in CPython, the __del__ still happens first:
>
> >>> class C:
> ... def __del__(self):
> ... print("__del__ called")
> ...
> >>> c = C()
> >>> import weakref
> >>> def cb():
> ... print("weakref callback called")
> ...
> >>> weakref.finalize(c, cb)
> 
> >>> del c
> __del__ called
> weakref callback called

Nick, it seems you're suggesting that before PEP 442 (in CPython 3.4)
the __del__ happened before the weakref-clearing operation as well.
That's not the case: before CPython 3.4, weakrefs are always cleared
first.  The situation became more muddy in 3.4, where weakrefs are
usually cleared after the __del__ is called---in the absence of
reference cycles (so it's a backward-incompatible change).  If there
are reference cycles, then the weakref is cleared before the __del__
is called.

This can be shown in your example by replacing "weakref.finalize(c,
cb)" with an old-style "wr = weakref.ref(c, cb)".  Then CPython <= 3.3
and >= 3.4 print the two lines in opposite order.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes, memory mapped files and context manager

[Armin Rigo]

Hi Hans-Peter,

On 6 January 2017 at 00:28, Hans-Peter Jansen  wrote:
> Leaves the question, how stable this "interface" is?

Another way to jump through hoops:

c_raw = ctypes.PYFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)(lambda p: p)

addr = c_raw(ctypes.pointer(T.from_buffer(m)))
b = ctypes.cast(addr, ctypes.POINTER(T)).contents

These lines give an object 'b' that is equivalent to
'T.from_buffer(m)', but doesn't hold any reference or any "opened
buffer" state to the original 'm'.  Your context manager can yield
that.  It should prevent all BufferErrors, at the price of segfaulting
if used incorrectly.  This means in your case that ``with
map_struct(..) as a:`` should not continue to use ``a`` after the
``with`` statement, which is pretty natural anyway.

(The same issue occurs with cffi instead of ctypes, but in this case a
simple cast is enough to detach the memoryview, instead of the hack
above.)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PySlice_GetIndicesEx annd stable ABI: bikeshedding

[Armin Rigo]

Hi Serhiy,

On 21 December 2016 at 15:51, Serhiy Storchaka  wrote:
> The code
>
> if (PySlice_GetIndicesEx(item, length,
> , , , ) < 0)
> return -1;
>
> should be replaced with
>
> if (foo(item, , , ) < 0)
> return -1;
> slicelength = bar(, , step, length);

As far as I can tell, as written, this change would not fix anything.
Shouldn't it be along the following lines instead?

if (foo(item, , , ) < 0)
return -1;
length = PyList_GET_SIZE(mylist);   /* <= after foo() */
slicelength = bar(, , , length);


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyPy progress: list of CPython 3.5 crashers and bugs

[Armin Rigo]

Hi Nick,

On 5 December 2016 at 13:22, Nick Coghlan  wrote:
> I think 3 omnibus issues would be a reasonable way to go, with the
> discussion on those issues then splitting things out to either new
> issue reports, or entries in
> https://hg.python.org/cpython/file/default/Lib/test/crashers/ (if any
> of the crashers can't be readily resolved).

Ok, added:

http://bugs.python.org/issue28883
http://bugs.python.org/issue28884
http://bugs.python.org/issue28885


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyPy progress: list of CPython 3.5 crashers and bugs

[Armin Rigo]

Hi Raymond,

On 6 December 2016 at 05:59, Raymond Hettinger
 wrote:
> Also, we use  {...} instead of OrderedDict(...).

No, at least CPython 3.5.2 uses ``...`` for OrderedDict, and not
``{...}``.  Following that example, deques should also use ``...``
instead of ``[...]``.  But I bow to your choice and remove my point
from cpython-crashers.rst anyway.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PyPy progress: list of CPython 3.5 crashers and bugs

[Armin Rigo]

Hi all,

PyPy 3.5 is progressing.  It's still alpha status, but we'll give a
progress report on morepypy.blogspot.com at some point.

Now the point of this mail is that when exploring the source code of
CPython 3.5.2+, we found a large number of crashers and bugs.  None of
them are essential---otherwise, they would already have been reported.
However, if the goal of python-dev is still to ensure that CPython
cannot normally be crashed and behaves as documented even in corner
cases, then you are probably interested in them.

I reported the first two crashers, http://bugs.python.org/issue27811
and http://bugs.python.org/issue27812, but then stopped to keep some
focus on PyPy.  I'm instead collecting them here as I find them:
http://bitbucket.org/pypy/extradoc/raw/extradoc/planning/py3.5/cpython-crashers.rst

I didn't systematically check the CPython trunk: the bugs are for
CPython 3.5.2+.  But I did check trunk a few times, and the same bug
was present there too.  So for now I assume that many items on that
list are still up-to-date.

In 3.5.2+ at least, I'm reasonably convinced that all crashers are
real, but I didn't spend the time to come up with actual examples or
patches, beyond the first two items on the list.  There are also
non-crasher bugs where the current behavior is clearly wrong according
to the documentation or the PEP.  I've also added a few points that
strike me as rather strange but not against the documentation.  What
should I do with this list?  From my point of view, I could drop it
all in a single issue, or possibly three of them (crashers, bugs,
"strange").  Alternatively I can go ahead and open one issue per
bullet point.  Which way would you prefer?  Or, if you think there is
no point in me filing issues without actual examples and patches, then
that's fine with me too and I will simply continue to expand my
cpython-crashers.rst file.


Thank you for your attention,

Armin Rigo
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Review request: issue 27350, compact ordered dict

[Armin Rigo]

Hi Inada,

On 10 August 2016 at 18:52, INADA Naoki  wrote:
> a. dict has one additional word and support ring internally.
> b. OrderedDict reimplements many APIs (iterating, resizing, etc...) to 
> support ring.

There is a solution "c." which might be simpler.  Let's think first
about what occurs with a normal dict (with your patch, or in PyPy) if
the user repeatedly pops the oldest item and adds a new item.  In this
case, the dict will accumulate dead entries at the start of the list,
and when it gets too many of them, it needs to make a complete copy of
the list to get rid of them.  (This is not a problem as it is
amortized over many pop-and-add operations.)

So now, when the user calls move_to_end(last=False), we can do the
opposite.  We look if there are already some deleted entries at the
start, and if so, we add the item at the place of the last deleted
entry.  If there aren't any, then we make a copy of the list that
*adds* some number of entries at the start, which are initially marked
as deleted...


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 520: Preserving Class Attribute Definition Order (round 5)

[Armin Rigo]

Hi,

On 24 June 2016 at 23:52, Eric Snow  wrote:
> Pending feedback, the impact on Python implementations is expected to
> be minimal.  If a Python implementation cannot support switching to
> `OrderedDict``-by-default then it can always set ``__definition_order__``
> to ``None``.

That's wishful thinking.  Any Python implementation that sets
``__definition_order__`` to None where CPython sets it to something
useful is likely going to break programs and be deemed not fully
compatible.  (Note: this PEP is not a problem for PyPy.)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyGC_Collect ignores state of `enabled`

[Armin Rigo]

Hi Lukasz,

On 10 May 2016 at 04:13, Łukasz Langa  wrote:
> However, because of PyGC_Collect() called in Py_Finalize(), during
> interpreter shutdown the collection is done anyway, Linux does CoW and the
> memory usage spikes. Which is ironic on process shutdown.

Try to call os._exit() to avoid doing all this work on shutdown (after
you have checked that it is indeed not doing anything interesting).


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Armin Rigo]

Hi Victor,

On 14 April 2016 at 17:19, Victor Stinner  wrote:
> Each time a dictionary is created, the global
> version is incremented and the dictionary version is initialized to the
> global version.

A detail, but why not set the version tag of new empty dictionaries to
zero, always?   Same after a clear().  This would satisfy the
condition: equality of the version tag is supposed to mean "the
dictionary content is precisely the same".


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Py_SETREF vs. Py_XSETREF

[Armin Rigo]

Hi,

On 3 April 2016 at 15:29, MRAB  wrote:
>> Should we rename Py_SETREF to Py_XSETREF and introduce new Py_SETREF
>> that uses Py_DECREF?
>
> Checking for NULL is convenient (and safer), but, on the other hand, it
> _would_ be consistent with the others.

My 2 cents would be to call the new macro Py_XSETREF for consistency,
at least, whether you decide to go with two macros or not.  Otherwise
it's kind of obvious that if you add Py_SETREF that checks for nulls,
in 2 or 3 releases people will really want a "fast" variant anyway,
and there will be no consistent name for that.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting references to Py_None

[Armin Rigo]

Hi,

On 20 March 2016 at 18:10, Brett Cannon  wrote:
> And if we didn't keep its count accurately it would eventually hit
> zero and constantly have its dealloc function checked for.

I think the idea is really consistency.  If we wanted to avoid all
"Py_INCREF(Py_None);", it would be possible: we could let the refcount
of None decrement to zero, at which point its deallocator is called;
but this deallocator can simply bumps the refcount to a large value
again.  The deallocator would end up being called very rarely.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "python.exe is not a valid Win32 app"

[Armin Rigo]

Hi all,

On Tue, Dec 1, 2015 at 8:13 PM, Laura Creighton  wrote:
> Python 3.5 is not supported on windows XP.  Upgrade your OS or
> stick with 3.4

Maybe this information should be written down somewhere more official?
 I can't find it in any of these pages:

https://www.python.org/downloads/windows/
https://www.python.org/downloads/release/python-350/
https://www.python.org/downloads/release/python-351/
https://docs.python.org/3/using/windows.html

It is found on the following page, to which googling "python 3.5
windows XP" does not point:

https://docs.python.org/3.5/whatsnew/3.5.html#unsupported-operating-systems

Instead, the google query above returns various threads on
stackoverflow and elsewhere where users wonder about that very
question.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Avoiding CPython performance regressions

[Armin Rigo]

Hi all,

Spending an hour with "hg bisect" is a good way to figure out some of
the worst speed regressions that occurred in the early days of 2.7
(which are still not fixed now).  Here's my favorite's pick:

* be4bec689de3  made bm_mako 15% slower, and spitfire_cstringio even much more

* ad030571e6c0  made ai 5% slower

Just thought it would be worth mentioning it here.  There is much more
waiting for someone with a little more patience if we believe
https://www.speedtin.com/public .


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Yet another "A better story for multi-core Python" comment

[Armin Rigo]

Hi Gary,

On Tue, Sep 8, 2015 at 4:12 PM, Gary Robinson  wrote:
> 1) More the reference counts away from data structures, so copy-on-write 
> isn’t an issue.

A general note about PyPy --- sorry, it probably doesn't help your use
case because SciPy is not supported right now...

Right now, PyPy hits the same problem as CPython, despite not being
based on reference counting, because every major collection needs to
write flag bits inside the header of every object.  However, fixing
this issue is much more straightforward here: there are
well-documented ways that other virtual machines (for other languages)
already do.  Mostly, instead of writing one bit in the GC header, we'd
write one bit in some compact out-of-line array of bits.  Moreover, it
is an issue that we hear about every now and again, so we may
eventually just do it.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] tp_finalize vs tp_del sematics

[Armin Rigo]

Hi Valentine,

On Thu, Sep 3, 2015 at 9:15 PM, Valentine Sinitsyn
 wrote:
>> That does not make it ok to have del called several time, does it?
>
> That's a tricky question.

If the Python documentation now says something like ``the __del__
method is never called more than once on the same instance'' without
acknowledging this corner case, then it could be regarded as
documentation bug.  I didn't check, though.  But feel free to open an
issue and mention everything I said above, if you want to.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] tp_finalize vs tp_del sematics

[Armin Rigo]

Hi Valentine,

On 24 August 2015 at 20:43, Valentine Sinitsyn
valentine.sinit...@gmail.com wrote:
 So you mean that this was to keep things backwards compatible for
 third-party extensions? I haven't thought about it this way, but this makes
 sense. However, the behavior of Python code using objects with __del__ has
 changed nevertheless: they are collectible now, and __del__ is always called
 exactly once, if I understand everything correctly.

Yes, I think so.  There is a *highly obscure* corner case: __del__
will still be called several times if you declare your class with
__slots__=().


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] tp_finalize vs tp_del sematics

[Armin Rigo]

Hi Valentine,

On 25 August 2015 at 09:56, Valentine Sinitsyn
valentine.sinit...@gmail.com wrote:
 Yes, I think so.  There is a *highly obscure* corner case: __del__
 will still be called several times if you declare your class with
 __slots__=().

 Even on post-PEP-0442 Python 3.4+? Could you share a link please?

class X(object):
__slots__=() # = try with and without this
def __del__(self):
global revive
revive = self
print(hi)

X()
revive = None
revive = None
revive = None


--Armin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] tp_finalize vs tp_del sematics

[Armin Rigo]

Hi Valentine,

On 19 August 2015 at 09:53, Valentine Sinitsyn
valentine.sinit...@gmail.com wrote:
 why it wasn't possible to
 implement proposed CI disposal scheme on top of tp_del?

I'm replying here as best as I understand the situation, which might
be incomplete or wrong.

From the point of view of someone writing a C extension module, both
tp_del and tp_finalize are called with the same guarantee that the
object is still valid at that point.  The difference is only that the
presence of tp_del prevents the object from being collected at all if
it is part of a cycle.  Maybe the same could have been done without
duplicating the function pointer (tp_del + tp_finalize) with a
Py_TPFLAGS_DEL_EVEN_IN_A_CYCLE.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 is here until 2020, please don't call it a waste.

[Armin Rigo]

Hi Larry,

On 31 May 2015 at 01:20, Larry Hastings la...@hastings.org wrote:
 p.s. Supporting this patch also helps cut into PyPy's reported performance
 lead--that is, if they ever upgrade speed.pypy.org from comparing against
 Python *2.7.2*.

Right, we should do this upgrade when 2.7.11 is out.

There is some irony in your comment which seems to imply PyPy is
cheating by comparing with an old Python 2.7.2: it is inside a thread
which started because we didn't backport performance improvements to
2.7.x so far.

Just to convince myself, I just ran a performance comparison.  I ran
the same benchmark suite as speed.pypy.org, with 2.7.2 against 2.7.10,
both freshly compiled with no configure options at all.  The
differences are usually in the noise, but range from +5% to... -60%.
If anything, this seems to show that CPython should take more care
about performance regressions.  If someone is interested:

* raytrace-simple is 1.19 times slower
* bm_mako is 1.29 times slower
* spitfire_cstringio is 1.60 times slower
* a number of other benchmarks are around 1.08.

The 7.0x faster number on speed.pypy.org would be significantly
*higher* if we upgraded the baseline to 2.7.10 now.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python-versus-CPython question for __mul__ dispatch

[Armin Rigo]

Hi Nick,

On 16 May 2015 at 10:31, Nick Coghlan ncogh...@gmail.com wrote:
 Oh, that's rather annoying that the PyPy team implemented bug-for-bug
 compatibility there, and didn't follow up on the operand precedence
 bug report to say that they had done so.

It's sadly not the only place, by far, where a behavior of CPython
could be considered an implementation detail, but people rely on it
and so we need to write a workaround.  We don't report all of them,
particularly not the ones that are clearly of the kind won't be
changed in CPython 2.7.  Maybe we should?

Another example where this same bug occurs is:

class T(tuple):
   def __radd__(self, other):
  return 42

lst = [ ]
lst += T()

which calls T.__radd__ in contradiction to all the general rules.
(Yes, if you print(lst) afterwards, you get 42.  And oops, trying this
out on PyPy does not give 42; only lst + T() does.  Probably another
corner case to fix...)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 488: elimination of PYO files

[Armin Rigo]

Hi Brett,

On 6 March 2015 at 19:11, Brett Cannon br...@python.org wrote:
 I disagree with your premise that .pyo files don't have a noticeable effect
 on performance. If you don't use asserts a lot then there is no effect, but
 if you use them heavily or have them perform expensive calculations then
 there is an impact.

Maybe you'd be interested to learn that PyPy (at least the 2.x branch)
uses a new bytecode, JUMP_IF_NOT_DEBUG, to conditionally jump over the
assert line.  In optimized mode PyPy follows the jumps; in
non-optimized mode it doesn't.  This mode is initialized with the -O
flag but can be changed dynamically, as the bytecode is the same.  We
introduced it as a simple solution to the mess of .pyc versus .pyo.
(We didn't consider the case of -OO very closely because PyPy is much
bigger than CPython as a binary to start with, so the demand for that
is lower.)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.

[Armin Rigo]

Hi Tim,

On 10 March 2015 at 18:22, Tim Peters tim.pet...@gmail.com wrote:
 1. Merge 2 at a time instead of just 1.  That is, first sort the
 next 2 elements to be merged (1 compare and a possible swap).  Then
 binary search to find where the smaller belongs, and a shorter binary
 search to find where the larger belongs.  Then shift both into place.

Good idea, but when I tried that it seemed to increase the total
number of comparisons (on random inputs with only up to 136 items).
The increase is on the order of 5%.  I'm not sure reduced data
movement can compensate for that in Python.

Test and code available here:
https://bitbucket.org/arigo/arigo/src/default/hack/pypy-hack/list_sort/

The change to insert two items at a time is here:
https://bitbucket.org/arigo/arigo/commits/68e04d143dc242cfd9e3934451321f685a68a8e2

(This is taken straight from PyPy's code.)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Encoding of PyFrameObject members

[Armin Rigo]

Hi,

On 6 February 2015 at 08:24, Maciej Fijalkowski fij...@gmail.com wrote:
 I don't think it's safe to assume f_code is properly filled by the
 time you might read it, depending a bit where you find the frame
 object. Are you sure it's not full of garbage?

Yes, before discussing how to do the utf8 decoding, we should realize
that it is really unsafe code starting from the line before.  From a
signal handler you're only supposed to read data that was written to
volatile fields.  So even PyEval_GetFrame(), which is done by
reading the thread state's frame field, is not safe: this is not a
volatile.  This means that the compiler is free to do crazy things
like *first* write into this field and *then* initialize the actual
content of the frame.  The uninitialized content may be garbage, not
just NULLs.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 468 (Ordered kwargs)

[Armin Rigo]

Hi all,

On 24 January 2015 at 11:50, Maciej Fijalkowski fij...@gmail.com wrote:
 I would like to point out that we implemented rhettingers idea in PyPy
 that makes all the dicts ordered by default and we don't have any
 adverse performance effects (in fact, there is quite significant
 memory saving coming from it). The measurments on CPython could be
 different, but in principle OrderedDict can be implemented as
 efficiently as normal dict.

I would like to add that http://bugs.python.org/issue16991 is the same
as today's dicts with an additional doubly-linked list for the order.
I'm unsure why you would do that after the 2012 thread started by
Raymond Hettinger, but anyway, don't conclude from only this that in
the CPython case ordered dictionaries would be slower and bigger.  My
guess is that, with a simple port of what is now in PyPy, they would
not be (but of course no-one can be sure at this point).  Let's say,
if you could imagine that CPython's dictionaries, tomorrow, are always
magically fully ordered, then would it still be a bad idea?

If such a discussion would resurface (soon or one day), and if other
related issues are resolved (like what to do in Jython and
IronPython), and if the conclusion would tentatively turn out
positive... then, provided there would at that point still be no
Raymond-style implementation of dicts, I would volunteer to port
PyPy's one to CPython[1].  As you may have guessed I don't consider
this particularly likely to occur, but it is a standing offer
nevertheless :-)


A bientôt,

Armin.


[1]  Someone could also do such a port for the goal of getting an
alternate `odictobject.c`.  He would be welcome to #pypy to get some
help from the PyPy guys, including me --- but my offer above doesn't
apply in this case.  I want to remove a thorn in the foot of
python-dev discussing about the language; I'm not really interested in
contributing to the `collections.OrderedDict` type.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

[Armin Rigo]

Hi all,

On 1 January 2015 at 14:52, Maciej Fijalkowski fij...@gmail.com wrote:
 PS. I wonder who came up with the idea first, PHP or rhettinger and
 who implemented it first (I'm pretty sure it was used in hippy before
 it was used in Zend PHP)

We'd need to look more in detail to that question, but a quick look
made me find this Java code from 2012:

https://code.google.com/r/wassermanlouis-guava/source/browse/guava/src/com/google/common/collect/CompactHashMap.java?name=refs/remotes/gcode-clone/compact-maps

which implements almost exactly the original idea of Raymond.  (It has
a twist because Java doesn't support arrays of (int, int, Object,
Object), and instead encodes it as one array of long and one array of
Objects.  It also uses a chain of buckets instead of open addressing.)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] results of id() and weakref.getweakrefs() sometimes break on object resurrection

[Armin Rigo]

Hi Stefan,

On 26 October 2014 02:50, Stefan Richthofer stefan.richtho...@gmx.de wrote:
 It appears weakrefs are only cleared if this is done by gc (where no
 resurrection can happen anyway). If a resurrection-performing-__del__ is
 just called by ref-count-drop-to-0, weakrefs persist -

How do you reach this conclusion?  The following test program seems to
show the opposite, by printing None on Python 2.7.6:

import weakref
   class X(object):
def __del__(self):
print ref()
x = X()
ref = weakref.ref(x)
del x


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] https:bugs.python.org -- Untrusted Connection (Firefox)

[Armin Rigo]

Hi,

On 18 August 2014 22:30, Oleg Broytman p...@phdru.name wrote:
Aha, I see now -- the signing certificate is CAcert, which I've
 installed manually.

I don't suppose anyone is particularly annoyed by this fact?  I know
for sure two classes of people that will never click Ignore.  The
first one is people that, for lack of a less negative term, I'll call
security freaks.  The second is serious business people to which
the shiny new look of python.org appeals; they are likely to heed the
warning Legitimate banks, stores, etc. will never ask you to do this
and would regard an official hint to ignore it as highly
unprofessional.

(The bug tracker of PyPy used to have the same problem.  We fixed the
situation recently, but previously, we used to argue that we didn't
have a lot of connections with either class of people...)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[Armin Rigo]

Hi all,

The core of the matter is that if we repeatedly __add__ strings from a
long list, we get O(n**2) behavior.  For one point of view, the
reason is that the additions proceed in left-to-right order.  Indeed,
sum() could proceed in a more balanced tree-like order: from [x0, x1,
x2, x3, ...], reduce the list to [x0+x1, x2+x3, ...]; then repeat
until there is only one item in the final list.  This order ensures
that sum(list_of_strings) is at worst O(n log n).  It might be in
practice close enough from linear to not matter.  It also improves a
lot the precision of sum(list_of_floats) (though not reaching the same
precision levels of math.fsum()).


Just a thought,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multiline with statement line continuation

[Armin Rigo]

Hi,

On 12 August 2014 01:08, Allen Li cyberdup...@gmail.com wrote:
 with (open('foo') as foo,
   open('bar') as bar,
   open('baz') as baz,
   open('spam') as spam,
   open('eggs') as eggs):
 pass

+1.  It's exactly the same grammar extension as for from import
statements, for the same reason.


Armin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

[Armin Rigo]

Hi Larry,

On 10 August 2014 08:11, Larry Hastings la...@hastings.org wrote:
 A small tip from my bzr days - cd into the directory before scanning it

 I doubt that's permissible for a library function like os.scandir().

Indeed, chdir() is notably not compatible with multithreading.  There
would be a non-portable but clean way to do that: the functions
openat() and fstatat().  They only exist on relatively modern Linuxes,
though.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 469: Restoring the iterkeys/values/items() methods

[Armin Rigo]

Hi Nick,

On 21 April 2014 07:39, Nick Coghlan ncogh...@gmail.com wrote:
 Notably, I recommend that hybrid code avoid calling mapping iteration
 methods directly, and instead rely on builtin functions where possible,
 and some additional helper functions for cases that would be a simple
 combination of a builtin and a mapping method in pure Python 3 code, but
 need to be handled slightly differently to get the exact same semantics in
 Python 2.

How about explicitly noting that in Python 2, a large fraction of
usages of the iterkeys(), itervalues() and iteritems() methods (that's
more than 99% in my experience, but I might be biased) should just be
written as keys(), values() and items() in the first place, with no
measurable difference of performance or memory usage?  I would
recommend to anyone with a large Python 2 code base to simply do a
textual search-and-replace and see if any test breaks.  If not,
problem solved.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dict and required hashing

[Armin Rigo]

Hi Jim,

On 18 April 2014 23:46, Jim J. Jewett jimjjew...@gmail.com wrote:
 (2)  Is the item will be hashed at least once a language guarantee?

I think that a reasonable implementation needs to hash at least once
all keys that are added to the dictionary.  Otherwise we end up, as
you said, with a dictionary that contains non-hashable keys: this
crashes when it grows and tries to convert from a list storage to a
hash table.  This is against the expectations and probably against the
language spec...

It's a bit unclear to me if the language spec requires if [ ] in
mydict: to raise TypeError, as the key doesn't get added to the dict
in this case.  Similarly for [ ] in { } where the dictionary is
empty anyway.  I would say yes, erring on the side of caution...


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Language Summit notes

[Armin Rigo]

Hi,

On 11 April 2014 19:55, Maciej Fijalkowski fij...@gmail.com wrote:
 Thanks, that clarification helps a lot. Does this mean that API-mode
 CFFI is competing with things like swig (which is not used much these
 days, as far as I know) and Cython (which is used a lot in the numeric
 community)? (ABI-mode CFFI is obviously directly competing with
 ctypes).

 Yes.

Funny how a high-level shift --- not really changing anything under
the hood --- changes how we look at a project :-)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Language Summit notes

[Armin Rigo]

Hi,

On 10 April 2014 22:12, Paul Moore p.f.mo...@gmail.com wrote:
 I agree. I'd like to see a clear explanation of what advantages (and
 disadvantages!) CFFI gives over ctypes, as well as the plan for
 inclusion and how the inevitable confusion over whether to use ctypes
 or cffi will be handled.  (...)

I can't judge exactly what was told in the Language Summit, but here
is my own position about CFFI. Code-wise, we're in precisely the same
spot as last year.  The usage of CFFI seems to be growing a lot.
However, it's not in any stdlib-ready state right now.

Why not?  Because we have a plan to go forward and fix the main issues
people seem to be having: when used in API mode there is some
building-C-sources-and-compiling-them going on under your feet;
however, explicit is better than implicit seems to apply here too.
Thus, it seems that the basic model might be changed toward a variant
in which you put your C declarations into some separate file that you
need to execute once, in order to build and compile a regular C
extension module.

This would be superficial, but change the perception of CFFI to be a
preprocessor that produces C extension modules.

This affects the API mode but not the ABI mode (which is basically
the same as ctypes, modulo the syntax).


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] version numbers mismatched in google search results.

[Armin Rigo]

Hi,

On 25 January 2014 17:26, Georg Brandl g.bra...@gmx.net wrote:
 Yep, and the URLs without version never served Python 3 docs as far as I can
 remember, so I don't know where Google has these titles from.

My guess would be that it's the title of the page that we (now) get
from the url http://docs.python.org/ .  Only my 2 cents, but this
bug as reported by Vincent Davis might be worth a workaround...


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4)

[Armin Rigo]

Hi Vajrasky,

On 28 January 2014 03:05, Vajrasky Kok sky@speaklikeaking.com wrote:
 I get your point. But strangely enough, I can still recover from
 list(repeat('a', 2**29)). It only slows down my computer. I can ^Z the
 application then kill it later. But with list(repeat('a', times=-1)),
 rebooting the machine is compulsory.

Actually you get the early OverflowError if the value doesn't fit a C
long, and any value up to sys.maxint gets past this check.  Try with
2**31-1.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: threading.RLock._acquire_restore() now raises a TypeError instead of a

[Armin Rigo]

Hi Serhiy,

On Fri, Jan 3, 2014 at 8:59 AM, Serhiy Storchaka storch...@gmail.com wrote:
 +if (!PyArg_ParseTuple(args, (kl):_acquire_restore, count, owner))

 Please don't use (...) in PyArg_ParseTuple, it is dangerous (see issue6083

I think that in this case it is fine, because the k and l are
returning C integers.  The refcounting issue occurs only when
PyObject* are returned.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_uuid.py on HP NonStop Python-2.7.5 fails (test case: testIssue8621)

[Armin Rigo]

Hi,

On Thu, Nov 28, 2013 at 7:14 AM, V S, Nagendra (Nonstop Filesystems Team)
 on NonStop uuid.py falls throw to  random.range() call to generate the 
 random number

And indeed, the random module gets identical results in both parent
and child after a fork().  This seems to mean that the random module
is not safe to use in uuid without taking special care of this fact
(e.g. feeding back os.getpid() into the seed).


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unicode Exception messages in py2.7

[Armin Rigo]

Hi,

FWIW, the pure Python traceback.py module has a slightly different
(and saner) behavior:

 e = Exception(uxx\u1234yy)
 traceback.print_exception(Exception, e, None)
Exception: xx\u1234yy

I'd suggest that the behavior of the two should be unified anyway.
The traceback module uses value.encode(ascii, backslashreplace)
for any unicode object.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unicode Exception messages in py2.7

[Armin Rigo]

Hi again,

I figured that even using the traceback.py module and getting
Exception: \u1234\u1235\u5321 is rather useless if you tried to
raise an exception with a message in Thai.  I believe this to also be
a bug, so I opened https://bugs.pypy.org/issue1634 .  According to
this thread, however, python-dev is against it, so I didn't bother
adding a CPython bug.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] An interesting exception handling quirk

[Armin Rigo]

Hi Nick,

On Sat, Oct 19, 2013 at 1:41 PM, Nick Coghlan ncogh...@gmail.com wrote:
 recreating the *exact* exception subclass check from
 Python is actually difficult these days.

Can't it be done roughly like that?

   def __exit__(self, typ, val, tb):
try:
raise typ, val
except self.exceptions:
return True
except:
return False


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make str/bytes hash algorithm pluggable?

[Armin Rigo]

Hi Guido,

On Thu, Oct 3, 2013 at 10:47 PM, Guido van Rossum gu...@python.org wrote:
 Sounds a bit like some security researchers drumming up business. If you can
 run the binary, presumably you can also recover the seed by looking in
 /proc, right? Or use ctypes or something. This demonstration seems of
 academic interest only.

I'll not try to defend the opposite point of view very actively, but
let me just say that, in my opinion, your objection is not valid.  It
is broken the same way as a different objection, which would claim
that Python can be made sandbox-safe without caring about the numerous
segfault cases.  They are all very obscure for sure; I tried at some
point to list them in Lib/test/crashers.  I gave up when people
started deleting the files because they no longer crashed on newer
versions, just because details changed --- but not because the general
crash they explained was in any way fixed...  Anyway, my point is that
most segfaults can, given enough effort, be transformed into a single,
well-documented tool to conduct a large class of attacks.

The hash issue is similar.  It should be IMHO either ignored (which is
fine for a huge fraction of users), or seriously fixed by people with
the correctly pessimistic approach.  The current hash randomization is
simply not preventing anything; someone posted long ago a way to
recover bit-by-bit the hash randomized used by a remote web program in
Python running on a server.  The only benefit of this hash
randomization option (-R) was to say to the press that Python fixed
very quickly the problem when it was mediatized :-/

This kind of security issues should never be classified as academic
interest only.  Instead they can be classified as it will take weeks
/ months / years before some crazy man manages to put together a
general attack script, but likely, someone will eventually.

From this point of view I'm saluting Christian's effort, even if I
prefer to stay far away from this kind of issues myself :-)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Revert #12085 fix for __del__ attribute error message

[Armin Rigo]

Hi Nick,

On Thu, Sep 26, 2013 at 6:59 AM, Nick Coghlan ncogh...@gmail.com wrote:
 I'm strongly in favour of Georg's one (Exception in __del__ caught and not 
 propagated).

 Such a change is highly unlikely to happen, as it would require
 changing every location where we call PyErr_WriteUnraisable.

Er, why?  It seems to me it's a matter of changing these three lines
in PyErr_WriteUnraisable():

-   PyFile_WriteString(Exception ignored in: , f);
+  PyFile_WriteString(Exception in , f);
PyFile_WriteObject(obj, f, 0);
-   PyFile_WriteString(\n, f);
+  PyFile_WriteString( caught and not propagated:\n, f);

I don't see what makes this technically different from the other
solution, Cannot propagate exception...


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 447: add type.__locallookup__

[Armin Rigo]

Hi Mark,

On Mon, Sep 9, 2013 at 11:18 PM, Mark Shannon m...@hotpy.org wrote:
 5. Other implementations. What do the Jython/IronPython/PyPy developers
 think?

Thanks for asking :-)  I'm fine with staying out of language design
issues like this one, and I believe it's the general concensus in
PyPy.  Whatever gets decided will probably be easy to port to PyPy and
have no measurable performance impact: the extra checks on the type,
if any, are all constant-folded by the JIT.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a transformdict to collections

[Armin Rigo]

Hi Richard,

On Tue, Sep 10, 2013 at 3:42 PM, Richard Oudkerk shibt...@gmail.com wrote:
 I guess another example is creating an identity dict (see
 http://code.activestate.com/lists/python-ideas/7161/) by doing

 d = transformdict(id)

This is bogus, because only the id will be stored, and the original
key object will be forgotten (and its id probably reused).


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] inspect and metaclasses

[Armin Rigo]

Hi Ethan,

Are you suggesting that inspect.get_mro(A) would return (A, object,
type)?  That seems very wrong to me.

If the goal is to fix `inspect.classify_class_attrs()`, then this
function only needs a specific fix, along the lines of looking in
`get_mro(A) + get_mro(type(A))`.  (A more minor issue is that the bug
report suggests `... + (type(A),)` only, but that's wrong: Python will
also look in all the base classes of type(A).)

Fixing inspect.get_mro() as suggested would break a lot of other usages of it.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] hg verify warnings

[Armin Rigo]

Hi Tim,

On Tue, Aug 20, 2013 at 5:12 PM, Tim Peters tim.pet...@gmail.com wrote:
 Try running hg verify -v - these warnings only appear when verify is
 run in verbose mode.

Indeed.  Ignore what I said then about a broken copy of the
repository: any copy will show these three warnings, and they can be
safely ignored it seems.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] hg verify warnings

[Armin Rigo]

Hi Tim,

On Tue, Aug 20, 2013 at 1:48 AM, Tim Peters tim.pet...@gmail.com wrote:
 warning: copy source of 'Modules/_threadmodule.c' not in parents of 
 60ad83716733
 warning: copy source of 'Objects/bytesobject.c' not in parents of 
 64bb1d258322
 warning: copy source of 'Objects/stringobject.c' not in parents of 
 357e268e7c5f

I've seen this once already (with another big repository).  The
problem I had was only these warnings when running hg verify, and it
was fixed by simply checking out a new copy of the repository.  It
seems that you have the same problem: for example my own copy of
CPython doesn't show any warning in hg verify.

I've deleted my slightly-broken repo by then (as it was already
several years old I suspected an old version of mercurial).  Maybe I
shouldn't have.  How about you send your repository to the Mercurial
bug tracker?


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Use a known unique object for the dummy entry.

[Armin Rigo]

Hi,

On Sat, Aug 17, 2013 at 8:42 PM, Antoine Pitrou solip...@pitrou.net wrote:
 summary:
   Use a known unique object for the dummy entry.

Another issue with this change: the dummy object should be of a dummy
subclass of 'object', rather than of 'object' itself.  When it is
'object' itself, a custom __eq__() method will be called, sending what
should be the dummy object to the pure Python code explicitly, as in
the example below.  This is bad because ---in addition to calling
__eq__() with unexpected arguments, which might break some code--- we
could then take the dummy object, and try to insert it into another
set...

class A(object):
def __init__(self, hash):
self.hash = hash
def __eq__(self, other):
print(seen!, self, other)
return False
def __hash__(self):
return self.hash

a1 = A(1)
a2 = A(2)
s = {a1, a2}
s.remove(a2)
A(2) in s


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks

[Armin Rigo]

Hi Arnaud,

On Mon, Aug 12, 2013 at 9:39 AM, Arnaud Fontaine
arnaud.fonta...@nexedi.com wrote:
   Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a
   package containing dynamic modules) handled by Import Hooks I
   implemented, so import lock is acquired before even running the hooks
   (Python/import.c:PyImport_ImportModuleLevel()). Then, these import
   hooks try to load objects from ZODB and a request is sent and handled
   by another thread (Thread 2) which itself tries to import another
   module.

A quick hack might be to call imp.release_lock() and
imp.acquire_lock() explicitly, from your import hook code, around
calls to ZODB.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The Return Of Argument Clinic

[Armin Rigo]

Hi Larry,

On Mon, Aug 5, 2013 at 10:48 AM, Larry Hastings la...@hastings.org wrote:
 Question 4: Return converters returning success/failure?

The option generally used elsewhere is: if we throw an exception, we
return some special value; but the special value doesn't necessarily
mean by itself that an exception was set.  It's a reasonable solution
because the caller only needs to call PyErr_Occurred() for one special
value, rather than every time.  See for example any call to
PyFloat_AsDouble().


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] stat module in C -- what to do with stat.py?

[Armin Rigo]

Hi,

On Fri, Jun 21, 2013 at 9:20 PM, Steven D'Aprano st...@pearwood.info wrote:
 process. Personally, I don't see the value in it; other implementations
 will need to do *something* special to use it anyway.

 That's not correct. Other implementations can do exactly what CPython 3.3
 does, namely just use stat.py as given. Not all implementations necessarily
 care about multiple platforms where stat constants are likely to change.

Thanks Steven.  That's PyPy's position: for now we really care only
about insert some list of common platform there where stat.py has
been obviously sufficient, as shown by CPython's long experience with
stat.py.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 443 - Single-dispatch generic functions

[Armin Rigo]

Hi,

On Thu, May 23, 2013 at 12:33 AM, Łukasz Langa luk...@langa.pl wrote:
 Alternative approaches
 ==

You could also mention pairtype, used in PyPy:
https://bitbucket.org/pypy/pypy/raw/default/rpython/tool/pairtype.py
(very short code).  It's originally about adding double-dispatch, but
the usage that grew out of it is for generic single-dispatch functions
that are bound to some common state object as follows (Python 2
syntax):

class MyRepr(object):
 ...state of my repr...

class __extend__(pairtype(MyRepr, int)):
def show((myrepr, x), y):
 print hi, I'm the integer %d, arg is %s % (x, y)

class __extend__(pairtype(MyRepr, list)):
def show((myrepr, x), y):
 print hi, I'm a list
 ...use myrepr to control the state...

pair(MyRepr(), [2,3,4]).show(42)


- Armin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Ordering keyword dicts

[Armin Rigo]

Hi all,

On Sun, May 19, 2013 at 4:59 PM, Maciej Fijalkowski fij...@gmail.com wrote:
 Note that raymonds proposal would make dicts and ordereddicts almost
 exactly the same speed.

Just checking: in view of Raymond's proposal, is there a good reason
against  having all dicts be systematically ordered?  It would
definitely improve the debugging experience, by making multiple runs
of the same program more like each other, instead of depending on the
random address-based ordering.  (Performance-wise, I guess it might be
a little bit slower or faster depending on cache issues and so on, but
the emphasis I'd put is on the little bit.)

I apologize if this was already shot down.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 442: Safe object finalization

[Armin Rigo]

Hi Antoine,

On Sat, May 18, 2013 at 10:59 AM, Antoine Pitrou solip...@pitrou.net wrote:
 Cyclic isolate (CI)
 A reference cycle in which no object is referenced from outside the
 cycle *and* whose objects are still in a usable, non-broken state:
 they can access each other from their respective finalizers.

Does this definition include more complicated cases?  For example:

A - B - Aand   A - C - A

Neither cycle is isolated.  If there is no reference from outside,
then the set of all three objects is isolated, but isn't strictly a
cycle.  I think the term is strongly connected component.

 1. Weakrefs to CI objects are cleared, and their callbacks called. At
this point, the objects are still safe to use.

 2. **The finalizers of all CI objects are called.**

You need to be very careful about what each call to a finalizer can do
to the object graph.  It may already be what you're doing, but the
most careful solution is to collect in 1. the complete list of
objects with finalizers that are in cycles; then incref them all; then
call the finalizer of each of them; then decref them all.  Such a
solution gives new cases to think about, which are slightly unexpected
for CPython's model: for example, if you have a cycle A - B - A,
let's say the GC calls A.__del__ first; it might cause it to store a
reference to B somewhere else, e.g. in some global; but then the GC
calls B.__del__ anyway.  This is probably fine but should be
considered.

 3. **The CI is traversed again to determine if it is still isolated.

How is this done?  I don't see a clear way to determine it by looking
only at the objects in the CI, given that arbitrary modifications of
the object graph may have occurred.  The solution I can think of
doesn't seem robust against minor changes done by the finalizer.  Take
the example A - lst - B - A, where the reference from A to B is
via a list (e.g. there is an attribute A.attr = [B]).  If A.__del__
does the seemingly innocent change of replacing the list with a copy
of itself, e.g. A.attr = A.attr[:], then after the finalizers are
called, lst is gone and we're left with A - lst2 - B - A.
Checking that this cycle is still isolated requires a possibly large
number of checks, as far as I can tell.  This can lead to O(n**2)
behavior if there are n objects in total and O(n) cycles.

The solution seems to be to simply wait for the next GC execution.
Assuming that a finalizer is only called once, this only delays a bit
freeing objects with finalizers in cycles (but your PEP still works to
call finalizers and eventually collect the objects).  Alternatively,
this might be done immediately: in the point 3. above we can forget
everything we found so far, and redo the tracking on all objects (this
time ignoring finalizers that were already called).  In fact, it may
be necessary anyway: anything found before might be invalid after the
finalizers are called, so forgetting it all and redoing the tracking
from scratch seems to be the only way.

 Type objects get a new ``tp_finalize`` slot to which ``__del__`` methods
 are bound.  Generators are also modified to use this slot, rather than
 ``tp_del``.  At the C level, a ``tp_finalize`` function is a normal
 function which will be called with a regular, alive object as its only
 argument.  It should not attempt to revive or collect the object.

Do you mean the opposite in the latest sentence?  ``tp_finalize`` can
do anything...


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 442: Safe object finalization

[Armin Rigo]

Hi Antoine,

On Sat, May 18, 2013 at 3:45 PM, Antoine Pitrou solip...@pitrou.net wrote:
 How is this done?  I don't see a clear way to determine it by looking
 only at the objects in the CI, given that arbitrary modifications of
 the object graph may have occurred.

 The same way a generation is traversed, but restricted to the CI.

 First the gc_refs field of each CI object is initialized to its
 ob_refcnt (again).

 Then, tp_traverse is called on each CI object, and each visited
 CI object has its gc_refs decremented. This substracts CI-internal
 references from the gc_refs fields.

 At the end of the traversal, if all CI objects have their gc_refs equal
 to 0, then the CI has no external reference to it and can be cleared.
 If at least one CI object has non-zero gc_refs, the CI cannot be
 cleared.

Ok, indeed.  Then you really should call finalizers only once: in case
one of the finalizers in a cycle did a trivial change like I
described, the algorithm above will conservatively assume the cycle
should be kept alive.  At the next GC collection we must not call the
finalizer again, because it's likely to just do a similar trivial
change.

(There are other open questions about calling finalizers multiple
times; e.g. an instance of this class has its finalizer called ad
infinitum and leaks, even though X() is never part of any cycle:

class X(object):
   def __del__(self):
  print tick
  lst = [self]
  lst.append(lst)

Try interactively: every gc.collect() prints tick, even if you make
only one instance.)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mysterious Python pyc file corruption problems

[Armin Rigo]

Hi all,

How about using the shared-or-exclusive advisory file locks (with
flock() or fcntl())?  It may only work on Posix though.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fighting the theoretical randomness of is on immutables

[Armin Rigo]

Hi Antoine,

On Tue, May 7, 2013 at 8:25 AM, Antoine Pitrou solip...@pitrou.net wrote:
 For me, a patch that mandated general-purpose containers (list, dict,
 etc.) respect object identity would be ok.

Thanks, that's also my opinion.

In PyPy's approach, in trying to emulate CPython vs. trying to
convince users that is is sometimes a bad idea, we might eventually
end up at the extreme side, which can be seen as where CPython would
be if it cached *all* ints, longs, floats, complexes, strings,
unicodes and tuples.

The original question in this thread was about if it's ok for two
objects x and y to satisfy x is y while at the same time id(x) !=
id(y).  I think by now that it would only create more confusion (even
if only in some very special cases).  We'll continue to maintain the
invariant then, and if it requires creating extremely large values for
id(), too bad. (1)


A bientôt,

Armin.


(1) the Jython approach of caching the id's is not applicable here:
the objects whose id are hard to get are precisely those that don't
have a long-living representation as object in memory.  You can't
cache an id with a key that is, say, a double-word long --- if this
double-word is not an object, but merely a value, it can't be used as
key in a weakdict.  You don't have a way of knowing when you can
remove it from the cache.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Fighting the theoretical randomness of is on immutables

[Armin Rigo]

Hi all,

In the context PyPy, we've recently seen again the issue of x is y
not being well-defined on immutable constants.  I've tried to
summarize the issues and possible solutions in a mail to pypy-dev [1]
and got some answers already.  Having been convinced that the core is
a language design issue, I'm asking for help from people on this list.
 (Feel free to cross-post.)

[1] http://mail.python.org/pipermail/pypy-dev/2013-May/011299.html

To summarize: the issue is a combination of various optimizations that
work great otherwise.  For example we can store integers directly in
lists of integers, so when we read them back, we need to put them into
fresh W_IntObjects (equivalent of PyIntObject).  We solved temporarily
the issue of I'm getting an object which isn't ``is``-identical to
the one I put in! by making all equal integers ``is``-identical.
This required hacking at ``id(x)`` as well to keep the requirement ``x
is y = id(x)==id(y)``.  This is getting annoying for strings, though
-- how do you compute the id() of a long string?  Give a unique long
integer?  And if we do the same for tuples, what about their id()?

The long-term solution that seems the most stable to me would be to
relax the requirement ``x is y = id(x)==id(y)``.  If we can get away
with only ``x is y = id(x)==id(y)`` then it would allow us to
implement ``is`` in a consistent way (e.g. two strings with equal
content would always be ``is``-identical) while keeping id()
reasonable (both in terms of complexity and of size of the resulting
long number).  Obviously ``x is y = id(x)==id(y)`` would still be
true if any of ``x`` or ``y`` is not an immutable by-value built-in
type.

This is clearly a language design issue though.  I can't really think
of a use case that would break if we relax the requirement, but I
might be wrong.  It seems to me that at most some modules like pickle
which use id()-keyed dictionaries will fail to find some
otherwise-identical objects, but would still work (even if tuples are
relaxed in this way, you can't have cycles with only tuples).


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Difference in RE between 3.2 and 3.3 (or Aaron Swartz memorial)

[Armin Rigo]

Hi Matej,

On Thu, Mar 7, 2013 at 11:08 AM, Matej Cepl mc...@redhat.com wrote:
  if c is not ' ' and c is not '  ':
 if c != ' ' and c != ' ':

Sorry for the delay in answering, but I just noticed what is wrong in
this fix: it compares c with the same single-character ' ' twice,
whereas the original compared it with ' ' and with the two-character '
 '.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Destructors and Closing of File Objects

[Armin Rigo]

Hi Jeff,

On Mon, Apr 29, 2013 at 11:58 PM, Jeff Allen ja...py@farowl.co.uk wrote:
 In Jython, (...)

Thanks Jeff for pointing this out.  Jython thus uses a custom
mechanism similar to PyPy's, which is also similar to atexit's.  It
should not be too hard to implement it in CPython 3 as well, if this
ends up classified as a bug.  This is what my bug report was about
(sorry if I failed to be clear enough about it).

Nikolaus: the bug report contains a failing test, is that what you're
looking for?


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Destructors and Closing of File Objects

[Armin Rigo]

Hi Nikolaus,

On Sat, Apr 27, 2013 at 4:39 AM, Nikolaus Rath nikol...@rath.org wrote:
 It's indeed very informative, but it doesn't fully address the question
 because of the _pyio module which certainly can't use any custom C code.
 Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose
 data in the write buffer when the interpreter exits without me calling
 x.close(), but when using x = io.BufferedWriter(), the buffer is
 guaranteed to get flushed?

I actually described the behavior of CPython 2 while not realizing
that CPython 3 silently dropped this guarantee.  (I also never
realized that Jython/IronPython don't have the same guarantee; they
could, if they implement 'atexit', like we did in PyPy.  That's
however more acceptable if the platform itself doesn't offer the
guarantee.)

Anyway, it's a guarantee that the C offers, so personally I find it
reasonable to expect CPython files to offer it too; not offering it is
kind of saying that there is a feature of C that is actually present
at a higher level than the exact same feature in Python, which looks
backward to me.

Additionally, this might be introducing subtle bugs in programs when
porting them to Python 3.

However I realize that the two arguments presented above might not be
accepted as relevant.  (http://bugs.python.org/issue17852)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] py2.7: dictobject not properly resizing

[Armin Rigo]

Hi Antoine,

On Sat, Mar 30, 2013 at 10:37 PM, Antoine Pitrou solip...@pitrou.net wrote:
 On Sat, 30 Mar 2013 17:31:26 -0400
 Micha Gorelick mynameisfi...@gmail.com wrote:
 I was taking a look at dictobject.c and realized that the logic
 controlling whether a resizedict will occur in
 dict_set_item_by_hash_or_entry disallows for the shrinking of a
 dictionary.

It doesn't disallow shrinking.  If you take a dictionary of size 1000,
remove of its elements, and continue to use it (write and delete more
items) for long enough, then eventually it is shrinked.  It just takes
a while because it needs to fill 2/3 of the slots of the big table
with deleted markers before it happens.

Python 3.3.0b1 (default:07ddf5ecaafa, Aug 12 2012, 17:47:28)
[GCC 3.4.6 (Gentoo 3.4.6-r2, ssp-3.4.6-1.0, pie-8.7.10)] on linux
Type help, copyright, credits or license for more information.
 d = {i:i for i in range(1000)}
 sys.getsizeof(d)
24624
 for i in range(1000): del d[i]
...
 sys.getsizeof(d)
24624
 for j in range(1001, 2000):
...d[j] = 0; del d[j]
...
 sys.getsizeof(d)
144



A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Armin Rigo]

Hi Gregory,

On Sat, Mar 2, 2013 at 8:40 AM, Gregory P. Smith g...@krypto.org wrote:

 On Wed, Feb 27, 2013 at 7:57 AM, Eli Bendersky eli...@gmail.com wrote:
 So would you say that the main use of the API level is provide an
 alternative for writing C API code to interface to C libraries. IOW, it's in
 competition with Swig?

 I'd hardly call it competition. The primary language I interface with is C++
 and cffi appears not see that giant elephant in the room

I don't think it's in competition with Swig, which does C++.  There
are certain workloads in which C++ is the elephant in the room; we
don't address such workloads.  If you want some more motivation, the
initial goal was to access the large number of standard Linux/Posix
libraries that are C (or have a C interface), but are too hard to
access for ctypes (macros, partially-documented structure types,
#define for constants, etc.).  For this goal, it works great.

 (it'd need to use clang for parsing if it were going to do that)...

I fear parsing is merely the tip of the iceberg when we talk about
interfacing with C++.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Armin Rigo]

Hi Stefan,

On Sat, Mar 2, 2013 at 10:10 AM, Stefan Behnel stefan...@behnel.de wrote:
 You say that the API is fairly stable. What about the implementation?
 Will users want to install a new version next to the stdlib one in a couple
 of months,

I think that the implementation is fairly stable as well.  The only
place I can foresee some potential changes is in details like the
location of temporary files, for example, which needs to be discussed
(probably with people from python-dev too) as some point.

 just because there was a bug in the parser in Python 3.4 that
 you still need to support because there's code that depends on it, or
 because there is this new feature that is required to make it work with
 library X, or ... ? What's the upgrade path in that case? How will you
 support this? What long-term guarantees do you give to users of the stdlib
 package?

I think these are general questions for any package that ends up in
the stdlib.  In the case of CFFI, it is now approaching a stability
point.  This is also because we are going to integrate it with the
stdlib of PyPy soon.

Bugs in the parser have not been found so far, but if there is any, we
will treat it like we treat any other bug in the stdlib.  For that
matter, there is actually no obvious solution for the user either: he
generally has to wait for the next micro release to have the bug
fixed.

 Or, in other words, will the normal fallback import for cffi look like this:

 try: import stdlib_cffi
 except ImportError: import external_cffi

 or will the majority of users end up prefering this order:

 try: import external_cffi
 except ImportError: import stdlib_cffi

I would rather drop the external CFFI entirely, or keep it only to
provide backports to older Python versions.  I personally see no
objection to call the stdlib one cffi too (but any other name is
fine as well).

 ... Wouldn't it be simpler to target Windows with a binary than
 with dynamically compiled C code? Is there a way to translate an API
 description into a static ABI description for a known platform ahead of
 time, or do I have to implement this myself in a separate ABI code path by
 figuring out a suitable ABI description myself?

No, I believe that you missed this point: when you make binary
distributions of a package with setup.py, it precompiles a library for
CFFI too.  So yes, you need a C compiler on machines where you develop
the program, but not on machines where you install it.  It's the same
needs as when writing custom C extension modules by hand.

 In which cases would users choose to use the C API support? And, is this
 dependency explicit or can I accidentally run into the dependency on a C
 compiler for my code without noticing?

A C compiler is clearly required: this is if and only if you call the
function verify(), and pass it arguments that are not the same ones as
the previous time (i.e. it's not in the cache).

 * We try to be complete. For now some C99 constructs are not
 supported, but all C89 should be, including macros (and including
 macro “abuses”, which you can manually wrap in saner-looking C
 functions).

 Ok, so the current status actually is that it's *not* complete, and that
 future versions will have to catch up in terms of C compatibility. So, why
 do you think it's a good time to get it into the stdlib *now*?

To be honest I don't have a precise list of C99 constructs missing.  I
used to know of a few of them, but these were eventually supported.
It is unlikely to get completed, or if it is, a fairly slow process
should be fine --- just like a substantial portion of the stdlib,
which gets occasional updates from one Python version to the next.

 You mentioned that it's fast under PyPy and slow under CPython, though.
 What would be the reason to use it under CPython then?

The reason is just ease of use.  I pretend that it takes less effort
(and little C knowledge), and is less prone to bugs and leaks, to
write a perfectly working prototype of a module to access a random C
library.  I do not pretend that you'll get the top-most performance.
For a lot of cases performance doesn't matter; and when it does, on
CPython, you can really write a C extension module by hand (as long as
you make sure to keep around the CFFI version for use by PyPy).  This
is how I see it, anyway.  The fact that we are busy rewriting existing
native well-tested CPython extensions with CFFI --- this is really
only of use for PyPy.

 Others have already mentioned the lack of C++ support. It's ok to say that
 you deliberately only want to support C, but it's also true that that's a
 substantial restriction.

I agree that it's a restriction, or rather a possible extension that
is not done.  I don't have plans to do it myself.  Please also keep in
mind that we pitch CFFI as a better ctypes, not as the ultimate tool
to access any foreign language.


A bientôt,

Armin.
___
Python-Dev mailing list

Re: [Python-Dev] cffi in stdlib

[Armin Rigo]

Hi Neil,

On Thu, Feb 28, 2013 at 12:34 AM, Neil Hodgson nyamaton...@me.com wrote:
Wouldn't it be better to understand the SAL annotations like _In_opt so 
 that spurious NULLs (for example) produce a good exception from cffi instead 
 of failing inside the system call?

Maybe.  Feel like adding an issue to
https://bitbucket.org/cffi/cffi/issues, with references?  This looks
like a Windows-specific extension, which means that I don't
automatically know about it.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Armin Rigo]

Hi Paul,

On Thu, Feb 28, 2013 at 9:27 AM, Paul Moore p.f.mo...@gmail.com wrote:
 Presumably ffi.NULL isn't needed and I can use 0? (After all, 0 and NULL are 
 equivalent in C, so that's
 not a correctness issue).

Indeed.  I created
https://bitbucket.org/cffi/cffi/issue/61/convert-0-to-a-null-pointer.
In C, NULL is typically #define NULL (void *)0, not just 0, which
means that there are not 100% equivalent.  But it's true that in C you
can pass the constant 0 to mean a NULL pointer argument.  The fact
that it's not working this way in CFFI is definitely a bug.

 That's a slightly unfair example, because in this case it happens to
 work with ctypes without specifying the argtypes and the restype.  I
 would argue that this feature of ctypes is not a good thing: it's
 mostly the same as saying you only need to declare argtypes and
 restype if you get nonsense results or segfaults.

 That's a bit unfair. I'd say you only need to declare argtypes if
 you're dealing with things more complex than integers, strings and
 null pointers. Which means you're fine for a huge proportion of the
 Windows API.

Yes, you're right, and the 32-bit Windows platform is still important.
 However, it only works on 32-bit.  On typical 64-bit Posix
environments, if you don't declare argtypes/restype, you usually end
up very quickly with confusion between int and long.  And I think
that even on 64-bit Windows, passing 0 as a NULL pointer is buggy,
because it will pass a 32-bit 0.  (It may be that it doesn't actually
make a difference and works anyway, but I'm not sure.)  Similarly, a
function that returns a pointer (e.g. a handle on Windows) will not
work without an explicit restype on any 64-bit platform I know of.
Explicit is better than implicit...


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Armin Rigo]

Hi Guido,

On Tue, Feb 26, 2013 at 8:24 PM, Guido van Rossum gu...@python.org wrote:
 From a software engineering perspective, 10 years is indistinguishable
 from infinity, so I don't care what happens 10 years from now -- as
 long as you don't blame me. :-)

I can't resist: around today it is the 10th anniversary of PyPy.  :-)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Armin Rigo]

Hi Paul,

On Wed, Feb 27, 2013 at 7:24 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 27 February 2013 11:53, Maciej Fijalkowski fij...@gmail.com wrote:
 I think it means you can't use the ABI version and specify the calling
 convention. It's a reasonable bug report (the calling convention on
 API version works though)

 That would be a deal-breaker for my use case of quick scripts working
 with the Windows API on a machine with no C compiler. I'd have to use
 ctypes in that case until cffi had this feature.

That's not correct: you can't indeed give the calling convention, but
it is not needed for the common case.  What is not supported is only
Python-defined callbacks using the Windows-specific convention --- as
documented, there is a workaround for that case.

And, in case you wonder, this automatic detection comes from ctypes.
I copied the hacked-up version of libffi for Windows from ctypes to
cffi, and the logic to detect the actual calling convention and
support both is there.  The difference is only that in ctypes, after
it did the call (always successfully), it checks the convention that
was found to be used by the C function, and if it differs from the one
specified by the user, then it complains.  I basically just removed
the complaining part.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Armin Rigo]

Hi Paul,

On Wed, Feb 27, 2013 at 9:31 PM, Paul Moore p.f.mo...@gmail.com wrote:
 from ctypes import windll
 MessageBox = windll.User32.MessageBoxW
 MessageBox(0, Hello, world!, Title, 0)

You are right that it's a bit cumbersome in cffi up to and including
0.5, but in the cffi trunk all standard Windows types are included.
So the general answer to your question is: we google MessageBox and
copy that line from the microsoft site, and manually remove the
unnecessary WINAPI and _In_opt declarations:

from cffi import FFI
ffi = FFI()
ffi.cdef(
int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType);
)
lib = ffi.dlopen(USER32.DLL)
lib.MessageBox(ffi.NULL, Hello, world!, Title, 0)

That's a slightly unfair example, because in this case it happens to
work with ctypes without specifying the argtypes and the restype.  I
would argue that this feature of ctypes is not a good thing: it's
mostly the same as saying you only need to declare argtypes and
restype if you get nonsense results or segfaults.

Note for completeness that the version with verify() simply replaces
lib = ffi.dlopen(USER32.DLL) with lib = ffi.verify() (no
#include necessary here).  Then you cannot misdeclare anything without
getting clear compile-time errors at the verify().  The compilation
occurs only once (it's cached) and by writing two lines in your
setup.py you can distribute binary installations, just like you do
with hand-written C extension modules; so the extra burden of
accessing the API level is in my opinion very small compared to its
segfault-avoiding gain.  But I know that either level of access can
make sense in different situations.  Typically, on Windows, the
ABI-level works fine; as argued elsewhere, on other platforms and/or
with some libraries, the API-level is definitely more suited.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Armin Rigo]

Hi,

On Tue, Feb 26, 2013 at 5:38 PM, Maciej Fijalkowski fij...@gmail.com wrote:
 Do you intend to actually maintain it inside the CPython repository?

 Once we put it in, yes, of course. Me Armin and Alex.

Yes, I confirm. :-)

Armin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] efficient string concatenation (yep, from 2004)

[Armin Rigo]

Hi Greg,

On Wed, Feb 13, 2013 at 10:17 PM, Greg Ewing
greg.ew...@canterbury.ac.nz wrote:
 If it's that unreliable, why was it ever implemented
 in the first place?

I was young and loved hacks and python-dev felt that it was a good
idea at the time
(http://mail.python.org/pipermail/python-dev/2004-August/046686.html).


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com