Re: [Python-ideas] PEP 505: None-aware operators

[Tim Peters]

[Steve Dower ]

> ...

* The "``None``-aware attribute access" operator ``?.`` evaluates the
> complete expression if the left hand side evaluates to a value that is not
> ``None``
>

And if the LHS does evaluate to `None` ...?  I'll assume the result is also
`None` then.


> ...
>  From ``inspect.py``::
>
>  for base in object.__bases__:
>  for name in getattr(base, "__abstractmethods__", ()):
>  value = getattr(object, name, None)
>  if getattr(value, "__isabstractmethod__", False):
>  return True
>
> After updating to use the ``?.`` operator (and deliberately not
> converting to use ``any()``)::
>
>  for base in object.__bases__:
>  for name in base?.__abstractmethods__ ?? ():
>  if object?.name?.__isabstractmethod__:
>  return True
>

I got lost on the `for` here.  The part following `in`:

for name in getattr(base, "__abstractmethods__", ()):

looks in `base` (regardless of whether `base` is `None`) for an attribute
named "_abstractmethods__"..  If such an attribute exists, the value of the
attribute is returned (`None` or not).  Else an AttributeError is swallowed
and `()` is returned.  It's hard to see how

 for name in base?.__abstractmethods__ ?? ():

does the same.  If `base` itself is `None`, I guess it returns `()`, or
if  `base` has an "_abstractmethods__" attribute then the value of that
attribute is returned - unless its value is None, in which case `()` is
again returned.  But if `base` is not `None` and the attribute does not
exist, doesn't this raise AttributeError?  The later "Exception-aware
operators" section seemed to explicitly reject the idea that `?.` and `?[]`
would suppress AttributeError and/or TypeError.

In short, the original getattr() didn't care at all whether `base` was
`None`, or whether the value of its "__abstractmethods__" attribute was
`None`, but cared a whole lot about whether that attribute exists.  I just
can't see how the updated code matches that in any of those respects.

Ignoring that and pressing on, I suffer the same kind of confusions on the
`if` part.  What am I missing?  For example, do these operators swallow
exceptions after all?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Revisiting str.rreplace()

[Graham Gott]

This was previously proposed here in 2014 <
https://mail.python.org/pipermail/python-ideas/2014-January/025091.html>,
but the discussion fizzled out. To me, str.rreplace() is an obvious and
necessary complement to str.replace(), just as str.rsplit() is a complement
to str.split(). It would make Python closer to the goal of the Zen of
Python: "There should be one-- and preferably only one --obvious way to do
it." To support its usefulness, this question has been asked on Stack
Overflow a number of times, (, <
https://stackoverflow.com/q/14496006>, )
with more than a trivial number of votes for the answers (>100 for the top
answer of the first two questions, and >50 for the third, not necessarily
mutually exclusive voters, but probably largely; even assuming the worst,
>100 is nothing to scoff at). While anonymous Stack Overflow votes are not
necessarily the best arbiter on what's a good idea, they at least show
interest.

My proposed behavior (though probably not the implementation details) would
be as follows (though the implementation would obviously be as a string
method, not a function, with 'self' replacing 's'):



def rreplace(s, old, new, count=-1):
'''
Return a copy with all occurrences of substring old replaced by new.

  count
Maximum number of occurrences to replace.
-1 (the default value) means replace all occurrences

Substrings are replaced starting at the end of the string and working
to the front.
'''
return new.join(s.rsplit(old, count))



Addressing some (paraphrased) issues from the previous discussion, in the
form of an FAQ:

rreplace() looks weird.
>

This is maintaining consistency with the trend of 'r' prefixes for
'reverse' methods. Introducing 'reverse_' as a prefix would be
inconsistent, but worse, it would also encourage backwards incompatible
changes into existing methods. I think such a prefix naming change warrants
its own separate discussion.

There are already ways to do this. Why should we add another string method?
>

My main motivation is having one clear and efficient way to do this. I
explain this in greater detail in my introduction above.

The default of count=-1 has the same behavior as str.replace(), right?
>

Actually, it doesn't have the same behavior, as MRAB pointed out in the
previous discussion <
https://mail.python.org/pipermail/python-ideas/2014-January/025102.html>.
The default of -1 also keeps the syntax consistent with str.rsplit()'s
syntax.

If we're adding this, shouldn't we also add bytes.rreplace,
> bytearray.rreplace, bytearray.rremove, tuple.rindex, list.rindex,
> list.rremove?
>

Honestly, I don't know. I would prefer not to dilute this discussion too
much, or start making a slippery slope argument, but if it's directly
relevant, I think any thoughts are welcome.

Couldn't we just add a traverse_backwards parameter to str.replace()? In
> fact, why don't we just get rid of str.rfind() and str.rindex() entirely
> and just add new parameters to str.find() and str.index()?
>

I think Steven D'Aprano explained this well in the previous discussion
here: <
https://mail.python.org/pipermail/python-ideas/2014-January/025132.html>.
And addressing counterarguments here: <
https://mail.python.org/pipermail/python-ideas/2014-January/025135.html>.
Basically, different functions for different purposes make the purpose
clearer (no confusing/complicated parameter names and functions), and
str.rreplace() and str.replace() are usually going to be used in situations
where the traversal direction is known at edit time, so there's no need to
make the method determine the direction at runtime.



Thoughts? Support/oppose?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Thu, Jul 19, 2018 at 2:06 AM Chris Angelico  wrote:
> > With all due respect (and I am sorry for being “vocal” about a PEP once
> > again) I find this simply ugly. To me this basically doesn’t look like
> > python anymore, so a strong -1 from me.
>
> I'd love to hear an explanation of WHY this doesn't look like Python
> any more.

Because it looks like Perl.

> For instance, is the + operator somehow wrong for Python,
> and it should have been the word "add"?

The meaning of "+" is obvious to anybody, including non programmers.
"?" is arbitrary so you cannot guess what it does or means, especially
when it can be spelled in so many different forms (?, ??, a?.b, ??=,
...), each form requiring a different mental replacement. Use this and
:= on the same line and the reader will practically be reading another
language.

>  [ please trim quotes, you just quoted the entire PEP in your post ]

Ouch! Sorry about that.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[David Mertz]

The examples in the PEP strengthen my opinion. I think every one of them
read much more clearly in the existing syntax. I can only understand any by
mentally inserting 'if Foo is None' everywhere... Basically mentally
replacing the "code golf" syntax with the actual intent that is spelled
like it is in current Python.

On Wed, Jul 18, 2018, 8:05 PM David Mertz  wrote:

> As before, I'm a strong -1 on this.
>
> I recognize the semantic utility (albeit, I've needed it much less than
> some folks testify). But the obscure characters are accumulating far too
> fast with this.
>
> Even with PEP 572, I would have preferred 'as' and restrictions on where
> it's allowed, but ':=' is more familiar from numerous languages and
> pseudo-code notations.  '??' and '?.' and ?[]' are really just marching
> into APL and Perl territory. Yes, I know such operators exist in other
> languages, but it feels very unpythonic.
>
> On Wed, Jul 18, 2018, 1:46 PM Steve Dower  wrote:
>
>> Possibly this is exactly the wrong time to propose the next big syntax
>> change, since we currently have nobody to declare on it, but since we're
>> likely to argue for a while anyway it probably can't hurt (and maybe
>> this will become the test PEP for whoever takes the reins?).
>>
>> FWIW, Guido had previously indicated that he was generally favourable
>> towards most of this proposal, provided we could figure out coherent
>> semantics. Last time we tried, that didn't happen, so this time I've
>> made the semantics much more precise, have implemented and verified
>> them, and made much stronger statements about why we are proposing these.
>>
>> Additional thanks to Mark Haase for writing most of the PEP. All the
>> fair and balanced parts are his - all the overly strong opinions are mine.
>>
>> Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we
>> went through this - if you're unhappy with "None" being treated as a
>> special kind of value, I recommend reading those before you start
>> repeating them.
>>
>> There is a formatted version of this PEP at
>> https://www.python.org/dev/peps/pep-0505/
>>
>> My current implementation is at
>> https://github.com/zooba/cpython/tree/pep-505 (though I'm considering
>> removing some of the new opcodes I added and just generating more
>> complex code - in any case, let's get hung up on the proposal rather
>> than the implementation :) )
>>
>> Let the discussions begin!
>>
>> ---
>>
>> PEP: 505
>> Title: None-aware operators
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Mark E. Haase , Steve Dower
>> 
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 18-Sep-2015
>> Python-Version: 3.8
>>
>> Abstract
>> 
>>
>> Several modern programming languages have so-called "``null``-coalescing"
>> or
>> "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift,
>> and PHP
>> (starting in version 7). These operators provide syntactic sugar for
>> common
>> patterns involving null references.
>>
>> * The "``null``-coalescing" operator is a binary operator that returns
>> its left
>>operand if it is not ``null``. Otherwise it returns its right operand.
>> * The "``null``-aware member access" operator accesses an instance
>> member only
>>if that instance is non-``null``. Otherwise it returns ``null``.
>> (This is also
>>called a "safe navigation" operator.)
>> * The "``null``-aware index access" operator accesses an element of a
>> collection
>>only if that collection is non-``null``. Otherwise it returns
>> ``null``. (This
>>is another type of "safe navigation" operator.)
>>
>> This PEP proposes three ``None``-aware operators for Python, based on the
>> definitions and other language's implementations of those above.
>> Specifically:
>>
>> * The "``None`` coalescing`` binary operator ``??`` returns the left
>> hand side
>>if it evaluates to a value that is not ``None``, or else it evaluates
>> and
>>returns the right hand side. A coalescing ``??=`` augmented assignment
>>operator is included.
>> * The "``None``-aware attribute access" operator ``?.`` evaluates the
>> complete
>>expression if the left hand side evaluates to a value that is not
>> ``None``
>> * The "``None``-aware indexing" operator ``?[]`` evaluates the complete
>>expression if the left hand site evaluates to a value that is not
>> ``None``
>>
>> Syntax and Semantics
>> 
>>
>> Specialness of ``None``
>> ---
>>
>> The ``None`` object denotes the lack of a value. For the purposes of these
>> operators, the lack of a value indicates that the remainder of the
>> expression
>> also lacks a value and should not be evaluated.
>>
>> A rejected proposal was to treat any value that evaluates to false in a
>> Boolean context as not having a value. However, the purpose of these
>> operators
>> is to propagate the "lack of value" state, rather that the "false" state.
>>
>> Some argue that this makes ``None`` 

Re: [Python-ideas] PEP 505: None-aware operators

[David Mertz]

As before, I'm a strong -1 on this.

I recognize the semantic utility (albeit, I've needed it much less than
some folks testify). But the obscure characters are accumulating far too
fast with this.

Even with PEP 572, I would have preferred 'as' and restrictions on where
it's allowed, but ':=' is more familiar from numerous languages and
pseudo-code notations.  '??' and '?.' and ?[]' are really just marching
into APL and Perl territory. Yes, I know such operators exist in other
languages, but it feels very unpythonic.

On Wed, Jul 18, 2018, 1:46 PM Steve Dower  wrote:

> Possibly this is exactly the wrong time to propose the next big syntax
> change, since we currently have nobody to declare on it, but since we're
> likely to argue for a while anyway it probably can't hurt (and maybe
> this will become the test PEP for whoever takes the reins?).
>
> FWIW, Guido had previously indicated that he was generally favourable
> towards most of this proposal, provided we could figure out coherent
> semantics. Last time we tried, that didn't happen, so this time I've
> made the semantics much more precise, have implemented and verified
> them, and made much stronger statements about why we are proposing these.
>
> Additional thanks to Mark Haase for writing most of the PEP. All the
> fair and balanced parts are his - all the overly strong opinions are mine.
>
> Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we
> went through this - if you're unhappy with "None" being treated as a
> special kind of value, I recommend reading those before you start
> repeating them.
>
> There is a formatted version of this PEP at
> https://www.python.org/dev/peps/pep-0505/
>
> My current implementation is at
> https://github.com/zooba/cpython/tree/pep-505 (though I'm considering
> removing some of the new opcodes I added and just generating more
> complex code - in any case, let's get hung up on the proposal rather
> than the implementation :) )
>
> Let the discussions begin!
>
> ---
>
> PEP: 505
> Title: None-aware operators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Mark E. Haase , Steve Dower
> 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 18-Sep-2015
> Python-Version: 3.8
>
> Abstract
> 
>
> Several modern programming languages have so-called "``null``-coalescing"
> or
> "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift,
> and PHP
> (starting in version 7). These operators provide syntactic sugar for common
> patterns involving null references.
>
> * The "``null``-coalescing" operator is a binary operator that returns
> its left
>operand if it is not ``null``. Otherwise it returns its right operand.
> * The "``null``-aware member access" operator accesses an instance
> member only
>if that instance is non-``null``. Otherwise it returns ``null``.
> (This is also
>called a "safe navigation" operator.)
> * The "``null``-aware index access" operator accesses an element of a
> collection
>only if that collection is non-``null``. Otherwise it returns
> ``null``. (This
>is another type of "safe navigation" operator.)
>
> This PEP proposes three ``None``-aware operators for Python, based on the
> definitions and other language's implementations of those above.
> Specifically:
>
> * The "``None`` coalescing`` binary operator ``??`` returns the left
> hand side
>if it evaluates to a value that is not ``None``, or else it evaluates
> and
>returns the right hand side. A coalescing ``??=`` augmented assignment
>operator is included.
> * The "``None``-aware attribute access" operator ``?.`` evaluates the
> complete
>expression if the left hand side evaluates to a value that is not
> ``None``
> * The "``None``-aware indexing" operator ``?[]`` evaluates the complete
>expression if the left hand site evaluates to a value that is not
> ``None``
>
> Syntax and Semantics
> 
>
> Specialness of ``None``
> ---
>
> The ``None`` object denotes the lack of a value. For the purposes of these
> operators, the lack of a value indicates that the remainder of the
> expression
> also lacks a value and should not be evaluated.
>
> A rejected proposal was to treat any value that evaluates to false in a
> Boolean context as not having a value. However, the purpose of these
> operators
> is to propagate the "lack of value" state, rather that the "false" state.
>
> Some argue that this makes ``None`` special. We contend that ``None`` is
> already special, and that using it as both the test and the result of these
> operators does not change the existing semantics in any way.
>
> See the `Rejected Ideas`_ section for discussion on the rejected
> approaches.
>
> Grammar changes
> ---
>
> The following rules of the Python grammar are updated to read::
>
>  augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
> '^=' |
>  '<<=' | '>>=' | '**=' | '//=' | 

Re: [Python-ideas] PEP 505: None-aware operators

[Chris Angelico]

On Thu, Jul 19, 2018 at 9:55 AM, Giampaolo Rodola'  wrote:
 [ please trim quotes, you just quoted the entire PEP in your post ]
> With all due respect (and I am sorry for being “vocal” about a PEP once
> again) I find this simply ugly. To me this basically doesn’t look like
> python anymore, so a strong -1 from me.

I'd love to hear an explanation of WHY this doesn't look like Python
any more. For instance, is the + operator somehow wrong for Python,
and it should have been the word "add"? People complain about
proposals with words like these, but the best explanation I've ever
had is "well, Python uses words and this proposal uses punctuation".

Personally, I'm +0 on this. It'd be a few small wins here and there,
nothing huge, and I could easily live without it; but it's something
that I know some people will love.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Wed, Jul 18, 2018 at 7:46 PM Steve Dower  wrote:
>
> Possibly this is exactly the wrong time to propose the next big syntax
> change, since we currently have nobody to declare on it, but since we're
> likely to argue for a while anyway it probably can't hurt (and maybe
> this will become the test PEP for whoever takes the reins?).
>
> FWIW, Guido had previously indicated that he was generally favourable
> towards most of this proposal, provided we could figure out coherent
> semantics. Last time we tried, that didn't happen, so this time I've
> made the semantics much more precise, have implemented and verified
> them, and made much stronger statements about why we are proposing these.
>
> Additional thanks to Mark Haase for writing most of the PEP. All the
> fair and balanced parts are his - all the overly strong opinions are mine.
>
> Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we
> went through this - if you're unhappy with "None" being treated as a
> special kind of value, I recommend reading those before you start
> repeating them.
>
> There is a formatted version of this PEP at
> https://www.python.org/dev/peps/pep-0505/
>
> My current implementation is at
> https://github.com/zooba/cpython/tree/pep-505 (though I'm considering
> removing some of the new opcodes I added and just generating more
> complex code - in any case, let's get hung up on the proposal rather
> than the implementation :) )
>
> Let the discussions begin!
>
> ---
>
> PEP: 505
> Title: None-aware operators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Mark E. Haase , Steve Dower
> 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 18-Sep-2015
> Python-Version: 3.8
>
> Abstract
> 
>
> Several modern programming languages have so-called "``null``-coalescing"
or
> "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift,
> and PHP
> (starting in version 7). These operators provide syntactic sugar for
common
> patterns involving null references.
>
> * The "``null``-coalescing" operator is a binary operator that returns
> its left
>operand if it is not ``null``. Otherwise it returns its right operand.
> * The "``null``-aware member access" operator accesses an instance
> member only
>if that instance is non-``null``. Otherwise it returns ``null``.
> (This is also
>called a "safe navigation" operator.)
> * The "``null``-aware index access" operator accesses an element of a
> collection
>only if that collection is non-``null``. Otherwise it returns
> ``null``. (This
>is another type of "safe navigation" operator.)
>
> This PEP proposes three ``None``-aware operators for Python, based on the
> definitions and other language's implementations of those above.
> Specifically:
>
> * The "``None`` coalescing`` binary operator ``??`` returns the left
> hand side
>if it evaluates to a value that is not ``None``, or else it evaluates
and
>returns the right hand side. A coalescing ``??=`` augmented assignment
>operator is included.
> * The "``None``-aware attribute access" operator ``?.`` evaluates the
> complete
>expression if the left hand side evaluates to a value that is not
> ``None``
> * The "``None``-aware indexing" operator ``?[]`` evaluates the complete
>expression if the left hand site evaluates to a value that is not
> ``None``
>
> Syntax and Semantics
> 
>
> Specialness of ``None``
> ---
>
> The ``None`` object denotes the lack of a value. For the purposes of these
> operators, the lack of a value indicates that the remainder of the
> expression
> also lacks a value and should not be evaluated.
>
> A rejected proposal was to treat any value that evaluates to false in a
> Boolean context as not having a value. However, the purpose of these
> operators
> is to propagate the "lack of value" state, rather that the "false" state.
>
> Some argue that this makes ``None`` special. We contend that ``None`` is
> already special, and that using it as both the test and the result of
these
> operators does not change the existing semantics in any way.
>
> See the `Rejected Ideas`_ section for discussion on the rejected
approaches.
>
> Grammar changes
> ---
>
> The following rules of the Python grammar are updated to read::
>
>  augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
> '^=' |
>  '<<=' | '>>=' | '**=' | '//=' | '??=')
>
>  power: coalesce ['**' factor]
>  coalesce: atom_expr ['??' factor]
>  atom_expr: ['await'] atom trailer*
>  trailer: ('(' [arglist] ')' |
>'[' subscriptlist ']' |
>'?[' subscriptlist ']' |
>'.' NAME |
>'?.' NAME)
>
> Inserting the ``coalesce`` rule in this location ensures that expressions
> resulting in ``None`` are natuarlly coalesced before they are used in
> operations that would typically raise ``TypeError``. Like ``and`` and

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Nathaniel Smith]

On Wed, Jul 18, 2018 at 11:49 AM, Stephan Houben  wrote:
> Basically, what I am suggesting is a direct translation of Javascript's
> Web Worker API
> (https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)
> to Python.
>
> The Web Worker API is generally considered a "share-nothing" approach,
> although
> as we will see some state can be shared.
>
> The basic principle is that any object lives in a single Worker (Worker =
> subinterpreter).
> If a message is send from Worker A to Worker B, the message is not shared,
> rather the so-called "structured clone" algorithm is used to create
> recursively a NEW message
> object in Worker B. This is roughly equivalent to pickling in A and then
> unpickling in B,
>
> Of course, this may become a bottleneck if large amounts of data need to be
> communicated.
> Therefore, there is a special object type designed to provide a view upon a
> piece
> of shared memory:  SharedArrayBuffer. Notable, this only provides a view
> upon
> raw "C"-style data (ints or floats or whatever), not on Javascript objects.

Note that this everything you said here also exactly describes the
programming model for the existing 'multiprocessing' module:
"structured clone" is equivalent to how multiprocessing uses pickle to
transfer arbitrary objects, or you can use multiprocessing.Array to
get a shared view on raw "C"-style data.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Steve Dower]

Thanks! Bit of discussion below about precedence, but thanks for 
spotting the typos.


On 18Jul2018 1318, MRAB wrote:

On 2018-07-18 18:43, Steve Dower wrote:

Grammar changes
---

The following rules of the Python grammar are updated to read::

  augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
'^=' |
  '<<=' | '>>=' | '**=' | '//=' | '??=')

  power: coalesce ['**' factor]
  coalesce: atom_expr ['??' factor]
  atom_expr: ['await'] atom trailer*
  trailer: ('(' [arglist] ')' |
    '[' subscriptlist ']' |
    '?[' subscriptlist ']' |
    '.' NAME |
    '?.' NAME)

The precedence is higher than I expected. I think of it more like 'or'. 
What is its precedence in the other languages?


Yes, I expected this to be the contentious part. I may have to add a bit 
of discussion.


Mostly, I applied intuition rather than copying other languages on 
precedence (and if you could go through my non-git history, you'd see I 
tried four other places ;) ). The most "obvious" cases were these::


a ?? 1 + b()

b ** a() ?? 2

In the first case, both "(a ?? 1) + b()" and "a ?? (1 + b())" make 
sense, so it's really just my own personal preference that I think it 
looks like the first. If you flip the operands to get "b() + a ?? 1" 
then you end up with either "b() + (a ?? 1)" or "(b() + a) ?? 1", then 
it's more obvious that the latter doesn't make any sense (why would 
__add__ return None?), and so binding more tightly than "+" helps write 
sensible expressions with fewer parentheses.


Similarly, I feel like "b ** (a() ?? 2)" makes more sense than "(b ** 
a()) ?? 2", where for the latter we would have to assume a __pow__ 
implementation that returns None, or one that handles being passed None 
without raising a TypeError.


Contrasting this with "or", it is totally legitimate for arithmetic 
operators to return falsey values.


As I open the text file to correct the typos, I see this is what I tried 
to capture with:



Inserting the ``coalesce`` rule in this location ensures that expressions
resulting in ``None`` are naturally coalesced before they are used in
operations that would typically raise ``TypeError``.


Take (2 ** a.b) ?? 0. The result of __pow__ is rarely going to be None, 
unless we train all the builtin types to do so (which, incidentally, I 
am not proposing and have no intention of proposing), whereas something 
like "2 ** coord?.exponent" attempting to call "2.__pow__(None)" seems 
comparatively likely. (Unfortunately, nobody writes code like this yet 
:) So there aren't any real-life examples. Originally I didn't include 
"??" in the proposal, but it became obvious in the examples that the 
presence of None-propagating operators ?. and ?[] just cause more pain 
without having the None-terminating operator ?? as well.)



Inserting the ``coalesce`` rule in this location ensures that expressions
resulting in ``None`` are natuarlly coalesced before they are used in


Typo "natuarlly".


Thanks.


  assert a == 'value'
  assert b == ''
  assert c == '0' and any(os.scandir('/'))


Wouldn't the last assertion fail, because c == 0?


Correct, another typo.

Cheers,
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Eric Snow]

On Wed, Jul 18, 2018 at 2:38 PM MRAB  wrote:
> What if an object is not going to be shared, but instead "moved" from
> one subinterpreter to another? The first subinterpreter would no longer
> have a reference to the object.
>
> If the object's refcount is 1 and the object doesn't refer to any other
> object, then copying would not be necessary.

Yeah, that's something that I'm sure we'll investigate at some point,
but it's not part of the short-term plans.  This belongs to a whole
class of possibilities that we'll explore once we have the basic
functionality established. :)  FWIW, I don't think that "moving" an
object like this would be to hard to implement.

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[MRAB]

On 2018-07-18 20:35, Eric Snow wrote:

On Wed, Jul 18, 2018 at 12:49 PM Stephan Houben  wrote:

Antoine said that what I proposed earlier was very similar to what Eric
is trying to do, but from the direction the discussion has taken so far
that appears not to be the case.


It looks like we are after the same thing actually. :)  Sorry for any confusion.


There are currently no provisions for actually sharing objects between
interpreters.  In fact, initially the plan is basically to support
sharing copies of basic builtin immuntable types.  The question of
refcounts comes in when we actually do share underlying data of
immutable objects (e.g. the buffer protocol).

What if an object is not going to be shared, but instead "moved" from 
one subinterpreter to another? The first subinterpreter would no longer 
have a reference to the object.


If the object's refcount is 1 and the object doesn't refer to any other 
object, then copying would not be necessary.


[snip]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[MRAB]

On 2018-07-18 18:43, Steve Dower wrote:

Possibly this is exactly the wrong time to propose the next big syntax
change, since we currently have nobody to declare on it, but since we're
likely to argue for a while anyway it probably can't hurt (and maybe
this will become the test PEP for whoever takes the reins?).

FWIW, Guido had previously indicated that he was generally favourable
towards most of this proposal, provided we could figure out coherent
semantics. Last time we tried, that didn't happen, so this time I've
made the semantics much more precise, have implemented and verified
them, and made much stronger statements about why we are proposing these.

Additional thanks to Mark Haase for writing most of the PEP. All the
fair and balanced parts are his - all the overly strong opinions are mine.

Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we
went through this - if you're unhappy with "None" being treated as a
special kind of value, I recommend reading those before you start
repeating them.

There is a formatted version of this PEP at
https://www.python.org/dev/peps/pep-0505/

My current implementation is at
https://github.com/zooba/cpython/tree/pep-505 (though I'm considering
removing some of the new opcodes I added and just generating more
complex code - in any case, let's get hung up on the proposal rather
than the implementation :) )

Let the discussions begin!


[snip]


Grammar changes
---

The following rules of the Python grammar are updated to read::

  augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
'^=' |
  '<<=' | '>>=' | '**=' | '//=' | '??=')

  power: coalesce ['**' factor]
  coalesce: atom_expr ['??' factor]
  atom_expr: ['await'] atom trailer*
  trailer: ('(' [arglist] ')' |
'[' subscriptlist ']' |
'?[' subscriptlist ']' |
'.' NAME |
'?.' NAME)

The precedence is higher than I expected. I think of it more like 'or'. 
What is its precedence in the other languages?



Inserting the ``coalesce`` rule in this location ensures that expressions
resulting in ``None`` are natuarlly coalesced before they are used in


Typo "natuarlly".


operations that would typically raise ``TypeError``. Like ``and`` and ``or``
the right-hand expression is not evaluated until the left-hand side is
determined to be ``None``. For example::

  a, b = None, None
  def c(): return None
  def ex(): raise Exception()

  (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
  (a * b ?? c // d) == a * (b ?? c) // d
  (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
  (c() ?? c() ?? True) == True
  (True ?? ex()) == True
  (c ?? ex)() == c()

Augmented coalescing assignment only rebinds the name if its current
value is
``None``. If the target name already has a value, the right-hand side is not
evaluated. For example::

  a = None
  b = ''
  c = 0

  a ??= 'value'
  b ??= undefined_name
  c ??= shutil.rmtree('/')# don't try this at home, kids

  assert a == 'value'
  assert b == ''
  assert c == '0' and any(os.scandir('/'))


Wouldn't the last assertion fail, because c == 0?

[snip]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Eric Snow]

On Wed, Jul 18, 2018 at 12:49 PM Stephan Houben  wrote:
> Antoine said that what I proposed earlier was very similar to what Eric
> is trying to do, but from the direction the discussion has taken so far
> that appears not to be the case.

It looks like we are after the same thing actually. :)  Sorry for any confusion.


There are currently no provisions for actually sharing objects between
interpreters.  In fact, initially the plan is basically to support
sharing copies of basic builtin immuntable types.  The question of
refcounts comes in when we actually do share underlying data of
immutable objects (e.g. the buffer protocol).

> I will therefore try to clarify my proposal.
>
> Basically, what I am suggesting is a direct translation of Javascript's
> Web Worker API 
> (https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)
> to Python.
>
> The Web Worker API is generally considered a "share-nothing" approach, 
> although
> as we will see some state can be shared.

Yes, there's a strong parallel to that model here.  In fact, I
mentioned web workers in my language summit talk at PyCon 2018.

> The basic principle is that any object lives in a single Worker (Worker = 
> subinterpreter).
> If a message is send from Worker A to Worker B, the message is not shared,
> rather the so-called "structured clone" algorithm is used to create 
> recursively a NEW message
> object in Worker B. This is roughly equivalent to pickling in A and then 
> unpickling in B,

That is exactly what the channels in the PEP 554 implementation do,
though much more efficiently than pickling.  Initial support will be
for basic builtin immutable types.  We can later consider support for
other (even arbitrary?) types, but anything beyond copying (e.g.
pickle) is way off my radar.  Python's C-API is so closely tied to
refcounting that we simply cannot support safely sharing actual Python
objects between interpreters once we no longer share the GIL between
them.

> Of course, this may become a bottleneck if large amounts of data need to be 
> communicated.
> Therefore, there is a special object type designed to provide a view upon a 
> piece
> of shared memory:  SharedArrayBuffer. Notable, this only provides a view upon
> raw "C"-style data (ints or floats or whatever), not on Javascript objects.

Yep, that translates to buffers in Python, which is covered by PEP 554
(see SendChannel.send_buffer).

In this case, where some underlying data is actually shared, the
implementation has to deal with keeping a reference to the original
object and releasing it when done, which is what all the talk of
refcounts has been about.  However, the PEP does not talk about it
because it is an implementation detail that is not exposed in Python.

> To translate this to the Python situation: each Python object is owned by a 
> single
> subinterpreter, and may only be manipulated by a thread which holds the GIL
> of that particular subinterpreter. Message sending between subinterpreters 
> will
> require the message objects to be "structured cloned".

Correct.  That is what PEP 554 does.

As an aside, your phrasing "may only be manipulated by a thread which
holds the GIL of that particular subinterpreter" did spark something
I'll consider later:  perhaps interpreters can acquire each other's
GIL when (infrequently) necessary.  That could simplify a few things.

> Certain C extension types may override what structured cloning means for them.
> In particular, some C extension types may have a two-layer structure where
> the Py_Object contains a refcounted pointer to the actual data.
> The structured cloning on such an object may create a second Py_Object which
> references the same underlying object.
> This secondary refcount will need to be properly atomic, since it may be 
> manipulated
> from multiple subinterpreters.

My implementation of PEP 554 supports this, though I have not made the
C-API for it public.  It's also not part of the PEP.  I was
considering adding it.

> In this way, interpreter-shared data structures can be implemented.
> However, all the "normal" Python objects are not shared and can continue
> to use the current, non-atomic refcounting implementation.

That is correct.  That entirely matches what I'm doing with PEP 554.
In fact, the isolation between interpreters is critical to my
multi-core Python project, of which PEP 554 is a part.  It's necessary
in order to stop sharing the GIL between interpreters.  So actual
objects will never be shared between interpreters.  They can't be.

> Hope this clarifies my proposal.

Yep.  Thanks!

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Jonathan Fine]

Hi

Python in the age of the multi-core processor is an important question. And
garbage collection is one of the many issues involved.

I've been thinking about the garbage collection problem, and lurking on
this list, for a while. I think it's now about time I showed myself, and
shared my thoughts. I intend to do this in a new thread, dealing only with
the problem of multi-core reference counting garbage collection. I hope you
don't mind my doing this. Expect the first instalment tomorrow.

with best regards

Jonathan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Eric Snow]

On Wed, Jul 18, 2018 at 1:37 AM Barry Scott  wrote:
> Let me try a longer answer. The inc+test and dec+test do not require a
> lock if coded correctly. All OS and run times have solved this to provide
> locks. All processors provide the instructions that are the building blocks
> for lock primitives.
>
> You cannot mutate a mutable python object that is not protected with the GIL 
> as
> the change of state involves multiple parts of the object changing.
>
> If you know that an object is immutable then you could only do a check on the
> ref count as you will never change the state of the object beyond its ref 
> count.
> To access the object you only have to ensure it will not be deleted, which the
> ref count guarantees. The delete of the immutable object is then the only job
> that the original interpreter must do.

Perhaps we're agreeing?  Other than the single decref at when
"releasing" the object, it won't ever be directly modified (even the
refcount) in the other interpreter.  In effect that interpreter holds
a reference to the object which prevents GC in the "owning"
interpreter (the corresponding incref happened in that original
interpreter before the object was "shared").  The only issue is how to
"release" the object in the other interpreter so that the decref
happens in the "owning" interpreter.  As earlier noted, I'm planning
on taking advantage of the exiting ceval "pending calls" machinery.

So I'm not sure where an atomic int would factor in.  If you mean
switching the exiting refcount to an atomic int for the sake of the
cross-interpreter decref then that's not going to happen, as Ronald
suggested.  Larry could tell you about his Gilectomy experience. :)

Are you suggesting something like a second "cross-interpreter
refcount", which would be atomic, and add a check in Py_DECREF?  That
would imply an extra cross-interpreter-oriented C-API to parallel
Py_DECREF.  It would also mean either adding another field to PyObject
(yikes!) or keeping a separate table for tracking cross-interpreter
references.  I'm not sure any of that would be better than the
alternative I'm pursuing.  Then again, I've considered tracking which
interpreters hold a "reference" to an object, which isn't that
different.

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] PEP 505: None-aware operators

[Steve Dower]

Possibly this is exactly the wrong time to propose the next big syntax 
change, since we currently have nobody to declare on it, but since we're 
likely to argue for a while anyway it probably can't hurt (and maybe 
this will become the test PEP for whoever takes the reins?).


FWIW, Guido had previously indicated that he was generally favourable 
towards most of this proposal, provided we could figure out coherent 
semantics. Last time we tried, that didn't happen, so this time I've 
made the semantics much more precise, have implemented and verified 
them, and made much stronger statements about why we are proposing these.


Additional thanks to Mark Haase for writing most of the PEP. All the 
fair and balanced parts are his - all the overly strong opinions are mine.


Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we 
went through this - if you're unhappy with "None" being treated as a 
special kind of value, I recommend reading those before you start 
repeating them.


There is a formatted version of this PEP at 
https://www.python.org/dev/peps/pep-0505/


My current implementation is at 
https://github.com/zooba/cpython/tree/pep-505 (though I'm considering 
removing some of the new opcodes I added and just generating more 
complex code - in any case, let's get hung up on the proposal rather 
than the implementation :) )


Let the discussions begin!

---

PEP: 505
Title: None-aware operators
Version: $Revision$
Last-Modified: $Date$
Author: Mark E. Haase , Steve Dower 


Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 18-Sep-2015
Python-Version: 3.8

Abstract


Several modern programming languages have so-called "``null``-coalescing" or
"``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift, 
and PHP

(starting in version 7). These operators provide syntactic sugar for common
patterns involving null references.

* The "``null``-coalescing" operator is a binary operator that returns 
its left

  operand if it is not ``null``. Otherwise it returns its right operand.
* The "``null``-aware member access" operator accesses an instance 
member only
  if that instance is non-``null``. Otherwise it returns ``null``. 
(This is also

  called a "safe navigation" operator.)
* The "``null``-aware index access" operator accesses an element of a 
collection
  only if that collection is non-``null``. Otherwise it returns 
``null``. (This

  is another type of "safe navigation" operator.)

This PEP proposes three ``None``-aware operators for Python, based on the
definitions and other language's implementations of those above. 
Specifically:


* The "``None`` coalescing`` binary operator ``??`` returns the left 
hand side

  if it evaluates to a value that is not ``None``, or else it evaluates and
  returns the right hand side. A coalescing ``??=`` augmented assignment
  operator is included.
* The "``None``-aware attribute access" operator ``?.`` evaluates the 
complete
  expression if the left hand side evaluates to a value that is not 
``None``

* The "``None``-aware indexing" operator ``?[]`` evaluates the complete
  expression if the left hand site evaluates to a value that is not 
``None``


Syntax and Semantics


Specialness of ``None``
---

The ``None`` object denotes the lack of a value. For the purposes of these
operators, the lack of a value indicates that the remainder of the 
expression

also lacks a value and should not be evaluated.

A rejected proposal was to treat any value that evaluates to false in a
Boolean context as not having a value. However, the purpose of these 
operators

is to propagate the "lack of value" state, rather that the "false" state.

Some argue that this makes ``None`` special. We contend that ``None`` is
already special, and that using it as both the test and the result of these
operators does not change the existing semantics in any way.

See the `Rejected Ideas`_ section for discussion on the rejected approaches.

Grammar changes
---

The following rules of the Python grammar are updated to read::

augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | 
'^=' |

'<<=' | '>>=' | '**=' | '//=' | '??=')

power: coalesce ['**' factor]
coalesce: atom_expr ['??' factor]
atom_expr: ['await'] atom trailer*
trailer: ('(' [arglist] ')' |
  '[' subscriptlist ']' |
  '?[' subscriptlist ']' |
  '.' NAME |
  '?.' NAME)

Inserting the ``coalesce`` rule in this location ensures that expressions
resulting in ``None`` are natuarlly coalesced before they are used in
operations that would typically raise ``TypeError``. Like ``and`` and ``or``
the right-hand expression is not evaluated until the left-hand side is
determined to be ``None``. For example::

a, b = None, None
def c(): return None
def ex(): raise Exception()

(a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
(a * 

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Eric Snow]

On Wed, Jul 18, 2018 at 3:31 AM Antoine Pitrou  wrote:
> Please read in context: we are not talking about making all refcounts
> atomic, only a couple refcounts on shared objects (which probably
> won't be Python objects, actually).

I have no plans to use refcounts for shared data (outside of Python objects).

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] grouping / dict of lists

[Michel Desmoulin]

Counter also consider any missing key has the value "0".

With the constructor (accepting any iterable) and the most_common(n),
it's just a very set of features if you need to count anything.

Le 13/07/2018 à 19:45, Michael Selik a écrit :
> On Mon, Jul 2, 2018 at 8:49 AM Chris Barker  > wrote:
> 
> On Fri, Jun 29, 2018 at 11:25 PM, Guido van Rossum  > wrote:
>  
> 
> Hm, this actually feels heavier to me. But then again I never
> liked or understood the need for Counter --
> 
> 
> actually, me neither -- and partly because it's too lightweight --
> that is, it's still a regular dict, and you pretty much have to know
> that to use it. That it, it provides a nice counting constructor,
> but after that, it's just a key:integer dict :-)
> 
> 
> Counter provides ``most_common`` which is often implemented
> inefficiently if written from scratch. People mistakenly use ``sorted``
> instead of ``heapq.nlargest``.
> 
> 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Antoine Pitrou]

On Wed, 18 Jul 2018 08:21:31 +0100
Ronald Oussoren via Python-ideas
 wrote:
> Some past attempts at getting rid of the GIL used atomic inc/dec, and that 
> resulted in bad performance because these instructions  aren’t cheap. 

Please read in context: we are not talking about making all refcounts
atomic, only a couple refcounts on shared objects (which probably
won't be Python objects, actually).

Regards

Antoine.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Barry Scott]

> On 18 Jul 2018, at 08:21, Ronald Oussoren  wrote:
> 
> Op 18 jul. 2018 om 08:02 heeft Barry  > het volgende geschreven:
> 
>> 
>> 
 On 17 Jul 2018, at 21:00, Eric Snow  wrote:
 
 On Tue, Jul 17, 2018 at 1:44 PM Barry  wrote:
 The decrement itself is not the problem, that can be made thread safe.
>>> 
>>> Yeah, by using the GIL.   Otherwise, please elaborate.  My
>>> understanding is that if the decrement itself were not the problem
>>> then we'd have gotten rid of the GIL already.
>> 
>> All processors have thread safe ways to inc and dec and test, integers 
>> without holding a lock.
>> 
>> That is the mechanism that locks themselves are built out of. You can use 
>> that to avoid holding the GIL until the ref count reaches 0.
>> 
>> In c++ they built it into the language with std::atomic_int, you would have 
>> to find the way to do this C, i don’t have an answer at my finger tips for C.
>> 
> Some past attempts at getting rid of the GIL used atomic inc/dec, and that 
> resulted in bad performance because these instructions  aren’t cheap. 

Isn't this class of problem what leads to the per-processor caches and other 
optimisations in Linux kernel?
I wonder if kernel optimisations could be applied to this problem?

> 
> My gut feeling is that you’d have to get rid of refcounts to get high 
> performance when getting rid of the GIL in a single interpreter, which would 
> almost certainly result in breaking the C API.

Working on the ref count costs might be the enabling tech.

We already have the problem of unchanging objects being copied after a fork 
because of the ref counts being inside the object.
It was suggested that the ref count would have to move out of the object to 
help with this problem.

If there is a desirable solution to the parallel problem we can think about the 
C API migration problem.

Barry

>  
> 
> Ronald
>> Barry
>> 
>>> 
 Do you mean that once the ref reaches 0 you have to make the delete happen 
 on the original interpreter?
>>> 
>>> Yep.  For one thing, GC can trigger __del__, which can do anything,
>>> including modifying other objects from the original interpreter (incl.
>>> decref'ing them).  __del__ should be run under the original
>>> interpreter.  For another thing, during GC containers often decref
>>> their items.  Also, separating the GIL between interpreters may mean
>>> we'll need an allocator per interpreter.  In that case the
>>> deallocation must happen relative to the interpreter where the object
>>> was allocated.
>>> 
>>> -eric
>>> 
>> 
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org 
>> https://mail.python.org/mailman/listinfo/python-ideas 
>> 
>> Code of Conduct: http://python.org/psf/codeofconduct/ 
>> 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Barry Scott]

> On 17 Jul 2018, at 21:00, Eric Snow  wrote:
> 
> On Tue, Jul 17, 2018 at 1:44 PM Barry  wrote:
>> The decrement itself is not the problem, that can be made thread safe.
> 
> Yeah, by using the GIL.   Otherwise, please elaborate.  My
> understanding is that if the decrement itself were not the problem
> then we'd have gotten rid of the GIL already.

Let me try a longer answer. The inc+test and dec+test do not require a
lock if coded correctly. All OS and run times have solved this to provide
locks. All processors provide the instructions that are the building blocks
for lock primitives.

You cannot mutate a mutable python object that is not protected with the GIL as
the change of state involves multiple parts of the object changing.

If you know that an object is immutable then you could only do a check on the
ref count as you will never change the state of the object beyond its ref count.
To access the object you only have to ensure it will not be deleted, which the
ref count guarantees. The delete of the immutable object is then the only job
that the original interpreter must do.

> 
>> Do you mean that once the ref reaches 0 you have to make the delete happen 
>> on the original interpreter?
> 
> Yep.  For one thing, GC can trigger __del__, which can do anything,
> including modifying other objects from the original interpreter (incl.
> decref'ing them).  __del__ should be run under the original
> interpreter.  For another thing, during GC containers often decref
> their items.  Also, separating the GIL between interpreters may mean
> we'll need an allocator per interpreter.  In that case the
> deallocation must happen relative to the interpreter where the object
> was allocated.

Yep that I understand.

Barry

> 
> -eric
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Ronald Oussoren via Python-ideas]

Op 18 jul. 2018 om 08:02 heeft Barry  het volgende 
geschreven:

> 
> 
>>> On 17 Jul 2018, at 21:00, Eric Snow  wrote:
>>> 
>>> On Tue, Jul 17, 2018 at 1:44 PM Barry  wrote:
>>> The decrement itself is not the problem, that can be made thread safe.
>> 
>> Yeah, by using the GIL.   Otherwise, please elaborate.  My
>> understanding is that if the decrement itself were not the problem
>> then we'd have gotten rid of the GIL already.
> 
> All processors have thread safe ways to inc and dec and test, integers 
> without holding a lock.
> 
> That is the mechanism that locks themselves are built out of. You can use 
> that to avoid holding the GIL until the ref count reaches 0.
> 
> In c++ they built it into the language with std::atomic_int, you would have 
> to find the way to do this C, i don’t have an answer at my finger tips for C.
> 
 Some past attempts at getting rid of the GIL used atomic inc/dec, and that 
resulted in bad performance because these instructions  aren’t cheap. 

My gut feeling is that you’d have to get rid of refcounts to get high 
performance when getting rid of the GIL in a single interpreter, which would 
almost certainly result in breaking the C API.  

Ronald
> Barry
> 
>> 
>>> Do you mean that once the ref reaches 0 you have to make the delete happen 
>>> on the original interpreter?
>> 
>> Yep.  For one thing, GC can trigger __del__, which can do anything,
>> including modifying other objects from the original interpreter (incl.
>> decref'ing them).  __del__ should be run under the original
>> interpreter.  For another thing, during GC containers often decref
>> their items.  Also, separating the GIL between interpreters may mean
>> we'll need an allocator per interpreter.  In that case the
>> deallocation must happen relative to the interpreter where the object
>> was allocated.
>> 
>> -eric
>> 
> 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Barry]

>> On 17 Jul 2018, at 21:00, Eric Snow  wrote:
>> 
>> On Tue, Jul 17, 2018 at 1:44 PM Barry  wrote:
>> The decrement itself is not the problem, that can be made thread safe.
> 
> Yeah, by using the GIL.   Otherwise, please elaborate.  My
> understanding is that if the decrement itself were not the problem
> then we'd have gotten rid of the GIL already.

All processors have thread safe ways to inc and dec and test, integers without 
holding a lock.

That is the mechanism that locks themselves are built out of. You can use that 
to avoid holding the GIL until the ref count reaches 0.

In c++ they built it into the language with std::atomic_int, you would have to 
find the way to do this C, i don’t have an answer at my finger tips for C.

Barry

> 
>> Do you mean that once the ref reaches 0 you have to make the delete happen 
>> on the original interpreter?
> 
> Yep.  For one thing, GC can trigger __del__, which can do anything,
> including modifying other objects from the original interpreter (incl.
> decref'ing them).  __del__ should be run under the original
> interpreter.  For another thing, during GC containers often decref
> their items.  Also, separating the GIL between interpreters may mean
> we'll need an allocator per interpreter.  In that case the
> deallocation must happen relative to the interpreter where the object
> was allocated.
> 
> -eric
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/