[Python-Dev] Re: Improving inspect capabilities for classes

2020-06-17 Thread Serhiy Storchaka 

16.06.20 21:02, Guido van Rossum пише:
It would certainly be much easier to get through the review process. 
Adding a `__filename__` (why not `__file__`?) attribute to classes is a 
major surgery, presumably requiring a PEP, and debating the pros and 
cons and performance implications and fixing a bunch of tests that don't 
expect this attribute, and so on. Adding an imperfect solution to 
inspect.getsource() would only require the cooperation of whoever 
maintains the inspect module.


If add the file name of the sources as a class attribute, we need also 
to add the line number (or the range of line numbers) of the class 
definition. Otherwise inspect.getsource() will still be ambiguous. Also, 
specifying the file name does not help in case of REPL or compiling a 
string, so maybe you need to attach full source text to a class?

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/3JUBWOYVLZGAOJJGHS7PM24WEOG24AJL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improving inspect capabilities for classes

2020-06-17 Thread Petr Viktorin 

On 2020-06-17 09:02, Serhiy Storchaka wrote:

16.06.20 21:02, Guido van Rossum пише:
It would certainly be much easier to get through the review process. 
Adding a `__filename__` (why not `__file__`?) attribute to classes is 
a major surgery, presumably requiring a PEP, and debating the pros and 
cons and performance implications and fixing a bunch of tests that 
don't expect this attribute, and so on. Adding an imperfect solution 
to inspect.getsource() would only require the cooperation of whoever 
maintains the inspect module.


If add the file name of the sources as a class attribute, we need also 
to add the line number (or the range of line numbers) of the class 
definition. Otherwise inspect.getsource() will still be ambiguous.


That, or even the entire __code__ of the internal function that sets up 
the class. That has all the needed information.


I did a small experiment with this, and indeed it breaks tests that 
either don't expect the attribute or expect anything with __code__ is a 
function: 
https://github.com/python/cpython/commit/3fddc0906f2e7b92ea0f7ff040560a10372f91ec


You can actually do this in pure Python, just to see what breaks. See 
the attachment.



Also, 
specifying the file name does not help in case of REPL or compiling a 
string, so maybe you need to attach full source text to a class?


You get the same problem with functions, already. But Jupyter Notebook 
apparently works around this issue.
import builtins

orig_build_class = builtins.__build_class__

def build_class(func, *args, **kwds):
result = orig_build_class(func, *args, **kwds)
result.__code__ = func.__code__
return result

builtins.__build_class__ = build_class


##
# Demo:

import inspect

class C():
def hello(self):
print('Hello world!')

print(inspect.getsource(C))

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/Q3EZXN6KLA6WPWEN5HV3T5NGC5PMFB62/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

2020-06-17 Thread Petr Viktorin 

On 2020-06-16 19:20, Guido van Rossum wrote:
Has anybody brought up the problem yet that if one subinterpreter 
encounters a hard crash (say, it segfaults due to a bug in a C extension 
module), all subinterpreters active at that moment in the same process 
are likely to lose all their outstanding work, without a chance of recovery?


(Of course once we have locks in shared memory, a crashed process 
leaving a lock behind may also screw up everybody else, though perhaps 
less severely.)


Not really.
Asyncio has the same problem; has anyone brought this issue up there? 
(Granted, asyncio probably didn't uncover too many issues in extension 
modules, but if it did, I assume they would get fixed.)


If you're worried about segfaults, then you should use multiple 
processes. That will always give you better isolation. But I don't think 
it's a reason to stop improving interpreter isolation.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/AGMIWNEGH2PJN473EGGB7J4LY4RYFQA5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Accepting PEP 618: zip(strict=True)

2020-06-17 Thread Victor Stinner 
Well done Brandt!

Even if only a few people had issues with zip(), I think that it was a
long awaited feature. It's great to have it in Python 3.10!

It wasn't trivial or convenient to emulate the feature (check manually
the length) on Python 3.9 and older.

zip(strict=True) should help to write more reliable code. Maybe it's
time to review stdlib code to check if some functions would deserve
the addition of strict=True? I had a look and found a few suspicious
usage of zip(). But I'm not sure if we want to make these functions
stricter.


(*) For example, ast._Unparse.visit_Compare() uses "for o, e in
zip(node.ops, node.comparators):" which expects that AST is correct.
But many projects modify AST and even generate AST from scratch. On
the other hand, tolerating minor inconsistencies can also be seen as a
feature for ast.unparse().


(*) Another example: dis.findlinestarts() expects co_lnotab has a
length which a multiple of 2, but PyCode_New() doesn't provide such
warranty:

byte_increments = code.co_lnotab[0::2]
line_increments = code.co_lnotab[1::2]
for byte_incr, line_incr in zip(byte_increments, line_increments):
...

Hum, maybe it's a bug in codeobject.c which should be stricter. The
file uses "Py_ssize_t size = PyBytes_Size(co->co_lnotab) / 2;".

Well, that's a minor issue. I don't expect a bug if co_lnotab has an
odd length, the last byte is simply ignored.


(*) Another example in inspect.getclosurevars():

nonlocal_vars = {
var : cell.cell_contents
for var, cell in zip(code.co_freevars, func.__closure__)
   }

I'm not sure that func.__closure__ and func.__code__.co_freevars are
always consistent. For example, PyFunction_SetClosure() doesn't
enforce.

Victor

Le mer. 17 juin 2020 à 01:14, Guido van Rossum  a écrit :
>
> After taking a break to recapitulate from the vigorous debate, Brandt Bucher 
> has revised PEP 618 and submitted it for review. I volunteered to be 
> PEP-Delegate (the new term for the outdated BDFL-Delegate) and the SC has 
> approved me for this role. (Note that Antoine, the PEP's sponsor, declined to 
> be the lightning rod, er, PEP-Delegate.)
>
> I have now reviewed the PEP and skimmed some of the more recent discussion 
> about the topic. It is clear that no solution will win everyone over. But 
> most seem to agree that offering *some* solution for the stated problem is 
> better than none.
>
> To spare us more heartache, I am hereby accepting PEP 618. I expect that the 
> implementation will land soon.
>
> I have two very minor editorial remarks, which Brandt may address at his 
> leisure:
>
> - The "Backward Compatibility" section could be beefed up slightly, e.g. by 
> pointing out that the default remains strict=False and that zip previously 
> did not take keyword arguments at all.
>
> - The error messages are somewhat odd: why is the error sometimes that one 
> iterator is too long, and other times that one iterator is too short? All we 
> really know is that not all iterators have the same length, but the current 
> phrasing seems to be assuming that the first iterator is never too short or 
> too long.
>
> Congratulations, Brandt!
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/[email protected]/message/NLWB7FVJGMBBMCF4P3ZKUIE53JPDOWJ3/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/QFKBMFP5SGW6DNKTYX3SGGARYWJMQKVK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Accepting PEP 618: zip(strict=True)

2020-06-17 Thread Antoine Pitrou 


Thank you Guido :-)

Regards

Antoine.


On Tue, 16 Jun 2020 16:07:43 -0700
Guido van Rossum  wrote:
> After taking a break to recapitulate from the vigorous debate, Brandt
> Bucher has revised PEP 618  and
> submitted it for review . I
> volunteered to be PEP-Delegate (the new term for the outdated
> BDFL-Delegate) and the SC has approved
> 
> me for this role. (Note that Antoine, the PEP's sponsor, declined to be the
> lightning rod, er, PEP-Delegate.)
> 
> I have now reviewed the PEP and skimmed some of the more recent discussion
> about the topic. It is clear that no solution will win everyone over. But
> most seem to agree that offering *some* solution for the stated problem is
> better than none.
> 
> To spare us more heartache, I am hereby accepting PEP 618. I expect that
> the implementation  will land
> soon.
> 
> I have two very minor editorial remarks, which Brandt may address at his
> leisure:
> 
> - The "Backward Compatibility" section could be beefed up slightly, e.g. by
> pointing out that the default remains strict=False and that zip previously
> did not take keyword arguments at all.
> 
> - The error messages are somewhat odd: why is the error sometimes that one
> iterator is too long, and other times that one iterator is too short? All
> we really know is that not all iterators have the same length, but the
> current phrasing seems to be assuming that the first iterator is never too
> short or too long.
> 
> Congratulations, Brandt!
> 


___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/RDJHHCKS2AYKY2TGBBLRHZDLD5IXAFL2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

2020-06-17 Thread Petr Viktorin 

On 2020-06-16 20:28, Mark Shannon wrote:


On 16/06/2020 1:24 pm, Nick Coghlan wrote:
Multiprocessing serialisation overheads are abysmal. With enough OS 
support you can attempt to mitigate that via shared memory mechanisms 
(which Davin added to the standard library), but it's impossible to 
get the overhead of doing that as low as actually using the address 
space of one OS process.


What does "multiprocessing serialisation" even mean? I assume you mean 
the overhead of serializing objects for communication between processes.


The cost of serializing an object has absolutely nothing to do with 
which process the interpreter is running in.


Separate interpreters within a single process will still need to 
serialize objects for communication.


The overhead of passing data through shared memory is the same for 
threads and processes. It's just memory.


Can we please stick to facts and not throw around terms like "abysmal" 
with no data whatsoever to back it up.



I'd like to get back to the facts. Let me quote the original mail from 
this thread:


On 2020-06-05 16:32, Mark Shannon wrote:

While I'm in favour of PEP 554, or some similar model for parallelism in 
Python, I am opposed to the changes we are currently making to support it.


Which changes?
There are several efforts in this general space. Personally, I also 
don't agree with them all. And I think the reason I wasn't able to 
formulate too many replies to you is that we don't have a common 
understanding of what is being discussed, and of the modivations behind 
the changes.




You seem to try convince everyone that multiple processes are better (at 
isolation, and at performance) than multiple interpreters in one 
process. And I see the point: if you can live with the restriction of 
multiple processes, they probably are a better choice!
But I don't think PEPs 554, 489, 573, etc. are about choosing between 
multiprocessing and multiple interpreters; they're about making multiple 
interpreters better than they currently are.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/MTPO3VZRTRR6R23X462QXJ2X74E5YX22/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Cython and incompatible C API changes

2020-06-17 Thread Victor Stinner 
re: [Python-Dev] When can we remove wchar_t* cache from string?

Le mar. 16 juin 2020 à 21:19, Steve Dower  a écrit :
> On 16Jun2020 1641, Inada Naoki wrote:
> > * This change doesn't affect to pure Python packages.
> > * Most of the rest uses Cython.  Since I already report an issue to Cython,
> >regenerating with new Cython release fixes them.
>
> The precedent set in our last release with tp_print was that
> regenerating Cython releases was too much to ask.
>
> Unless we're going to overrule that immediately, we should leave
> everything there and give users/developers a full release cycle with
> updated Cython version to make new releases without causing any breakage.

I already made changes in Python 3.10 which require again to
regenerate C code generated by Cython:
https://docs.python.org/dev/whatsnew/3.10.html#id2

Py_TYPE(), Py_REFCNT() and Py_SIZE() can no longer be used as l-value.
These changes also broke numpy. I helped to fix Cython and numpy (and
they are already fixed).

You can expect further incompatible changes in the C API. For example,
I would like to make the PyThreadState structure opaque, whereas
Cython currently accesses directly to PyThreadState members.

There is an ongoing discussion about always requiring to run Cython
when installing a C extension which uses Cython.

Maybe we can find a way to use pre-generated C files for Python up to
version N, but require to run Cython for Python newer than version N?
It would prevent to require running Cython on stable Python versions,
but help to upgrade to newer Python and also test the "next Python"
(current master branch).

Note: if the Py_TYPE() & cie changes are causing too many issues, we
can also reconsider to postpone/revert these changes. IMO it's
important that we remain able to push incompatible changes to the C
API, because there are multiple known flaws in the C API.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/RCORRG7TAJHOTVQ7R65OI6BTODCG4OMC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: When can we remove wchar_t* cache from string?

2020-06-17 Thread Inada Naoki 
On Mon, Jun 15, 2020 at 4:25 PM Serhiy Storchaka  wrote:
>
> I have a plan for more graduate removing of this feature. I created a PR
> which adds several compile options, so Python can be built in one of
> three modes:
>
> 1. Support wchar_t* cache and use it. It is the current mode.
>
> 2. Support wchar_t* cache, but do not use it internally in CPython. It
> can be used to test whether getting rid of the wchar_t* cache can have
> negative effects.
>
> 3. Do not support wchar_t* cache. It is binary incompatible build. Its
> purpose is to allow authors of third-party libraries to prepare to
> future breakage.
>
[snip]
> https://github.com/python/cpython/pull/12409

I like your pull request, although I am not sure option 2 is really needed.

With your compile-time option, we can remove wstr in early alpha stage
(e.g. Python 3.11a1), and get it back again if it breaks too many packages.


> The plan is:
>
> 1. Add support of the above compile options. Unfortunately I did not
> have time to do this before feature freeze in 3.9, but maybe make an
> exception?
> 2. Make option 2 default.
> 3. Remove option 1.
> 4. Enable compiler deprecations for all legacy C API. Currently they are
> silenced for the C API used internally.
> 5. Make legacy C API always failing.
> 6. Remove legacy C API from header files.
>
> There is a long way to steps 5 and 6. I think 3.11 is too early.
>

Note that compiler deprecation (4) is approved by Łukasz Langa.
So Python 3.9 will have compiler deprecation.
https://github.com/python/cpython/pull/20878#issuecomment-644830032

On the other hand, PyArg_ParseTuple(AndKeywords) with u/Z format
doesn't have any deprecation yet.  I'm not sure we can backport the
runtime DeprecationWarning to 3.9, because we need to fix Argument
Clinic too.  (Serhiy's pull request fix the Argument Clinic.)

Regards,

-- 
Inada Naoki  
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/76K5ASJTINWIHOB2KASQ5RMGIB6S2OGV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Cython and incompatible C API changes

2020-06-17 Thread Petr Viktorin 



On 2020-06-17 12:03, Victor Stinner wrote:

re: [Python-Dev] When can we remove wchar_t* cache from string?

Le mar. 16 juin 2020 à 21:19, Steve Dower  a écrit :

On 16Jun2020 1641, Inada Naoki wrote:

* This change doesn't affect to pure Python packages.
* Most of the rest uses Cython.  Since I already report an issue to Cython,
regenerating with new Cython release fixes them.


The precedent set in our last release with tp_print was that
regenerating Cython releases was too much to ask.

Unless we're going to overrule that immediately, we should leave
everything there and give users/developers a full release cycle with
updated Cython version to make new releases without causing any breakage.


I already made changes in Python 3.10 which require again to
regenerate C code generated by Cython:
https://docs.python.org/dev/whatsnew/3.10.html#id2

Py_TYPE(), Py_REFCNT() and Py_SIZE() can no longer be used as l-value.
These changes also broke numpy. I helped to fix Cython and numpy (and
they are already fixed).


Those are not all the projects that were broken by the change -- they're 
just the most popular ones. Right?



You can expect further incompatible changes in the C API. For example,
I would like to make the PyThreadState structure opaque, whereas
Cython currently accesses directly to PyThreadState members.

There is an ongoing discussion about always requiring to run Cython
when installing a C extension which uses Cython.


Do you have a link to that discussion?


Maybe we can find a way to use pre-generated C files for Python up to
version N, but require to run Cython for Python newer than version N?
It would prevent to require running Cython on stable Python versions,
but help to upgrade to newer Python and also test the "next Python"
(current master branch).

Note: if the Py_TYPE() & cie changes are causing too many issues, we
can also reconsider to postpone/revert these changes. IMO it's
important that we remain able to push incompatible changes to the C
API, because there are multiple known flaws in the C API.


If PEP 387 (Backwards Compatibility Policy) is accepted, all the 
incompatible changes changes will require a two-year deprecation period. 
Right?

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/FWJPUTHFPOSNYXRNDYIO3VUNFPGWK5QW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Cython and incompatible C API changes

2020-06-17 Thread Victor Stinner 
Le mer. 17 juin 2020 à 12:38, Petr Viktorin  a écrit :
> > Py_TYPE(), Py_REFCNT() and Py_SIZE() can no longer be used as l-value.
> > These changes also broke numpy. I helped to fix Cython and numpy (and
> > they are already fixed).
>
> Those are not all the projects that were broken by the change -- they're
> just the most popular ones. Right?

I don't know. So far, I'm only aware of 4 broken projects: Cython,
numpy and immutables, pycurl.

Some references:

* These changes are part of https://bugs.python.org/issue39573
* Cython change:
https://github.com/cython/cython/commit/d8e93b332fe7d15459433ea74cd29178c03186bd
* numpy change:
https://github.com/numpy/numpy/commit/a96b18e3d4d11be31a321999cda4b795ea9eccaa

I'm not sure how Cython works, but affected functions are:

* generate_usr_dealloc_call() (something related to "__dealloc__")
* __Pyx_PyLong_AbsNeg()
* __Pyx_PyList_Append()
* __Pyx_PyList_Pop()
* __Pyx__PyList_PopIndex()
* __Pyx_ListComp_Append()


> > You can expect further incompatible changes in the C API. For example,
> > I would like to make the PyThreadState structure opaque, whereas
> > Cython currently accesses directly to PyThreadState members.
> >
> > There is an ongoing discussion about always requiring to run Cython
> > when installing a C extension which uses Cython.
>
> Do you have a link to that discussion?

Hum, I forgot where the discussion happened. Maybe it wasn't a proper
"discussion", but just a few tweets:
https://twitter.com/tacaswell/status/1266472526806474752

Thomas A Caswell wrote: "also, if you use cython please make it a
build-time dependency and please don't put the generated c code in the
sdists. cython can only handle the changes in the CPyhon c-api if you
let it!"


> If PEP 387 (Backwards Compatibility Policy) is accepted, all the
> incompatible changes changes will require a two-year deprecation period.
> Right?

For Py_TYPE() which cannot be used as an l-value, I don't see how to
emit a deprecation warning. I don't think that anyone would pay
attention if you only add a small deprecation note in the C API
documentation.

For structures made opaque like PyInterepreterState (done in Python
3.8) or PyThreadState (my next target!), again, I don't see how to
mark the usage of a structure as "deprecated".

My assumption is that only a minority of C extensions will be
affected. If this assumption is wrong, we can revert or postpone the
incompatible change.

So far, it doesn't seem to be a giant disaster :-) Only a few C
extensions had to be modified, and only a few lines. For example, the
l-value change only required to modify exactly 5 lines in the whole
numpy project!

--

By the way, I already made C API changes in Python 3.8, especially
macros converted to static inline functions:

* https://docs.python.org/dev/whatsnew/3.8.html#build-and-c-api-changes
* https://vstinner.github.io/split-include-directory-python38.html

As far as I recall, these changes didn't cause any issue.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/SCXC5CYU66BOT2VXW67BXTEQZBZ6VWSY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Cython and incompatible C API changes

2020-06-17 Thread Paul Moore 
On Wed, 17 Jun 2020 at 12:27, Victor Stinner  wrote:

> > If PEP 387 (Backwards Compatibility Policy) is accepted, all the
> > incompatible changes changes will require a two-year deprecation period.
> > Right?
>
[...]
>
> So far, it doesn't seem to be a giant disaster :-) Only a few C
> extensions had to be modified, and only a few lines. For example, the
> l-value change only required to modify exactly 5 lines in the whole
> numpy project!

Personally, I'm completely fine with the C API changes. But it does
seem to me that it exposes a flaw (or maybe just over-simplification)
in PEP 387, as I don't see how to reconcile the approach you took (and
the arguments you make here) with the wording of the PEP. Unless you
make fairly free use of the "if in doubt, ask the SC" clause - and I
suspect Victor has a somewhat above average instinct as to the SC's
views on such matters ;-))

My concern here is that PEP 387 could become a blocker for proposals
by people who are not "in the know" as to how to confirm if
something's backward incompatible. It would maybe be good if these
changes could be incorporated into that PEP as a case study in how the
new process would have applied.

I'll link this message into the PEP 387 discussion, to close the loop.
Paul
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/ZQ33UCSUNFJOGHUREA6DFFIWMFQZZZ2B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

2020-06-17 Thread Nick Coghlan 
On Wed., 17 Jun. 2020, 4:28 am Mark Shannon,  wrote:

>
> On 16/06/2020 1:24 pm, Nick Coghlan wrote:
> > Multiprocessing serialisation overheads are abysmal. With enough OS
> > support you can attempt to mitigate that via shared memory mechanisms
> > (which Davin added to the standard library), but it's impossible to get
> > the overhead of doing that as low as actually using the address space of
> > one OS process.
>
> What does "multiprocessing serialisation" even mean? I assume you mean
> the overhead of serializing objects for communication between processes.
>
> The cost of serializing an object has absolutely nothing to do with
> which process the interpreter is running in.
>
> Separate interpreters within a single process will still need to
> serialize objects for communication.
>
> The overhead of passing data through shared memory is the same for
> threads and processes. It's just memory.
>

No, it's not. With multiple processes, you have to instruct the OS to poke
holes in the isolated-by-default behavior in order to give multiple Python
interpreters access to a common memory store.

When the interpreters are in the same process, that isn't true - to give
multiple Python interpreters access, you just give them all a pointer to
the common data.

This will work most easily when the state being shared is not itself a
Python object. PEP 3118 buffers will be one example of that (including when
using pickle protocol 5 for data passing between interpreters), but the
application embedding use case (where there's no real "main" interpreter,
just multiple subinterpreters manipulating the application state) is the
other one I expect to be reasonably common.

This is the Ceph/mod_wsgi/hexchat plugin use case, which is beneficial
enough for people to have pursued it *despite* the significant usability
problems with the current state of the subinterpreter support.

Doing full blown zero-copy ownership transfer of actual Python objects
would be more difficult, since the current plan is to have separate memory
allocation pools per interpreter to avoid excessive locking overhead, so I
don't currently expect to see that any time soon, even if PEP 554 is
accepted. Assuming that remains the case, I'd expect multiprocessing to
remain the default choice for CPU bound use cases where all the interesting
state is held in Python objects (if you're going to have to mess about with
a separate heap of shared objects anyway, you may as well also enjoy the
benefits of greater process isolation).

Cheers,
Nick.

>

>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/GFBTU7MLX5V4KQYDSO6WYGLUH6XO2SIA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improving inspect capabilities for classes

2020-06-17 Thread Thomas Viehmann 

On 16/06/2020 20:02, Guido van Rossum wrote:

Very few stars. This suggests not many people care about this problem, and
that in turn might explain the lukewarm response you find everywhere.


This seems to be the core, and combined with the cost of measuring 
performance impacts of adding a new field, it may well not be worth it 
from a Python developer's perspective.

I see it clearly now.


Lastly, I have to ask: Why is this so important to you? What does this
prevent you from doing? You have illustrated the problem with toy examples
-- but what is the real-world problem you're encountering (apparently
regularly) that causes you to keep pushing on this? This needs to be
explored especially since so few other people appear to need this to work.


I do all nearly my Python development work on Jupyter notebooks.
One thing I miss is getting the class source code (via Jupyter 
Notebook's ??). When things get large and complicated enough, it seems 
that I end up trying to look at my own code.


The other part is that I might be overly fond of manipulating Python 
programs themselves. For example, PyTorch (a library sometimes used for 
machine learning) sports a JIT for a subset of Python and I spent some 
time trying to see why they can parse functions but not classes (instead 
they look at the methods one by one on instances of the class, which 
works, but always feels like a work-around). Before that method came 
about, I tried for a while to work with classes directly, and this is 
was a large part of the original motivation of looking to fix access to 
source code.


In hindsight, it would seem that this feature is mostly interesting to 
tool developers, not the general population, and again, I can see why 
it's a very niche feature that likely isn't worth going through the 
process of adding support for from Python's perspective.


Thank you for taking the time to consider my request, I sincerely 
appreciate it and I learned a great deal from our conversation and it 
makes me feel much better about the ill fate of my proposed patch.


Best regards

Thomas
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/KV2YRETS4ZMQ7VIBZRL32ZGMHF6W2LYO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Design and Architecture of python

2020-06-17 Thread Shakil Khan 
I am well versed in Python and now trying to understand and learn the Design 
philosophy of Python as a Programming Language.
Is there any document related to Design/Archoitecture of Python language itself 
where I can learn about implementation of garbage collection,
top level of python objects and different other features.

Bottom line is I want to understand and get hold of the Python source code 
itself in C. Any document or pointer towards that would be very much 
appreciated.

Thanks
Shakil
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/QW26GRLCHI7CCSK3G35YLWXAMYR7E7IW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improving inspect capabilities for classes

2020-06-17 Thread Guido van Rossum 
I presume Jupyter also lets you import code from a file, which you edit
outside, Jupyter? Is,that not an option for you?

On Wed, Jun 17, 2020 at 04:09 Thomas Viehmann  wrote:

> On 16/06/2020 20:02, Guido van Rossum wrote:
> > Very few stars. This suggests not many people care about this problem,
> and
> > that in turn might explain the lukewarm response you find everywhere.
>
> This seems to be the core, and combined with the cost of measuring
> performance impacts of adding a new field, it may well not be worth it
> from a Python developer's perspective.
> I see it clearly now.
>
> > Lastly, I have to ask: Why is this so important to you? What does this
> > prevent you from doing? You have illustrated the problem with toy
> examples
> > -- but what is the real-world problem you're encountering (apparently
> > regularly) that causes you to keep pushing on this? This needs to be
> > explored especially since so few other people appear to need this to
> work.
>
> I do all nearly my Python development work on Jupyter notebooks.
> One thing I miss is getting the class source code (via Jupyter
> Notebook's ??). When things get large and complicated enough, it seems
> that I end up trying to look at my own code.
>
> The other part is that I might be overly fond of manipulating Python
> programs themselves. For example, PyTorch (a library sometimes used for
> machine learning) sports a JIT for a subset of Python and I spent some
> time trying to see why they can parse functions but not classes (instead
> they look at the methods one by one on instances of the class, which
> works, but always feels like a work-around). Before that method came
> about, I tried for a while to work with classes directly, and this is
> was a large part of the original motivation of looking to fix access to
> source code.
>
> In hindsight, it would seem that this feature is mostly interesting to
> tool developers, not the general population, and again, I can see why
> it's a very niche feature that likely isn't worth going through the
> process of adding support for from Python's perspective.
>
> Thank you for taking the time to consider my request, I sincerely
> appreciate it and I learned a great deal from our conversation and it
> makes me feel much better about the ill fate of my proposed patch.
>
> Best regards
>
> Thomas
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/UD666BQRCRJEZG2WHQGXGMTZZDRF273B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Design and Architecture of python

2020-06-17 Thread Steve Dower 
If you're not willing to slog through the source code and many PEPs and 
past discussions, Anthony Shaw has done it for you and written a book: 
https://realpython.com/products/cpython-internals-book/


All the other write-ups I'm aware of are very dated, so I don't have any 
free suggestions I'm afraid.


Cheers,
Steve

On 17Jun2020 1437, Shakil Khan wrote:

I am well versed in Python and now trying to understand and learn the Design 
philosophy of Python as a Programming Language.
Is there any document related to Design/Archoitecture of Python language itself 
where I can learn about implementation of garbage collection,
top level of python objects and different other features.

Bottom line is I want to understand and get hold of the Python source code 
itself in C. Any document or pointer towards that would be very much 
appreciated.

Thanks
Shakil

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/AA2RRW4RMFM2KZ7V2CAATWVVI4DQGPZZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

2020-06-17 Thread Jeff Allen 

On 12/06/2020 12:55, Eric V. Smith wrote:

On 6/11/2020 6:59 AM, Mark Shannon wrote:
Different interpreters need to operate in their own isolated address 
space, or there will be horrible race conditions.

Regardless of whether that separation is done in software or hardware,
it has to be done.


I realize this is true now, but why must it always be true? Can't we 
fix this? At least one solution has been proposed: passing around a 
pointer to the current interpreter. I realize there issues here, like 
callbacks and signals that will need to be worked out. But I don't 
think it's axiomatically true that we'll always have race conditions 
with multiple interpreters in the same address space.


Eric


Axiomatically? No, but let me rise to the challenge.

If (1) interpreters manage the life-cycle of objects, and (2) a race 
condition arises when the life-cycle or state of an object is accessed 
by the interpreter that did not create it, and (3) an object will 
sometimes be passed to an interpreter that did not create it, and (4) an 
interpreter with a reference to an object will sometimes access its 
life-cycle or state, then (5) a race condition will sometimes arise. 
This seems to be true (as a deduction) if all the premises hold.


(1) and (2) are true in CPython as we know it. (3) is prevented 
(completely?) by the Python API, but not at all by the C API. (4) is 
implicit in an interpreter having access to an object, the way CPython 
and its extensions are written, so (5) follows in the case that the C 
API is used. You could change (1) and/or (2), maybe (4).


"Passing around a pointer to the current interpreter" sounds like an 
attempt to break (2) or maybe (4). But I don't understand "current". 
What you need at any time is the interpreter (state and life-cycle 
manager) for the object you're about to handle, so that the receiving 
interpreter can delegate the action, instead of crashing ahead itself. 
This suggests a reference to the interpreter must be embedded in each 
object, but it could be implicit in the memory address.


There is then still an issue that the owning interpreter has to be 
thread-safe (if there are threads) in the sense that it can serialise 
access to object state or life-cycle. If serialisation is by a GIL, the 
receiving interpreter must take the GIL of the owning interpreter, and 
we are somewhat back where we started. Note that the "current 
interpreter" is not a function of the current thread (or vice-versa). 
The current thread is running in both interpreters, and by hypothesis, 
so are the competing threads.


Can I just point out that, while most of this argument concerns a 
particular implementation, we have a reason in Python (the language) for 
an interpreter construct: it holds the current module context, so that 
whenever code is executing, we can give definite meaning to the 'import' 
statement. Here "current interpreter" does have a meaning, and I suggest 
it needs to be made a property of every function object as it is 
defined, and picked up when the execution frame is created. This *may* 
help with the other, internal, use of interpreter, for life-cycle and 
state management, because it provides a recognisable point (function 
call) where one may police object ownership, but that isn't why you need it.


Jeff Allen


___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/GACVQJNCZLT4P3YX5IISRBOQTXXTJVMB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Design and Architecture of python

2020-06-17 Thread Shakil Khan 
Thanks Steve. That would definitely help.


> On Jun 17, 2020, at 9:23 AM, Steve Dower  wrote:
> 
> If you're not willing to slog through the source code and many PEPs and past 
> discussions, Anthony Shaw has done it for you and written a book: 
> https://realpython.com/products/cpython-internals-book/
> 
> All the other write-ups I'm aware of are very dated, so I don't have any free 
> suggestions I'm afraid.
> 
> Cheers,
> Steve
> 
> On 17Jun2020 1437, Shakil Khan wrote:
>> I am well versed in Python and now trying to understand and learn the Design 
>> philosophy of Python as a Programming Language.
>> Is there any document related to Design/Archoitecture of Python language 
>> itself where I can learn about implementation of garbage collection,
>> top level of python objects and different other features.
>> Bottom line is I want to understand and get hold of the Python source code 
>> itself in C. Any document or pointer towards that would be very much 
>> appreciated.
>> Thanks
>> Shakil
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/2CO6QT4UQ3WQTV45ETKDIQV7UTIYBE6B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

2020-06-17 Thread Emily Bowman 
On Wed, Jun 17, 2020 at 5:56 AM Nick Coghlan  wrote:

>
> Doing full blown zero-copy ownership transfer of actual Python objects
> would be more difficult, since the current plan is to have separate memory
> allocation pools per interpreter to avoid excessive locking overhead, so I
> don't currently expect to see that any time soon, even if PEP 554 is
> accepted. Assuming that remains the case, I'd expect multiprocessing to
> remain the default choice for CPU bound use cases where all the interesting
> state is held in Python objects (if you're going to have to mess about with
> a separate heap of shared objects anyway, you may as well also enjoy the
> benefits of greater process isolation).
>

So most likely there wouldn't be any way to share something like a
bytearray or another buffer interface-compatible type for some time. That's
too bad, I was hoping to have shared arrays that I could put a memoryview
on in each thread/interpreter and deal with locking if I need to, but I
suppose I can work through an extension once the changes stabilize.
Packages like NumPy have had their own opaque C types and C-only routines
to handle all the big threading outside of Python as a workaround for a
long time now.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/7MR4MJDTFQIEEMY6WQ2IE4EYFYU6PPKJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

2020-06-17 Thread Eric V. Smith 

On 6/17/2020 12:07 PM, Jeff Allen wrote:

On 12/06/2020 12:55, Eric V. Smith wrote:

On 6/11/2020 6:59 AM, Mark Shannon wrote:
Different interpreters need to operate in their own isolated address 
space, or there will be horrible race conditions.

Regardless of whether that separation is done in software or hardware,
it has to be done.


I realize this is true now, but why must it always be true? Can't we 
fix this? At least one solution has been proposed: passing around a 
pointer to the current interpreter. I realize there issues here, like 
callbacks and signals that will need to be worked out. But I don't 
think it's axiomatically true that we'll always have race conditions 
with multiple interpreters in the same address space.


Eric


Axiomatically? No, but let me rise to the challenge.

If (1) interpreters manage the life-cycle of objects, and (2) a race 
condition arises when the life-cycle or state of an object is accessed 
by the interpreter that did not create it, and (3) an object will 
sometimes be passed to an interpreter that did not create it, and (4) 
an interpreter with a reference to an object will sometimes access its 
life-cycle or state, then (5) a race condition will sometimes arise. 
This seems to be true (as a deduction) if all the premises hold.


(1) and (2) are true in CPython as we know it. (3) is prevented 
(completely?) by the Python API, but not at all by the C API. (4) is 
implicit in an interpreter having access to an object, the way CPython 
and its extensions are written, so (5) follows in the case that the C 
API is used. You could change (1) and/or (2), maybe (4).


I'm assuming that passing an object between interpreters would not be 
supported. It would require that the object somehow be marshalled 
between interpreters, so that no object would be operated on outside the 
interpreter that created it. So 2-5 couldn't happen in valid code.


"Passing around a pointer to the current interpreter" sounds like an 
attempt to break (2) or maybe (4). But I don't understand "current". 
What you need at any time is the interpreter (state and life-cycle 
manager) for the object you're about to handle, so that the receiving 
interpreter can delegate the action, instead of crashing ahead itself. 
This suggests a reference to the interpreter must be embedded in each 
object, but it could be implicit in the memory address.


Sorry for being loose with terms. If I want to create an interpreter and 
execute it, then I'd allocate and initialize an interpreter state 
object, then call it, passing the interpreter state object in to 
whatever Python functions I want to call. They would in turn pass that 
pointer to whatever they call, or access the state through it directly. 
That pointer is the "current interpreter".


There is then still an issue that the owning interpreter has to be 
thread-safe (if there are threads) in the sense that it can serialise 
access to object state or life-cycle. If serialisation is by a GIL, 
the receiving interpreter must take the GIL of the owning interpreter, 
and we are somewhat back where we started. Note that the "current 
interpreter" is not a function of the current thread (or vice-versa). 
The current thread is running in both interpreters, and by hypothesis, 
so are the competing threads.


Agreed that an interpreter shouldn't belong to a thread, but since an 
interpreter couldn't access objects of another interpreter, there'd be 
no need for cross-intepreter locking. There would be a GIL per 
interpreter, protecting access to that interpreter's state.


Can I just point out that, while most of this argument concerns a 
particular implementation, we have a reason in Python (the language) 
for an interpreter construct: it holds the current module context, so 
that whenever code is executing, we can give definite meaning to the 
'import' statement. Here "current interpreter" does have a meaning, 
and I suggest it needs to be made a property of every function object 
as it is defined, and picked up when the execution frame is created. 
This *may* help with the other, internal, use of interpreter, for 
life-cycle and state management, because it provides a recognisable 
point (function call) where one may police object ownership, but that 
isn't why you need it.


There's a lot of state per interpreter, including the module state. See 
"struct _is" in Include/internal/pycore_interp.h.


Eric


Jeff Allen



___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/GACVQJNCZLT4P3YX5IISRBOQTXXTJVMB/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

2020-06-17 Thread Eric Snow 
On Wed, Jun 17, 2020 at 11:42 AM Emily Bowman  wrote:
> So most likely there wouldn't be any way to share something like a bytearray 
> or another
> buffer interface-compatible type for some time. That's too bad, I was hoping 
> to have
> shared arrays that I could put a memoryview on in each thread/interpreter and 
> deal with
> locking if I need to,

Earlier versions of PEP 554 did have a "SendChannel.send_buffer()"
method for this but we tabled it in the interest of simplifying.  That
said, I expect we'll add something like that separately later.

> but I suppose I can work through an extension once the changes stabilize.

Yep.  This should be totally doable in an extension and hopefully
without much effort.

> Packages like NumPy have had their own opaque C types and C-only routines to 
> handle all the big threading outside of Python as a workaround for a long 
> time now.

As a workaround for what?  This sounds interesting. :)

-eric
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/D2APLOLR4UL7VXLNRFGFWOUN5MPIO2BV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improving inspect capabilities for classes

2020-06-17 Thread Thomas Viehmann 

On 17/06/2020 17:25, Guido van Rossum wrote:

I presume Jupyter also lets you import code from a file, which you edit
outside, Jupyter? Is,that not an option for you?


It's not the file that is the problem, but the lack of it. If I didn't 
want to cover classes within the __main__ module, I wouldn't have gotten 
myself into this in the first place.


If it were just the technical aspects, I'd still think recording the 
filename would be the a good solution, I just never thought about the 
cost of the process enough.


Maybe the other fundamentally sound solution would be to treat the 
__main__ module of an interactive session as a single file with ever 
incrementing line numbers. Making it more explicit that Jupyter/IPython 
is a world of amending only would solve so many other problems as well 
and create a world of entirely new ones.


Best regards

Thomas
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/QHSINZZQRPV2HQROSXOLG3AUQ5QKKG6V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

2020-06-17 Thread Jeff Allen 

On 17/06/2020 19:28, Eric V. Smith wrote:

On 6/17/2020 12:07 PM, Jeff Allen wrote:
If (1) interpreters manage the life-cycle of objects, and (2) a race 
condition arises when the life-cycle or state of an object is 
accessed by the interpreter that did not create it, and (3) an object 
will sometimes be passed to an interpreter that did not create it, 
and (4) an interpreter with a reference to an object will sometimes 
access its life-cycle or state, then (5) a race condition will 
sometimes arise. This seems to be true (as a deduction) if all the 
premises hold. 
I'm assuming that passing an object between interpreters would not be 
supported. It would require that the object somehow be marshalled 
between interpreters, so that no object would be operated on outside 
the interpreter that created it. So 2-5 couldn't happen in valid code.


The Python level doesn't support it, prevents it I think, and perhaps 
the implementation doesn't support it, but nothing can stop C actually 
doing it. I would agree that with sufficient discipline in the code it 
should be possible to  prevent the worlds from colliding. But it is 
difficult, so I think that is why Mark is arguing for a separate address 
space. Marshalling the value across is supported, but that's just the 
value, not a shared object.


Sorry for being loose with terms. If I want to create an interpreter 
and execute it, then I'd allocate and initialize an interpreter state 
object, then call it, passing the interpreter state object in to 
whatever Python functions I want to call. They would in turn pass that 
pointer to whatever they call, or access the state through it 
directly. That pointer is the "current interpreter".


I think that can work if you have disciplined separation, which you are 
assuming. I think you would pass the function to the interpreter, not 
the other way around. I'm assuming this is described from the 
perspective of some C code and your Python functions are PyFunction 
objects, not just text? What, however, prevents you creating that 
function in one interpreter and giving it to another? The function, and 
any closure or defaults are owned by the creating interpreter.


There's a lot of state per interpreter, including the module state. 
See "struct _is" in Include/internal/pycore_interp.h.


So much more than when I last looked! Look back in time and interpreter 
state mostly contains the module context (in a broad sense that includes 
shortcuts to sys, builtins, codec state, importlib). Ok, there's some 
stuff about exit handling and debugging too. The recent huge growth is 
to shelter previously singleton object allocation mechanisms, a 
consequence of the implementation choice that gives the interpreter 
object that responsibility too. I'm not saying this is wrong, just that 
it's not a concept in Python-the-language, while the module state is.


Jeff


___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/ESNK7A5UFBQOQXKUDWCUMS2372AL7ZPU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?)

2020-06-17 Thread Eric V. Smith 

On 6/17/2020 6:03 PM, Jeff Allen wrote:

On 17/06/2020 19:28, Eric V. Smith wrote:

On 6/17/2020 12:07 PM, Jeff Allen wrote:
If (1) interpreters manage the life-cycle of objects, and (2) a race 
condition arises when the life-cycle or state of an object is 
accessed by the interpreter that did not create it, and (3) an 
object will sometimes be passed to an interpreter that did not 
create it, and (4) an interpreter with a reference to an object will 
sometimes access its life-cycle or state, then (5) a race condition 
will sometimes arise. This seems to be true (as a deduction) if all 
the premises hold. 
I'm assuming that passing an object between interpreters would not be 
supported. It would require that the object somehow be marshalled 
between interpreters, so that no object would be operated on outside 
the interpreter that created it. So 2-5 couldn't happen in valid code.


The Python level doesn't support it, prevents it I think, and perhaps 
the implementation doesn't support it, but nothing can stop C actually 
doing it. I would agree that with sufficient discipline in the code it 
should be possible to  prevent the worlds from colliding. But it is 
difficult, so I think that is why Mark is arguing for a separate 
address space. Marshalling the value across is supported, but that's 
just the value, not a shared object.


Yes, it's difficult to have the discipline in C, just as multi-threaded 
is difficult in C. I agree separate address spaces makes isolation much 
easier, but I think there are use cases that don't align with separate 
address spaces, and we should support those.


Sorry for being loose with terms. If I want to create an interpreter 
and execute it, then I'd allocate and initialize an interpreter state 
object, then call it, passing the interpreter state object in to 
whatever Python functions I want to call. They would in turn pass 
that pointer to whatever they call, or access the state through it 
directly. That pointer is the "current interpreter".


I think that can work if you have disciplined separation, which you 
are assuming. I think you would pass the function to the interpreter, 
not the other way around. I'm assuming this is described from the 
perspective of some C code and your Python functions are PyFunction 
objects, not just text? What, however, prevents you creating that 
function in one interpreter and giving it to another? The function, 
and any closure or defaults are owned by the creating interpreter.


In the C API (which is what I think we're discussing), I think it would 
be passing the interpreter state to the function. And nothing would 
prevent you from getting it wrong.


There's a lot of state per interpreter, including the module state. 
See "struct _is" in Include/internal/pycore_interp.h.


So much more than when I last looked! Look back in time and 
interpreter state mostly contains the module context (in a broad sense 
that includes shortcuts to sys, builtins, codec state, importlib). Ok, 
there's some stuff about exit handling and debugging too. The recent 
huge growth is to shelter previously singleton object allocation 
mechanisms, a consequence of the implementation choice that gives the 
interpreter object that responsibility too. I'm not saying this is 
wrong, just that it's not a concept in Python-the-language, while the 
module state is.


I think most of these changes are Victor's, and I think they're a step 
in the right direction. Since Python globals are really module state, it 
makes sense that that's the part that's visible to Python.


Eric
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/TRML3XIT7IN2XZZT3MDJGP5NSZ63A6EC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.7.8rc1 and 3.6.11rc1 are now available for testing

2020-06-17 Thread Ned Deily 
Details here:

https://discuss.python.org/t/python-3-7-8rc1-and-3-6-11rc1-are-now-available-for-testing/4467


https://www.python.org/downloads/release/python-378rc1/
https://www.python.org/downloads/release/python-3611rc1/

--
  Ned Deily
  [email protected] -- []
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/2FY3TKOR3ULMJ26MQESAO5Y44ZCDXNE5/
Code of Conduct: http://python.org/psf/codeofconduct/