Re: [Python-Dev] PEP 554 v3 (new interpreters module)

[Eric Snow]

Thanks for the feedback, Antoine.  Sorry for the delay; it's been a
busy week for me.  I just pushed an updated PEP to the repo.  Once
I've sorted out the question of passing bytes through channels I plan
on posting the PEP to the list again for another round of discussion.
In the meantime, I've replied below in-line.

-eric

On Mon, Sep 18, 2017 at 4:46 AM, Antoine Pitrou  wrote:
> First my high-level opinion about the PEP: the CSP model can probably
> be already implemented using Queues.  To me, the interesting promise of
> subinterpreters is if they allow to remove the GIL while sharing memory
> for big objects (such as Numpy arrays).  This means the PEP should
> probably focus on potential concurrency improvements rather than try to
> faithfully follow the CSP model.

Please elaborate.  I'm interested in understanding what you mean here.
Do you have some subinterpreter-based concurrency improvements in
mind?  What aspect of CSP is the PEP following too faithfully?

>> ``list_all()``::
>>
>>Return a list of all existing interpreters.
>
> See my naming proposal in the previous thread.

Sorry, your previous comment slipped through the cracks.  You suggested:

As for the naming, let's make it both unconfusing and explicit?
How about three functions: `all_interpreters()`, `running_interpreters()`
and `idle_interpreters()`, for example?

As to "all_interpreters()", I suppose it's the difference between
"interpreters.all_interpreters()" and "interpreters.list_all()".  To
me the latter looks better.

As to "running_interpreters()" and "idle_interpreters()", I'm not sure
what the benefit would be.  You can compose either list manually with
a simple comprehension:

[interp for interp in interpreters.list_all() if interp.is_running()]
[interp for interp in interpreters.list_all() if not interp.is_running()]

>>run(source_str, /, **shared):
>>
>>   Run the provided Python source code in the interpreter.  Any
>>   keyword arguments are added to the interpreter's execution
>>   namespace.
>
> "Execution namespace" specifically means the __main__ module in the
> target interpreter, right?

Right.  It's explained in more detail a little further down and
elsewhere in the PEP.  I've updated the PEP to explicitly mention
__main__ here too.

>>  If any of the values are not supported for sharing
>>   between interpreters then RuntimeError gets raised.  Currently
>>   only channels (see "create_channel()" below) are supported.
>>
>>   This may not be called on an already running interpreter.  Doing
>>   so results in a RuntimeError.
>
> I would distinguish between both error cases: RuntimeError for calling
> run() on an already running interpreter, ValueError for values which
> are not supported for sharing.

Good point.

>>   Likewise, if there is any uncaught
>>   exception, it propagates into the code where "run()" was called.
>
> That makes it a bit harder to differentiate with errors raised by run()
> itself (see above), though how much of an annoyance this is remains
> unclear.  The more litigious implication, though, is that it forces the
> interpreter to support migration of arbitrary objects from one
> interpreter to another (since a traceback keeps all local variables
> alive).

Yeah, the proposal to propagate exceptions out of the subinterpreter
is still rather weak.  I've added some notes the the PEP about this
open issue.

>> The mechanism for passing objects between interpreters is through
>> channels.  A channel is a simplex FIFO similar to a pipe.  The main
>> difference is that channels can be associated with zero or more
>> interpreters on either end.
>
> So it seems channels have become more complicated now?  Is it important
> to support multi-producer multi-consumer channels?

To me it made the API simpler.  The change did introduce the "close()"
method, which I suppose could be confusing.  However, I'm sure that in
practice it won't be.  In contrast, the FIFO/pipe-based API that I had
before required passing names around, required more calls, required
managing the channel/interpreter relationship more carefully, and made
it hard to follow that relationship.

>>  Unlike queues, which are also many-to-many,
>> channels have no buffer.
>
> How does it work?  Does send() block until someone else calls recv()?
> That does not sound like a good idea to me.

Correct "send()" blocks until the other end receives (if ever).
Likewise "recv()" blocks until the other end sends.  This specific
behavior is probably the main thing I borrowed from CSP.  It is *the*
synchronization mechanism.  Given the isolated nature of
subinterpreters, I consider using this concept from CSP to be a good
fit.

>  I don't think it's a
> coincidence that the most varied kinds of I/O (from socket or file IO
> to threading Queues to multiprocessing Pipes) have non-blocking send().

Interestingly, you can set sockets to blocking mode, in which case
send() 

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2017-09-15 - 2017-09-22)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open6196 (+25)
  closed 37107 (+42)
  total  43303 (+67)

Open issues with patches: 2376 


Issues opened (56)
==

#13487: inspect.getmodule fails when module imports change sys.modules
https://bugs.python.org/issue13487  reopened by merwok

#31486: calling a _json.Encoder object raises a SystemError in case ob
https://bugs.python.org/issue31486  opened by Oren Milman

#31488: IDLE: Update feature classes when options are changed.
https://bugs.python.org/issue31488  opened by terry.reedy

#31489: Signal delivered to a subprocess triggers parent's handler
https://bugs.python.org/issue31489  opened by Ilya.Kulakov

#31490: assertion failure in ctypes in case an _anonymous_ attr appear
https://bugs.python.org/issue31490  opened by Oren Milman

#31491: Add is_closing() to asyncio.StreamWriter.
https://bugs.python.org/issue31491  opened by aymeric.augustin

#31492: assertion failures in case a module has a bad __name__ attribu
https://bugs.python.org/issue31492  opened by Oren Milman

#31493: IDLE cond context: fix code update and font update timers
https://bugs.python.org/issue31493  opened by terry.reedy

#31494: Valgrind suppression file
https://bugs.python.org/issue31494  opened by Aaron Michaux

#31495: Wrong offset with IndentationError ("expected an indented bloc
https://bugs.python.org/issue31495  opened by blueyed

#31496: IDLE: test_configdialog failed
https://bugs.python.org/issue31496  opened by serhiy.storchaka

#31500: IDLE: Tiny font on HiDPI display
https://bugs.python.org/issue31500  opened by serhiy.storchaka

#31502: IDLE: Config dialog again deletes custom themes and keysets.
https://bugs.python.org/issue31502  opened by terry.reedy

#31503: Enhance dir(module) to be informed by __all__ by updating modu
https://bugs.python.org/issue31503  opened by codypiersall

#31504: Documentation for return value for string.rindex is missing wh
https://bugs.python.org/issue31504  opened by kgashok

#31505: assertion failure in json, in case _json.make_encoder() receiv
https://bugs.python.org/issue31505  opened by Oren Milman

#31506: Improve the error message logic for object_new & object_init
https://bugs.python.org/issue31506  opened by ncoghlan

#31507: email.utils.parseaddr has no docstring
https://bugs.python.org/issue31507  opened by mark.dickinson

#31508: Running test_ttk_guionly logs "test_widgets.py:1562: UserWarni
https://bugs.python.org/issue31508  opened by haypo

#31509: test_subprocess hangs randomly on AMD64 Windows10 3.x
https://bugs.python.org/issue31509  opened by haypo

#31510: test_many_processes() of test_multiprocessing_spawn failed on 
https://bugs.python.org/issue31510  opened by haypo

#31511: test_normalization: test.support.open_urlresource() doesn't ha
https://bugs.python.org/issue31511  opened by haypo

#31512: Add non-elevated symlink support for dev mode Windows 10
https://bugs.python.org/issue31512  opened by vidartf

#31516: current_thread() becomes "dummy" thread during shutdown
https://bugs.python.org/issue31516  opened by pitrou

#31517: MainThread association logic is fragile
https://bugs.python.org/issue31517  opened by pitrou

#31518: ftplib, urllib2, poplib, httplib, urllib2_localnet use ssl.PRO
https://bugs.python.org/issue31518  opened by doko

#31520: ResourceWarning: unclosed  w
https://bugs.python.org/issue31520  opened by haypo

#31521: segfault in PyBytes_AsString
https://bugs.python.org/issue31521  opened by Tim Smith

#31522: _mboxMMDF.get_string() fails to pass param to get_bytes()
https://bugs.python.org/issue31522  opened by bpoaugust

#31523: Windows build file fixes
https://bugs.python.org/issue31523  opened by steve.dower

#31524: mailbox._mboxMMDF.get_message throws away From envelope
https://bugs.python.org/issue31524  opened by bpoaugust

#31526: Allow setting timestamp in gzip-compressed tarfiles
https://bugs.python.org/issue31526  opened by randombit

#31527: Report details of excess arguments in object_new & object_init
https://bugs.python.org/issue31527  opened by ncoghlan

#31528: Let ConfigParser parse systemd units
https://bugs.python.org/issue31528  opened by johnlinp

#31529: IDLE: Add docstrings and tests for editor.py reload functions
https://bugs.python.org/issue31529  opened by csabella

#31530: Python 2.7 readahead feature of file objects is not thread saf
https://bugs.python.org/issue31530  opened by haypo

#31531: crash and SystemError in case of a bad zipimport._zip_director
https://bugs.python.org/issue31531  opened by Oren Milman

#31534: python 3.6.2 installation failed 0x80070002 error
https://bugs.python.org/issue31534  opened by roie

#31535: configparser unable to write comment with a upper cas letter
https://bugs.python.org/issue31535  opened by philippewagnie...@hispeed.ch

#31536: 

Re: [Python-Dev] SK-CSIRT identified malicious software libraries in the official Python package repository, PyPI

[Victor Stinner]

Hi,

FYI I just sent a public advisory for the PyPI typo squatting issue to
the new security-announce list:

[Security-announce] Typo squatting and malicious packages on PyPI
https://mail.python.org/pipermail/security-announce/2017-September/00.html

Please subscribe to this newly created mailing list to stay tuned!

https://mail.python.org/mailman/listinfo/security-announce

Victor

2017-09-15 22:28 GMT+02:00 Victor Stinner :
> Hi,
>
> Last week, the National Security Authority of Slovakia contacted the
> Python Security Response Team (PSRT) to report that the Python Package
> Index (PyPI) was hosting malicious packages. Installing these packages
> send user data to a HTTP server, but also install the expected module
> so it was an easy to notice the attack.
>
> Advisory: http://www.nbu.gov.sk/skcsirt-sa-20170909-pypi/
>
> Kudos to them to report the issue!
>
>
> It's not a compromise of the PyPI server nor a third-party project,
> but the "typo squatting" issue which is known since at least June 2016
> (for PyPI). The issue is not specific to Python, npmjs.com or
> rubygems.org are vulnerable to the same issue.
>
> For example, a malicious package used the names "urllib" (no 3) and
> "urlib3" (1 L) instead of "urllib3" (2 L). These packages were
> downloaded by users, so the attack was effective.
>
> More information on typo squatting and Python package security:
> https://python-security.readthedocs.io/packages.html#pypi-typo-squatting
>
> The PRST contacted PyPI administrators and all identified packages
> were taken down, only 1h10 after the PSRT received the email from the
> National Security Authority of Slovakia!
>
>
> The typo squatting issue is known and discussed, but not solution was
> found yet. See for example this warehouse issue:
> https://github.com/pypa/warehouse/issues/2151
>
> It seems like the consensus is that pip is not responsible to detect
> malicious code, it's more the responsability of PyPI.
>
> The problem is to decide how to detect malicious code and/or prevent
> typo squatting on PyPI.
>
>
> The issue has been discussed privately on the PSRT list last week. The
> National Security Authority of Slovakia just published their advisory,
> and a public discussion started on reddit:
> https://news.ycombinator.com/item?id=15256121
>
> I consider that it's now time to find a solution on the public
> python-dev mailing list.
>
>
> Let's try to find a solution!
>
> Can we learn something from the Update Framework (TUF)?
>
> How does Javascript, Ruby, Perl and other programming languages deal
> with these security issues on their package manager?
>
>
> See also my other notes on Python security and the list of known
> CPython vulnerabilities:
> https://python-security.readthedocs.io/
>
> Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com