Re: [Python-Dev] Python startup time - daemon

[Guido van Rossum]

On Fri, May 11, 2018 at 11:57 PM, Barry Warsaw  wrote:

> On May 11, 2018, at 12:23, Guido van Rossum  wrote:
> >
> > Indeed, we have an implementation of this specific to mypy.
>
> Is there anything in mypy’s implementation that can be generalized into a
> library?
>

Not sure, here's the code:
https://github.com/python/mypy/blob/master/mypy/dmypy.py
https://github.com/python/mypy/blob/master/mypy/dmypy_server.py
(also dmypy_util.py there)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time - daemon

[Barry Warsaw]

On May 11, 2018, at 12:23, Guido van Rossum  wrote:
> 
> Indeed, we have an implementation of this specific to mypy.

Is there anything in mypy’s implementation that can be generalized into a 
library?

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (Looking for) A Retrospective on the Move to Python 3

[Chris Barker - NOAA Federal via Python-Dev]

> while the changes introduced by Python 3
> affect pretty much everyone, even people who only write small simple
> scripts.

Sure they do, but the *hard stuff* not so much.

I have found 2to3 conversion to be remarkably easy and painless.

And the whole Unicode thing is much easier.

CHB


> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time - daemon

[Guido van Rossum]

Indeed, we have an implementation of this specific to mypy.

On Fri, May 11, 2018 at 11:34 AM, Antoine Pitrou 
wrote:

>
> Yes, you don't want this to be a generic utility, rather a helper
> library that people can integrate into their command-line applications
> to enable such startup caching.
>
> Regards
>
> Antoine.
>
>
> On Fri, 11 May 2018 17:27:35 +0200
> Oleg Broytman  wrote:
> > On Fri, May 11, 2018 at 07:38:05AM -0700, Chris Barker - NOAA Federal
> via Python-Dev  wrote:
> > > Could one make a little startup utility that, when invoked the first
> > > time, starts up a raw python interpreter, keeps it running somewhere,
> > > and then forks it to run the actual python code.
> > >
> > > Then every invocation after that would make a new fork.
> >
> >Used to be implemented (and discussed in this list) many times. Just
> > a few examples:
> >
> > http://readyexec.sourceforge.net/
> > https://blogs.gnome.org/johan/2007/01/18/introducing-python-launcher/
> >
> >Proven to be hard and never gain any traction.
> >
> > a) you don't want the daemon to import all possible modules so you need
> >to run a separate copy of the daemon for every Python version, every
> >user and every client program;
> > b) you need to find "your" daemon - using TCP? unix sockets? named pipes?
> > b) need to redirect stdio to/from the daemon;
> > c) need to redirect signals and exceptions;
> > d) have problems with elevated privileges (how do you elevate the daemon
> >if the client was started with `sudo -H`?);
> > e) not portable (there is a popular GUI that cannot fork).
> >
> > > -CHB
> > > Sent from my iPhone
> >
> > Oleg.
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2018-05-04 - 2018-05-11)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open6644 (+15)
  closed 38571 (+24)
  total  45215 (+39)

Open issues with patches: 2604 


Issues opened (34)
==

#14384: Add "default" kw argument to operator.itemgetter and operator.
https://bugs.python.org/issue14384  reopened by gvanrossum

#33426: Behavior of os.path.join does not match documentation
https://bugs.python.org/issue33426  opened by Michael Klatt

#33427: Dead link in "The Python Standard Library" page
https://bugs.python.org/issue33427  opened by lighthawk

#33428: pathlib.Path.glob does not follow symlinks
https://bugs.python.org/issue33428  opened by brianmsheldon

#33430: Import secrets module in secrets examples
https://bugs.python.org/issue33430  opened by dchimeno

#33431: Change description about doc in programming, faq.
https://bugs.python.org/issue33431  opened by lvhuiyang

#33433: ipaddress is_private misleading for IPv4 mapped IPv6 addresses
https://bugs.python.org/issue33433  opened by Thomas Kriechbaumer

#33435: incorrect detection of information of some distributions
https://bugs.python.org/issue33435  opened by mrandybu

#33436: Add an interactive shell for Sqlite3
https://bugs.python.org/issue33436  opened by rhettinger

#33437: Defining __init__ in enums
https://bugs.python.org/issue33437  opened by killerrex

#33438: pkg-config file misses flags for static linking
https://bugs.python.org/issue33438  opened by pitrou

#33439: python-config.py should be part of the stdlib
https://bugs.python.org/issue33439  opened by pitrou

#33440: Possible lazy import opportunities in `pathlib`
https://bugs.python.org/issue33440  opened by ncoghlan

#33441: Expose the sigset_t converter via private API
https://bugs.python.org/issue33441  opened by serhiy.storchaka

#33442: Python 3 doc sidebar dosnt follow page scrolling like 2.7 doc 
https://bugs.python.org/issue33442  opened by pete312

#33443: Typo in Python/import.c
https://bugs.python.org/issue33443  opened by ukwksk

#33446: destructors of local variables are not traced
https://bugs.python.org/issue33446  opened by xdegaye

#33447: Asynchronous lambda syntax
https://bugs.python.org/issue33447  opened by Noah Simon

#33448: Output_Typo_Error
https://bugs.python.org/issue33448  opened by vishva_11

#33449: Documentation for email.charset confusing about the location o
https://bugs.python.org/issue33449  opened by Francois Labelle

#33450: unexpected EPROTOTYPE returned by sendto on MAC OSX
https://bugs.python.org/issue33450  opened by racitup

#33451: Start pyc file lock the file
https://bugs.python.org/issue33451  opened by Jean-Louis Tamburini

#33452: add user notification that parent init will not be called in d
https://bugs.python.org/issue33452  opened by Ricyteach

#33453: from __future__ import annotations breaks dataclasses ClassVar
https://bugs.python.org/issue33453  opened by Ricyteach

#33454: Mismatched C function signature in _xxsubinterpreters.channel_
https://bugs.python.org/issue33454  opened by serhiy.storchaka

#33456: site.py: by default, a virtual environment is *not* isolated f
https://bugs.python.org/issue33456  opened by meribold

#33457: python-config ldflags, PEP 513 and explicit linking to libpyth
https://bugs.python.org/issue33457  opened by dimi

#33458: pdb.run() does not trace destructors of __main__
https://bugs.python.org/issue33458  opened by xdegaye

#33459: Define "tuple display" in the docs
https://bugs.python.org/issue33459  opened by adelfino

#33460: ... used to indicate many different things in chapter 3, some 
https://bugs.python.org/issue33460  opened by lew18

#33461: json.loads(encoding=) does not emit deprecation warn
https://bugs.python.org/issue33461  opened by mbussonn

#33462: reversible dict
https://bugs.python.org/issue33462  opened by selik

#33463: Can namedtuple._asdict return a regular dict instead of Ordere
https://bugs.python.org/issue33463  opened by selik

#33464: AIX and "specialized downloads" links
https://bugs.python.org/issue33464  opened by Michael.Felt



Most recent 15 issues with no replies (15)
==

#33462: reversible dict
https://bugs.python.org/issue33462

#33460: ... used to indicate many different things in chapter 3, some 
https://bugs.python.org/issue33460

#33457: python-config ldflags, PEP 513 and explicit linking to libpyth
https://bugs.python.org/issue33457

#33456: site.py: by default, a virtual environment is *not* isolated f
https://bugs.python.org/issue33456

#33454: Mismatched C function signature in _xxsubinterpreters.channel_
https://bugs.python.org/issue33454

#33451: Start pyc file lock the file
https://bugs.python.org/issue33451

#33443: Typo in Python/import.c
https://bugs.python.org/issue33443

#33442: Python 3 doc sidebar dosnt follow page scrolling like 2.7 

Re: [Python-Dev] Python startup time - daemon

[Antoine Pitrou]

Yes, you don't want this to be a generic utility, rather a helper
library that people can integrate into their command-line applications
to enable such startup caching.

Regards

Antoine.


On Fri, 11 May 2018 17:27:35 +0200
Oleg Broytman  wrote:
> On Fri, May 11, 2018 at 07:38:05AM -0700, Chris Barker - NOAA Federal via 
> Python-Dev  wrote:
> > Could one make a little startup utility that, when invoked the first
> > time, starts up a raw python interpreter, keeps it running somewhere,
> > and then forks it to run the actual python code.
> > 
> > Then every invocation after that would make a new fork.  
> 
>Used to be implemented (and discussed in this list) many times. Just
> a few examples:
> 
> http://readyexec.sourceforge.net/
> https://blogs.gnome.org/johan/2007/01/18/introducing-python-launcher/
> 
>Proven to be hard and never gain any traction.
> 
> a) you don't want the daemon to import all possible modules so you need
>to run a separate copy of the daemon for every Python version, every
>user and every client program;
> b) you need to find "your" daemon - using TCP? unix sockets? named pipes?
> b) need to redirect stdio to/from the daemon;
> c) need to redirect signals and exceptions;
> d) have problems with elevated privileges (how do you elevate the daemon
>if the client was started with `sudo -H`?);
> e) not portable (there is a popular GUI that cannot fork).
> 
> > -CHB
> > Sent from my iPhone  
> 
> Oleg.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time - daemon

[Oleg Broytman]

On Fri, May 11, 2018 at 07:38:05AM -0700, Chris Barker - NOAA Federal via 
Python-Dev  wrote:
> Could one make a little startup utility that, when invoked the first
> time, starts up a raw python interpreter, keeps it running somewhere,
> and then forks it to run the actual python code.
> 
> Then every invocation after that would make a new fork.

   Used to be implemented (and discussed in this list) many times. Just
a few examples:

http://readyexec.sourceforge.net/
https://blogs.gnome.org/johan/2007/01/18/introducing-python-launcher/

   Proven to be hard and never gain any traction.

a) you don't want the daemon to import all possible modules so you need
   to run a separate copy of the daemon for every Python version, every
   user and every client program;
b) you need to find "your" daemon - using TCP? unix sockets? named pipes?
b) need to redirect stdio to/from the daemon;
c) need to redirect signals and exceptions;
d) have problems with elevated privileges (how do you elevate the daemon
   if the client was started with `sudo -H`?);
e) not portable (there is a popular GUI that cannot fork).

> -CHB
> Sent from my iPhone

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Ryan Gonzalez]

 https://refi64.com/uprocd/ 


On May 11, 2018 9:39:28 AM Chris Barker - NOAA Federal via Python-Dev 
 wrote:



Inspired by chg:

Could one make a little startup utility that, when invoked the first
time, starts up a raw python interpreter, keeps it running somewhere,
and then forks it to run the actual python code.

Then every invocation after that would make a new fork. I presume
forking is a LOT faster than re-invoking the entire startup.

I suspect that many of the cases where startup time really matters is
when a command line utility is likely to be invoked many times — often
in the same shell instance.

So having a “pre-built” warm interpreter ready to go could really help.

This is way past my technical expertise to know if it’s possible, or
to try to prototype it, but I’m sure many of you would know.

-CHB

Sent from my iPhone


On May 7, 2018, at 12:28 PM, Neil Schemenauer  wrote:

On 2018-05-03, Lukasz Langa wrote:

On May 2, 2018, at 8:57 PM, INADA Naoki  wrote:
* Add lazy compiling API or flag in `re` module.  The pattern is compiled
when first used.


How about go the other way and allow compiling at Python
*compile*-time? That would actually make things faster instead of
just moving the time spent around.


Lisp has a special form 'eval-when'.  It can be used to cause
evaluation of the body expression at compile time.

In Carl's "A fast startup patch" post, he talks about getting rid of
the unmarshal step and storing objects in the heap segment of the
executable.  Those would be the objects necessary to evaluate code.
The marshal module has a limited number of types that it handle.
I believe they are: bool, bytes, code objects, complex, Ellipsis
float, frozenset, int, None, tuple and str.

If the same mechanism could handle more types, rather than storing
the code to be evaluated, we could store the objects created after
evaluation of the top-level module body.  Or, have a mechanism to
mark which code should be evaluated at compile time (much like the
eval-when form).

For the re.compile example, the compiled regex could be what is
stored after compiling the Python module (i.e. the re.compile gets
run at compile time).  The objects created by re.compile (e.g.
SRE_Pattern) would have to be something that the heap dumper could
handle.

Traditionally, Python has had the model "there is only runtime".
So, starting to do things at compile time complicates that model.

Regards,

 Neil
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Chris Barker - NOAA Federal via Python-Dev]

Inspired by chg:

Could one make a little startup utility that, when invoked the first
time, starts up a raw python interpreter, keeps it running somewhere,
and then forks it to run the actual python code.

Then every invocation after that would make a new fork. I presume
forking is a LOT faster than re-invoking the entire startup.

I suspect that many of the cases where startup time really matters is
when a command line utility is likely to be invoked many times — often
in the same shell instance.

So having a “pre-built” warm interpreter ready to go could really help.

This is way past my technical expertise to know if it’s possible, or
to try to prototype it, but I’m sure many of you would know.

-CHB

Sent from my iPhone

> On May 7, 2018, at 12:28 PM, Neil Schemenauer  wrote:
>
> On 2018-05-03, Lukasz Langa wrote:
>>> On May 2, 2018, at 8:57 PM, INADA Naoki  wrote:
>>> * Add lazy compiling API or flag in `re` module.  The pattern is compiled
>>> when first used.
>>
>> How about go the other way and allow compiling at Python
>> *compile*-time? That would actually make things faster instead of
>> just moving the time spent around.
>
> Lisp has a special form 'eval-when'.  It can be used to cause
> evaluation of the body expression at compile time.
>
> In Carl's "A fast startup patch" post, he talks about getting rid of
> the unmarshal step and storing objects in the heap segment of the
> executable.  Those would be the objects necessary to evaluate code.
> The marshal module has a limited number of types that it handle.
> I believe they are: bool, bytes, code objects, complex, Ellipsis
> float, frozenset, int, None, tuple and str.
>
> If the same mechanism could handle more types, rather than storing
> the code to be evaluated, we could store the objects created after
> evaluation of the top-level module body.  Or, have a mechanism to
> mark which code should be evaluated at compile time (much like the
> eval-when form).
>
> For the re.compile example, the compiled regex could be what is
> stored after compiling the Python module (i.e. the re.compile gets
> run at compile time).  The objects created by re.compile (e.g.
> SRE_Pattern) would have to be something that the heap dumper could
> handle.
>
> Traditionally, Python has had the model "there is only runtime".
> So, starting to do things at compile time complicates that model.
>
> Regards,
>
>  Neil
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Associated images in PEPs broken?

[Chris Angelico]

On Fri, May 11, 2018 at 9:33 PM, Chris Angelico  wrote:
> https://www.python.org/dev/peps/pep-0495/
>
> All the images seem to be missing - showing up 404. They're in the
> peps repository, but aren't showing up in the page. Who's in charge of
> the HTML rendering there? Infrastructure?
>
> ChrisA

Never mind! They just came good. I probably saw the page right as a
build was taking place or something.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Associated images in PEPs broken?

[Chris Angelico]

https://www.python.org/dev/peps/pep-0495/

All the images seem to be missing - showing up 404. They're in the
peps repository, but aren't showing up in the page. Who's in charge of
the HTML rendering there? Infrastructure?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com