Re: [Python-ideas] Move optional data out of pyc files

On Sat, 14 Apr 2018 at 17:01 Neil Schemenauer wrote: > On 2018-04-12, M.-A. Lemburg wrote: > > This leaves the proposal to restructure pyc files into a sectioned > > file and possibly indexed file to make access to (lazily) loaded > > parts faster. > > I would like

Re: [Python-ideas] Move optional data out of pyc files

On 2018-04-12, M.-A. Lemburg wrote: > This leaves the proposal to restructure pyc files into a sectioned > file and possibly indexed file to make access to (lazily) loaded > parts faster. I would like to see a format can hold one or more modules in a single file. Something like the zip format

Re: [Python-ideas] Move optional data out of pyc files

On Fri, 13 Apr 2018 at 03:47, M.-A. Lemburg wrote: > I think moving data out of pyc files is going in a wrong direction: > more stat calls means slower import and slower startup time. > > Trying to make pycs smaller also isn't really worth it (they > compress quite well). > >

Re: [Python-ideas] Move optional data out of pyc files

I've been playing a bit with this trying to collect some data and measure how useful this would be. You can take a look at the script I'm using at: https://github.com/dmoisset/pycstats What I'm measuring is: 1. Number of objects in the pyc, and how many of those are: * docstrings (I'm using a

Re: [Python-ideas] Move optional data out of pyc files

One implementation difficulty specifically related to annotations, is that they are quite hard to find/extract from the code objects. Both docstrings and lnotab are within specific fields of the code object for their function/class/module; annotations are spread as individual constants (assuming

Re: [Python-ideas] Move optional data out of pyc files

> Finally, loading docstrings and other optional components can be made lazy. > This was not in my original idea, and this will significantly complicate the > implementation, but in principle it is possible. This will require larger > changes in the marshal format and bytecode. I'm +1 on this

Re: [Python-ideas] Move optional data out of pyc files

10.04.18 20:38, Chris Angelico пише: On Wed, Apr 11, 2018 at 2:14 AM, Serhiy Storchaka wrote: A deployed Python distribution generally has .pyc files for all of the standard library. I don't think people want to lose the ability to call help(), and unless I'm

Re: [Python-ideas] Move optional data out of pyc files

On Thu, Apr 12, 2018 at 11:59 AM, Steven D'Aprano wrote: > On Thu, Apr 12, 2018 at 12:09:38AM +1000, Chris Angelico wrote: > > [...] >> >> Consider a very common use-case: an OS-provided >> >> Python interpreter whose files are all owned by 'root'. Those will be >> >>

Re: [Python-ideas] Move optional data out of pyc files

On Thu, Apr 12, 2018 at 12:09:38AM +1000, Chris Angelico wrote: [...] > >> Consider a very common use-case: an OS-provided > >> Python interpreter whose files are all owned by 'root'. Those will be > >> distributed with .pyc files for performance, but you don't want to > >> deprive the users of

Re: [Python-ideas] Move optional data out of pyc files

On 4/11/2018 4:26 AM, Petr Viktorin wrote: Currently in Fedora, we ship *both* optimized and non-optimized pycs to make sure both -O and non--O will work nicely without root privilieges. So splitting the docstrings into a separate file would be, for us, a benefit in terms of file size.

Re: [Python-ideas] Move optional data out of pyc files

On Wed, Apr 11, 2018 at 4:06 PM, Steven D'Aprano wrote: > On Wed, Apr 11, 2018 at 02:21:17PM +1000, Chris Angelico wrote: > > [...] >> > Yes, it will double the number of files. Actually quadruple it, if the >> > annotations and line numbers are in separate files too. But if

Re: [Python-ideas] Move optional data out of pyc files

On Tue, Apr 10, 2018 at 9:50 PM, Eric V. Smith wrote: > >>> 3. Annotations. They are used mainly by third party tools that >>> statically analyze sources. They are rarely used at runtime. >> >> Even less used than docstrings probably. > > typing.NamedTuple and dataclasses use

Re: [Python-ideas] Move optional data out of pyc files

On 04/11/18 08:06, Steven D'Aprano wrote: On Wed, Apr 11, 2018 at 02:21:17PM +1000, Chris Angelico wrote: [...] Yes, it will double the number of files. Actually quadruple it, if the annotations and line numbers are in separate files too. But if most of those extra files never need to be

Re: [Python-ideas] Move optional data out of pyc files

On 04/11/18 06:21, Chris Angelico wrote: On Wed, Apr 11, 2018 at 1:02 PM, Steven D'Aprano wrote: On Wed, Apr 11, 2018 at 10:08:58AM +1000, Chris Angelico wrote: File system limits aren't usually an issue; as you say, even FAT32 can store a metric ton of files in a single

Re: [Python-ideas] Move optional data out of pyc files

On 10/04/2018 18:54, Zachary Ware wrote: > On Tue, Apr 10, 2018 at 12:38 PM, Chris Angelico wrote: >> A deployed Python distribution generally has .pyc files for all of the >> standard library. I don't think people want to lose the ability to >> call help(), and unless I'm

Re: [Python-ideas] Move optional data out of pyc files

On Wed, Apr 11, 2018 at 02:21:17PM +1000, Chris Angelico wrote: [...] > > Yes, it will double the number of files. Actually quadruple it, if the > > annotations and line numbers are in separate files too. But if most of > > those extra files never need to be opened, then there's no cost to them.

Re: [Python-ideas] Move optional data out of pyc files

On Wed, Apr 11, 2018 at 1:02 PM, Steven D'Aprano wrote: > On Wed, Apr 11, 2018 at 10:08:58AM +1000, Chris Angelico wrote: > >> File system limits aren't usually an issue; as you say, even FAT32 can >> store a metric ton of files in a single directory. I'm more interested >>

Re: [Python-ideas] Move optional data out of pyc files

On Wed, Apr 11, 2018 at 10:08:58AM +1000, Chris Angelico wrote: > File system limits aren't usually an issue; as you say, even FAT32 can > store a metric ton of files in a single directory. I'm more interested > in how long it takes to open a file, and whether doubling that time > will have a

Re: [Python-ideas] Move optional data out of pyc files

On Tue, Apr 10, 2018 at 5:03 PM, Steven D'Aprano wrote: > > __pycache__/spam.cpython-38.pyc > __pycache__/spam.cpython-38-doc.pyc > __pycache__/spam.cpython-38-lno.pyc > __pycache__/spam.cpython-38-ann.pyc > ​Our product uses the doc strings for

Re: [Python-ideas] Move optional data out of pyc files

On Tue, Apr 10, 2018 at 12:51 PM Eric V. Smith wrote: > > >> 3. Annotations. They are used mainly by third party tools that > >> statically analyze sources. They are rarely used at runtime. > > > > Even less used than docstrings probably. > > typing.NamedTuple and dataclasses

Re: [Python-ideas] Move optional data out of pyc files

On Wed, Apr 11, 2018 at 10:03 AM, Steven D'Aprano wrote: > On Wed, Apr 11, 2018 at 03:38:08AM +1000, Chris Angelico wrote: >> A deployed Python distribution generally has .pyc files for all of the >> standard library. I don't think people want to lose the ability to >> call

Re: [Python-ideas] Move optional data out of pyc files

On Wed, Apr 11, 2018 at 03:38:08AM +1000, Chris Angelico wrote: > On Wed, Apr 11, 2018 at 2:14 AM, Serhiy Storchaka wrote: > > Currently pyc files contain data that is useful mostly for developing and is > > not needed in most normal cases in stable program. There is even an

Re: [Python-ideas] Move optional data out of pyc files

>> 3. Annotations. They are used mainly by third party tools that >> statically analyze sources. They are rarely used at runtime. > > Even less used than docstrings probably. typing.NamedTuple and dataclasses use annotations at runtime. Eric ___

Re: [Python-ideas] Move optional data out of pyc files

On Tue, 10 Apr 2018 19:14:58 +0300 Serhiy Storchaka wrote: > Currently pyc files contain data that is useful mostly for developing > and is not needed in most normal cases in stable program. There is even > an option that allows to exclude a part of this information from

Re: [Python-ideas] Move optional data out of pyc files

On Tue, 10 Apr 2018 11:13:01 -0700 Ethan Furman wrote: > On 04/10/2018 10:54 AM, Zachary Ware wrote: > > On Tue, Apr 10, 2018 at 12:38 PM, Chris Angelico wrote: > >> A deployed Python distribution generally has .pyc files for all of the > >> standard

Re: [Python-ideas] Move optional data out of pyc files

I'm not sure I understand the benefit of this, perhaps you can clarify. What I see is two scenarios Scenario A) External files are present In this case, the data is loaded from the pyc and then from external file, so there are no savings in memory, startup time, disk space, or network load time,

Re: [Python-ideas] Move optional data out of pyc files

There are libraries out there like this: https://docopt.readthedocs.io/en/0.2.0/ which use docstrings for runtime info. Today we already have -OO which allows us to create docstring-less bytecode files in case we have, after careful consideration, established that it is safe to do so. I think

Re: [Python-ideas] Move optional data out of pyc files

On 04/10/2018 10:54 AM, Zachary Ware wrote: On Tue, Apr 10, 2018 at 12:38 PM, Chris Angelico wrote: A deployed Python distribution generally has .pyc files for all of the standard library. I don't think people want to lose the ability to call help(), and unless I'm

Re: [Python-ideas] Move optional data out of pyc files

On Tue, Apr 10, 2018 at 12:38 PM, Chris Angelico wrote: > A deployed Python distribution generally has .pyc files for all of the > standard library. I don't think people want to lose the ability to > call help(), and unless I'm misunderstanding, that requires > docstrings. So

Re: [Python-ideas] Move optional data out of pyc files

On Wed, Apr 11, 2018 at 2:14 AM, Serhiy Storchaka wrote: > Currently pyc files contain data that is useful mostly for developing and is > not needed in most normal cases in stable program. There is even an option > that allows to exclude a part of this information from pyc

Re: [Python-ideas] Move optional data out of pyc files

On Tue, 10 Apr 2018 19:14:58 +0300 Serhiy Storchaka wrote: > Currently pyc files contain data that is useful mostly for developing > and is not needed in most normal cases in stable program. There is even > an option that allows to exclude a part of this information from