Re: RFC: Python minimization in Fedora

2020-03-28 Thread Miro Hrončok
On 15. 01. 20 23:59, Zbigniew Jędrzejewski-Szmek wrote: ### Solution 5: Stop shipping mandatory bytecode cache This solution sounds simple: We do no longer ship the bytecode cache mandatorily. Technically, we move the `.pyc` files to a subpackage of `python3-libs` (or three different

Re: RFC: Python minimization in Fedora

2020-03-28 Thread Miro Hrončok
On 15. 01. 20 23:59, Zbigniew Jędrzejewski-Szmek wrote: ### Solution 5: Stop shipping mandatory bytecode cache This solution sounds simple: We do no longer ship the bytecode cache mandatorily. Technically, we move the `.pyc` files to a subpackage of `python3-libs` (or three different

Re: RFC: Python minimization in Fedora

2020-01-19 Thread Zbigniew Jędrzejewski-Szmek
On Sat, Jan 18, 2020 at 03:35:29PM -0500, James Cassell wrote: > > On Thu, Jan 16, 2020, at 5:16 AM, Zbigniew Jędrzejewski-Szmek wrote: > > > > A quick benchmark: > > $ time python3 -c 'import importlib as i, pydoc_data.topics as t; > > [i.reload(t) for _ in range(1)]' > > python3 -c

Re: RFC: Python minimization in Fedora

2020-01-18 Thread James Cassell
On Thu, Jan 16, 2020, at 5:16 AM, Zbigniew Jędrzejewski-Szmek wrote: > > A quick benchmark: > $ time python3 -c 'import importlib as i, pydoc_data.topics as t; > [i.reload(t) for _ in range(1)]' > python3 -c 4.16s user 0.45s system 99% cpu 4.646 total [...] > sudo rm

Re: RFC: Python minimization in Fedora

2020-01-18 Thread Barry Scott
> On 15 Jan 2020, at 17:05, Miro Hrončok wrote: > > In Python Maint, we sat down and we came up with several ideas how to > minimize the filesystem footprint of Python. Unfortunately, the result is > horribly long, sorry about that. Did you calculate file sizes including rounding up by the

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Miro Hrončok
On 16. 01. 20 21:55, David Malcolm wrote: If a traceback for an exception includes files from the .zip, can the traceback-printing machinery still print the pertinent lines of source? Apparently no: $ echo 0/0 > t.py $ zip t.zip t.py adding: t.py (stored 0%) $ python -c 'import t' Traceback

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Nicolas Mailhot via devel
Le jeudi 16 janvier 2020 à 22:00 +0100, Felix Schwarz a écrit : > > If I understood Nicolas correctly this was about installing multiple > versions > of the same *library* in the global Python site-packages directory? Whatever you wish to call it :) The non stdlib parts projects do not seem to

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Felix Schwarz
Am 16.01.20 um 21:15 schrieb Zbigniew Jędrzejewski-Szmek: >> Accommodating component versioning would mean deploying >> >> /usr/lib/pythonxx/site-packages/something-semver.zip > > This path includes xx, which contains the major and minor numbers. So > adding "semver" would only allow

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Felix Schwarz
Am 16.01.20 um 21:15 schrieb Zbigniew Jędrzejewski-Szmek: >> Accommodating component versioning would mean deploying >> >> /usr/lib/pythonxx/site-packages/something-semver.zip > > This path includes xx, which contains the major and minor numbers. So > adding "semver" would only allow

Re: RFC: Python minimization in Fedora

2020-01-16 Thread David Malcolm
On Thu, 2020-01-16 at 10:27 +0100, Miro Hrončok wrote: > On 15. 01. 20 23:11, Victor Stinner wrote: > > > Solution 4: ZIP the entire standard library > > > (...) > > > Nevertheless, this might (in theory) **save 17.8 MiB / 47 %**. > > > > It's my favorite option. Almost 50% smaller is quite good!

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Zbigniew Jędrzejewski-Szmek
On Thu, Jan 16, 2020 at 03:36:11PM +0100, Nicolas Mailhot via devel wrote: > Le 2020-01-16 15:10, Felix Schwarz a écrit : > >Am 16.01.20 um 13:37 schrieb Nicolas Mailhot via devel: > >>If we start messing with the Python tree it would be nice to put > >>each shared > >>python component in a

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Zbigniew Jędrzejewski-Szmek
On Thu, Jan 16, 2020 at 03:36:11PM +0100, Nicolas Mailhot via devel wrote: > Le 2020-01-16 15:10, Felix Schwarz a écrit : > >Am 16.01.20 um 13:37 schrieb Nicolas Mailhot via devel: > >>If we start messing with the Python tree it would be nice to put > >>each shared > >>python component in a

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Neal Gompa
On Thu, Jan 16, 2020 at 2:13 PM Christian Glombek wrote: > > On a side note (and without reading all of the above in detail), I'd like to > note that Fedora CoreOS (aka FCOS) is completely Python free by now - > probably not achievable for Desktop, but it may be for IoT. > Unfortunately

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Christian Glombek
On a side note (and without reading all of the above in detail), I'd like to note that Fedora CoreOS (aka FCOS) is completely Python free by now - probably not achievable for Desktop, but it may be for IoT. Miroslav Suchý schrieb am Do., 16. Jan. 2020, 16:35: > Dne 15. 01. 20 v 18:05 Miro

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Miroslav Suchý
Dne 15. 01. 20 v 18:05 Miro Hrončok napsal(a): > ### Solution 2: Move developer oriented modules to python3-devel (or split > the stdlib into pieces) +1 > ### Solution 5: Stop shipping mandatory bytecode cache +1 > Problem 5.1: Slower starts without bytecode cache Especially in

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Nicolas Mailhot via devel
Le 2020-01-16 15:10, Felix Schwarz a écrit : Am 16.01.20 um 13:37 schrieb Nicolas Mailhot via devel: If we start messing with the Python tree it would be nice to put each shared python component in a separate zip/xz/whatever, and allow versioning those archives (ie use the highest semver zip

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Felix Schwarz
Am 16.01.20 um 13:37 schrieb Nicolas Mailhot via devel: > If we start messing with the Python tree it would be nice to put each shared > python component in a separate zip/xz/whatever, and allow versioning those > archives > > (ie use the highest semver zip present unless the code explicitely

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Nicolas Mailhot via devel
Le 2020-01-16 11:18, Felix Schwarz a écrit : Am 15.01.20 um 23:11 schrieb Victor Stinner: This solution is well supported by unmodified Python: it's part of the default sys.path search path: $ python3 Python 3.7.6 (default, Dec 19 2019, 22:52:49) import sys; sys.path ['',

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Miro Hrončok
On 16. 01. 20 11:16, Zbigniew Jędrzejewski-Szmek wrote: So we clearly need to keep the possibility of installing .pyc files, at least optionally. To clarify: We would keep the .pyc files by default. We would only provide an opt-out. No, it will not (TTBMK). The file (or the lack of it)

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Felix Schwarz
Am 15.01.20 um 23:11 schrieb Victor Stinner: > This solution is well supported by unmodified Python: it's part of the > default sys.path search path: > > $ python3 > Python 3.7.6 (default, Dec 19 2019, 22:52:49) import sys; sys.path > ['', '/usr/lib64/python37.zip', ...] > > It's the

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Zbigniew Jędrzejewski-Szmek
On Thu, Jan 16, 2020 at 10:33:32AM +0100, Miro Hrončok wrote: > On 15. 01. 20 23:59, Zbigniew Jędrzejewski-Szmek wrote: > >On Wed, Jan 15, 2020 at 06:05:42PM +0100, Miro Hrončok wrote: > >>### File types (and bytecode caches) > >> > >>The orthogonal dimension is the file type. Python standard

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Miro Hrončok
On 15. 01. 20 23:59, Zbigniew Jędrzejewski-Szmek wrote: On Wed, Jan 15, 2020 at 06:05:42PM +0100, Miro Hrončok wrote: ### File types (and bytecode caches) The orthogonal dimension is the file type. Python standard library contains directories with both "extension modules" (written in C

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Miro Hrončok
On 15. 01. 20 23:11, Victor Stinner wrote: Solution 4: ZIP the entire standard library (...) Nevertheless, this might (in theory) **save 17.8 MiB / 47 %**. It's my favorite option. Almost 50% smaller is quite good! It would be very efficient to have such disk space gain! Using a ZIP file for

Re: RFC: Python minimization in Fedora

2020-01-16 Thread Miro Hrončok
On 15. 01. 20 18:56, Chris wrote: That's an amazing amount of work! Thanks. My only criticism would be: - the quest for reducing disk space is getting a bit over the top.  I mean to make comparisons to 3.5" floppy disks which haven't been around for 20 years? That is obviously only used

Re: RFC: Python minimization in Fedora

2020-01-15 Thread Zbigniew Jędrzejewski-Szmek
On Wed, Jan 15, 2020 at 06:05:42PM +0100, Miro Hrončok wrote: > ### File types (and bytecode caches) > > The orthogonal dimension is the file type. Python standard library > contains directories with both "extension modules" (written in C > (usually) and compiled to

Re: RFC: Python minimization in Fedora

2020-01-15 Thread Zbigniew Jędrzejewski-Szmek
On Wed, Jan 15, 2020 at 06:05:42PM +0100, Miro Hrončok wrote: > ### File types (and bytecode caches) > > The orthogonal dimension is the file type. Python standard library > contains directories with both "extension modules" (written in C > (usually) and compiled to

Re: RFC: Python minimization in Fedora

2020-01-15 Thread Victor Stinner
> Solution 4: ZIP the entire standard library > (...) > Nevertheless, this might (in theory) **save 17.8 MiB / 47 %**. It's my favorite option. Almost 50% smaller is quite good! It would be very efficient to have such disk space gain! Using a ZIP file for the stdlib is commonly suggested

Re: RFC: Python minimization in Fedora

2020-01-15 Thread Victor Stinner
> Solution 4: ZIP the entire standard library > (...) > Nevertheless, this might (in theory) **save 17.8 MiB / 47 %**. It's my favorite option. Almost 50% smaller is quite good! It would be very efficient to have such disk space gain! Using a ZIP file for the stdlib is commonly suggested

Re: RFC: Python minimization in Fedora

2020-01-15 Thread Łukasz Posadowski
Wed, 15 Jan 2020 18:05:42 +0100 Miro Hrončok : > Hello Fedora! > > In Python Maint, we sat down and we came up with several ideas how to > minimize the filesystem footprint of Python. Unfortunately, the > result is horribly long, sorry about that. It was delightfull to read. I have some better

Re: RFC: Python minimization in Fedora

2020-01-15 Thread Przemek Klosowski via devel
On 1/15/20 12:56 PM, Chris wrote: That's an amazing amount of work! My only criticism would be: - the quest for reducing disk space is getting a bit over the top.  I mean to make comparisons to 3.5" floppy disks which haven't been around for 20 years? Why is ~100MB so much? If you scale up

Re: RFC: Python minimization in Fedora

2020-01-15 Thread Chris
That's an amazing amount of work! My only criticism would be: - the quest for reducing disk space is getting a bit over the top. I mean to make comparisons to 3.5" floppy disks which haven't been around for 20 years? Why is ~100MB so much? If you scale up from floppy disks and even reference a

RFC: Python minimization in Fedora

2020-01-15 Thread Miro Hrončok
Hello Fedora! In Python Maint, we sat down and we came up with several ideas how to minimize the filesystem footprint of Python. Unfortunately, the result is horribly long, sorry about that. Please, share your feedback, additional solutions, comments etc. Version with formatting and

RFC: Python minimization in Fedora

2020-01-15 Thread Miro Hrončok
Hello Fedora! In Python Maint, we sat down and we came up with several ideas how to minimize the filesystem footprint of Python. Unfortunately, the result is horribly long, sorry about that. Please, share your feedback, additional solutions, comments etc. Version with formatting and