Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Matthew Miller]

On Fri, Sep 25, 2015 at 10:09:50AM -0400, Paul Wouters wrote:
> It's nice to be able to edit the .py for testing without going through
> hoops or building/installing rpms.

Right, but you know that if you're doing that in /usr/lib, that's
*gross*, right? :)





-- 
Matthew Miller

Fedora Project Leader
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Orion Poplawski]

On 09/25/2015 08:36 AM, Matthew Miller wrote:

On Fri, Sep 25, 2015 at 10:09:50AM -0400, Paul Wouters wrote:

It's nice to be able to edit the .py for testing without going through
hoops or building/installing rpms.


Right, but you know that if you're doing that in /usr/lib, that's
*gross*, right? :)


To heck with gross, it's incredibly efficient.  I've done it numerous 
times to track down problems.  But that said, I'd be happy to install 
-pysource sub-package in order to do it.


--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA/CoRA DivisionFAX: 303-415-9702
3380 Mitchell Lane  or...@cora.nwra.com
Boulder, CO 80301  http://www.cora.nwra.com
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Christopher Meng]

On 9/25/15, Matthew Miller  wrote:
> On Fri, Sep 25, 2015 at 11:30:38AM -0400, Bohuslav Kabrda wrote:
>> Maybe we could utilize weak dependencies here. The python-foo package
>> would have only bytecompiled files and would Recommend
>> python-foo-sourcefiles. That way python-foo-sourcefiles would be
>> installed in standard setting, but it would be possible to opt out of
>> it.

Just my 2 cents here, I'd seek another solution for this but
implemented as a plugin or an option of dnf, whereas introducing new
subpkg of large amount of packages sounds tedious to me.

For example define a new kind of file as %cache in %files, and process
them like find-lang, then let users decide whether files marked as
%cache will be installed in default or not.

-- 

Yours sincerely,
Christopher Meng

http://awk.io
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Matthew Miller]

On Fri, Sep 25, 2015 at 11:30:38AM -0400, Bohuslav Kabrda wrote:
> Maybe we could utilize weak dependencies here. The python-foo package
> would have only bytecompiled files and would Recommend
> python-foo-sourcefiles. That way python-foo-sourcefiles would be
> installed in standard setting, but it would be possible to opt out of
> it.

Great idea!

-- 
Matthew Miller

Fedora Project Leader
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Przemek Klosowski]

On 09/25/2015 10:04 AM, Matthew Miller wrote:

On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote:

Also, you might consider to ship the precompiled bytecode just
optionally, using recommends.

On contrary, if you insist on shipping the bytecode, why you don't drop
the .py files? I see a lot of duplication all around python packages 

Wait, we can do that? Why don't we?

Everything I see in online discussion is centered on, basically,
transparency. But we wouldn't be doing it for obfuscation. The srpms
would still be there, and for that matter we could ship the .py files
in a subpackage.

I wondered how much of a saving it'd be: I have around 250 python 
packages installed (plus python code from assorted other packages, of 
course).


The total size of .py files on my system is 320MB! There are even 24 .py 
files that are larger than 500kB (mostly from third party packages, like 
e.g. gnuradio, freecad).



-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Bohuslav Kabrda]

- Original Message -
> On 09/25/2015 08:36 AM, Matthew Miller wrote:
> > On Fri, Sep 25, 2015 at 10:09:50AM -0400, Paul Wouters wrote:
> >> It's nice to be able to edit the .py for testing without going through
> >> hoops or building/installing rpms.
> >
> > Right, but you know that if you're doing that in /usr/lib, that's
> > *gross*, right? :)
> 
> To heck with gross, it's incredibly efficient.  I've done it numerous
> times to track down problems.  But that said, I'd be happy to install
> -pysource sub-package in order to do it.

Maybe we could utilize weak dependencies here. The python-foo package would 
have only bytecompiled files and would Recommend python-foo-sourcefiles. That 
way python-foo-sourcefiles would be installed in standard setting, but it would 
be possible to opt out of it.

> --
> Orion Poplawski
> Technical Manager 303-415-9701 x222
> NWRA/CoRA DivisionFAX: 303-415-9702
> 3380 Mitchell Lane  or...@cora.nwra.com
> Boulder, CO 80301  http://www.cora.nwra.com
> --
> devel mailing list
> devel@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

-- 
Regards,
Slavek Kabrda
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Orion Poplawski]

On 09/25/2015 10:01 AM, Christopher Meng wrote:
> On 9/25/15, Matthew Miller  wrote:
>> On Fri, Sep 25, 2015 at 11:30:38AM -0400, Bohuslav Kabrda wrote:
>>> Maybe we could utilize weak dependencies here. The python-foo package
>>> would have only bytecompiled files and would Recommend
>>> python-foo-sourcefiles. That way python-foo-sourcefiles would be
>>> installed in standard setting, but it would be possible to opt out of
>>> it.
> 
> Just my 2 cents here, I'd seek another solution for this but
> implemented as a plugin or an option of dnf, whereas introducing new
> subpkg of large amount of packages sounds tedious to me.

Yes, that would be tedious.  I'd only be for it if it could be made automatic
- which I think could be possible, similar to how debuginfo is produced.  It
would be a lot trickier though as the files live side-by-side.

> 
> For example define a new kind of file as %cache in %files, and process
> them like find-lang, then let users decide whether files marked as
> %cache will be installed in default or not.
> 

This is an interesting idea too though.

-- 
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane   or...@nwra.com
Boulder, CO 80301   http://www.nwra.com
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Florian Weimer]

* Matthew Miller:

> On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote:
>> Also, you might consider to ship the precompiled bytecode just
>> optionally, using recommends.
>> 
>> On contrary, if you insist on shipping the bytecode, why you don't drop
>> the .py files? I see a lot of duplication all around python packages 
>
> Wait, we can do that? Why don't we?

It alters backtraces:

Here's a silly example:

Traceback (most recent call last):
  File "u.py", line 2, in 
t.t()
  File "/tmp/t.py", line 2, in t
a()
  File "/tmp/t.py", line 5, in a
b()
  File "/tmp/t.py", line 8, in b
c()
NameError: global name 'c' is not defined

This turns into:

Traceback (most recent call last):
  File "u.py", line 2, in 
t.t()
  File "/tmp/t.py", line 2, in t
  File "/tmp/t.py", line 5, in a
  File "/tmp/t.py", line 8, in b
NameError: global name 'c' is not defined

Such a change will not be universally well-received.

In any case, source code (with comments stripped and whitespace
normalized) is often smaller than bytecode (or most other forms of
serializing syntax), and compresses better.
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Vít Ondruch]

Dne 25.9.2015 v 16:04 Matthew Miller napsal(a):
> On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote:
>> Also, you might consider to ship the precompiled bytecode just
>> optionally, using recommends.

Just to be clear, not/optional shipping the bytecode is the preferred
method IMO 

>>
>> On contrary, if you insist on shipping the bytecode, why you don't drop
>> the .py files? I see a lot of duplication all around python packages 

I proposed this because some might complain that not shipping the
bytecode might result in some performance penalty, but I'd say that this
applies just for first run, so it is negligible

Vít

> Wait, we can do that? Why don't we?
>
> Everything I see in online discussion is centered on, basically,
> transparency. But we wouldn't be doing it for obfuscation. The srpms
> would still be there, and for that matter we could ship the .py files
> in a subpackage.
>


-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Vít Ondruch]

Dne 25.9.2015 v 16:15 Mathieu Bridon napsal(a):
> On Fri, 2015-09-25 at 10:04 -0400, Matthew Miller wrote:
>> On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote:
>>> Also, you might consider to ship the precompiled bytecode just
>>> optionally, using recommends.
>>>
>>> On contrary, if you insist on shipping the bytecode, why you don't
>>> drop
>>> the .py files? I see a lot of duplication all around python
>>> packages 
>> Wait, we can do that? Why don't we?
>>
>> Everything I see in online discussion is centered on, basically,
>> transparency. But we wouldn't be doing it for obfuscation. The srpms
>> would still be there, and for that matter we could ship the .py files
>> in a subpackage.
> Or maybe rpm could have a macro a bit like %{_install_langs} (which
> controls what language files are installed, even though they are all in
> the packages) to control what gets installed for Python stuff, even
> though everything is in the packages.
>
> On Workstation, that macro would be set so that both the byte-code and
> the code would be installed 


> (it is invaluable for learning and
> debugging purposes to be able to read/edit the code).

Come on, this is not an argument. We don't install source code for any
other language which produces some binary libraries except Python while
you cannot deny that it would be invaluable for the same purposes. I
understand you sentiment, but I rather appreciate Orion's position, let
me quote: "But that said, I'd be happy to install -pysource sub-package
in order to do it."


Vít


>
> And on Cloud, that macro would be set to have only the bytecode
> installed, so that you'd save the space.
>
>


-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Neal Gompa]

On Fri, Sep 25, 2015 at 3:25 PM, Vít Ondruch  wrote:

> Dne 25.9.2015 v 16:15 Mathieu Bridon napsal(a):
> > (it is invaluable for learning and
> > debugging purposes to be able to read/edit the code).
>
> Come on, this is not an argument. We don't install source code for any
> other language which produces some binary libraries except Python while
> you cannot deny that it would be invaluable for the same purposes. I
> understand you sentiment, but I rather appreciate Orion's position, let
> me quote: "But that said, I'd be happy to install -pysource sub-package
> in order to do it."
>
>
> Vít
>
>
​If it's possible to auto-generate -pysource subpackages like we do with
debuginfo packages, then I would certainly welcome it. If my understanding
is correct, Python module importing works fine with only *.pyc files,
right?​


-- 
真実はいつも一つ！/ Always, there's only one truth!
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Matthew Miller]

On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote:
> Also, you might consider to ship the precompiled bytecode just
> optionally, using recommends.
> 
> On contrary, if you insist on shipping the bytecode, why you don't drop
> the .py files? I see a lot of duplication all around python packages 

Wait, we can do that? Why don't we?

Everything I see in online discussion is centered on, basically,
transparency. But we wouldn't be doing it for obfuscation. The srpms
would still be there, and for that matter we could ship the .py files
in a subpackage.

-- 
Matthew Miller

Fedora Project Leader
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Paul Wouters]

On Fri, 25 Sep 2015, Matthew Miller wrote:


On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote:

Also, you might consider to ship the precompiled bytecode just
optionally, using recommends.

On contrary, if you insist on shipping the bytecode, why you don't drop
the .py files? I see a lot of duplication all around python packages 


Wait, we can do that? Why don't we?

Everything I see in online discussion is centered on, basically,
transparency. But we wouldn't be doing it for obfuscation. The srpms
would still be there, and for that matter we could ship the .py files
in a subpackage.


It's nice to be able to edit the .py for testing without going through
hoops or building/installing rpms.

It's also nice to be able to read the .py code. That is one reason
people use script languages :P

Paul
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Mathieu Bridon]

On Fri, 2015-09-25 at 10:04 -0400, Matthew Miller wrote:
> On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote:
> > Also, you might consider to ship the precompiled bytecode just
> > optionally, using recommends.
> > 
> > On contrary, if you insist on shipping the bytecode, why you don't
> > drop
> > the .py files? I see a lot of duplication all around python
> > packages 
> 
> Wait, we can do that? Why don't we?
> 
> Everything I see in online discussion is centered on, basically,
> transparency. But we wouldn't be doing it for obfuscation. The srpms
> would still be there, and for that matter we could ship the .py files
> in a subpackage.

Or maybe rpm could have a macro a bit like %{_install_langs} (which
controls what language files are installed, even though they are all in
the packages) to control what gets installed for Python stuff, even
though everything is in the packages.

On Workstation, that macro would be set so that both the byte-code and
the code would be installed (it is invaluable for learning and
debugging purposes to be able to read/edit the code).

And on Cloud, that macro would be set to have only the bytecode
installed, so that you'd save the space.


-- 
Mathieu
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Fabian Deutsch]

On Fri, Sep 25, 2015 at 4:04 PM, Matthew Miller
 wrote:
>> On contrary, if you insist on shipping the bytecode, why you don't drop
>> the .py files? I see a lot of duplication all around python packages 

Don't do that, it has impact on functionality.
We did this for years [0], but you wonder how many components actually
glob for *.py files i.e. to load plugins.
Blacklisting files is not worth the space.

- fabian

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1233106

-- 
Fabian Deutsch 
RHEV Hypervisor
Red Hat
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Stephen Gallagher]

On Fri, 2015-09-25 at 16:17 +0200, Fabian Deutsch wrote:
> On Fri, Sep 25, 2015 at 4:04 PM, Matthew Miller
>  wrote:
> > > On contrary, if you insist on shipping the bytecode, why you
> > > don't drop
> > > the .py files? I see a lot of duplication all around python
> > > packages 
> 
> Don't do that, it has impact on functionality.
> We did this for years [0], but you wonder how many components
> actually
> glob for *.py files i.e. to load plugins.
> Blacklisting files is not worth the space.
> 


Just as an FYI, rolekit does exactly this.

signature.asc
Description: This is a digitally signed message part
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat

[Fabian Deutsch]

On Fri, Sep 25, 2015 at 4:26 PM, Stephen Gallagher  wrote:
> On Fri, 2015-09-25 at 16:17 +0200, Fabian Deutsch wrote:
>> On Fri, Sep 25, 2015 at 4:04 PM, Matthew Miller
>>  wrote:
>> > > On contrary, if you insist on shipping the bytecode, why you
>> > > don't drop
>> > > the .py files? I see a lot of duplication all around python
>> > > packages 
>>
>> Don't do that, it has impact on functionality.
>> We did this for years [0], but you wonder how many components
>> actually
>> glob for *.py files i.e. to load plugins.
>> Blacklisting files is not worth the space.
>>
>
>
> Just as an FYI, rolekit does exactly this.

virt-who, sosreport, and python-urllib3 are just other examples of
packages doing this.

- fabian

-- 
Fabian Deutsch 
RHEV Hypervisor
Red Hat
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct