Re: [yocto] Reducing the size of the image by optimizing python

[Mike Looijmans]

> Optimising to just pyc files is an optimisation further than most
> people find they need and will be much harder to do.

It's actually quite simple, just add a bbappend to the python recipe that puts 
all .py files into the "dbg" or newly created "src" package. Or simply delete 
all .py files in a do_install_append.


Here's an example of the "src" approach:

https://github.com/OpenPLi/openpli-oe-core/blob/develop/meta-openpli/recipes-devtools/python/python_2.7.13.bbappend


Simply doing this in the bbappend might also work (untested):

do_install_append() {
find ${D}${libdir}/python${PYTHON_MAJMIN}/ -name '*.py' -remove
}


Removing the ".py" files roughly cuts the disk space in half.


The "src" approach works on any package, and has the advantage that you can 
still install the source files for debugging and development.

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
Postbus 440, 5680 AK Best
The Netherlands

T: +31 (0) 499 33 69 69
E: {E-mail
W: www.topicproducts.com

Please consider the environment before printing this e-mail
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[Ross Burton]

On 19/10/2019 13:45, Abhi Arora wrote:

IMAGE_INSTALL_append = " \
     python3 \




This is installing "all of python".  If you want to trim your Python 
installation then don't do that, instead determine what libraries you 
*actually* need and explicitly add those.


Ross
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[Abhi Arora]

Hello,
I have listed the python modules and packages that I don't want. Someone 
recommended to list them with BAD_RECOMMENDATION. Can someone help me where 
this variable should go?

Thank you.

Get Outlook for Android


From: Josef Holzmayr 
Sent: Sunday, October 20, 2019 4:29:00 PM
To: Abhi Arora 
Cc: Yann Dirson ; 
richard.pur...@linuxfoundation.org ; 
yocto@yoctoproject.org 
Subject: Re: [yocto] Reducing the size of the image by optimizing python

On Sat, Oct 19, 2019 at 04:32:26PM +, Abhi Arora wrote:
> I am planning to install it in my host machine and see if I am able to run my 
> scripts and manually add or remove modules and python packages to get minimal 
> required system for my scripts.
>
> For that I need to access python3-misc and what it installs.

Its not something that directly corresponds to a standard desktop
distribution. Rather, its "everything that would otherwise be left
behind", see:

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/python/python3_3.7.4.bb#n323

So its just like the advice was already: remove it, see what breaks, and
then add smaller bits and pieces again until everything works as
desired.

Greetz
>
>
>
> Get Outlook for Android
>
> 
> From: Yann Dirson 
> Sent: Saturday, October 19, 2019 8:51:52 PM
> To: richard.pur...@linuxfoundation.org 
> Cc: Abhi Arora ; Khem Raj ; 
> yocto@yoctoproject.org 
> Subject: Re: [yocto] Reducing the size of the image by optimizing python
>
> You can also look at the package dependency graph (see the file
> generated by bitbake -g) to get a better idea of what pulls what.
>
> 2019-10-19 17:10 UTC+02:00, richard.pur...@linuxfoundation.org
> :
> > On Sat, 2019-10-19 at 14:55 +, Abhi Arora wrote:
> >> Thanks your for the suggestion.
> >>
> >> From where I can find out what modules and packages python3-misc
> >> downloads and installs? I am new to yocto and Don't know where to
> >> look for. I tried grep but didn't help me.
> >
> > python3-misc is a package. You can look at its dependencies to see what
> > it adds to the image.
> >
> >> And how about optimization using pyc file? Is it doable?
> >
> > Start simple. If you have python3-misc installed its using a lot of
> > space and is the sensible place to start with trimming things down.
> >
> > Optimising to just pyc files is an optimisation further than most
> > people find they need and will be much harder to do.
> >
> > Cheers,
> >
> > Richard
> >
> >
> >
> > --
> > ___
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
>
>
> --
> Yann Dirson 
> Blade / Shadow -- http://shadow.tech

> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto


--
———
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548

_
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[Josef Holzmayr]

On Sat, Oct 19, 2019 at 04:32:26PM +, Abhi Arora wrote:
> I am planning to install it in my host machine and see if I am able to run my 
> scripts and manually add or remove modules and python packages to get minimal 
> required system for my scripts.
> 
> For that I need to access python3-misc and what it installs.

Its not something that directly corresponds to a standard desktop
distribution. Rather, its "everything that would otherwise be left
behind", see:

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/python/python3_3.7.4.bb#n323

So its just like the advice was already: remove it, see what breaks, and
then add smaller bits and pieces again until everything works as
desired.

Greetz
> 
> 
> 
> Get Outlook for Android
> 
> 
> From: Yann Dirson 
> Sent: Saturday, October 19, 2019 8:51:52 PM
> To: richard.pur...@linuxfoundation.org 
> Cc: Abhi Arora ; Khem Raj ; 
> yocto@yoctoproject.org 
> Subject: Re: [yocto] Reducing the size of the image by optimizing python
> 
> You can also look at the package dependency graph (see the file
> generated by bitbake -g) to get a better idea of what pulls what.
> 
> 2019-10-19 17:10 UTC+02:00, richard.pur...@linuxfoundation.org
> :
> > On Sat, 2019-10-19 at 14:55 +, Abhi Arora wrote:
> >> Thanks your for the suggestion.
> >>
> >> From where I can find out what modules and packages python3-misc
> >> downloads and installs? I am new to yocto and Don't know where to
> >> look for. I tried grep but didn't help me.
> >
> > python3-misc is a package. You can look at its dependencies to see what
> > it adds to the image.
> >
> >> And how about optimization using pyc file? Is it doable?
> >
> > Start simple. If you have python3-misc installed its using a lot of
> > space and is the sensible place to start with trimming things down.
> >
> > Optimising to just pyc files is an optimisation further than most
> > people find they need and will be much harder to do.
> >
> > Cheers,
> >
> > Richard
> >
> >
> >
> > --
> > ___
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
> 
> 
> --
> Yann Dirson 
> Blade / Shadow -- http://shadow.tech

> -- 
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto


-- 
———
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548 

_
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[Abhi Arora]

Thank you for being patient.

I am unable to find out python3-misc package online. Neither "sudo apt-get 
install python3-misc" helps me nor "pip install python3-misc".

I am a newbie, please help me what kind of package it is and where to find it.

I am planning to install it in my host machine and see if I am able to run my 
scripts and manually add or remove modules and python packages to get minimal 
required system for my scripts.

For that I need to access python3-misc and what it installs.



Get Outlook for Android


From: Yann Dirson 
Sent: Saturday, October 19, 2019 8:51:52 PM
To: richard.pur...@linuxfoundation.org 
Cc: Abhi Arora ; Khem Raj ; 
yocto@yoctoproject.org 
Subject: Re: [yocto] Reducing the size of the image by optimizing python

You can also look at the package dependency graph (see the file
generated by bitbake -g) to get a better idea of what pulls what.

2019-10-19 17:10 UTC+02:00, richard.pur...@linuxfoundation.org
:
> On Sat, 2019-10-19 at 14:55 +, Abhi Arora wrote:
>> Thanks your for the suggestion.
>>
>> From where I can find out what modules and packages python3-misc
>> downloads and installs? I am new to yocto and Don't know where to
>> look for. I tried grep but didn't help me.
>
> python3-misc is a package. You can look at its dependencies to see what
> it adds to the image.
>
>> And how about optimization using pyc file? Is it doable?
>
> Start simple. If you have python3-misc installed its using a lot of
> space and is the sensible place to start with trimming things down.
>
> Optimising to just pyc files is an optimisation further than most
> people find they need and will be much harder to do.
>
> Cheers,
>
> Richard
>
>
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>


--
Yann Dirson 
Blade / Shadow -- http://shadow.tech
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[Yann Dirson]

You can also look at the package dependency graph (see the file
generated by bitbake -g) to get a better idea of what pulls what.

2019-10-19 17:10 UTC+02:00, richard.pur...@linuxfoundation.org
:
> On Sat, 2019-10-19 at 14:55 +, Abhi Arora wrote:
>> Thanks your for the suggestion.
>>
>> From where I can find out what modules and packages python3-misc
>> downloads and installs? I am new to yocto and Don't know where to
>> look for. I tried grep but didn't help me.
>
> python3-misc is a package. You can look at its dependencies to see what
> it adds to the image.
>
>> And how about optimization using pyc file? Is it doable?
>
> Start simple. If you have python3-misc installed its using a lot of
> space and is the sensible place to start with trimming things down.
>
> Optimising to just pyc files is an optimisation further than most
> people find they need and will be much harder to do.
>
> Cheers,
>
> Richard
>
>
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>


-- 
Yann Dirson 
Blade / Shadow -- http://shadow.tech
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[richard . purdie]

On Sat, 2019-10-19 at 14:55 +, Abhi Arora wrote:
> Thanks your for the suggestion. 
> 
> From where I can find out what modules and packages python3-misc
> downloads and installs? I am new to yocto and Don't know where to
> look for. I tried grep but didn't help me.

python3-misc is a package. You can look at its dependencies to see what
it adds to the image.

> And how about optimization using pyc file? Is it doable?

Start simple. If you have python3-misc installed its using a lot of
space and is the sensible place to start with trimming things down.

Optimising to just pyc files is an optimisation further than most
people find they need and will be much harder to do.

Cheers,

Richard



-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[Abhi Arora]

Thanks your for the suggestion.

>From where I can find out what modules and packages python3-misc downloads and 
>installs? I am new to yocto and Don't know where to look for. I tried grep but 
>didn't help me.

And how about optimization using pyc file? Is it doable?

Get Outlook for Android


From: Richard Purdie 
Sent: Saturday, October 19, 2019 8:04:58 PM
To: Abhi Arora ; Khem Raj 
Cc: yocto@yoctoproject.org 
Subject: Re: [yocto] Reducing the size of the image by optimizing python

On Sat, 2019-10-19 at 12:45 +, Abhi Arora wrote:
> Thanks for the reply. Can you help me which file to modify to prevent
> downloading certain packages and modules using BAD_RECOMMENDATION?
>
> And how to have only pyc files in the final image to improve speed
> and reduce footprint?
>
> Following are the lines in my yocto project which install python
> packages.
>
> IMAGE_INSTALL_append = " \
> python3 \
> python3-core \
> python3-misc \
> python3-async \
> python3-six \
> python3-websockets \
> python3-requests \
> python3-urllib3 \
> python3-chardet \
> python3-websocket-client \
> python3-pyopenssl \
> python3-simplejson \
> python3-cython \
>
> Pardon for some stupid questions as I am new to yocto.

Take python3-misc out that list as its huge. See if anything breaks, if
so, try and install just what you need, not the large "catchall"
package.

Cheers,

Richard

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[Richard Purdie]

On Sat, 2019-10-19 at 12:45 +, Abhi Arora wrote:
> Thanks for the reply. Can you help me which file to modify to prevent
> downloading certain packages and modules using BAD_RECOMMENDATION? 
> 
> And how to have only pyc files in the final image to improve speed
> and reduce footprint?
> 
> Following are the lines in my yocto project which install python
> packages.
> 
> IMAGE_INSTALL_append = " \
> python3 \
> python3-core \
> python3-misc \
> python3-async \
> python3-six \
> python3-websockets \
> python3-requests \
> python3-urllib3 \
> python3-chardet \
> python3-websocket-client \
> python3-pyopenssl \
> python3-simplejson \
> python3-cython \
> 
> Pardon for some stupid questions as I am new to yocto.

Take python3-misc out that list as its huge. See if anything breaks, if
so, try and install just what you need, not the large "catchall"
package.

Cheers,

Richard

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[Abhi Arora]

Thanks for the reply. Can you help me which file to modify to prevent 
downloading certain packages and modules using BAD_RECOMMENDATION?

And how to have only pyc files in the final image to improve speed and reduce 
footprint?

Following are the lines in my yocto project which install python packages.

IMAGE_INSTALL_append = " \
python3 \
python3-core \
python3-misc \
python3-async \
python3-six \
python3-websockets \
python3-requests \
python3-urllib3 \
python3-chardet \
python3-websocket-client \
python3-pyopenssl \
python3-simplejson \
python3-cython \

Pardon for some stupid questions as I am new to yocto.

Get Outlook for Android


From: Khem Raj 
Sent: Friday, October 18, 2019 2:28:23 PM
To: Abhi Arora 
Cc: yocto@yoctoproject.org 
Subject: Re: [yocto] Reducing the size of the image by optimizing python



On Fri, Oct 18, 2019 at 1:21 PM Abhi Arora 
mailto:abhiaro...@live.com>> wrote:
Hello,
I am having an embedded system. We have bsp with python 3.5 but we want to 
reduce its size further. I am planning to have only oyc files but not sure how 
it can be achieved in recipes. Also, I want to know if there is a other way to 
reduce the footprint.

Please start by eliminating modules first so find which modules are required 
for your workloads and remove all other modules from image by adding them to 
BAD_RECOMMENDATIONS from IMAGE_INSTALL


Please help.

Get Outlook for Android

--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Reducing the size of the image by optimizing python

[Khem Raj]

On Fri, Oct 18, 2019 at 1:21 PM Abhi Arora  wrote:

> Hello,
> I am having an embedded system. We have bsp with python 3.5 but we want to
> reduce its size further. I am planning to have only oyc files but not sure
> how it can be achieved in recipes. Also, I want to know if there is a other
> way to reduce the footprint.
>

Please start by eliminating modules first so find which modules are
required for your workloads and remove all other modules from image by
adding them to BAD_RECOMMENDATIONS from IMAGE_INSTALL

>
>
> Please help.
>
> Get Outlook for Android 
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto