Re: [OE-core] [RFC PATCH] package.bbclass: omit .pyc and .pyo file
On Wed, 2015-01-07 at 17:32 +0800, Robert Yang wrote: On 01/07/2015 05:23 PM, Mike Looijmans wrote: On 01/07/2015 09:07 AM, Richard Purdie wrote: On Tue, 2015-01-06 at 17:07 -0800, Robert Yang wrote: We should not ship .pyc or .pyo file, but there are a few packages ship .pyc, should we: Why should we not ship them? Doesn't python create these at runtime if they're not present? What happens on a read only filesystem? You definitely SHOULD ship the .pyc files. If they don't exist, the interpreter is forced to re-compile the .py source, and will attempt to write the result to the filesystem. It won't cause harm, it won't fail, but it's very inefficient. It's better to let the host do the py-pyc conversion anyway. AFAIK, the .pyc is not version compatible, the .pyc created with the build host's python may not work with the target python. I thought that was true for pyo but not pyc? Do you have a pointer to documentation on that? Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC PATCH] package.bbclass: omit .pyc and .pyo file
On 01/07/2015 09:07 AM, Richard Purdie wrote: On Tue, 2015-01-06 at 17:07 -0800, Robert Yang wrote: We should not ship .pyc or .pyo file, but there are a few packages ship .pyc, should we: Why should we not ship them? Doesn't python create these at runtime if they're not present? What happens on a read only filesystem? You definitely SHOULD ship the .pyc files. If they don't exist, the interpreter is forced to re-compile the .py source, and will attempt to write the result to the filesystem. It won't cause harm, it won't fail, but it's very inefficient. It's better to let the host do the py-pyc conversion anyway. The opposite works just fine: You can omit the .py files and ship only .pyc files. We do that on settopboxes that use Python for the GUI, this saves several megabytes of flash space. To accomplish that, we put the .py files into a $PN-src package. There has been general agreement that .pyo files are utterly pointless. I'm sure we've had issues raised by someone with a read only filesystem before FWIW. I agree there is probably an issue here but deleting them may not be the best option. I'm open to ideas though. My idea would be to standardize on shipping ONLY compiled files, and put the source .py files into a separate package named $PN-src by default. There is no need to install megabytes of python source files that neither users nor the interpreter will ever read. Met vriendelijke groet / kind regards, Mike Looijmans System Expert TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best Telefoon: (+31) (0) 499 33 69 79 Telefax: (+31) (0) 499 33 69 70 E-mail: mike.looijm...@topic.nl Website: www.topic.nl Please consider the environment before printing this e-mail Topic zoekt gedreven (embedded) software specialisten! http://topic.nl/vacatures/topic-zoekt-software-engineers/ -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC PATCH] package.bbclass: omit .pyc and .pyo file
On 01/07/2015 05:23 PM, Mike Looijmans wrote: On 01/07/2015 09:07 AM, Richard Purdie wrote: On Tue, 2015-01-06 at 17:07 -0800, Robert Yang wrote: We should not ship .pyc or .pyo file, but there are a few packages ship .pyc, should we: Why should we not ship them? Doesn't python create these at runtime if they're not present? What happens on a read only filesystem? You definitely SHOULD ship the .pyc files. If they don't exist, the interpreter is forced to re-compile the .py source, and will attempt to write the result to the filesystem. It won't cause harm, it won't fail, but it's very inefficient. It's better to let the host do the py-pyc conversion anyway. AFAIK, the .pyc is not version compatible, the .pyc created with the build host's python may not work with the target python. // Robert The opposite works just fine: You can omit the .py files and ship only .pyc files. We do that on settopboxes that use Python for the GUI, this saves several megabytes of flash space. To accomplish that, we put the .py files into a $PN-src package. There has been general agreement that .pyo files are utterly pointless. I'm sure we've had issues raised by someone with a read only filesystem before FWIW. I agree there is probably an issue here but deleting them may not be the best option. I'm open to ideas though. My idea would be to standardize on shipping ONLY compiled files, and put the source .py files into a separate package named $PN-src by default. There is no need to install megabytes of python source files that neither users nor the interpreter will ever read. Met vriendelijke groet / kind regards, Mike Looijmans System Expert TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best Telefoon: (+31) (0) 499 33 69 79 Telefax: (+31) (0) 499 33 69 70 E-mail: mike.looijm...@topic.nl Website: www.topic.nl Please consider the environment before printing this e-mail Topic zoekt gedreven (embedded) software specialisten! http://topic.nl/vacatures/topic-zoekt-software-engineers/ -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC PATCH] package.bbclass: omit .pyc and .pyo file
On Tue, 2015-01-06 at 17:07 -0800, Robert Yang wrote: We should not ship .pyc or .pyo file, but there are a few packages ship .pyc, should we: Why should we not ship them? Doesn't python create these at runtime if they're not present? What happens on a read only filesystem? I'm sure we've had issues raised by someone with a read only filesystem before FWIW. I agree there is probably an issue here but deleting them may not be the best option. I'm open to ideas though. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC PATCH] package.bbclass: omit .pyc and .pyo file
On 7 January 2015 at 09:23, Mike Looijmans mike.looijm...@topic.nl wrote: You definitely SHOULD ship the .pyc files. If they don't exist, the interpreter is forced to re-compile the .py source, and will attempt to write the result to the filesystem. It won't cause harm, it won't fail, but it's very inefficient. It's better to let the host do the py-pyc conversion anyway. The opposite works just fine: You can omit the .py files and ship only .pyc files. We do that on settopboxes that use Python for the GUI, this saves several megabytes of flash space. To accomplish that, we put the .py files into a $PN-src package. I agree with Mike, there is a use-case for just shipping .pyc. Whether oe-core should do something to assist with this or not is debatable. There's an open bug report about this: https://bugzilla.yoctoproject.org/show_bug.cgi?id=6434. https://docs.python.org/2/tutorial/modules.html#compiled-python-files has the details we're after. Summary: .pyc is Python bytecode, which is an implementation detail of CPython. As such it's version-dependent but explicitly architecture-independent. .pyo generated with -O is simply .pyc but with asserts removed. .pyo generated with -O -O has asserts and docstrings removed. I think it's fair to say we should just ignore .pyo - no point generating and shipping it. It does seem that if we want to ship usable .pyc we need to ensure that they are compiled with python-native. I guess this could be done by calling python-native's compileall module to recompile the sources at package time. Ross -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC PATCH] package.bbclass: omit .pyc and .pyo file
On 01/07/2015 12:16 PM, Burton, Ross wrote:
On 7 January 2015 at 09:23, Mike Looijmans mike.looijm...@topic.nl
mailto:mike.looijm...@topic.nl wrote:
You definitely SHOULD ship the .pyc files. If they don't exist, the
interpreter is forced to re-compile the .py source, and will attempt to
write the result to the filesystem. It won't cause harm, it won't fail,
but it's very inefficient. It's better to let the host do the py-pyc
conversion anyway.
The opposite works just fine: You can omit the .py files and ship only
.pyc files. We do that on settopboxes that use Python for the GUI, this
saves several megabytes of flash space.
To accomplish that, we put the .py files into a $PN-src package.
I agree with Mike, there is a use-case for just shipping .pyc. Whether
oe-core should do something to assist with this or not is debatable.
We currently have this in a python_2.7.%.bbappend:
PACKAGES =+ ${PN}-src
RDEPENDS_{PN}-src = ${PN}
FILES_${PN}-src += ${libdir}/python${PYTHON_MAJMIN}/*.py
FILES_${PN}-src += ${libdir}/python${PYTHON_MAJMIN}/*/*.py
FILES_${PN}-src += ${libdir}/python${PYTHON_MAJMIN}/*/*/*.py
This moves all sources into a single package. I experimented with creating src
packages for each python sub-package, but all that accomplished was adding a
lot of packages to the feed that no one would ever install.
I patched the larger Python recipes (e.g. twisted) in similar ways to reduce
the image size (and network traffic).
A generic please remove or split all .py files flag or class would be a
useful addition.
There's an open bug report about this:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=6434.
https://docs.python.org/2/tutorial/modules.html#compiled-python-files has the
details we're after. Summary:
.pyc is Python bytecode, which is an implementation detail of CPython. As
such it's version-dependent but explicitly architecture-independent.
.pyo generated with -O is simply .pyc but with asserts removed.
.pyo generated with -O -O has asserts and docstrings removed.
For horrible historic reasons OpenPLi (and its many forks) still use a patch
that makes .pyo files the default instead of .pyc. It's been on my todo list
for over a year now to get rid of it, but too many plugins and 3rd party stuff
still count on this being the case.
I think it's fair to say we should just ignore .pyo - no point generating and
shipping it. It does seem that if we want to ship usable .pyc we need to
ensure that they are compiled with python-native. I guess this could be done
by calling python-native's compileall module to recompile the sources at
package time.
I'd expect any halfway decent recipe to have done that already in the
do_compile step. Most, if not all, Python recipes already do so. If anything,
the core could provide a class that provides a simple do_compile that calls
compileall on the source dir.
Moving this to packaging will lead to unexpected failures, there are some
situations where the source .py file must be installed and the .pyc will be
ignored. A simple example is when the .py script is the application entry
point, those aren't compiled into pyc at runtime, and adding a pyc would be a
waste. Only the package's makefile (or whatever) can be expected to know that.
Mike.
Met vriendelijke groet / kind regards,
Mike Looijmans
System Expert
TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax: (+31) (0) 499 33 69 70
E-mail: mike.looijm...@topic.nl
Website: www.topic.nl
Please consider the environment before printing this e-mail
Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC PATCH] package.bbclass: omit .pyc and .pyo file
On 01/07/2015 09:07 AM, Robert Yang wrote:
We should not ship .pyc or .pyo file, but there are a few packages
ship .pyc, should we:
1) Ignore them in package.bbclass as this patch showes ?
Or
2) Add a qa check then fix it by hand one by one ?
3) Fix it in a bbclass that python related recipes would generally inherit.
Regards,
Chen Qi
Here is the list of oe-core's world build:
python-smartpm-1.4.1
nativesdk-python-smartpm-1.4.1
python3-distribute-0.6.32
python-pycurl-7.19.5
python-pyrex-0.9.9
python-numpy-1.7.0
python-distribute-0.6.32
python-async-0.6.1
python-docutils-0.12
python-pycairo-1.10.0
python-scons-2.3.2
python-imaging-1.1.7
python-gitdb-0.5.4
Signed-off-by: Robert Yang liezhi.y...@windriver.com
---
meta/classes/package.bbclass |6 ++
1 file changed, 6 insertions(+)
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index fc501fc..6960221 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1022,6 +1022,9 @@ python populate_packages () {
files.append(file)
for file in files:
+# Skip .pyc and .pyo file.
+if file.endswith('.pyc') or file.endswith('.pyo'):
+continue
if not cpath.islink(file):
if cpath.isdir(file):
newfiles = [ os.path.join(file,x) for x in
os.listdir(file) ]
@@ -1083,6 +1086,9 @@ python populate_packages () {
if not dir:
dir = os.sep
for f in (files + dirs):
+# Skip .pyc and .pyo file.
+if f.endswith('.pyc') or f.endswith('.pyo'):
+continue
path = os.path.join(dir, f)
if ('.' + path) not in seen:
unshipped.append(path)
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [RFC PATCH] package.bbclass: omit .pyc and .pyo file
We should not ship .pyc or .pyo file, but there are a few packages
ship .pyc, should we:
1) Ignore them in package.bbclass as this patch showes ?
Or
2) Add a qa check then fix it by hand one by one ?
Here is the list of oe-core's world build:
python-smartpm-1.4.1
nativesdk-python-smartpm-1.4.1
python3-distribute-0.6.32
python-pycurl-7.19.5
python-pyrex-0.9.9
python-numpy-1.7.0
python-distribute-0.6.32
python-async-0.6.1
python-docutils-0.12
python-pycairo-1.10.0
python-scons-2.3.2
python-imaging-1.1.7
python-gitdb-0.5.4
Signed-off-by: Robert Yang liezhi.y...@windriver.com
---
meta/classes/package.bbclass |6 ++
1 file changed, 6 insertions(+)
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index fc501fc..6960221 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1022,6 +1022,9 @@ python populate_packages () {
files.append(file)
for file in files:
+# Skip .pyc and .pyo file.
+if file.endswith('.pyc') or file.endswith('.pyo'):
+continue
if not cpath.islink(file):
if cpath.isdir(file):
newfiles = [ os.path.join(file,x) for x in
os.listdir(file) ]
@@ -1083,6 +1086,9 @@ python populate_packages () {
if not dir:
dir = os.sep
for f in (files + dirs):
+# Skip .pyc and .pyo file.
+if f.endswith('.pyc') or f.endswith('.pyo'):
+continue
path = os.path.join(dir, f)
if ('.' + path) not in seen:
unshipped.append(path)
--
1.7.9.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
