Re: [OE-core] [PATCH 00/45] Move recipes to use Python 3 whenever possible
On Tue, 2016-05-24 at 14:53 +0300, Alexander Kanavin wrote: > This patchset updates recipes to use Python 3 whenever possible. A > few items > cannot be moved at the moment for various reasons, here they are: I put this through a round of testing on the autobuilder overnight. I just replied to Maxin about the gst-plugins-bad/bluez issue, the world build also is a bit of a mess: https://autobuilder.yoctoproject.org/main/builders/nightly-world-lsb/bu ilds/528/steps/BuildImages/logs/stdio python3-numpy looks like it can't find various symbols it thinks it should be able to. There are more gdbus-codegen issues - perhaps that isn't bluez but something else causing them? (triggered gtk3+ and libsecret to fail too) Since various pieces did build is this a dependency issue? python3-dbus doesn't build on x32: https://autobuilder.yoctoproject.org/main/builders/nightly-x32/builds/7 98/steps/BuildImages/logs/stdio Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 00/45] Move recipes to use Python 3 whenever possible
On 5/26/16 8:49 AM, Alexander Kanavin wrote: > On 05/26/2016 04:39 PM, Mark Hatle wrote: > >> The interchange format of the newer createrepo is very different then the >> older >> format the smartpm uses. Thus they have to be kept in sync. The new format >> also does not have the extensions we added for the recommended package >> syntax. >> >> createrepo is very small, and it's not YUM dependent. It simply reads the >> metadata (via the rpm interface) and dumps out data in an XML format.. then >> does >> a few more things as well. So porting the existing one forward shouldn't be >> difficult. It's just not clear at this point what the best behavior will be >> to >> do.. port createrepo forward -- or update smartpm to support the newer format >> (with appropriate extensions.) > > I still don't understand why introducing yum/dnf as a replacement for > smartpm should be avoided at all costs. Like smartpm, it's fully written > in Python, so that theoretically should be less painful that building > working binaries. I never said it should be avoided at all costs. Yum is dead. So any discussions would be on DNF. DNF has many library dependencies that smartpm does not have. It also has a larger overhead for other resources. (But with that said, it also has other advantages in some cases, like support for DeltaRPM.) We should investigate DNF, but at present DNF [and the index format] do not understand the recommended syntax that is used by OE/RPM5. DNF also has some issues with the other items that RPM5 has implemented (self-signed packages, package arch is not 'statically defined', etc.) Smartpm does not have a problem with any of these, and is quite small. Smartpm however does have an issue (maybe) in that it's resolution code is written in python and likely is much slower then using libsolv [which DNF uses]. So it's not that I'm dismissing DNF, it's that we need to determine what is necessary to move smartpm forward. We need to investigate what it will take to port over DNF... and we need to understand performance impacts (runtime, disk space, and dependency needs.) --Mark > Alex > -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 00/45] Move recipes to use Python 3 whenever possible
On 05/26/2016 04:39 PM, Mark Hatle wrote: The interchange format of the newer createrepo is very different then the older format the smartpm uses. Thus they have to be kept in sync. The new format also does not have the extensions we added for the recommended package syntax. createrepo is very small, and it's not YUM dependent. It simply reads the metadata (via the rpm interface) and dumps out data in an XML format.. then does a few more things as well. So porting the existing one forward shouldn't be difficult. It's just not clear at this point what the best behavior will be to do.. port createrepo forward -- or update smartpm to support the newer format (with appropriate extensions.) I still don't understand why introducing yum/dnf as a replacement for smartpm should be avoided at all costs. Like smartpm, it's fully written in Python, so that theoretically should be less painful that building working binaries. Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 00/45] Move recipes to use Python 3 whenever possible
On 5/25/16 8:01 AM, alexander.kana...@linux.intel.com wrote: >>> Consequently, rpm's python bindings (required by smartpm) stay at python >>> 2 as well >>> for now, even tough python 3 seems to be supported. >> >> Is there a porting guide or similar that can used to help identify what >> types of >> changes are needed. I am not at all familiar with what python 3 needs >> specifically, but I do expect we will need to port these items. > > Of course, there's a ton of such guides on the net. Here's the official one: > https://docs.python.org/3/howto/pyporting.html > >> The yum version of createrepo will not work. We need to port forward the >> component as appropriate -- or modify smartpm to use a newer version -- >> assuming >> we can add missing pieces, as necessary. (key is smartpm and createrepo >> need to >> be kept in sync.) > > Can you say why it won't work, specifically, if we also replace smartpm > with yum or dnf at the same time? The only alternative I see is that we > have to fork the old version of createrepo and the abandoned upstream of > smartpm, port them to Python 3, and maintain them going forward - not a > light undertaking. The interchange format of the newer createrepo is very different then the older format the smartpm uses. Thus they have to be kept in sync. The new format also does not have the extensions we added for the recommended package syntax. createrepo is very small, and it's not YUM dependent. It simply reads the metadata (via the rpm interface) and dumps out data in an XML format.. then does a few more things as well. So porting the existing one forward shouldn't be difficult. It's just not clear at this point what the best behavior will be to do.. port createrepo forward -- or update smartpm to support the newer format (with appropriate extensions.) --Mark >> Please create one (or three) defects and assign them to me for smartpm, >> createrepo, and rpm python3 support. (I doubt I'll be the one doing the >> work, >> but I'll do my best to find someone to do the work.) > > Yes, I'll do that now. > >>> 5) LSB spec is implicitly requiring Python 2 (via requirement for >>> '/usr/bin/python'). >> >> I expect the LSB will require python 2 for a while. So we will need to >> continue >> to ensure that python 2 works, as well as python3. > > Of course, Python 2 will be supported by upstream at least until 2020, and > so it'll be provided by oe-core at least until then. 2.7.12 should appear > soon. > > Alex > -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 00/45] Move recipes to use Python 3 whenever possible
>> Please create one (or three) defects and assign them to me for smartpm, >> createrepo, and rpm python3 support. (I doubt I'll be the one doing the >> work, >> but I'll do my best to find someone to do the work.) > > Yes, I'll do that now. Filed at https://bugzilla.yoctoproject.org/show_bug.cgi?id=9675 Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 00/45] Move recipes to use Python 3 whenever possible
>> Consequently, rpm's python bindings (required by smartpm) stay at python >> 2 as well >> for now, even tough python 3 seems to be supported. > > Is there a porting guide or similar that can used to help identify what > types of > changes are needed. I am not at all familiar with what python 3 needs > specifically, but I do expect we will need to port these items. Of course, there's a ton of such guides on the net. Here's the official one: https://docs.python.org/3/howto/pyporting.html > The yum version of createrepo will not work. We need to port forward the > component as appropriate -- or modify smartpm to use a newer version -- > assuming > we can add missing pieces, as necessary. (key is smartpm and createrepo > need to > be kept in sync.) Can you say why it won't work, specifically, if we also replace smartpm with yum or dnf at the same time? The only alternative I see is that we have to fork the old version of createrepo and the abandoned upstream of smartpm, port them to Python 3, and maintain them going forward - not a light undertaking. > Please create one (or three) defects and assign them to me for smartpm, > createrepo, and rpm python3 support. (I doubt I'll be the one doing the > work, > but I'll do my best to find someone to do the work.) Yes, I'll do that now. >> 5) LSB spec is implicitly requiring Python 2 (via requirement for >> '/usr/bin/python'). > > I expect the LSB will require python 2 for a while. So we will need to > continue > to ensure that python 2 works, as well as python3. Of course, Python 2 will be supported by upstream at least until 2020, and so it'll be provided by oe-core at least until then. 2.7.12 should appear soon. Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 00/45] Move recipes to use Python 3 whenever possible
On 5/24/16 6:53 AM, Alexander Kanavin wrote: > This patchset updates recipes to use Python 3 whenever possible. A few items > cannot be moved at the moment for various reasons, here they are: > > 1) Smartpm package manager has not been ported to Python 3 and it is unlikely > to happen: > https://github.com/smartpm/smart/issues/4 > We need to look into alternatives such as yum or dnf (the 'next generation' > fork of yum). > > Consequently, rpm's python bindings (required by smartpm) stay at python 2 as > well > for now, even tough python 3 seems to be supported. Is there a porting guide or similar that can used to help identify what types of changes are needed. I am not at all familiar with what python 3 needs specifically, but I do expect we will need to port these items. > 2) Scons build system is python 2 only, due to lack of resources. > > 3) createrepo recipe is using an upstream version from 2007, before > createrepo was > rewritten on top of yum's python modules. > > We need to look into adding yum recipe which will allow updating createrepo > to latest upstream. > Also, libxml2 is currently a createrepo dependency, and so stays at Python 2 > for now. The yum version of createrepo will not work. We need to port forward the component as appropriate -- or modify smartpm to use a newer version -- assuming we can add missing pieces, as necessary. (key is smartpm and createrepo need to be kept in sync.) Please create one (or three) defects and assign them to me for smartpm, createrepo, and rpm python3 support. (I doubt I'll be the one doing the work, but I'll do my best to find someone to do the work.) > 4) varous recipes that haven't been ported: kconfig-frontends, mklibs, > opkg-utils, > ltp, mc, parted, libepoxy, mesa, perf, rt-tests, webkitgtk. > > 5) LSB spec is implicitly requiring Python 2 (via requirement for > '/usr/bin/python'). I expect the LSB will require python 2 for a while. So we will need to continue to ensure that python 2 works, as well as python3. Thanks for the update above. --Mark > 6) gobject introspection has been ported to Python 3 in release 1.48. I'll > send this > update separately, as a part of normal version update patches. > > 7) piglit has been ported to Python 3 as well in the latest upstream > releases, Jussi > Kukkonen should take care of it. I've added piglit's python dependencies in > their > Python 3 versions - python3-mako and python3-numpy, so that the task should > be a little > easier. > > The following changes since commit c7e614c438706fb3ed7520b4990ebb3973366942: > > useradd: Fix infinite build loop (2016-05-23 10:33:45 +0100) > > are available in the git repository at: > > git://git.yoctoproject.org/poky-contrib akanavin/deprecate-python2 > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/deprecate-python2 > > Alexander Kanavin (45): > python-native, python3-native: remove the use of exported HOST_SYS and > BUILD_SYS variables > distutils-native-base.bbclass, distutils3-native-base.bbclass: remove > python3-dir.bbclass: add a separate class for Python 3 > default-versions.inc: drop python-related defaults > sip.bbclass: remove > avahi-ui: remove support for building a python module > bind: switch Python dependency to Python 3.x > python-dbus: update to 1.2.4, port to python 3 > python3: manipulate all of the config*/Makefile files, not just > config/Makefile > python3: drop 110-enable-zlib.patch > glib: move to Python 3 > dbus-test: remove unneeded pygobject dependency > python-pygobject: port to Python 3 > neard: do not package python test scripts > bluez5: switch to Python 3 > connman: do not install Python test scripts > ofono: drop the custom-made revert to Python 2 from Python 3 > packagegroup-core-full-cmdline: drop python-dbus from the list of > services > nfs-utils: switch to Python 3 > systemd: drop python dependency for ptests > util-linux: move to Python 3 > python-pycairo: move to Python 3 > bootchart2: move to Python 3 > gdb: move to Python 3 > git: remove Python package (to which nothing was packaged) > qemu: remove runtime python dependency > subversion: remove unnecessary python dependency > swig: move to Python 3 > python-pyrex: remove unused recipe > python-imaging: remove unused recipe > python-docutils: move to Python 3 > cracklib: disable building the python module > libuser: move to Python 3 > libnewt-python: move to Python 3 > gnome-doc-utils: remove recipe > lttng-tools: move to Python 3 > lttng-ust: move to Python 3 > systemtap: move to Python 3 > libcap-ng: move to Python 3 > hwlatdetect: move to Python 3 > python3-mako: add a Python 3 recipe > python3-nose: add a recipe > python3: add = to -L linking option only when the path is absolute > python-numpy: move recipe to own directory > python3-numpy: add a recipe > > meta/classes/distutils-common-base.bbclass |
[OE-core] [PATCH 00/45] Move recipes to use Python 3 whenever possible
This patchset updates recipes to use Python 3 whenever possible. A few items cannot be moved at the moment for various reasons, here they are: 1) Smartpm package manager has not been ported to Python 3 and it is unlikely to happen: https://github.com/smartpm/smart/issues/4 We need to look into alternatives such as yum or dnf (the 'next generation' fork of yum). Consequently, rpm's python bindings (required by smartpm) stay at python 2 as well for now, even tough python 3 seems to be supported. 2) Scons build system is python 2 only, due to lack of resources. 3) createrepo recipe is using an upstream version from 2007, before createrepo was rewritten on top of yum's python modules. We need to look into adding yum recipe which will allow updating createrepo to latest upstream. Also, libxml2 is currently a createrepo dependency, and so stays at Python 2 for now. 4) varous recipes that haven't been ported: kconfig-frontends, mklibs, opkg-utils, ltp, mc, parted, libepoxy, mesa, perf, rt-tests, webkitgtk. 5) LSB spec is implicitly requiring Python 2 (via requirement for '/usr/bin/python'). 6) gobject introspection has been ported to Python 3 in release 1.48. I'll send this update separately, as a part of normal version update patches. 7) piglit has been ported to Python 3 as well in the latest upstream releases, Jussi Kukkonen should take care of it. I've added piglit's python dependencies in their Python 3 versions - python3-mako and python3-numpy, so that the task should be a little easier. The following changes since commit c7e614c438706fb3ed7520b4990ebb3973366942: useradd: Fix infinite build loop (2016-05-23 10:33:45 +0100) are available in the git repository at: git://git.yoctoproject.org/poky-contrib akanavin/deprecate-python2 http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/deprecate-python2 Alexander Kanavin (45): python-native, python3-native: remove the use of exported HOST_SYS and BUILD_SYS variables distutils-native-base.bbclass, distutils3-native-base.bbclass: remove python3-dir.bbclass: add a separate class for Python 3 default-versions.inc: drop python-related defaults sip.bbclass: remove avahi-ui: remove support for building a python module bind: switch Python dependency to Python 3.x python-dbus: update to 1.2.4, port to python 3 python3: manipulate all of the config*/Makefile files, not just config/Makefile python3: drop 110-enable-zlib.patch glib: move to Python 3 dbus-test: remove unneeded pygobject dependency python-pygobject: port to Python 3 neard: do not package python test scripts bluez5: switch to Python 3 connman: do not install Python test scripts ofono: drop the custom-made revert to Python 2 from Python 3 packagegroup-core-full-cmdline: drop python-dbus from the list of services nfs-utils: switch to Python 3 systemd: drop python dependency for ptests util-linux: move to Python 3 python-pycairo: move to Python 3 bootchart2: move to Python 3 gdb: move to Python 3 git: remove Python package (to which nothing was packaged) qemu: remove runtime python dependency subversion: remove unnecessary python dependency swig: move to Python 3 python-pyrex: remove unused recipe python-imaging: remove unused recipe python-docutils: move to Python 3 cracklib: disable building the python module libuser: move to Python 3 libnewt-python: move to Python 3 gnome-doc-utils: remove recipe lttng-tools: move to Python 3 lttng-ust: move to Python 3 systemtap: move to Python 3 libcap-ng: move to Python 3 hwlatdetect: move to Python 3 python3-mako: add a Python 3 recipe python3-nose: add a recipe python3: add = to -L linking option only when the path is absolute python-numpy: move recipe to own directory python3-numpy: add a recipe meta/classes/distutils-common-base.bbclass |2 - meta/classes/distutils-native-base.bbclass |3 - meta/classes/distutils-tools.bbclass |4 - meta/classes/distutils.bbclass |4 - meta/classes/distutils3-base.bbclass |3 - meta/classes/distutils3-native-base.bbclass|4 - meta/classes/distutils3.bbclass| 24 - meta/classes/gobject-introspection.bbclass |2 - meta/classes/python-dir.bbclass|6 +- meta/classes/python3-dir.bbclass |5 + meta/classes/python3native.bbclass |4 +- meta/classes/sip.bbclass | 61 - meta/conf/distro/include/default-versions.inc |5 - meta/conf/distro/include/distro_alias.inc |2 - meta/conf/distro/include/security_flags.inc|1 - meta/recipes-connectivity/avahi/avahi-ui_0.6.32.bb | 11 +- meta/recipes-connectivity/bind/bind_9.10.3-P3.bb |4 +- meta/recipes-connectivity/bluez5/bluez5.inc|6 +- meta/recipes-connectivity/connman/connman.inc
