On Wed, Feb 23, 2022 at 8:50 PM Tim Orling via lists.openembedded.org <[email protected]> wrote:
> > > On Wed, Feb 23, 2022 at 8:07 PM Khem Raj <[email protected]> wrote: > >> Right now approx 150 out of 470 recipes [1] in meta-python are directly >> failing, >> perhaps there will be more once these are fixed then there are other >> layers >> which might be carrying some impacted python modules. > > > The issue has mostly been detection of the wheel filename and this should > be improved by: > > https://lore.kernel.org/openembedded-core/[email protected]/T/#t > > Also YP compatibility >> is broken due to python3-nose removal. > > > We can try to run 2to3 on the python3-nose plugins or remove them > surgically to reintroduce cleanly Python 3 only > python3-nose. I propose that we move it to meta-python even so because > nothing in oe-core depends on nose and the > code has been completely bit-rotting since 2016. It could not be built > anymore in oe-core and something had to give. > Perhaps we can switch to this repo: https://github.com/atsb/nose-py3 > > >> If we are planning >> to include this in the upcoming release,we would need some commitment >> and strategy to fix these and other ripple effects throughout the >> ecosystem >> especially since this is an LTS release. I wish this changeset was >> proposed >> in a release prior to LTS release, so it could have given enough time to >> adjust for the ecosystem. This is will take some effort in stabilizing >> > > This has been months of work on my part to attempt to get us to where > upstream > Python Packaging Authority has been pushing since PEP-517 was announced in > 2015. > We are very far behind the curve of upstream Python packaging. > > While I too would love to have had this ready sooner, that was not > possible for many reasons. > But no matter what, this would have been painful until recipes could be > migrated, even if had > been available at M1. > > >> I also wonder, what do we lose if we do not put this into the LTS >> release? > > > We already have recipes in oe-core such as python3-jsonschema which can no > longer be upgraded > without moving to pyproject.toml based package builds. I'm sure there are > others in meta-python and > other layers in a similar story. We once again are in a situation that is > a bad image for the project because > we are using the now deprecated "setup.py install" and still building > eggs. To step into an LTS release > with the python packaging already entirely deprecated is just not a good > impression. > > I understand your frustration with the build failures, but it has been a > very solid week of getting oe-core > stabilized (see > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3280). > > I am running a build now for meta-python with the improved wheel filename > patch as well as a patch to > check for whether the current recipe already has pyproject.toml back-end > declared: > > https://lore.kernel.org/openembedded-core/[email protected]/T/#u > > >> > [1] >> https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/1460 > > >> >> On Tue, Feb 22, 2022 at 11:23 AM Tim Orling <[email protected]> wrote: >> > >> > >> > >> > On Tue, Feb 22, 2022 at 11:18 AM Khem Raj <[email protected]> wrote: >> >> >> >> On Tue, Feb 22, 2022 at 10:40 AM Tim Orling <[email protected]> >> wrote: >> >> > >> >> > >> >> > >> >> > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <[email protected]> >> wrote: >> >> >> >> >> >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <[email protected]> >> wrote: >> >> >> > >> >> >> > >> >> >> > >> >> >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <[email protected]> >> wrote: >> >> >> >> >> >> >> >> this is going into dep loops Try >> >> >> >> >> >> >> >> bitbake python3-wheel-native >> >> >> > >> >> >> > >> >> >> > It shouldn’t >> >> >> > >> https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels >> >> >> > >> >> >> >> >> >> I have been trying whats on master-next >> >> >> >> >> > >> >> > I can't reproduce. Can you point me at an error log please? >> >> > It might have something to do with sstate cache, but that's just a >> hunch. >> >> > >> >> >> >> This is because python3-wheel was moved to core and modified but not >> >> deleted from meta-python at same time. I have sent a patch for that >> >> >> > Already sent one: >> > >> https://patchwork.yoctoproject.org/project/oe/patch/[email protected]/ >> > >> >> >> >> >> >> >> >> > DEPENDS:remove:class-native = "python3-pip-native" >> >> >> > >> >> >> > do_install:class-native () { >> >> >> > # We need to bootstrap python3-wheel-native >> >> >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} >> >> >> > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" >> >> >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ >> >> >> > bbfatal_log "Failed to install" >> >> >> > } >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <[email protected]> >> wrote: >> >> >> >> > >> >> >> >> > Upstream Python is moving to pyproject.toml rather than >> setup.py and >> >> >> >> > wheels rather than eggs. The main place this is documented is >> in >> >> >> >> > PEP-517[1]. Installing packages with setup.py install is >> deprecated. >> >> >> >> > >> >> >> >> > We have a choice with the 'kirkstone' LTS release of >> introducing this >> >> >> >> > series (and fixing any fall out during the stabilization >> period) or >> >> >> >> > living with deprecated Python packaging for the duration of >> the LTS (at >> >> >> >> > a minimum 2 years). This seems to be a compelling enough >> reason to bring >> >> >> >> > in this series, even if it results in a slight slip of the >> 'feature >> >> >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. >> >> >> >> > >> >> >> >> > This series bootstraps python3-setuptools-native, >> >> >> >> > python3-flit-core-native, python3-wheel-native and >> python3-pip-native, >> >> >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This >> allows all >> >> >> >> > other recipes to be installed with pip, as intended by >> upstream. >> >> >> >> > >> >> >> >> > Three new classes are introduced: >> >> >> >> > (1) flit_core which builds PEP-517 compliant packages which use >> >> >> >> > flit_core.buildapi as their build-backend. >> >> >> >> > (2) setuptools_build_meta which builds PEP-517 compliant >> packages which use >> >> >> >> > setuptools_build_meta as their build-backend. >> >> >> >> > (3) pip_install_wheel, which simply installs wheels built by >> other >> >> >> >> > classes. >> >> >> >> > >> >> >> >> > The original setuptools3.bbclass do_compile is modified to >> >> >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than >> the deprecated >> >> >> >> > 'setup.py build' mechanism. >> >> >> >> > >> >> >> >> > The original setuptools3.bbclass do_install is modified to >> >> >> >> > use the pip_install_wheel.bbclass to install wheels with pip >> rather than >> >> >> >> > the deprecated 'setup.py install' mechanism. >> >> >> >> > >> >> >> >> > For pip install, we disable dependency checking (--no-deps) >> because it >> >> >> >> > proves to be brittle in our environment without providing full >> wheel >> >> >> >> > cache that seems like overhead that we do not need. We also >> disable >> >> >> >> > fetching wheels from pypi (--no-index). This is no different >> than our >> >> >> >> > normal workflow of requiring dependencies to be defined in a >> recipe. >> >> >> >> > >> >> >> >> > Missing from this series is documentation of the new features, >> classes >> >> >> >> > and variables (this will be done under separate cover). >> >> >> >> > >> >> >> >> > Also desired and previously discussed in YP meetings is a >> helper >> >> >> >> > function to warn that a recipe which currently inherits >> setuptools3 >> >> >> >> > but has a pyproject.toml and defines a >> build-system.build-backend should >> >> >> >> > instead use of the new PEP-517 classes. This function is still >> in progress. >> >> >> >> > >> >> >> >> > During testing, recipes which use python3-setuptools-scm are >> failing to >> >> >> >> > determine the version of the python package, For several >> recipes, the fix >> >> >> >> > was to use pyproject.toml and PEP-517 build backends. To be >> sure, other recipes >> >> >> >> > are still going to fail for the short term, until the root >> cause can be >> >> >> >> > determined. This was not seen again in v2 builds. >> >> >> >> > >> >> >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has been >> >> >> >> > identified in recipes in meta-python, but not in oe-core. The >> >> >> >> > python3-poetry-core recipe is WIP, but brings in a number of >> >> >> >> > dependencies that are perhaps not worth it to bring into >> oe-core at this >> >> >> >> > time. Rather we can support these recipes in meta-python? >> >> >> >> > >> >> >> >> > Fixes in v2 include passing --root=${D} to pip install (rather >> than --target) >> >> >> >> > and also passing --prefix so that scripts are (usually) >> installed in the proper >> >> >> >> > location. pip install also 'helpfully' overwrites python3 >> intepreter with nativepython3 >> >> >> >> > in #!, so add a fix to return the values to what is actually >> correct (similar to what is >> >> >> >> > done in setuptools3.bbclass). Also several recipes still >> needed PIP_INSTALL_PACKAGE defined. >> >> >> >> > >> >> >> >> > Also added self as maintainer of new recipes. >> >> >> >> > >> >> >> >> > [YOCTO #14638] >> >> >> >> > >> >> >> >> > [1] https://www.python.org/dev/peps/pep-0517/ >> >> >> >> > >> >> >> >> > The following changes since commit >> 41ae26ecab731f2fb6593df143476c94183be209: >> >> >> >> > >> >> >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) >> >> >> >> > >> >> >> >> > are available in the Git repository at: >> >> >> >> > >> >> >> >> > git://git.yoctoproject.org/git/poky-contrib >> timo/bootstrap-wheels >> >> >> >> > >> http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels >> >> >> >> > >> >> >> >> > Tim Orling (32): >> >> >> >> > python3-wheel: move 0.37.0 from meta-python >> >> >> >> > python3-flit-core: add recipe for 3.6.0 >> >> >> >> > flit_core.bbclass: add helper for newer python packaging >> >> >> >> > python3-wheel: inherit flit_core >> >> >> >> > pip_install_wheel.bbclass: add helper class >> >> >> >> > setuptools_build_meta.bbclass: add helper class >> >> >> >> > python3-pip: inherit setuptools_build_meta >> >> >> >> > python3-attrs: inherit setuptools_build_meta >> >> >> >> > python3-git: inherit setuptools_build_meta >> >> >> >> > python3-pytest: inherit setuptools_build_meta >> >> >> >> > python3-setuptools-scm: inherit setuptools_build_meta >> >> >> >> > python3-zipp: inherit setuptools_build_meta >> >> >> >> > python3-iniconfig: inherit setuptools_build_meta >> >> >> >> > python3-py: inherit setuptools_build_meta >> >> >> >> > python3-pluggy: inherit setuptools_build_meta >> >> >> >> > python3-setuptools: inherit setuptools_base_meta >> >> >> >> > setuptools3.bbclass: refactor for wheels >> >> >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE >> >> >> >> > meson: inherit setuptools_build_meta >> >> >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE >> >> >> >> > python3-smartypants: patch hash bang to python3 >> >> >> >> > python3-scons: merge -native recipe >> >> >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE >> >> >> >> > python3-magic: set PIP_INSTALL_PACKAGE >> >> >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER >> >> >> >> > asciidoc: set PIP_INSTALL_PACKAGE >> >> >> >> > gi-docgen: set PIP_INSTALL_PACKAGE >> >> >> >> > numpy: set PIP_INSTALL_PACKAGE >> >> >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE >> >> >> >> > python3-mako: inherit setuptools_build_meta >> >> >> >> > python3-packaging: inherit setuptools_build_meta >> >> >> >> > python3-nose: drop recipe >> >> >> >> > >> >> >> >> > meta/classes/flit_core.bbclass | 16 +++ >> >> >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ >> >> >> >> > meta/classes/setuptools3.bbclass | 20 ++-- >> >> >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ >> >> >> >> > meta/conf/distro/include/maintainers.inc | 3 +- >> >> >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- >> >> >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- >> >> >> >> > .../python/python-subunit.inc | 14 --- >> >> >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- >> >> >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + >> >> >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ >> >> >> >> > .../python/python3-git_3.1.26.bb | 2 +- >> >> >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- >> >> >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + >> >> >> >> > .../python/python3-magic_0.4.25.bb | 1 + >> >> >> >> > .../python/python3-mako_1.1.6.bb | 2 +- >> >> >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + >> >> >> >> > .../python/python3-nose_1.3.7.bb | 2 - >> >> >> >> > .../python/python3-numpy_1.22.2.bb | 1 + >> >> >> >> > .../python/python3-packaging_21.3.bb | 2 +- >> >> >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- >> >> >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- >> >> >> >> > .../python/python3-py_1.11.0.bb | 2 +- >> >> >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- >> >> >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- >> >> >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- >> >> >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- >> >> >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- >> >> >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ >> >> >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + >> >> >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ >> >> >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- >> >> >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- >> >> >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 >> ++++++++++++++++++ >> >> >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ >> >> >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- >> >> >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + >> >> >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + >> >> >> >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- >> >> >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) >> >> >> >> > create mode 100644 meta/classes/flit_core.bbclass >> >> >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass >> >> >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass >> >> >> >> > delete mode 100644 >> meta/recipes-devtools/python/python-nose.inc >> >> >> >> > delete mode 100644 >> meta/recipes-devtools/python/python-subunit.inc >> >> >> >> > create mode 100644 meta/recipes-devtools/python/ >> python3-flit-core_3.6.0.bb >> >> >> >> > delete mode 100644 meta/recipes-devtools/python/ >> python3-nose_1.3.7.bb >> >> >> >> > delete mode 100644 meta/recipes-devtools/python/ >> python3-scons-native_4.3.0.bb >> >> >> >> > create mode 100644 >> meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch >> >> >> >> > create mode 100644 >> meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch >> >> >> >> > create mode 100644 >> meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch >> >> >> >> > create mode 100644 meta/recipes-devtools/python/ >> python3-wheel_0.37.1.bb >> >> >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => >> bmap-tools_git.bb} (86%) >> >> >> >> > >> >> >> >> > -- >> >> >> >> > 2.30.2 >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#162295): https://lists.openembedded.org/g/openembedded-core/message/162295 Mute This Topic: https://lists.openembedded.org/mt/89304007/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
