Right now approx 150 out of 470 recipes [1] in meta-python are directly failing, perhaps there will be more once these are fixed then there are other layers which might be carrying some impacted python modules. Also YP compatibility is broken due to python3-nose removal. If we are planning to include this in the upcoming release,we would need some commitment and strategy to fix these and other ripple effects throughout the ecosystem especially since this is an LTS release. I wish this changeset was proposed in a release prior to LTS release, so it could have given enough time to adjust for the ecosystem. This is will take some effort in stabilizing
I also wonder, what do we lose if we do not put this into the LTS release? [1] https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/1460 On Tue, Feb 22, 2022 at 11:23 AM Tim Orling <[email protected]> wrote: > > > > On Tue, Feb 22, 2022 at 11:18 AM Khem Raj <[email protected]> wrote: >> >> On Tue, Feb 22, 2022 at 10:40 AM Tim Orling <[email protected]> wrote: >> > >> > >> > >> > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <[email protected]> wrote: >> >> >> >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <[email protected]> wrote: >> >> > >> >> > >> >> > >> >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <[email protected]> wrote: >> >> >> >> >> >> this is going into dep loops Try >> >> >> >> >> >> bitbake python3-wheel-native >> >> > >> >> > >> >> > It shouldn’t >> >> > https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels >> >> > >> >> >> >> I have been trying whats on master-next >> >> >> > >> > I can't reproduce. Can you point me at an error log please? >> > It might have something to do with sstate cache, but that's just a hunch. >> > >> >> This is because python3-wheel was moved to core and modified but not >> deleted from meta-python at same time. I have sent a patch for that >> > Already sent one: > https://patchwork.yoctoproject.org/project/oe/patch/[email protected]/ > >> >> >> >> >> > DEPENDS:remove:class-native = "python3-pip-native" >> >> > >> >> > do_install:class-native () { >> >> > # We need to bootstrap python3-wheel-native >> >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} >> >> > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" >> >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ >> >> > bbfatal_log "Failed to install" >> >> > } >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <[email protected]> wrote: >> >> >> > >> >> >> > Upstream Python is moving to pyproject.toml rather than setup.py and >> >> >> > wheels rather than eggs. The main place this is documented is in >> >> >> > PEP-517[1]. Installing packages with setup.py install is deprecated. >> >> >> > >> >> >> > We have a choice with the 'kirkstone' LTS release of introducing this >> >> >> > series (and fixing any fall out during the stabilization period) or >> >> >> > living with deprecated Python packaging for the duration of the LTS >> >> >> > (at >> >> >> > a minimum 2 years). This seems to be a compelling enough reason to >> >> >> > bring >> >> >> > in this series, even if it results in a slight slip of the 'feature >> >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. >> >> >> > >> >> >> > This series bootstraps python3-setuptools-native, >> >> >> > python3-flit-core-native, python3-wheel-native and >> >> >> > python3-pip-native, >> >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows >> >> >> > all >> >> >> > other recipes to be installed with pip, as intended by upstream. >> >> >> > >> >> >> > Three new classes are introduced: >> >> >> > (1) flit_core which builds PEP-517 compliant packages which use >> >> >> > flit_core.buildapi as their build-backend. >> >> >> > (2) setuptools_build_meta which builds PEP-517 compliant packages >> >> >> > which use >> >> >> > setuptools_build_meta as their build-backend. >> >> >> > (3) pip_install_wheel, which simply installs wheels built by other >> >> >> > classes. >> >> >> > >> >> >> > The original setuptools3.bbclass do_compile is modified to >> >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than the >> >> >> > deprecated >> >> >> > 'setup.py build' mechanism. >> >> >> > >> >> >> > The original setuptools3.bbclass do_install is modified to >> >> >> > use the pip_install_wheel.bbclass to install wheels with pip rather >> >> >> > than >> >> >> > the deprecated 'setup.py install' mechanism. >> >> >> > >> >> >> > For pip install, we disable dependency checking (--no-deps) because >> >> >> > it >> >> >> > proves to be brittle in our environment without providing full wheel >> >> >> > cache that seems like overhead that we do not need. We also disable >> >> >> > fetching wheels from pypi (--no-index). This is no different than our >> >> >> > normal workflow of requiring dependencies to be defined in a recipe. >> >> >> > >> >> >> > Missing from this series is documentation of the new features, >> >> >> > classes >> >> >> > and variables (this will be done under separate cover). >> >> >> > >> >> >> > Also desired and previously discussed in YP meetings is a helper >> >> >> > function to warn that a recipe which currently inherits setuptools3 >> >> >> > but has a pyproject.toml and defines a build-system.build-backend >> >> >> > should >> >> >> > instead use of the new PEP-517 classes. This function is still in >> >> >> > progress. >> >> >> > >> >> >> > During testing, recipes which use python3-setuptools-scm are failing >> >> >> > to >> >> >> > determine the version of the python package, For several recipes, >> >> >> > the fix >> >> >> > was to use pyproject.toml and PEP-517 build backends. To be sure, >> >> >> > other recipes >> >> >> > are still going to fail for the short term, until the root cause can >> >> >> > be >> >> >> > determined. This was not seen again in v2 builds. >> >> >> > >> >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has been >> >> >> > identified in recipes in meta-python, but not in oe-core. The >> >> >> > python3-poetry-core recipe is WIP, but brings in a number of >> >> >> > dependencies that are perhaps not worth it to bring into oe-core at >> >> >> > this >> >> >> > time. Rather we can support these recipes in meta-python? >> >> >> > >> >> >> > Fixes in v2 include passing --root=${D} to pip install (rather than >> >> >> > --target) >> >> >> > and also passing --prefix so that scripts are (usually) installed in >> >> >> > the proper >> >> >> > location. pip install also 'helpfully' overwrites python3 intepreter >> >> >> > with nativepython3 >> >> >> > in #!, so add a fix to return the values to what is actually correct >> >> >> > (similar to what is >> >> >> > done in setuptools3.bbclass). Also several recipes still needed >> >> >> > PIP_INSTALL_PACKAGE defined. >> >> >> > >> >> >> > Also added self as maintainer of new recipes. >> >> >> > >> >> >> > [YOCTO #14638] >> >> >> > >> >> >> > [1] https://www.python.org/dev/peps/pep-0517/ >> >> >> > >> >> >> > The following changes since commit >> >> >> > 41ae26ecab731f2fb6593df143476c94183be209: >> >> >> > >> >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) >> >> >> > >> >> >> > are available in the Git repository at: >> >> >> > >> >> >> > git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels >> >> >> > >> >> >> > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels >> >> >> > >> >> >> > Tim Orling (32): >> >> >> > python3-wheel: move 0.37.0 from meta-python >> >> >> > python3-flit-core: add recipe for 3.6.0 >> >> >> > flit_core.bbclass: add helper for newer python packaging >> >> >> > python3-wheel: inherit flit_core >> >> >> > pip_install_wheel.bbclass: add helper class >> >> >> > setuptools_build_meta.bbclass: add helper class >> >> >> > python3-pip: inherit setuptools_build_meta >> >> >> > python3-attrs: inherit setuptools_build_meta >> >> >> > python3-git: inherit setuptools_build_meta >> >> >> > python3-pytest: inherit setuptools_build_meta >> >> >> > python3-setuptools-scm: inherit setuptools_build_meta >> >> >> > python3-zipp: inherit setuptools_build_meta >> >> >> > python3-iniconfig: inherit setuptools_build_meta >> >> >> > python3-py: inherit setuptools_build_meta >> >> >> > python3-pluggy: inherit setuptools_build_meta >> >> >> > python3-setuptools: inherit setuptools_base_meta >> >> >> > setuptools3.bbclass: refactor for wheels >> >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE >> >> >> > meson: inherit setuptools_build_meta >> >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE >> >> >> > python3-smartypants: patch hash bang to python3 >> >> >> > python3-scons: merge -native recipe >> >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE >> >> >> > python3-magic: set PIP_INSTALL_PACKAGE >> >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER >> >> >> > asciidoc: set PIP_INSTALL_PACKAGE >> >> >> > gi-docgen: set PIP_INSTALL_PACKAGE >> >> >> > numpy: set PIP_INSTALL_PACKAGE >> >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE >> >> >> > python3-mako: inherit setuptools_build_meta >> >> >> > python3-packaging: inherit setuptools_build_meta >> >> >> > python3-nose: drop recipe >> >> >> > >> >> >> > meta/classes/flit_core.bbclass | 16 +++ >> >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ >> >> >> > meta/classes/setuptools3.bbclass | 20 ++-- >> >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ >> >> >> > meta/conf/distro/include/maintainers.inc | 3 +- >> >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- >> >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- >> >> >> > .../python/python-subunit.inc | 14 --- >> >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- >> >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + >> >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ >> >> >> > .../python/python3-git_3.1.26.bb | 2 +- >> >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- >> >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + >> >> >> > .../python/python3-magic_0.4.25.bb | 1 + >> >> >> > .../python/python3-mako_1.1.6.bb | 2 +- >> >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + >> >> >> > .../python/python3-nose_1.3.7.bb | 2 - >> >> >> > .../python/python3-numpy_1.22.2.bb | 1 + >> >> >> > .../python/python3-packaging_21.3.bb | 2 +- >> >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- >> >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- >> >> >> > .../python/python3-py_1.11.0.bb | 2 +- >> >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- >> >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- >> >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- >> >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- >> >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- >> >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ >> >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + >> >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ >> >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- >> >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- >> >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 >> >> >> > ++++++++++++++++++ >> >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ >> >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- >> >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + >> >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + >> >> >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- >> >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) >> >> >> > create mode 100644 meta/classes/flit_core.bbclass >> >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass >> >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass >> >> >> > delete mode 100644 meta/recipes-devtools/python/python-nose.inc >> >> >> > delete mode 100644 meta/recipes-devtools/python/python-subunit.inc >> >> >> > create mode 100644 >> >> >> > meta/recipes-devtools/python/python3-flit-core_3.6.0.bb >> >> >> > delete mode 100644 >> >> >> > meta/recipes-devtools/python/python3-nose_1.3.7.bb >> >> >> > delete mode 100644 >> >> >> > meta/recipes-devtools/python/python3-scons-native_4.3.0.bb >> >> >> > create mode 100644 >> >> >> > meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch >> >> >> > create mode 100644 >> >> >> > meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch >> >> >> > create mode 100644 >> >> >> > meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch >> >> >> > create mode 100644 >> >> >> > meta/recipes-devtools/python/python3-wheel_0.37.1.bb >> >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => >> >> >> > bmap-tools_git.bb} (86%) >> >> >> > >> >> >> > -- >> >> >> > 2.30.2 >> >> >> > >> >> >> > >> >> >> > >> >> >> >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#162289): https://lists.openembedded.org/g/openembedded-core/message/162289 Mute This Topic: https://lists.openembedded.org/mt/89304007/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
