Re: [OE-core] [PATCH] oeqa parselogs.py: load ignore files from sys.path
On Wed, 2024-01-10 at 15:03 +0200, Mikko Rapeli wrote:
> python import.resources open_text() loads files from the module paths
> but this requires layers to set "addpylib ${LAYERDIR}/lib oeqa"
> which is not needed to find the plain .py test files to run the tests.
> Also an empty __init__.py file in a layer will break the resource file
> loading completely as only that path with __init__.py file will
> be used to search the resource files. Then open_text() is marked
> as deprecated from python 3.11 onwards
> https://docs.python.org/3/library/importlib.resources.html
>
> So replace open_text() by iterating over sys.path to find the ignore
> files. This works since paths like ${LAYERDIR}/lib/oeqa/runtime/cases are
> already in sys.path. Add debug prints for found and not found files
> while at it.
>
> Signed-off-by: Mikko Rapeli
> ---
> meta/lib/oeqa/runtime/cases/parselogs.py | 29 +++-
> 1 file changed, 13 insertions(+), 16 deletions(-)
For completeness for the archives, after some discussion we're
preferring layers use addpylib explicitly to include these. The older
way of handling things will likely get removed so this would break when
that happens.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196274):
https://lists.openembedded.org/g/openembedded-core/message/196274
Mute This Topic: https://lists.openembedded.org/mt/103639917/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] oeqa parselogs.py: load ignore files from sys.path
Hi,
I think this patch should be applied as it aligns oeqa .py and .text file
searches
across layers. Without this layers need to add "addpylib ${LAYERDIR}/lib oeqa"
in layer.conf which doesn't have any other uses and debugging this is really
hard.
There may be addition things wrong in bitbake python environment when oeqa
tests run
but this is already an improvement.
Ross, do you agree?
Cheers,
-Mikko
On Wed, Jan 10, 2024 at 04:23:22PM +0200, Mikko Rapeli via
lists.openembedded.org wrote:
> Hi,
>
> On Wed, Jan 10, 2024 at 02:01:36PM +, Ross Burton wrote:
> > On 10 Jan 2024, at 13:03, Mikko Rapeli via lists.openembedded.org
> > wrote:
> > > python import.resources open_text() loads files from the module paths
> > > but this requires layers to set "addpylib ${LAYERDIR}/lib oeqa"
> > > which is not needed to find the plain .py test files to run the tests.
> > > Also an empty __init__.py file in a layer will break the resource file
> > > loading completely as only that path with __init__.py file will
> > > be used to search the resource files. Then open_text() is marked
> > > as deprecated from python 3.11 onwards
> > > https://docs.python.org/3/library/importlib.resources.html
> >
> > Deprecated but replaceable with files(package).joinpath(resource).open()
> > (which is all open_text does internally).
>
> Yes, but that seems to break if __init__.py file is found from
> ${LAYERDIR}/lib/oeqa/runtime/cases.
>
> > > So replace open_text() by iterating over sys.path to find the ignore
> > > files. This works since paths like ${LAYERDIR}/lib/oeqa/runtime/cases are
> > > already in sys.path. Add debug prints for found and not found files
> > > while at it.
> >
> > I’m confused why lib/oeqa/runtime/cases is on sys.path directly, that
> > sounds… unusual. Do you have any idea where that comes from? I’m guessing
> > oeqa is throwing it on the path whilst searching for test cases, but that
> > sounds like bad behaviour that should be removed to me.
>
> No idea where this comes from, but it's there. addpylib is doing similar
> things,
> adding paths to sys.path and for this use case this not really necessary as
> ${LAYERDIR}/lib should not need to be, it only has oeqa directory. I presume
> test case loading via testimage.bbclass or OERuntim are adding it. These are
> kind
> of hard to debug since "bitbake -e" show any of this.
>
> It's a bit odd that layer.conf would need "addpylib ${LAYERDIR}/lib oeqa" for
> loading
> .txt files but that's not needed for finding and using the oeqa .py test
> files.
> With this change addpylib magic is not needed, at least for finding these
> .txt files.
>
> Cheers,
>
> -Mikko
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#194016):
https://lists.openembedded.org/g/openembedded-core/message/194016
Mute This Topic: https://lists.openembedded.org/mt/103639917/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] oeqa parselogs.py: load ignore files from sys.path
Hi,
On Wed, Jan 10, 2024 at 02:01:36PM +, Ross Burton wrote:
> On 10 Jan 2024, at 13:03, Mikko Rapeli via lists.openembedded.org
> wrote:
> > python import.resources open_text() loads files from the module paths
> > but this requires layers to set "addpylib ${LAYERDIR}/lib oeqa"
> > which is not needed to find the plain .py test files to run the tests.
> > Also an empty __init__.py file in a layer will break the resource file
> > loading completely as only that path with __init__.py file will
> > be used to search the resource files. Then open_text() is marked
> > as deprecated from python 3.11 onwards
> > https://docs.python.org/3/library/importlib.resources.html
>
> Deprecated but replaceable with files(package).joinpath(resource).open()
> (which is all open_text does internally).
Yes, but that seems to break if __init__.py file is found from
${LAYERDIR}/lib/oeqa/runtime/cases.
> > So replace open_text() by iterating over sys.path to find the ignore
> > files. This works since paths like ${LAYERDIR}/lib/oeqa/runtime/cases are
> > already in sys.path. Add debug prints for found and not found files
> > while at it.
>
> I’m confused why lib/oeqa/runtime/cases is on sys.path directly, that sounds…
> unusual. Do you have any idea where that comes from? I’m guessing oeqa is
> throwing it on the path whilst searching for test cases, but that sounds like
> bad behaviour that should be removed to me.
No idea where this comes from, but it's there. addpylib is doing similar things,
adding paths to sys.path and for this use case this not really necessary as
${LAYERDIR}/lib should not need to be, it only has oeqa directory. I presume
test case loading via testimage.bbclass or OERuntim are adding it. These are
kind
of hard to debug since "bitbake -e" show any of this.
It's a bit odd that layer.conf would need "addpylib ${LAYERDIR}/lib oeqa" for
loading
.txt files but that's not needed for finding and using the oeqa .py test files.
With this change addpylib magic is not needed, at least for finding these
.txt files.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193513):
https://lists.openembedded.org/g/openembedded-core/message/193513
Mute This Topic: https://lists.openembedded.org/mt/103639917/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] oeqa parselogs.py: load ignore files from sys.path
On 10 Jan 2024, at 13:03, Mikko Rapeli via lists.openembedded.org
wrote:
> python import.resources open_text() loads files from the module paths
> but this requires layers to set "addpylib ${LAYERDIR}/lib oeqa"
> which is not needed to find the plain .py test files to run the tests.
> Also an empty __init__.py file in a layer will break the resource file
> loading completely as only that path with __init__.py file will
> be used to search the resource files. Then open_text() is marked
> as deprecated from python 3.11 onwards
> https://docs.python.org/3/library/importlib.resources.html
Deprecated but replaceable with files(package).joinpath(resource).open() (which
is all open_text does internally).
> So replace open_text() by iterating over sys.path to find the ignore
> files. This works since paths like ${LAYERDIR}/lib/oeqa/runtime/cases are
> already in sys.path. Add debug prints for found and not found files
> while at it.
I’m confused why lib/oeqa/runtime/cases is on sys.path directly, that sounds…
unusual. Do you have any idea where that comes from? I’m guessing oeqa is
throwing it on the path whilst searching for test cases, but that sounds like
bad behaviour that should be removed to me.
Ross
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193511):
https://lists.openembedded.org/g/openembedded-core/message/193511
Mute This Topic: https://lists.openembedded.org/mt/103639917/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] oeqa parselogs.py: load ignore files from sys.path
python import.resources open_text() loads files from the module paths
but this requires layers to set "addpylib ${LAYERDIR}/lib oeqa"
which is not needed to find the plain .py test files to run the tests.
Also an empty __init__.py file in a layer will break the resource file
loading completely as only that path with __init__.py file will
be used to search the resource files. Then open_text() is marked
as deprecated from python 3.11 onwards
https://docs.python.org/3/library/importlib.resources.html
So replace open_text() by iterating over sys.path to find the ignore
files. This works since paths like ${LAYERDIR}/lib/oeqa/runtime/cases are
already in sys.path. Add debug prints for found and not found files
while at it.
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/runtime/cases/parselogs.py | 29 +++-
1 file changed, 13 insertions(+), 16 deletions(-)
diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py
b/meta/lib/oeqa/runtime/cases/parselogs.py
index 6966923c94..47583dbb5d 100644
--- a/meta/lib/oeqa/runtime/cases/parselogs.py
+++ b/meta/lib/oeqa/runtime/cases/parselogs.py
@@ -6,27 +6,19 @@
import collections
import os
+import pathlib
import sys
from shutil import rmtree
from oeqa.runtime.case import OERuntimeTestCase
from oeqa.core.decorator.depends import OETestDepends
-# importlib.resources.open_text in Python <3.10 doesn't search all directories
-# when a package is split across multiple directories. Until we can rely on
-# 3.10+, reimplement the searching logic.
-if sys.version_info < (3, 10):
-def _open_text(package, resource):
-import importlib, pathlib
-module = importlib.import_module(package)
-for path in module.__path__:
-candidate = pathlib.Path(path) / resource
-if candidate.exists():
-return candidate.open(encoding='utf-8')
-raise FileNotFoundError
-else:
-from importlib.resources import open_text as _open_text
-
+def open_syspath_text(resource):
+for path in sys.path:
+candidate = pathlib.Path(path) / resource
+if candidate.exists():
+return candidate.open(encoding='utf-8')
+raise FileNotFoundError
class ParseLogsTest(OERuntimeTestCase):
@@ -61,11 +53,16 @@ class ParseLogsTest(OERuntimeTestCase):
for candidate in ["common", cls.td.get("TARGET_ARCH")] +
cls.td.get("MACHINEOVERRIDES").split(":"):
try:
name = f"parselogs-ignores-{candidate}.txt"
-for line in _open_text("oeqa.runtime.cases", name):
+print_once = True
+for line in open_syspath_text(name):
+if print_once:
+bb.debug(1, "parselogs: ignore file %s found" % (name))
+print_once = False
line = line.strip()
if line and not line.startswith("#"):
cls.ignore_errors.append(line.casefold())
except FileNotFoundError:
+bb.debug(1, "parselogs: ignore file %s not found" % (name))
pass
# Go through the log locations provided and if it's a folder
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193499):
https://lists.openembedded.org/g/openembedded-core/message/193499
Mute This Topic: https://lists.openembedded.org/mt/103639917/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
