Re: [OE-core][PATCH] classes: cve-check: Get shared database lock
On Thu, Sep 1, 2022 at 10:51 PM Marta Rybczynska wrote:
>
> On Fri, Sep 2, 2022 at 9:09 AM Marta Rybczynska via
> lists.openembedded.org
> wrote:
> >
> > On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt wrote:
> > >
> > > The CVE check database needs to have a shared lock acquired on it before
> > > it is accessed. This to prevent cve-update-db-native from deleting the
> > > database file out from underneath it.
> > >
> > > [YOCTO #14899]
> > >
> > > Signed-off-by: Joshua Watt
> > > +cve_data = get_cve_info(d, patched + unpatched + ignored)
> > > +cve_write_data(d, patched, unpatched, ignored, cve_data,
> > > status)
> > > +else:
> > > +bb.note("No CVE database found, skipping CVE check")
> > >
> >
> > With this commit in kirkstone-nut, we're getting an error with
> > meta-zephyr builds:
> >
> > RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
> > do_cve_check: Error executing a python function in exec_func_python()
> > autogenerated:
> > The stack trace of python calls that resulted in this exception/failure was:
> > File: 'exec_func_python() autogenerated', lineno: 2, function:
> > 0001:
> > *** 0002:do_cve_check(d)
> > 0003:
> > File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
> > lineno: 142, function: do_cve_check
> > 0138: Check recipe for patched and unpatched CVEs
> > 0139: """
> > 0140: from oe.cve_check import get_patched_cves
> > 0141:
> > *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
> > shared=True):
> > 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
> > 0144: try:
> > 0145: patched_cves = get_patched_cves(d)
> > 0146: except FileNotFoundError:
> > File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper
> > 0236:
> > 0237: """
> > 0238: @wraps(func)
> > 0239: def helper(*args, **kwds):
> > *** 0240: return _GeneratorContextManager(func, args, kwds)
> > 0241: return helper
> > 0242:
> > 0243:
> > 0244:def asynccontextmanager(func):
> > File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__
> > 0079:class _GeneratorContextManagerBase:
> > 0080: """Shared functionality for @contextmanager and
> > @asynccontextmanager."""
> > 0081:
> > 0082: def __init__(self, func, args, kwds):
> > *** 0083: self.gen = func(*args, **kwds)
> > 0084: self.func, self.args, self.kwds = func, args, kwds
> > 0085: # Issue 19330: ensure context manager instances have good docstrings
> > 0086: doc = getattr(func, "__doc__", None)
> > 0087: if doc is None:
> > Exception: TypeError: fileslocked() got an unexpected keyword argument
> > 'shared'
> > ERROR: Logfile of failure stored in:
> > /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
> > NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
> > task do_cve_check: Failed
> > ERROR: Task
> > (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check)
> > failed with exit code '1'
> >
>
> This is a mismatch with bitbake, because one commit hasn't reached 2.0.
> Steve, should I be using
> https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for
> testing?
If you are going to try to use oe-core stable/kirkstone-nut then, yes,
you should also use bitbake stable/2.0-nut since occasionally changes
in core must be paired with a bitbake change.
However . . .
I highly discourage anyone from using the -nut branches! They are
guaranteed to be rebased (and broken!), sometimes several times a day.
So be prepared for pain!
Patches move to stable/kirkstone-next (and stable2.0-next) a day after
I post a patchset for review. At this point they have passed all
testing and are unlikely to change unless the community finds an issue
during the review period. So it would be safer to use the
stable/*-next branch (but I still discourage it since it is possible
it will change)
Steve
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170248):
https://lists.openembedded.org/g/openembedded-core/message/170248
Mute This Topic: https://lists.openembedded.org/mt/93352038/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH] classes: cve-check: Get shared database lock
On Fri, 2022-09-02 at 10:50 +0200, Marta Rybczynska wrote:
> On Fri, Sep 2, 2022 at 9:09 AM Marta Rybczynska via
> lists.openembedded.org
> wrote:
> >
> > On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt wrote:
> > >
> > > The CVE check database needs to have a shared lock acquired on it before
> > > it is accessed. This to prevent cve-update-db-native from deleting the
> > > database file out from underneath it.
> > >
> > > [YOCTO #14899]
> > >
> > > Signed-off-by: Joshua Watt
> > > +cve_data = get_cve_info(d, patched + unpatched + ignored)
> > > +cve_write_data(d, patched, unpatched, ignored, cve_data,
> > > status)
> > > +else:
> > > +bb.note("No CVE database found, skipping CVE check")
> > >
> >
> > With this commit in kirkstone-nut, we're getting an error with
> > meta-zephyr builds:
> >
> > RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
> > do_cve_check: Error executing a python function in exec_func_python()
> > autogenerated:
> > The stack trace of python calls that resulted in this exception/failure was:
> > File: 'exec_func_python() autogenerated', lineno: 2, function:
> > 0001:
> > *** 0002:do_cve_check(d)
> > 0003:
> > File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
> > lineno: 142, function: do_cve_check
> > 0138: Check recipe for patched and unpatched CVEs
> > 0139: """
> > 0140: from oe.cve_check import get_patched_cves
> > 0141:
> > *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
> > shared=True):
> > 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
> > 0144: try:
> > 0145: patched_cves = get_patched_cves(d)
> > 0146: except FileNotFoundError:
> > File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper
> > 0236:
> > 0237: """
> > 0238: @wraps(func)
> > 0239: def helper(*args, **kwds):
> > *** 0240: return _GeneratorContextManager(func, args, kwds)
> > 0241: return helper
> > 0242:
> > 0243:
> > 0244:def asynccontextmanager(func):
> > File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__
> > 0079:class _GeneratorContextManagerBase:
> > 0080: """Shared functionality for @contextmanager and
> > @asynccontextmanager."""
> > 0081:
> > 0082: def __init__(self, func, args, kwds):
> > *** 0083: self.gen = func(*args, **kwds)
> > 0084: self.func, self.args, self.kwds = func, args, kwds
> > 0085: # Issue 19330: ensure context manager instances have good docstrings
> > 0086: doc = getattr(func, "__doc__", None)
> > 0087: if doc is None:
> > Exception: TypeError: fileslocked() got an unexpected keyword argument
> > 'shared'
> > ERROR: Logfile of failure stored in:
> > /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
> > NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
> > task do_cve_check: Failed
> > ERROR: Task
> > (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check)
> > failed with exit code '1'
> >
>
> This is a mismatch with bitbake, because one commit hasn't reached 2.0.
> Steve, should I be using
> https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for
> testing?
I've gone ahead and backported that bitbake change to make things
easier for people.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170241):
https://lists.openembedded.org/g/openembedded-core/message/170241
Mute This Topic: https://lists.openembedded.org/mt/93352038/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH] classes: cve-check: Get shared database lock
On Fri, Sep 2, 2022 at 9:09 AM Marta Rybczynska via
lists.openembedded.org
wrote:
>
> On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt wrote:
> >
> > The CVE check database needs to have a shared lock acquired on it before
> > it is accessed. This to prevent cve-update-db-native from deleting the
> > database file out from underneath it.
> >
> > [YOCTO #14899]
> >
> > Signed-off-by: Joshua Watt
> > +cve_data = get_cve_info(d, patched + unpatched + ignored)
> > +cve_write_data(d, patched, unpatched, ignored, cve_data,
> > status)
> > +else:
> > +bb.note("No CVE database found, skipping CVE check")
> >
>
> With this commit in kirkstone-nut, we're getting an error with
> meta-zephyr builds:
>
> RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
> do_cve_check: Error executing a python function in exec_func_python()
> autogenerated:
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_func_python() autogenerated', lineno: 2, function:
> 0001:
> *** 0002:do_cve_check(d)
> 0003:
> File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
> lineno: 142, function: do_cve_check
> 0138: Check recipe for patched and unpatched CVEs
> 0139: """
> 0140: from oe.cve_check import get_patched_cves
> 0141:
> *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
> shared=True):
> 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
> 0144: try:
> 0145: patched_cves = get_patched_cves(d)
> 0146: except FileNotFoundError:
> File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper
> 0236:
> 0237: """
> 0238: @wraps(func)
> 0239: def helper(*args, **kwds):
> *** 0240: return _GeneratorContextManager(func, args, kwds)
> 0241: return helper
> 0242:
> 0243:
> 0244:def asynccontextmanager(func):
> File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__
> 0079:class _GeneratorContextManagerBase:
> 0080: """Shared functionality for @contextmanager and @asynccontextmanager."""
> 0081:
> 0082: def __init__(self, func, args, kwds):
> *** 0083: self.gen = func(*args, **kwds)
> 0084: self.func, self.args, self.kwds = func, args, kwds
> 0085: # Issue 19330: ensure context manager instances have good docstrings
> 0086: doc = getattr(func, "__doc__", None)
> 0087: if doc is None:
> Exception: TypeError: fileslocked() got an unexpected keyword argument
> 'shared'
> ERROR: Logfile of failure stored in:
> /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
> NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
> task do_cve_check: Failed
> ERROR: Task
> (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check)
> failed with exit code '1'
>
This is a mismatch with bitbake, because one commit hasn't reached 2.0.
Steve, should I be using
https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for
testing?
Kind regards,
Marta
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170239):
https://lists.openembedded.org/g/openembedded-core/message/170239
Mute This Topic: https://lists.openembedded.org/mt/93352038/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH] classes: cve-check: Get shared database lock
On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt wrote:
>
> The CVE check database needs to have a shared lock acquired on it before
> it is accessed. This to prevent cve-update-db-native from deleting the
> database file out from underneath it.
>
> [YOCTO #14899]
>
> Signed-off-by: Joshua Watt
> +cve_data = get_cve_info(d, patched + unpatched + ignored)
> +cve_write_data(d, patched, unpatched, ignored, cve_data,
> status)
> +else:
> +bb.note("No CVE database found, skipping CVE check")
>
With this commit in kirkstone-nut, we're getting an error with
meta-zephyr builds:
RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
do_cve_check: Error executing a python function in exec_func_python()
autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function:
0001:
*** 0002:do_cve_check(d)
0003:
File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
lineno: 142, function: do_cve_check
0138: Check recipe for patched and unpatched CVEs
0139: """
0140: from oe.cve_check import get_patched_cves
0141:
*** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
shared=True):
0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
0144: try:
0145: patched_cves = get_patched_cves(d)
0146: except FileNotFoundError:
File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper
0236:
0237: """
0238: @wraps(func)
0239: def helper(*args, **kwds):
*** 0240: return _GeneratorContextManager(func, args, kwds)
0241: return helper
0242:
0243:
0244:def asynccontextmanager(func):
File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__
0079:class _GeneratorContextManagerBase:
0080: """Shared functionality for @contextmanager and @asynccontextmanager."""
0081:
0082: def __init__(self, func, args, kwds):
*** 0083: self.gen = func(*args, **kwds)
0084: self.func, self.args, self.kwds = func, args, kwds
0085: # Issue 19330: ensure context manager instances have good docstrings
0086: doc = getattr(func, "__doc__", None)
0087: if doc is None:
Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared'
ERROR: Logfile of failure stored in:
/tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
task do_cve_check: Failed
ERROR: Task
(/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check)
failed with exit code '1'
Kind regards,
Marta
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170235):
https://lists.openembedded.org/g/openembedded-core/message/170235
Mute This Topic: https://lists.openembedded.org/mt/93352038/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core][PATCH] classes: cve-check: Get shared database lock
The CVE check database needs to have a shared lock acquired on it before
it is accessed. This to prevent cve-update-db-native from deleting the
database file out from underneath it.
[YOCTO #14899]
Signed-off-by: Joshua Watt
---
meta/classes/cve-check.bbclass | 23 ---
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index d95465775d..5c8b512c11 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -145,17 +145,18 @@ python do_cve_check () {
"""
from oe.cve_check import get_patched_cves
-if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
-try:
-patched_cves = get_patched_cves(d)
-except FileNotFoundError:
-bb.fatal("Failure in searching patches")
-ignored, patched, unpatched, status = check_cves(d, patched_cves)
-if patched or unpatched or (d.getVar("CVE_CHECK_COVERAGE") == "1" and
status):
-cve_data = get_cve_info(d, patched + unpatched + ignored)
-cve_write_data(d, patched, unpatched, ignored, cve_data, status)
-else:
-bb.note("No CVE database found, skipping CVE check")
+with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
shared=True):
+if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
+try:
+patched_cves = get_patched_cves(d)
+except FileNotFoundError:
+bb.fatal("Failure in searching patches")
+ignored, patched, unpatched, status = check_cves(d, patched_cves)
+if patched or unpatched or (d.getVar("CVE_CHECK_COVERAGE") == "1"
and status):
+cve_data = get_cve_info(d, patched + unpatched + ignored)
+cve_write_data(d, patched, unpatched, ignored, cve_data,
status)
+else:
+bb.note("No CVE database found, skipping CVE check")
}
--
2.33.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170076):
https://lists.openembedded.org/g/openembedded-core/message/170076
Mute This Topic: https://lists.openembedded.org/mt/93352038/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
