[OE-core] [RFC PATCH][RESEND] cve-update-nvd2-native: make an error if the DB file is not present at the end of do_fetch
From: Yuta Hayama Signed-off-by: Yuta Hayama --- Please do not apply this. For now, this is just my question. I think the issue of [YOCTO #14929] is partially still there. https://bugzilla.yoctoproject.org/show_bug.cgi?id=14929 Certainly, now that 8efe99214d has been applied, do_cve_check no longer runs on broken DB file. However, if the first cve-update-nvd2-native:do_fetch run after creating the build environment fails to save CVE_CHECK_DB_FILE, do_fetch will have succeeded even though the DB file does not exist. This may be confusing. * do_cve_check seems to exit without doing anything if CVE_CHECK_DB_FILE is not existing. At this time, do_cve_check only logs in bb.note() and nothing seems to appear in the console. This may make it difficult for the user to understand what happened. * Similar to cve-update-nvd2-native:do_fetch, do_cve_check is also successful even though it does not actually do anything. Yes, a check like this patch may be needed for cve-check.bbclass... I have written a task in our own recipe (sorry, we can't publish it) that depends on do_cve_check and expects ${WORKDIR}/temp/cve.log to be present, but I have found that the task may not work because of this. I don't know if I didn't understand do_cve_check well or if the current behavior is not good... First, I would like to ask why cve-update-nvd2-native:do_fetch and do_cve_check are completing successfully even when the DB file does not exist. Well, NVD API frequently fails with errors, so I don't think it is necessary to fail the do_fetch task if it "just failed to update". But other than that, I feel it's better to make an error... Thanks, meta/recipes-core/meta/cve-update-nvd2-native.bb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb index bfe48b27e7..42e50d9cd1 100644 --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb @@ -83,6 +83,9 @@ python do_fetch() { # Update failed, do not modify the database bb.warn("CVE database update failed") os.remove(db_tmp_file) + +if not os.path.isfile(db_file): +bb.error("Failed to fetch the CVE DB. do_cve_check run after this will not work.") } do_fetch[lockfiles] += "${CVE_CHECK_DB_FILE_LOCK}" -- 2.25.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#194339): https://lists.openembedded.org/g/openembedded-core/message/194339 Mute This Topic: https://lists.openembedded.org/mt/103968560/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH] cve-update-nvd2-native: make an error if the DB file is not present at the end of do_fetch
From: Yuta Hayama Signed-off-by: Yuta Hayama --- Please do not apply this. For now, this is just my question. I think the issue of [YOCTO #14929] is partially still there. https://bugzilla.yoctoproject.org/show_bug.cgi?id=14929 Certainly, now that 8efe99214d has been applied, do_cve_check no longer runs on broken DB file. However, if the first cve-update-nvd2-native:do_fetch run after creating the build environment fails to save CVE_CHECK_DB_FILE, do_fetch will have succeeded even though the DB file does not exist. This may be confusing. * do_cve_check seems to exit without doing anything if CVE_CHECK_DB_FILE is not existing. At this time, do_cve_check only logs in bb.note() and nothing seems to appear in the console. This may make it difficult for the user to understand what happened. * Similar to cve-update-nvd2-native:do_fetch, do_cve_check is also successful even though it does not actually do anything. Yes, a check like this patch may be needed for cve-check.bbclass... I have written a task in our own recipe (sorry, we can't publish it) that depends on do_cve_check and expects ${WORKDIR}/temp/cve.log to be present, but I have found that the task may not work because of this. I don't know if I didn't understand do_cve_check well or if the current behavior is not good... First, I would like to ask why cve-update-nvd2-native:do_fetch and do_cve_check are completing successfully even when the DB file does not exist. Well, NVD API frequently fails with errors, so I don't think it is necessary to fail the do_fetch task if it "just failed to update". But other than that, I feel it's better to make an error... Thanks, meta/recipes-core/meta/cve-update-nvd2-native.bb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb index bfe48b27e7..42e50d9cd1 100644 --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb @@ -83,6 +83,9 @@ python do_fetch() { # Update failed, do not modify the database bb.warn("CVE database update failed") os.remove(db_tmp_file) + +if not os.path.isfile(db_file): +bb.error("Failed to fetch the CVE DB. do_cve_check run after this will not work.") } do_fetch[lockfiles] += "${CVE_CHECK_DB_FILE_LOCK}" -- 2.25.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193342): https://lists.openembedded.org/g/openembedded-core/message/193342 Mute This Topic: https://lists.openembedded.org/mt/103538437/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [master] [PATCH] cve-update-nvd2-native: increase the delay between subsequent request failures
On 2023/12/12 11:54, Yuta Hayama via lists.openembedded.org wrote: > I think you are right that HTTP 403 does not look like reaching the rate limit > (As Yoann noted, 429 would feel right). The HTTP 403 error returned may be > because the request is being sent from a single PC, so I will try > cve-update-nvd2-native:do_fetch on three different PCs. This would be closer > to the actual use case I have indicated. I have been trying to test running cve-update-nvd2-native:do_fetch on three PCs at the same time. The result was that only one machine continued to make successful requests until do_fetch completed, while the other two failed with . It seems that only one person can fetch in my assumed situation, no matter how much we try to adjust the delay time... >>>> As a specific example, multiple cve-update-nvd2-native:do_fetch runs >>>> in parallel can easily reach the rate limit. It can be assumed that >>>> this situation will occur if several people run bitbake in one office. >>>> (often unaware of each >>>> other...) Also, the following was my misunderstanding. > Perhaps someone has not read the documentation > about rate limiting, so the server is putting a delay before returning a > response. > That is, I think a single bitbake will not cause the issue. The reason it takes a few seconds for the server to respond is probably that the response is too long and the server is struggling. In fact, if we send requests repeatedly that shorten the response, the response will come back in a relatively short time, eventually reaching the rate limit. And the HTTP error code at that time seems to be 403 for some reason. This is the operation that clearly causes the rate limit to be reached, but it still does not look like it will be 429. Below is an example of test code: --- import urllib.request import time cves = \ ['CVE-2019-14899', 'CVE-2021-3714', 'CVE-2021-3864', 'CVE-2022-0400', 'CVE-2022-1247', 'CVE-2022-3219', 'CVE-2022-36402', 'CVE-2022-38096', 'CVE-2022-4543', 'CVE-2022-46456', 'CVE-2023-0687', 'CVE-2023-1386', 'CVE-2023-25584', 'CVE-2023-3019', 'CVE-2023-3397', 'CVE-2023-3640', 'CVE-2023-38559', 'CVE-2023-40030', 'CVE-2023-4010', 'CVE-2023-4039', 'CVE-2023-42363', 'CVE-2023-42364', 'CVE-2023-42365', 'CVE-2023-42366', 'CVE-2023-46407', 'CVE-2023-47100', 'CVE-2023-49292', 'CVE-2023-5088', 'CVE-2023-5156', 'CVE-2023-6238',] for cve in cves: url = 'https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=' + cve print('URL:', url) res = urllib.request.urlopen(url) print('.urlopen() done.') #time.sleep(6) --- Anyway, the question of HTTP error codes still remains, but unfortunately the situation of multiple people fetching at the same time seems to be difficult to deal with. Regards, Yuta Hayama -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#192254): https://lists.openembedded.org/g/openembedded-core/message/192254 Mute This Topic: https://lists.openembedded.org/mt/103049968/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [master] [PATCH] cve-update-nvd2-native: increase the delay between subsequent request failures
Hi, On 2023/12/11 19:28, Dhairya Nagodra via lists.openembedded.org wrote: >>> I have observed the do_fetch logs and found that HTTP 403 errors are >>> returned if the request is blocked, probably due to rate limitation. > > As per my knowledge, HTTP 403 is related to a permission issue rather than a > rate limitation. > I have not seen an HTTP 403 error, anytime. Can you please help clarify how > was it generated? Is it reproducible? > I tried removing both sleep delays altogether (and without API keys) to try > and generate an error. In that, I couldn't generate any errors instead, got > the response with a 15-20 sec delay. So, I guess it stayed within the rate > limit. > Yesterday I also tried to fetch with sleep removed, the result was the same, no error of any kind occurred. Perhaps someone has not read the documentation about rate limiting, so the server is putting a delay before returning a response. That is, I think a single bitbake will not cause the issue. The HTTP 403 error should be reproducible by running multiple cve-update-nvd2-native:do_fetch at the same time on a single PC. Here, I noticed that yesterday I could reproduce the error by executing two tasks in parallel, but today I had to execute three tasks in parallel to reproduce the error. Somehow, the delay that may have been provided by the server may have become longer. I think you are right that HTTP 403 does not look like reaching the rate limit (As Yoann noted, 429 would feel right). The HTTP 403 error returned may be because the request is being sent from a single PC, so I will try cve-update-nvd2-native:do_fetch on three different PCs. This would be closer to the actual use case I have indicated. >>> As a specific example, multiple cve-update-nvd2-native:do_fetch runs >>> in parallel can easily reach the rate limit. It can be assumed that >>> this situation will occur if several people run bitbake in one office. >>> (often unaware of each >>> other...) Regards, Yuta Hayama -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#192188): https://lists.openembedded.org/g/openembedded-core/message/192188 Mute This Topic: https://lists.openembedded.org/mt/103049968/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [master] [PATCH] cve-update-nvd2-native: increase the delay between subsequent request failures
Hi, On 2023/12/08 14:04, Dhairya Nagodra via lists.openembedded.org wrote: > Sometimes NVD servers are unstable and return too many errors. > There is an option to have higher fetch attempts to increase the chances > of successfully fetching the CVE data. > > Additionally, it also makes sense to progressively increase the delay > after a failed request to an already unstable or busy server. > The increase in delay is reset after every successful request and > the maximum delay is limited to 30 seconds. > > Also, the logs are improved to give more clarity. > > Signed-off-by: Dhairya Nagodra I was just working on a similar issue. As a specific example, multiple cve-update-nvd2-native:do_fetch runs in parallel can easily reach the rate limit. It can be assumed that this situation will occur if several people run bitbake in one office. (often unaware of each other...) I have observed the do_fetch logs and found that HTTP 403 errors are returned if the request is blocked, probably due to rate limitation. NOTE: Requesting https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=6000 NOTE: CVE database: received error (HTTP Error 403: Forbidden), retrying NOTE: CVE database: received error (HTTP Error 403: Forbidden), retrying NOTE: CVE database: received error (HTTP Error 403: Forbidden), retrying NOTE: CVE database: received error (HTTP Error 403: Forbidden), retrying NOTE: CVE database: received error (HTTP Error 403: Forbidden), retrying WARNING: CVE database update failed DEBUG: Python function do_fetch finished Other times a request may fail with IncompleteRead, but this is clearly distinguishable from an HTTP error. Here, we can think of the following ideas. If an HTTP error occurs, assume that the rate limit has already been reached and wait 30 seconds to ensure that the next window starts. The patch will be something like this. --- meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb index 9ab8dc6050..34fcc0317e 100644 --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb @@ -121,6 +121,7 @@ def nvd_request_next(url, attempts, api_key, args): import urllib.request import urllib.parse +import urllib.error import gzip import http import time @@ -142,10 +143,12 @@ def nvd_request_next(url, attempts, api_key, args): r.close() +except urllib.error.HTTPError as e: +bb.note("CVE database: received error (%s), wait until the next window" % (e)) +time.sleep(30) except Exception as e: bb.note("CVE database: received error (%s), retrying" % (e)) time.sleep(6) -pass else: return raw_data else: -- The time taken to fetch is likely to increase further, but the probability of failure due to error is expected to decrease greatly. Unfortunately, if the number of parallel executions is too large, this is still not good enough... I will consider what to do with this patch after your patches are merged. Since it may be enough to just extend the delay each time. Regards, Yuta Hayama -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#192145): https://lists.openembedded.org/g/openembedded-core/message/192145 Mute This Topic: https://lists.openembedded.org/mt/103049968/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/3] linux/generate-cve-exclusions: fix mishandling of boundary values
On 2023/09/08 19:57, Ross Burton via lists.openembedded.org wrote: > On 6 Sep 2023, at 13:30, Ross Burton via lists.openembedded.org > wrote: >>>> On 5 Sep 2023, at 08:29, Yuta Hayama wrote: >>>>> >>>>> affected_versions in kernel_cves.json does not mean "first affected >>>>> version >>>>> to last affected version" but actually "first affected version to fixed >>>>> version". Therefore, the variable names, conditional expressions, and >>>>> CVE_STATUS descriptions should be fixed. >>>> >>>> I’m happy to believe you on this, but do you have a source? >>> >>> Unfortunately, I have not found any official explanation for this. All I >>> know >>> is what I wrote in the following message. And that is what I have been able >>> to >>> confirm empirically. >>> >>> https://lists.openembedded.org/g/openembedded-core/message/186994 >> >> Based on that evidence you appear to be right, yes. I’ve just mailed the >> maintainer of the JSON to see if he’d like to make a statement either way. > > I got a reply: > > "The code takes the breaking_cmt to fixing_cmt. So it would be First Affected > version to First Fixed version in the mainline.” > > Yes, you’re correct. Thank you, Ross. I am relieved to hear that I was not mistaken. Also, thank you to the maintainers for applying the patch to the master. Yuta Hayama -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187474): https://lists.openembedded.org/g/openembedded-core/message/187474 Mute This Topic: https://lists.openembedded.org/mt/101164830/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/3] linux/generate-cve-exclusions: fix mishandling of boundary values
Hi, On 2023/09/05 20:14, Ross Burton wrote: > On 5 Sep 2023, at 08:29, Yuta Hayama wrote: >> >> affected_versions in kernel_cves.json does not mean "first affected version >> to last affected version" but actually "first affected version to fixed >> version". Therefore, the variable names, conditional expressions, and >> CVE_STATUS descriptions should be fixed. > > I’m happy to believe you on this, but do you have a source? Unfortunately, I have not found any official explanation for this. All I know is what I wrote in the following message. And that is what I have been able to confirm empirically. https://lists.openembedded.org/g/openembedded-core/message/186994 I looked again to see if there is any information on this in the Linux Kernel CVEs official, but it seems that at best there is only a hint in the README.md in the git repository. https://github.com/nluedtke/linux_kernel_cves/blob/master/README.md >## Overview of Process > >The process for generating these documents is focused on being as >automated as possible. Below is the general outline of steps. > > 1) Take list of all kernel CVEs > 2) If the issue is marked as Vendor specific, ignore it. > 3) Get the Breaking/Fixing Commits. This is retrieved from the > internal cache first, if not present it pulls from Ubuntu, Debian, > etc to try and fill that information in. > 4) Using those commit ids, get the first tags in the mainline that > they appear. > 5) Using that version timeline, for each stream that would be > vulnerable perform steps 6 through 8. > 6) Find the commit who has the commit message that matches the commit > message from the mainline. This is the fixing commit in that stream. > 7) Record the commit id and get the earliest tag in the stream which > has that commit. > 8) Output information to stream document. > 9) Update JSONs. In 4), it says to look for the tag where the breaking/fixing commits first appears, so we can assume that this is looking for the fixed version, not the last affected version. Well, it doesn't say which field in the json that would be... In addition, there seems to be a key "last_affected_version" in kernel_cves.json. I have no idea how this is obtained, but it seems to literally represent the "last affected version", unlike the second value of "affected_versions". Regards, Yuta Hayama -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187267): https://lists.openembedded.org/g/openembedded-core/message/187267 Mute This Topic: https://lists.openembedded.org/mt/101164830/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/3] linux/generate-cve-exclusions: print the generated time in UTC
Allow time comparisons to be made regardless of where the script was run. Signed-off-by: Yuta Hayama --- meta/recipes-kernel/linux/generate-cve-exclusions.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-kernel/linux/generate-cve-exclusions.py b/meta/recipes-kernel/linux/generate-cve-exclusions.py index b936f843e2..aa9195aab4 100755 --- a/meta/recipes-kernel/linux/generate-cve-exclusions.py +++ b/meta/recipes-kernel/linux/generate-cve-exclusions.py @@ -44,7 +44,7 @@ def main(argp=None): print(f""" # Auto-generated CVE metadata, DO NOT EDIT BY HAND. -# Generated at {datetime.datetime.now()} for version {version} +# Generated at {datetime.datetime.now(datetime.timezone.utc)} for version {version} python check_kernel_cve_status_version() {{ this_version = "{version}" -- 2.25.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187201): https://lists.openembedded.org/g/openembedded-core/message/187201 Mute This Topic: https://lists.openembedded.org/mt/101164836/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/3] linux/generate-cve-exclusions: fix mishandling of boundary values
affected_versions in kernel_cves.json does not mean "first affected version to last affected version" but actually "first affected version to fixed version". Therefore, the variable names, conditional expressions, and CVE_STATUS descriptions should be fixed. For example, when the script was run against v6.1, if affected_versions was "xxx to 6.1", the output was "cpe-stable-backport: Backported in 6.1", but this should be "fixed-version: Fixed from version 6.1". Signed-off-by: Yuta Hayama --- .../linux/generate-cve-exclusions.py | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/meta/recipes-kernel/linux/generate-cve-exclusions.py b/meta/recipes-kernel/linux/generate-cve-exclusions.py index 3bc1c7096f..b936f843e2 100755 --- a/meta/recipes-kernel/linux/generate-cve-exclusions.py +++ b/meta/recipes-kernel/linux/generate-cve-exclusions.py @@ -62,17 +62,17 @@ do_cve_check[prefuncs] += "check_kernel_cve_status_version" continue affected = data["affected_versions"] -first_affected, last_affected = re.search(r"(.+) to (.+)", affected).groups() +first_affected, fixed = re.search(r"(.+) to (.+)", affected).groups() first_affected = parse_version(first_affected) -last_affected = parse_version(last_affected) +fixed = parse_version(fixed) -if not last_affected: +if not fixed: print(f"# {cve} has no known resolution") elif first_affected and version < first_affected: print(f'CVE_STATUS[{cve}] = "fixed-version: only affects {first_affected} onwards"') -elif last_affected < version: +elif fixed <= version: print( -f'CVE_STATUS[{cve}] = "fixed-version: Fixed after version {last_affected}"' +f'CVE_STATUS[{cve}] = "fixed-version: Fixed from version {fixed}"' ) else: if cve in stream_data: @@ -87,9 +87,9 @@ do_cve_check[prefuncs] += "check_kernel_cve_status_version" # TODO print a note that the kernel needs bumping print(f"# {cve} needs backporting (fixed from {backport_ver})") else: -print(f"# {cve} needs backporting (fixed from {last_affected})") +print(f"# {cve} needs backporting (fixed from {fixed})") else: -print(f"# {cve} needs backporting (fixed from {last_affected})") +print(f"# {cve} needs backporting (fixed from {fixed})") print() -- 2.25.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187200): https://lists.openembedded.org/g/openembedded-core/message/187200 Mute This Topic: https://lists.openembedded.org/mt/101164830/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] generate-cve-exclusions.py: boundary-value error?
Sorry, I thought that a fixed CVE would be incorrectly marked as "has no known resolution" if (last_affected == version) unless the following patch is applied, but this was a misunderstanding. --- a/meta/recipes-kernel/linux/generate-cve-exclusions.py +++ b/meta/recipes-kernel/linux/generate-cve-exclusions.py @@ -70,7 +70,7 @@ do_cve_check[prefuncs] += "check_kernel_cve_status_version" print(f"# {cve} has no known resolution") elif first_affected and version < first_affected: print(f'CVE_STATUS[{cve}] = "fixed-version: only affects {first_affected} onwards"') -elif last_affected < version: +elif last_affected <= version: print( f'CVE_STATUS[{cve}] = "fixed-version: Fixed after version {last_affected}"' ) Comparing the output for version 6.1 before and after applying above patch, the following results were obtained: --- 6.1.0_unpatched.inc 2023-09-05 09:59:54.166291339 +0900 +++ 6.1.0_patched.inc 2023-09-05 09:58:28.179817233 +0900 (... snip ...) @@ -4422,7 +4422,7 @@ # CVE-2022-3642 has no known resolution -CVE_STATUS[CVE-2022-3643] = "cpe-stable-backport: Backported in 6.1" +CVE_STATUS[CVE-2022-3643] = "fixed-version: Fixed after version 6.1" CVE_STATUS[CVE-2022-3646] = "fixed-version: Fixed after version 6.1rc1" If the range check for affected_versions is not hit, the script checks for backports to the stable version, so the result is not "has no known resolution". Since I didn't read the last part of the script properly, I missed it... On the other hand, there still seems to be an issue that the meaning of CVE_STATUS seems to be incorrect, so I will try to write a patch. * I think "fixed-version: Fixed after version {last_affected}" is exactly "fixed-version: Fixed *from* version {last_affected}". (i.e., {last_affected} should have already be fixed) * Since the backported patches are not included in version X.Y, "cpe-stable-backport: Backported in X.Y" should be "fixed-version: Fixed from version X.Y". Regards, Yuta Hayama On 2023/09/01 15:03, Yuta Hayama via lists.openembedded.org wrote: > Hi, > > I have been trying Linux Kernel CVEs for a while now and found a weird part in > generate-cve-exclusions.py. > > https://git.openembedded.org/openembedded-core/tree/meta/recipes-kernel/linux/generate-cve-exclusions.py#n64 > >>affected = data["affected_versions"] >>first_affected, last_affected = re.search(r"(.+) to (.+)", >> affected).groups() >>first_affected = parse_version(first_affected) >>last_affected = parse_version(last_affected) >> >>if not last_affected: >>print(f"# {cve} has no known resolution") >>elif first_affected and version < first_affected: >>print(f'CVE_STATUS[{cve}] = "fixed-version: only affects >> {first_affected} onwards"') >>elif last_affected < version: >>print( >>f'CVE_STATUS[{cve}] = "fixed-version: Fixed after version >> {last_affected}"' > > generate-cve-exclusions.py interprets "affected_versions" in kernel_cves.json > literally as "first affected version" to "last affected version". However, > "affected_versions" may actually mean "first affected version" to "fixed > version". > > I honestly don't know what's going on here, as I can't find any documentation > for Linux Kernel CVEs. But, picking up a random CVE and checking it in detail, > I think the second element in "affected_versions" is actually a "fixed > version". > > Some examples. > > [CVE-2023-28327] > * In kernel_cves.json, "affected_versions" is described as "v5.3-rc1 to v6.1" > * the patch is > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b3abe42e94900bdd045c472f9c9be620ba5ce553 > * v6.1-rc8 source is > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/unix/diag.c?h=v6.1-rc8#n117 > * v6.1 source is > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/unix/diag.c?h=v6.1#n117 > * It seems to be fixed in 6.1 (not after 6.1). > > [CVE-2021-3772] > * In kernel_cves.json, "affected_versions" is described as "v2.6.12-rc2 to > v5.15" > * the patch is > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4f7019c7eb33967eb87766e0e4602b5576873680 > * v5.15-rc7 source is > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sctp/sm_statefun
[OE-core] generate-cve-exclusions.py: boundary-value error?
Hi, I have been trying Linux Kernel CVEs for a while now and found a weird part in generate-cve-exclusions.py. https://git.openembedded.org/openembedded-core/tree/meta/recipes-kernel/linux/generate-cve-exclusions.py#n64 >affected = data["affected_versions"] >first_affected, last_affected = re.search(r"(.+) to (.+)", > affected).groups() >first_affected = parse_version(first_affected) >last_affected = parse_version(last_affected) > >if not last_affected: >print(f"# {cve} has no known resolution") >elif first_affected and version < first_affected: >print(f'CVE_STATUS[{cve}] = "fixed-version: only affects > {first_affected} onwards"') >elif last_affected < version: >print( >f'CVE_STATUS[{cve}] = "fixed-version: Fixed after version > {last_affected}"' generate-cve-exclusions.py interprets "affected_versions" in kernel_cves.json literally as "first affected version" to "last affected version". However, "affected_versions" may actually mean "first affected version" to "fixed version". I honestly don't know what's going on here, as I can't find any documentation for Linux Kernel CVEs. But, picking up a random CVE and checking it in detail, I think the second element in "affected_versions" is actually a "fixed version". Some examples. [CVE-2023-28327] * In kernel_cves.json, "affected_versions" is described as "v5.3-rc1 to v6.1" * the patch is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b3abe42e94900bdd045c472f9c9be620ba5ce553 * v6.1-rc8 source is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/unix/diag.c?h=v6.1-rc8#n117 * v6.1 source is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/unix/diag.c?h=v6.1#n117 * It seems to be fixed in 6.1 (not after 6.1). [CVE-2021-3772] * In kernel_cves.json, "affected_versions" is described as "v2.6.12-rc2 to v5.15" * the patch is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4f7019c7eb33967eb87766e0e4602b5576873680 * v5.15-rc7 source is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sctp/sm_statefuns.c?h=v5.15-rc7#n6350 * v5.15 source is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sctp/sm_statefuns.c?h=v5.15#n6380 * It seems to be fixed in 5.15 (not after 5.15). [CVE-2022-39190] * In kernel_cves.json, "affected_versions" is described as "v5.9-rc1 to v6.0-rc3" * the patch is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e02f0d3970404bfea385b6edb86f2d936db0ea2b * v6.0-rc2 source is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/netfilter/nf_tables_api.c?h=v6.0-rc2#n9708 * v6.0-rc3 source is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/netfilter/nf_tables_api.c?h=v6.0-rc3#n9712 * It seems to be fixed in v6.0-rc3 (not after v6.0-rc3). If the assumption is correct, I will try to write a patch. Note that in reality, I think there are very few cases of error. >elif last_affected < version: >print( >f'CVE_STATUS[{cve}] = "fixed-version: Fixed after version > {last_affected}"' The last_affected here is the major version (in short, the last_affected is x.y, not x.y.z), so there is a possibility of error only if version is the major version. But actually, in most cases, version is x.y.z. Regards, Yuta Hayama -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186994): https://lists.openembedded.org/g/openembedded-core/message/186994 Mute This Topic: https://lists.openembedded.org/mt/101089426/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [kirkstone][dunfell][PATCH] cve-update-nvd2-native: always pass str for json.loads()
Hi Steve, On 2023/07/28 4:01, Steve Sakoman wrote: > Will this change also work on master and mickledore? If so, please submit > for master and I will cherry-pick to the other branches. > > I've been trying to keep the cve checking in the stable branches in sync > with master as much as possible. This patch can be applied to master and mickledore, but this patch is actually only needed for build environments where the build host Python3 is earlier than 3.6. I referred to the Yocto Project Reference Manual to find out which Linux distributions are supported by each version of Yocto (OE-Core). The Linux distributions supported by mickledore are listed below, but I think that none of them will install Python3 earlier than 3.6 by default. This should be the same for master. * Ubuntu 18.04, 20.04, 22.04 * Fedora 36, 37 * AlmaLinux 8.7, 9.1 * Debian GNU/Linux 11.x * OpenSUSE Leap 15.3, 15.4 On the other hand, kirkstone supports Debian 9.x (with python 3.5 installed by default) and dunfell supports Debian 9.x and Ubuntu 16.04 (also with python 3.5 installed by default). Therefore, I think strictly speaking only kirkstone and dunfell need this patch. For other branches (i.e., those that are clearly intended to run on Python 3.6 or later), there is no difference in behavior whether this patch is applied or not. Rather, it may be better performance-wise to not apply this patch, since it eliminates unnecessary decode() calls. Yes, of course it is important to sync each stable branch with master. What should we do in such a case... Regards, Yuta Hayama > > Thanks! > > Steve > > On Wed, Jul 26, 2023 at 10:56 PM Yuta Hayama wrote: > >> Currently json.loads() accepts one of the types str, bytes, or bytearray >> as an argument, but bytes and bytearrays have only been allowed since >> python 3.6. The version of Python3 provided by default on Ubuntu 16.04 >> and Debian 9.x is 3.5, so make raw_data type str to work correctly on >> these build hosts. >> >> Signed-off-by: Yuta Hayama >> --- >> meta/recipes-core/meta/cve-update-nvd2-native.bb | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb >> b/meta/recipes-core/meta/cve-update-nvd2-native.bb >> index 2f7dad7e82..67d76f75dd 100644 >> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb >> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb >> @@ -136,7 +136,7 @@ def nvd_request_next(url, api_key, args): >> >> if (r.headers['content-encoding'] == 'gzip'): >> buf = r.read() >> -raw_data = gzip.decompress(buf) >> +raw_data = gzip.decompress(buf).decode("utf-8") >> else: >> raw_data = r.read().decode("utf-8") >> >> -- >> 2.25.1 >> >> > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#185006): https://lists.openembedded.org/g/openembedded-core/message/185006 Mute This Topic: https://lists.openembedded.org/mt/100387035/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [kirkstone][dunfell][PATCH] cve-update-nvd2-native: always pass str for json.loads()
Currently json.loads() accepts one of the types str, bytes, or bytearray as an argument, but bytes and bytearrays have only been allowed since python 3.6. The version of Python3 provided by default on Ubuntu 16.04 and Debian 9.x is 3.5, so make raw_data type str to work correctly on these build hosts. Signed-off-by: Yuta Hayama --- meta/recipes-core/meta/cve-update-nvd2-native.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb index 2f7dad7e82..67d76f75dd 100644 --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb @@ -136,7 +136,7 @@ def nvd_request_next(url, api_key, args): if (r.headers['content-encoding'] == 'gzip'): buf = r.read() -raw_data = gzip.decompress(buf) +raw_data = gzip.decompress(buf).decode("utf-8") else: raw_data = r.read().decode("utf-8") -- 2.25.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184924): https://lists.openembedded.org/g/openembedded-core/message/184924 Mute This Topic: https://lists.openembedded.org/mt/100387035/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [mickledore][kirkstone][dunfell][PATCH] systemd-systemctl: fix errors in instance name expansion
Hi Steve, Sorry for rushing you. If the patch obviously needs to be backported to stable branch, could I post a backport request email right away? This issue is not listed in bugzilla, so I was worried that the patch might have been missed. Regards, Yuta Hayama On 2023/07/27 17:19, Yuta Hayama via lists.openembedded.org wrote: > If the instance name indicated by %i begins with a number, the meaning of the > replacement string "\\1{}".format(instance) is ambiguous. > > To indicate group number 1 regardless of the instance name, use "\g<1>". > > (From OE-Core rev: d18b939fb08b37380ce95934da38e6522392621c) > > Signed-off-by: Yuta Hayama > Signed-off-by: Richard Purdie > --- > meta/recipes-core/systemd/systemd-systemctl/systemctl | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/recipes-core/systemd/systemd-systemctl/systemctl > b/meta/recipes-core/systemd/systemd-systemctl/systemctl > index 514f747fe6..7fe751b397 100755 > --- a/meta/recipes-core/systemd/systemd-systemctl/systemctl > +++ b/meta/recipes-core/systemd/systemd-systemctl/systemctl > @@ -202,7 +202,7 @@ class SystemdUnit(): > try: > for dependent in config.get('Install', prop): > # expand any %i to instance (ignoring escape sequence %%) > -dependent = re.sub("([^%](%%)*)%i", > "\\1{}".format(instance), dependent) > +dependent = re.sub("([^%](%%)*)%i", > "\\g<1>{}".format(instance), dependent) > wants = systemdir / "{}.{}".format(dependent, dirstem) / > service > add_link(wants, target) > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184919): https://lists.openembedded.org/g/openembedded-core/message/184919 Mute This Topic: https://lists.openembedded.org/mt/100386774/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [mickledore][kirkstone][dunfell][PATCH] systemd-systemctl: fix errors in instance name expansion
If the instance name indicated by %i begins with a number, the meaning of the replacement string "\\1{}".format(instance) is ambiguous. To indicate group number 1 regardless of the instance name, use "\g<1>". (From OE-Core rev: d18b939fb08b37380ce95934da38e6522392621c) Signed-off-by: Yuta Hayama Signed-off-by: Richard Purdie --- meta/recipes-core/systemd/systemd-systemctl/systemctl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-core/systemd/systemd-systemctl/systemctl b/meta/recipes-core/systemd/systemd-systemctl/systemctl index 514f747fe6..7fe751b397 100755 --- a/meta/recipes-core/systemd/systemd-systemctl/systemctl +++ b/meta/recipes-core/systemd/systemd-systemctl/systemctl @@ -202,7 +202,7 @@ class SystemdUnit(): try: for dependent in config.get('Install', prop): # expand any %i to instance (ignoring escape sequence %%) -dependent = re.sub("([^%](%%)*)%i", "\\1{}".format(instance), dependent) +dependent = re.sub("([^%](%%)*)%i", "\\g<1>{}".format(instance), dependent) wants = systemdir / "{}.{}".format(dependent, dirstem) / service add_link(wants, target) -- 2.25.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184918): https://lists.openembedded.org/g/openembedded-core/message/184918 Mute This Topic: https://lists.openembedded.org/mt/100386774/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] systemd service@ bug
Hi, On 2023/07/19 16:47, Vyacheslav Yurkov wrote: > Hey, > Usually Steve Sakoman cherry-picks patches from master. If it's not the case, > it was probably not applied clearly or simply overlooked. You can submit a > backport with the branch tag. Thank you! I understood. I checked git log and it looked like it usually takes around a month to be backported, so I will wait until then. Regards, Yuta Hayama > > Vyacheslav > > On 19.07.2023 03:48, Yuta Hayama wrote: >> Hi, >> >> This issue has been fixed in master. >> https://git.openembedded.org/openembedded-core/commit/?id=d18b939fb08b37380ce95934da38e6522392621c >> >> But, yes. This patch has not yet been backported to the any stable branch. >> >> >> I don't know about the maintenance process for the stable branch, but I >> expect that patch will probably be queued and backported to dunfell in >> a month or so. >> >> Please let me know if anyone knows anything about this. Should we simply >> wait? Or do I have to submit a backport request? >> >> >> Regards, >> >> Yuta Hayama > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184620): https://lists.openembedded.org/g/openembedded-core/message/184620 Mute This Topic: https://lists.openembedded.org/mt/100140897/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] systemd service@ bug
Hi, This issue has been fixed in master. https://git.openembedded.org/openembedded-core/commit/?id=d18b939fb08b37380ce95934da38e6522392621c But, yes. This patch has not yet been backported to the any stable branch. I don't know about the maintenance process for the stable branch, but I expect that patch will probably be queued and backported to dunfell in a month or so. Please let me know if anyone knows anything about this. Should we simply wait? Or do I have to submit a backport request? Regards, Yuta Hayama On 2023/07/14 22:42, Dvorkin Dmitry wrote: > Hello! > > (sorry, previously posted as the reply to wrong message) > > starting from 2aa82324d43467e7c8bfbbb59570ee3306264b75 commit (dunfell and > probably other branches) > > https://git.openembedded.org/openembedded-core/commit/?h=dunfell=2aa82324d43467e7c8bfbbb59570ee3306264b75 > > the > > SYSTEMD_SERVICE:${PN} = "php-fpm@9000.service" > > syntax in the recipe brings OE to error at do_rootfs step: > > https://pastebin.com/WbDCPy4V > > > + for service in php-fpm@9000.service > + systemctl > --root=/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac > -anpr/1.0-r0/rootfs enable php-fpm@9000.service > Traceback (most recent call last): > File "/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac-anpr/1.0-r > 0/recipe-sysroot-native/usr/bin/systemctl", line 331, in > main() > File "/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac-anpr/1.0-r > 0/recipe-sysroot-native/usr/bin/systemctl", line 319, in main > SystemdUnit(root, service).enable() > File "/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac-anpr/1.0-r > 0/recipe-sysroot-native/usr/bin/systemctl", line 232, in enable > self._process_deps(config, service, path, 'WantedBy', 'wants', instance) > File "/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac-anpr/1.0-r > 0/recipe-sysroot-native/usr/bin/systemctl", line 192, in _process_deps > dependent = re.sub("([^%](%%)*)%i", "\\1{}".format(instance), dependent) > File "/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac-anpr/1.0-r > 0/recipe-sysroot-native/usr/lib/python3.8/re.py", line 210, in sub > return _compile(pattern, flags).sub(repl, string, count) > File "/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac-anpr/1.0-r > 0/recipe-sysroot-native/usr/lib/python3.8/re.py", line 327, in _subx > template = _compile_repl(template, pattern) > File "/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac-anpr/1.0-r > 0/recipe-sysroot-native/usr/lib/python3.8/re.py", line 318, in _compile_repl > return sre_parse.parse_template(repl, pattern) > File "/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac-anpr/1.0-r > 0/recipe-sysroot-native/usr/lib/python3.8/sre_parse.py", line 1036, in > parse_tem > plate > addgroup(int(this[1:]), len(this) - 1) > File "/workdir/build.tppg2/tmp/work/tppg2-tps-linux-gnueabi/img-oac-anpr/1.0-r > 0/recipe-sysroot-native/usr/lib/python3.8/sre_parse.py", line 980, in addgroup > raise s.error("invalid group reference %d" % index, pos) > re.error: invalid group reference 19 at position 1 > %post(oac-php-1.0-r0.noarch): waitpid(3549888) rc 3549888 status 100 > warning: %post(oac-php-1.0-r0.noarch) scriptlet failed, exit status 1 > Error in POSTIN scriptlet in rpm package oac-php > Installing : curl-7.69.1-r0.cortexa9hf_neon 849/863 > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184564): https://lists.openembedded.org/g/openembedded-core/message/184564 Mute This Topic: https://lists.openembedded.org/mt/100140897/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] systemd-systemctl: fix errors in instance name expansion
If the instance name indicated by %i begins with a number, the meaning of the replacement string "\\1{}".format(instance) is ambiguous. To indicate group number 1 regardless of the instance name, use "\g<1>". Signed-off-by: Yuta Hayama --- meta/recipes-core/systemd/systemd-systemctl/systemctl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-core/systemd/systemd-systemctl/systemctl b/meta/recipes-core/systemd/systemd-systemctl/systemctl index 514f747fe6..7fe751b397 100755 --- a/meta/recipes-core/systemd/systemd-systemctl/systemctl +++ b/meta/recipes-core/systemd/systemd-systemctl/systemctl @@ -202,7 +202,7 @@ class SystemdUnit(): try: for dependent in config.get('Install', prop): # expand any %i to instance (ignoring escape sequence %%) -dependent = re.sub("([^%](%%)*)%i", "\\1{}".format(instance), dependent) +dependent = re.sub("([^%](%%)*)%i", "\\g<1>{}".format(instance), dependent) wants = systemdir / "{}.{}".format(dependent, dirstem) / service add_link(wants, target) -- 2.25.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#183875): https://lists.openembedded.org/g/openembedded-core/message/183875 Mute This Topic: https://lists.openembedded.org/mt/99960048/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-