On Thu Jun 25, 2026 at 1:54 PM CEST, Awais Belal wrote: > Hi Yoann, > > Thanks for picking this up > >> What value are you using? Especially in the last few days... > > I've seen much better results with a cap of 500/1000 than the default of > 2000. > >> I'm not sure CVE_DB_RESULTS_PER_PAGE_MAX should be a bitbake variable (I >> don't see a reason for it to be configurable). Can you simplify the code >> a little by making it a constant in the code instead? We already have a >> handfull of hardcoded limits from NVD. > > Well it is a valid request parameter for starters. This allows the users to > control it from the configurations rather than hard coding to any limits > someone sees fitting for their environment so more flexibility. Not setting > the variable defaults to the limits recommended by NVD however my analysis > shows that these days due to bulk updates on the NVD side the fetch has > failed most of the time due to an "incomplete read" error.
I agree that a configurable CVE_DB_RESULTS_PER_PAGE looks like a good idea. I was questionning why CVE_DB_RESULTS_PER_PAGE*_MAX* (notice the _MAX) should be. Seems to me that this is a limit imposed by NVD itself and I don't see a reason to set it to a different value than 2000 (as given by NVD). > > On Thu, Jun 25, 2026 at 4:43 PM Yoann Congal <[email protected]> wrote: > >> Hello, >> >> Thanks for looking at this! >> >> On Thu Jun 25, 2026 at 1:15 PM CEST, Awais Belal via >> lists.openembedded.org wrote: >> > From: Awais B <[email protected]> >> > >> > It is seen that during bulk updates on the NVD side the server >> > struggles to keep up with the default/max of 2000 entries per >> > page and we see a lot of incomplete read errors resulting in >> > proper db sync failures most of the times. Lowering the per >> > page value noticably increases the reliability of the process >> > and hence should ideally be configurable. >> >> What value are you using? Especially in the last few days... >> >> > >> > Signed-off-by: Awais B <[email protected]> >> > --- >> > meta/recipes-core/meta/cve-update-nvd2-native.bb | 14 ++++++++++++++ >> > 1 file changed, 14 insertions(+) >> > >> > diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb >> b/meta/recipes-core/meta/cve-update-nvd2-native.bb >> > index 945bd1d927..5d8b76e62c 100644 >> > --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb >> > +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb >> > @@ -34,6 +34,11 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000" >> > # Number of attempts for each http query to nvd server before giving up >> > CVE_DB_UPDATE_ATTEMPTS ?= "5" >> > >> > +# Maximum number of CVE records per API response. Default/max is 2000. >> > +# Lowering this value can help avoid incomplete read errors during bulk >> NVD updates. >> > +CVE_DB_RESULTS_PER_PAGE ?= "" >> > +CVE_DB_RESULTS_PER_PAGE_MAX ?= "2000" >> > + >> > CVE_CHECK_DB_DLDIR_FILE ?= >> "${DL_DIR}/CVE_CHECK/${CVE_CHECK_DB_FILENAME}" >> > CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock" >> > CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp" >> > @@ -217,6 +222,15 @@ def update_db_file(db_tmp_file, d, database_time): >> > api_key = d.getVar("NVDCVE_API_KEY") or None >> > attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS")) >> > >> > + results_per_page = d.getVar("CVE_DB_RESULTS_PER_PAGE") >> > + results_per_page_max = >> int(d.getVar("CVE_DB_RESULTS_PER_PAGE_MAX")) >> >> I'm not sure CVE_DB_RESULTS_PER_PAGE_MAX should be a bitbake variable (I >> don't see a reason for it to be configurable). Can you simplify the code >> a little by making it a constant in the code instead? We already have a >> handfull of hardcoded limits from NVD. >> >> > + if results_per_page: >> > + results_per_page = int(results_per_page) >> > + if results_per_page > results_per_page_max: >> > + bb.warn("CVE_DB_RESULTS_PER_PAGE exceeds maximum of %d, >> capping" % results_per_page_max) >> > + results_per_page = results_per_page_max >> > + req_args['resultsPerPage'] = results_per_page >> > + >> > # Recommended by NVD >> > wait_time = 6 >> > if api_key: >> >> >> -- >> Yoann Congal >> Smile ECS >> >> -- Yoann Congal Smile ECS
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#239557): https://lists.openembedded.org/g/openembedded-core/message/239557 Mute This Topic: https://lists.openembedded.org/mt/119971554/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
