> I was questionning why CVE_DB_RESULTS_PER_PAGE*_MAX* (notice the _MAX) > should be. Seems to me that this is a limit imposed by NVD itself and I > don't see a reason to set it to a different value than 2000 (as given by > NVD).
Ah, sorry for misreading your comment earlier. I went with a configurable max for flexibility purposes as well. Consider a scenario where someone would want to limit the max as well for whatever reason. Even in this scenario if the NVD limits change all we'd need to do is update a single variable rather than having to change multiple locations, we could also use a local var for that but then it won't cover point 1. Still if you think this is over engineered please let me know. On Thu, Jun 25, 2026 at 5:22 PM Yoann Congal <[email protected]> wrote: > On Thu Jun 25, 2026 at 1:54 PM CEST, Awais Belal wrote: > > Hi Yoann, > > > > Thanks for picking this up > > > >> What value are you using? Especially in the last few days... > > > > I've seen much better results with a cap of 500/1000 than the default of > > 2000. > > > >> I'm not sure CVE_DB_RESULTS_PER_PAGE_MAX should be a bitbake variable (I > >> don't see a reason for it to be configurable). Can you simplify the code > >> a little by making it a constant in the code instead? We already have a > >> handfull of hardcoded limits from NVD. > > > > Well it is a valid request parameter for starters. This allows the users > to > > control it from the configurations rather than hard coding to any limits > > someone sees fitting for their environment so more flexibility. Not > setting > > the variable defaults to the limits recommended by NVD however my > analysis > > shows that these days due to bulk updates on the NVD side the fetch has > > failed most of the time due to an "incomplete read" error. > > I agree that a configurable CVE_DB_RESULTS_PER_PAGE looks like a good idea. > > I was questionning why CVE_DB_RESULTS_PER_PAGE*_MAX* (notice the _MAX) > should be. Seems to me that this is a limit imposed by NVD itself and I > don't see a reason to set it to a different value than 2000 (as given by > NVD). > > > > > On Thu, Jun 25, 2026 at 4:43 PM Yoann Congal <[email protected]> > wrote: > > > >> Hello, > >> > >> Thanks for looking at this! > >> > >> On Thu Jun 25, 2026 at 1:15 PM CEST, Awais Belal via > >> lists.openembedded.org wrote: > >> > From: Awais B <[email protected]> > >> > > >> > It is seen that during bulk updates on the NVD side the server > >> > struggles to keep up with the default/max of 2000 entries per > >> > page and we see a lot of incomplete read errors resulting in > >> > proper db sync failures most of the times. Lowering the per > >> > page value noticably increases the reliability of the process > >> > and hence should ideally be configurable. > >> > >> What value are you using? Especially in the last few days... > >> > >> > > >> > Signed-off-by: Awais B <[email protected]> > >> > --- > >> > meta/recipes-core/meta/cve-update-nvd2-native.bb | 14 ++++++++++++++ > >> > 1 file changed, 14 insertions(+) > >> > > >> > diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb > >> b/meta/recipes-core/meta/cve-update-nvd2-native.bb > >> > index 945bd1d927..5d8b76e62c 100644 > >> > --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb > >> > +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb > >> > @@ -34,6 +34,11 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000" > >> > # Number of attempts for each http query to nvd server before giving > up > >> > CVE_DB_UPDATE_ATTEMPTS ?= "5" > >> > > >> > +# Maximum number of CVE records per API response. Default/max is > 2000. > >> > +# Lowering this value can help avoid incomplete read errors during > bulk > >> NVD updates. > >> > +CVE_DB_RESULTS_PER_PAGE ?= "" > >> > +CVE_DB_RESULTS_PER_PAGE_MAX ?= "2000" > >> > + > >> > CVE_CHECK_DB_DLDIR_FILE ?= > >> "${DL_DIR}/CVE_CHECK/${CVE_CHECK_DB_FILENAME}" > >> > CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock" > >> > CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp" > >> > @@ -217,6 +222,15 @@ def update_db_file(db_tmp_file, d, > database_time): > >> > api_key = d.getVar("NVDCVE_API_KEY") or None > >> > attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS")) > >> > > >> > + results_per_page = d.getVar("CVE_DB_RESULTS_PER_PAGE") > >> > + results_per_page_max = > >> int(d.getVar("CVE_DB_RESULTS_PER_PAGE_MAX")) > >> > >> I'm not sure CVE_DB_RESULTS_PER_PAGE_MAX should be a bitbake variable (I > >> don't see a reason for it to be configurable). Can you simplify the code > >> a little by making it a constant in the code instead? We already have a > >> handfull of hardcoded limits from NVD. > >> > >> > + if results_per_page: > >> > + results_per_page = int(results_per_page) > >> > + if results_per_page > results_per_page_max: > >> > + bb.warn("CVE_DB_RESULTS_PER_PAGE exceeds maximum of > %d, > >> capping" % results_per_page_max) > >> > + results_per_page = results_per_page_max > >> > + req_args['resultsPerPage'] = results_per_page > >> > + > >> > # Recommended by NVD > >> > wait_time = 6 > >> > if api_key: > >> > >> > >> -- > >> Yoann Congal > >> Smile ECS > >> > >> > > > -- > Yoann Congal > Smile ECS > > -- Regards, Awais Belal
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#239558): https://lists.openembedded.org/g/openembedded-core/message/239558 Mute This Topic: https://lists.openembedded.org/mt/119971554/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
