Re: [OE-core] [meta-oe][PATCH v5] cve-update-db: do_populate_cve_db depends on do_fetch
Hello, > Did you try > > $ bitbake universe --runall=fetch > > Please? It still doesn't work on latest master branch. > I sent a patch to skip cve-update-db recipe if the cve-check class is not loaded. This should fix the problem. Pierre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [meta-oe][PATCH v5] cve-update-db: do_populate_cve_db depends on do_fetch
Hi Pierre, Did you try $ bitbake universe --runall=fetch Please? It still doesn't work on latest master branch. ERROR: Running idle function Traceback (most recent call last): File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 1500, in RunQueue.execute_runqueue(): try: >return self._execute_runqueue() except bb.runqueue.TaskFailure: File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 1420, in RunQueue._execute_runqueue(): [43, 967, 4, 3, 1, 5, 3, 7, 13, 1, 2, 1, 1, 246, 35, 1, 38, 1, 35, 2, 338, 204, 142, 3, 3, 37, 244]) >if self.rqdata.prepare() == 0: self.state = runQueueComplete File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 906, in RunQueueData.prepare(): for tid in list(runall_tids): >mark_active(tid,1) File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 837, in mark_active(tid='/buildarea1/lyang1/poky/meta/recipes-core/meta/cve-update-db.bb:do_fetch', depth=1): for depend in depends: >mark_active(depend, depth+1) File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 835, in mark_active(tid='/buildarea1/lyang1/poky/meta/recipes-core/meta/cve-update-db.bb:do_populate_cve_db', depth=2): >depends = self.runtaskentries[tid].depends for depend in depends: KeyError: '/buildarea1/lyang1/poky/meta/recipes-core/meta/cve-update-db.bb:do_populate_cve_db' // Robert On 7/3/19 5:44 PM, Pierre Le Magourou wrote: Hello, This patch broke "bitbake universe --runall=fetch" totally: The line which caused the problem is: addtask do_populate_cve_db before do_fetch Would you please fix it? Or I can help if needed. This seems related to the NVD website down problem. I sent a patch to manage inaccessible URLs. cve-update-db will not fail anymore if the NVD data feeds cannot be downloaded. Pierre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [meta-oe][PATCH v5] cve-update-db: do_populate_cve_db depends on do_fetch
Hi Pierre, Did you try $ bitbake universe --runall=fetch Please? It still doesn't work on latest master branch. ERROR: Running idle function Traceback (most recent call last): File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 1500, in RunQueue.execute_runqueue(): try: >return self._execute_runqueue() except bb.runqueue.TaskFailure: File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 1420, in RunQueue._execute_runqueue(): [43, 967, 4, 3, 1, 5, 3, 7, 13, 1, 2, 1, 1, 246, 35, 1, 38, 1, 35, 2, 338, 204, 142, 3, 3, 37, 244]) >if self.rqdata.prepare() == 0: self.state = runQueueComplete File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 906, in RunQueueData.prepare(): for tid in list(runall_tids): >mark_active(tid,1) File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 837, in mark_active(tid='/buildarea1/lyang1/poky/meta/recipes-core/meta/cve-update-db.bb:do_fetch', depth=1): for depend in depends: >mark_active(depend, depth+1) File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 835, in mark_active(tid='/buildarea1/lyang1/poky/meta/recipes-core/meta/cve-update-db.bb:do_populate_cve_db', depth=2): >depends = self.runtaskentries[tid].depends for depend in depends: KeyError: '/buildarea1/lyang1/poky/meta/recipes-core/meta/cve-update-db.bb:do_populate_cve_db' // Robert On 7/3/19 5:44 PM, Pierre Le Magourou wrote: Hello, This patch broke "bitbake universe --runall=fetch" totally: The line which caused the problem is: addtask do_populate_cve_db before do_fetch Would you please fix it? Or I can help if needed. This seems related to the NVD website down problem. I sent a patch to manage inaccessible URLs. cve-update-db will not fail anymore if the NVD data feeds cannot be downloaded. Pierre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [meta-oe][PATCH v5] cve-update-db: do_populate_cve_db depends on do_fetch
Hello, > This patch broke "bitbake universe --runall=fetch" totally: > > The line which caused the problem is: > addtask do_populate_cve_db before do_fetch > > Would you please fix it? Or I can help if needed. > This seems related to the NVD website down problem. I sent a patch to manage inaccessible URLs. cve-update-db will not fail anymore if the NVD data feeds cannot be downloaded. Pierre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [meta-oe][PATCH v5] cve-update-db: do_populate_cve_db depends on do_fetch
Hi Pierre,
This patch broke "bitbake universe --runall=fetch" totally:
ERROR: Running idle function
Traceback (most recent call last):
File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 1500, in
RunQueue.execute_runqueue():
try:
>return self._execute_runqueue()
except bb.runqueue.TaskFailure:
File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 1420, in
RunQueue._execute_runqueue():
[43, 967, 4,
3, 1, 5, 3, 7, 13, 1, 2, 1, 1, 246, 35, 1, 38, 1, 35, 2, 338, 204, 142, 3, 3,
37, 244])
>if self.rqdata.prepare() == 0:
self.state = runQueueComplete
File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 906, in
RunQueueData.prepare():
for tid in list(runall_tids):
>mark_active(tid,1)
File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 837, in
mark_active(tid='/buildarea1/lyang1/poky/meta/recipes-core/meta/cve-update-db.bb:do_fetch',
depth=1):
for depend in depends:
>mark_active(depend, depth+1)
File "/buildarea1/lyang1/poky/bitbake/lib/bb/runqueue.py", line 835, in
mark_active(tid='/buildarea1/lyang1/poky/meta/recipes-core/meta/cve-update-db.bb:do_populate_cve_db',
depth=2):
>depends = self.runtaskentries[tid].depends
for depend in depends:
KeyError:
'/buildarea1/lyang1/poky/meta/recipes-core/meta/cve-update-db.bb:do_populate_cve_db'
The line which caused the problem is:
addtask do_populate_cve_db before do_fetch
Would you please fix it? Or I can help if needed.
// Robert
On 6/27/19 10:59 PM, Pierre Le Magourou wrote:
From: Pierre Le Magourou
To be able to populate NVD database on a fetchall
(bitbake --run-all=fetch), set the do_populate_cve_db task to be
executed before do_fetch.
Do not get CVE_CHECK_DB_DIR, CVE_CHECK_DB_FILE and CVE_CHECK_TMP_FILE
variable because do_populate_cve_db can be called in a context where
cve-check class is not loaded.
Signed-off-by: Pierre Le Magourou
---
meta/recipes-core/meta/cve-update-db.bb | 21 +
1 file changed, 13 insertions(+), 8 deletions(-)
diff --git a/meta/recipes-core/meta/cve-update-db.bb
b/meta/recipes-core/meta/cve-update-db.bb
index 4c896dc880..3e5bae8b1d 100644
--- a/meta/recipes-core/meta/cve-update-db.bb
+++ b/meta/recipes-core/meta/cve-update-db.bb
@@ -6,7 +6,6 @@ PACKAGES = ""
inherit nopackages
-deltask do_fetch
deltask do_unpack
deltask do_patch
deltask do_configure
@@ -24,11 +23,16 @@ python do_populate_cve_db() {
BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-;
YEAR_START = 2002
-JSON_TMPFILE = d.getVar("CVE_CHECK_DB_DIR") + '/nvd.json.gz'
+
+db_dir = d.getVar("DL_DIR") + '/CVE_CHECK'
+db_file = db_dir + '/nvd-json.db'
+json_tmpfile = db_dir + '/nvd.json.gz'
proxy = d.getVar("https_proxy")
+if not os.path.isdir(db_dir):
+os.mkdir(db_dir)
+
# Connect to database
-db_file = d.getVar("CVE_CHECK_DB_FILE")
conn = sqlite3.connect(db_file)
c = conn.cursor()
@@ -55,9 +59,9 @@ python do_populate_cve_db() {
req = urllib.request.Request(json_url)
if proxy:
req.set_proxy(proxy, 'https')
-with urllib.request.urlopen(req) as r, open(JSON_TMPFILE, 'wb') as
tmpfile:
+with urllib.request.urlopen(req) as r, open(json_tmpfile, 'wb') as
tmpfile:
shutil.copyfileobj(r, tmpfile)
-with gzip.open(JSON_TMPFILE, 'rt') as jsonfile:
+with gzip.open(json_tmpfile, 'rt') as jsonfile:
update_db(c, jsonfile)
c.execute("insert or replace into META values (?, ?)",
[year, last_modified])
@@ -65,8 +69,9 @@ python do_populate_cve_db() {
conn.commit()
conn.close()
-with open(d.getVar("CVE_CHECK_TMP_FILE"), 'a'):
-os.utime(d.getVar("CVE_CHECK_TMP_FILE"), None)
+cve_check_tmp_file = d.getVar("TMPDIR") + '/cve_check'
+with open(cve_check_tmp_file, 'a'):
+os.utime(cve_check_tmp_file, None)
}
# DJB2 hash algorithm
@@ -120,7 +125,7 @@ def update_db(c, json_filename):
-addtask do_populate_cve_db before do_cve_check
+addtask do_populate_cve_db before do_fetch
do_populate_cve_db[nostamp] = "1"
EXCLUDE_FROM_WORLD = "1"
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
