Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
Hi Pierre, I found that the hash function is causing collisions in the generated database such that some CVEs are being overwritten because of the UNIQUE constraint on the HASH column. For example, CVE-2018-1000873 has the same hash of 623198722 as CVE-2018-18338. This results in one of the two CVEs not appearing in the database. -- Kevin Weng -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
Hi Kevin, > I found that the hash function is causing collisions in the generated > database such that some CVEs are being overwritten because of the UNIQUE > constraint on the HASH column. For example, CVE-2018-1000873 has the same > hash of 623198722 as CVE-2018-18338. This results in one of the two CVEs not > appearing in the database. This is problematic. I kept using djb2 hash function, because it was the one used in the previous cve-check-tool and it was fast. But it might not be the right hash function to use. Do you have a better hash function in mind ? I can also drop hash function, remove everything from the database and recreate all entries at each update but it will increase database update time. I don't have the same hash as you for CVE-2018-1000873 and CVE-2018-18338, do you use my latest patches from master ? I did several changes recently. Pierre Le Magourou -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
Hi, > It looks like CVE_CHECK_DB_DIR has no default value which resulted in: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/988/steps/7/logs/step1b > > We only started seeing that error after your later patch to add back > the do_fetch task. build-appliance is trying to collect up all the > sources it may need. > I see the problem, it happens when cve-update-db do_fetch task is executed and cve-check class is not inherited. I sent a v2 patch that sets default value to CVE_CHECK_DB_DIR. Pierre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
On Wed, 2019-06-19 at 15:59 +0200, Pierre Le Magourou wrote: > From: Pierre Le Magourou > > cve-check-tool-native do_populate_cve_db task was using deprecated > NVD > xml data feeds, cve-update-db uses NVD json data feeds. > > Sqlite database schema was updated to take into account CVSSv3 CVE > scores and operator in affected product versions. > A new META table was added to store the last modification date of the > NVD json data feeds. > > Signed-off-by: Pierre Le Magourou < > pierre.lemagou...@softbankrobotics.com> > --- > meta/recipes-core/meta/cve-update-db.bb | 121 > > 1 file changed, 121 insertions(+) > create mode 100644 meta/recipes-core/meta/cve-update-db.bb > > diff --git a/meta/recipes-core/meta/cve-update-db.bb b/meta/recipes- > core/meta/cve-update-db.bb > new file mode 100644 > index 00..522fd23807 > --- /dev/null > +++ b/meta/recipes-core/meta/cve-update-db.bb > @@ -0,0 +1,121 @@ > +SUMMARY = "Updates the NVD CVE database" > +LICENSE = "MIT" > + > +INHIBIT_DEFAULT_DEPS = "1" > +PACKAGES = "" > + > +inherit nopackages > + > +deltask do_fetch > +deltask do_unpack > +deltask do_patch > +deltask do_configure > +deltask do_compile > +deltask do_install > +deltask do_populate_sysroot > + > +python do_populate_cve_db() { > +""" > +Update NVD database with json data feed > +""" > + > +import sqlite3, urllib3, shutil, gzip, re > +from datetime import date > + > +BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-; > +YEAR_START = 2002 > +JSON_TMPFILE = d.getVar("CVE_CHECK_DB_DIR") + '/nvd.json.gz' It looks like CVE_CHECK_DB_DIR has no default value which resulted in: https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/988/steps/7/logs/step1b We only started seeing that error after your later patch to add back the do_fetch task. build-appliance is trying to collect up all the sources it may need. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
On Tue, 25 Jun 2019 at 09:49, Pierre Le Magourou wrote: > > Also, the CVE db is updated using this custom task without link to > > do_fetch, which means a fetchall task would not update the database for > > off line NO_NETWORK builds. > > > > Could the task be added as dependency to do_fetch() or are there some other > > side effects? > > > > Yes I can do that, I don't think this will cause side effects. Oh, you'll also want to set the proxies appropriately. Ross -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
Hi, > Also, the CVE db is updated using this custom task without link to > do_fetch, which means a fetchall task would not update the database for > off line NO_NETWORK builds. > > Could the task be added as dependency to do_fetch() or are there some other > side effects? > Yes I can do that, I don't think this will cause side effects. Pierre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
Thanks! :) Ross On Mon, 24 Jun 2019 at 09:33, Pierre Le Magourou wrote: > > Hi, > > > > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment > > > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe > > > documentation could be updated too, e.g. > > > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages > > > > Somehow I didn't notice it, thanks. > > > > Pierre, can you rewrite this to use standard library instead of urllib3? > > Yes of course, > I only need urrlib to fetch the NVD json feeds, so I don't need > urllib3, I can use standard library. > > Pierre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
Hi, > > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment > > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe > > documentation could be updated too, e.g. > > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages > > Somehow I didn't notice it, thanks. > > Pierre, can you rewrite this to use standard library instead of urllib3? Yes of course, I only need urrlib to fetch the NVD json feeds, so I don't need urllib3, I can use standard library. Pierre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
On Fri, Jun 21, 2019 at 01:29:18PM +0100, Burton, Ross wrote: > On Fri, 21 Jun 2019 at 12:11, wrote: > > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment > > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe > > documentation could be updated too, e.g. > > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages > > Somehow I didn't notice it, thanks. > > Pierre, can you rewrite this to use standard library instead of urllib3? Also, the CVE db is updated using this custom task without link to do_fetch, which means a fetchall task would not update the database for off line NO_NETWORK builds. Could the task be added as dependency to do_fetch() or are there some other side effects? -Mikko -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
On Fri, 21 Jun 2019 at 12:11, wrote: > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe > documentation could be updated too, e.g. > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages Somehow I didn't notice it, thanks. Pierre, can you rewrite this to use standard library instead of urllib3? Ross -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
On Fri, Jun 21, 2019 at 02:03:36PM +0200, Alexander Kanavin wrote: > On Fri, 21 Jun 2019 at 13:48, wrote: > > > > > Hmm, possibly? I cherry-picked the patches to sumo and saw this missing > > dependency in my container. > > > > Did poky master switch from using host python to native after sumo? > > > > poky uses host python for some things and native python for other things. > Generally where 'core python' is enough then we use the host python, but > when additional libraries are needed, then native python plus those > libraries is a better choice, as this doesn't require adding to the > required host packages list, and allows better control over how they're > built. Currently do_populate_cve_db is executed using host python, and the DEPENDS field is empty, and INHIBIT_DEFAULT_DEPS = "1". python-urllib3 is also not available in poky, but only in meta-openembedded. I'm fine with additional python3-urllib3 dependency to bitbake build host tools when running CVE check builds. Just wanted to document it. -Mikko -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
On Fri, 21 Jun 2019 at 13:48, wrote: > > Hmm, possibly? I cherry-picked the patches to sumo and saw this missing > dependency in my container. > > Did poky master switch from using host python to native after sumo? > poky uses host python for some things and native python for other things. Generally where 'core python' is enough then we use the host python, but when additional libraries are needed, then native python plus those libraries is a better choice, as this doesn't require adding to the required host packages list, and allows better control over how they're built. Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
On Fri, Jun 21, 2019 at 01:42:11PM +0200, Alexander Kanavin wrote: > On Fri, 21 Jun 2019 at 13:11, wrote: > > > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment > > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe > > documentation could be updated too, e.g. > > > > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages > > > > On my Debian 9 build container python3-urllib3 wasn't installed by default > > or via other dependencies. > > > > Should this rather be provided via python3-native? Hmm, possibly? I cherry-picked the patches to sumo and saw this missing dependency in my container. Did poky master switch from using host python to native after sumo? -Mikko -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
On Fri, 21 Jun 2019 at 13:11, wrote: > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe > documentation could be updated too, e.g. > > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages > > On my Debian 9 build container python3-urllib3 wasn't installed by default > or via other dependencies. > Should this rather be provided via python3-native? Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
Hi, This adds python3 urllib3 (python3-urllib3 in Debian) to build environment dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe documentation could be updated too, e.g. https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages On my Debian 9 build container python3-urllib3 wasn't installed by default or via other dependencies. Thanks for these features! -Mikko -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
> Not sure which of the changes is responsible, but this is new: > WARNING: flex-native-2.6.0-r0 do_cve_check: Found unpatched CVE > (CVE-2015-1773) > > https://nvd.nist.gov/vuln/detail/CVE-2015-1773 > > Note that the flex tool is completely unrelated to Apache Flex. > > I see, the 4/4 patch is responsible for that (Consider CVE that affects versions with less than operator). It takes into account the comparison operator in the json NVD file (new 'version_affected' field that was not in the XML data feed). So this CVE matches because 2.6.0 <= 4.14.0. But it should not match because it concerns another product (flex_project/flex vs Apache/flex). There is indeed a problem I didn't manage. The CVE_PRODUCT variable we use in cve-check only takes the product name (here 'flex') into account, we should also consider the vendor name (here 'flex_project'). Without this patch (4/4), the behaviour should be the same as before. Pierre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
Not sure which of the changes is responsible, but this is new: WARNING: flex-native-2.6.0-r0 do_cve_check: Found unpatched CVE (CVE-2015-1773) https://nvd.nist.gov/vuln/detail/CVE-2015-1773 Note that the flex tool is completely unrelated to Apache Flex. cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database
From: Pierre Le Magourou cve-check-tool-native do_populate_cve_db task was using deprecated NVD xml data feeds, cve-update-db uses NVD json data feeds. Sqlite database schema was updated to take into account CVSSv3 CVE scores and operator in affected product versions. A new META table was added to store the last modification date of the NVD json data feeds. Signed-off-by: Pierre Le Magourou --- meta/recipes-core/meta/cve-update-db.bb | 121 1 file changed, 121 insertions(+) create mode 100644 meta/recipes-core/meta/cve-update-db.bb diff --git a/meta/recipes-core/meta/cve-update-db.bb b/meta/recipes-core/meta/cve-update-db.bb new file mode 100644 index 00..522fd23807 --- /dev/null +++ b/meta/recipes-core/meta/cve-update-db.bb @@ -0,0 +1,121 @@ +SUMMARY = "Updates the NVD CVE database" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" +PACKAGES = "" + +inherit nopackages + +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install +deltask do_populate_sysroot + +python do_populate_cve_db() { +""" +Update NVD database with json data feed +""" + +import sqlite3, urllib3, shutil, gzip, re +from datetime import date + +BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-; +YEAR_START = 2002 +JSON_TMPFILE = d.getVar("CVE_CHECK_DB_DIR") + '/nvd.json.gz' + +# Connect to database +db_file = d.getVar("CVE_CHECK_DB_FILE") +conn = sqlite3.connect(db_file) +c = conn.cursor() + +initialize_db(c) + +http = urllib3.PoolManager() + +for year in range(YEAR_START, date.today().year + 1): +year_url = BASE_URL + str(year) +meta_url = year_url + ".meta" +json_url = year_url + ".json.gz" + +# Retrieve meta last modified date +with http.request('GET', meta_url, preload_content=False) as r: +date_line = str(r.data.splitlines()[0]) +last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1) + +# Compare with current db last modified date +c.execute("select DATE from META where YEAR = '%d'" % year) +meta = c.fetchone() +if not meta or meta[0] != last_modified: +# Update db with current year json file +with http.request('GET', json_url, preload_content=False) as r, open(JSON_TMPFILE, 'wb') as tmpfile: +shutil.copyfileobj(r, tmpfile) +with gzip.open(JSON_TMPFILE, 'rt') as jsonfile: +update_db(c, jsonfile) +c.execute("insert or replace into META values (?, ?)", +[year, last_modified]) + +conn.commit() +conn.close() + +with open(d.getVar("CVE_CHECK_TMP_FILE"), 'a'): +os.utime(d.getVar("CVE_CHECK_TMP_FILE"), None) +} + +# DJB2 hash algorithm +def hash_djb2(s): +hash = 5381 +for x in s: +hash = (( hash << 5) + hash) + ord(x) + +return hash & 0x + +def initialize_db(c): +c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") +c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ +SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") +c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (HASH INTEGER UNIQUE, ID TEXT, \ +VENDOR TEXT, PRODUCT TEXT, VERSION TEXT, OPERATOR TEXT)") +c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_IDX ON PRODUCTS \ +(PRODUCT, VERSION)") + +def update_db(c, json_filename): +import json +root = json.load(json_filename) + +for elt in root['CVE_Items']: +if not elt['impact']: +continue + +cveId = elt['cve']['CVE_data_meta']['ID'] +cveDesc = elt['cve']['description']['description_data'][0]['value'] +date = elt['lastModifiedDate'] +accessVector = elt['impact']['baseMetricV2']['cvssV2']['accessVector'] +cvssv2 = elt['impact']['baseMetricV2']['cvssV2']['baseScore'] + +try: +cvssv3 = elt['impact']['baseMetricV3']['cvssV3']['baseScore'] +except: +cvssv3 = 0.0 + +c.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)", +[cveId, cveDesc, cvssv2, cvssv3, date, accessVector]) + +for vendor in elt['cve']['affects']['vendor']['vendor_data']: +for product in vendor['product']['product_data']: +for version in product['version']['version_data']: +product_str = cveId+vendor['vendor_name']+product['product_name']+version['version_value'] +hashstr = hash_djb2(product_str) +c.execute("insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?)", +[ hashstr, cveId, vendor['vendor_name'], +product['product_name'], version['version_value'], +version['version_affected']]) + + + +addtask