Hi Gavin, Michael, > From: Gavin Schenk <g.sch...@eckelmann.de> > Sent: Monday, September 11, 2023 15:11 > Hi, > > > On Thu, Sep 07, 2023 at 03:03:47PM +0000, Simon Falsig wrote: > >> I saw a post from 2021 to the mailing list on generating SBOMs from > ptxdist. > >> Has there been any further work on this? > > > > I've not worked on this and I'm not aware of any other efforts in this > > direction. > > > >> We've been looking at implementing this internally - plan would be to > >> generate the SBOM in CycloneDX format, and consume it with > >> Dependency-Track (https://dependencytrack.org) for automatic > >> vulnerability and license monitoring. > >> > >> Looks like we're quite close to having a working setup, but it'd make > >> a lot more sense to have it upstreamed rather than as local patches, > >> so would like to get a bit of input on the approach, and see if we > >> can make that happen :) > > we have a similar task. There is a rudimentary prototype running on our > site. It uses licencse-complicance.yaml and a CPE-Dictionary from > https://nvd.nist.gov/products/cpe to generate a bom.json. This bom.json > can than be feed into dependency tracker (We are using docker-compose for > demonstration). > > <snip bom> > { > "bomFormat": "CycloneDX", > "specVersion": "1.4", > "serialNumber": "urn:uuid:f20af5d1-0480-477d-8bea-8cbcf6d9268c", > "components": [ > { > "name": "attr", > "version": "2.4.47", > "cpe": "cpe:2.3:a:attr_project:attr:2.4.47:*:*:*:*:*:*:*", > "licenses": [ > { > "license": { > "id": "GPL-2.0-only" > } > }, > { > "license": { > "id": "LGPL-2.0-only" > } > } > ], > "hashes": [ > { > "alg": "MD5", > "content": "84f58dec00b60f2dc8fd1c9709291cc7" > } > ] > }, > ... > </snip> >
That sounds exactly like what we're working on :) I like how it also splits up the multi-licensed specifications into separate statements. How does that show up in dependency-track? (I just know that SPDX expressions with e.g. AND and OR don't really work for setting up policies.) > > > I know very little about this stuff, but I'm open to add support for > this. > > So please sent patches once they are ready. > > > >> We've identified two main steps: > >> 1. Generate the SBOM itself. A minimal version of this can be created > from the > >> output of the existing fast-bsp-report in 40 lines of Python, using > the > >> CycloneDX library. > >> I'd assume that such a script would just go into the scripts folder > in ptxdist? > > > > Yes, just put it in the scripts/ folder. > > > > Our script has 110 lines python. It patches some package names that does > not match with the db e.g. > > expat -> libexpat > gdbserver -> gdb > kernel -> linux_kernel > libmod -> kmod > ... > > I am unsure, if this is too error prone. Adding CPE vars to the recepies, > where only the VERSION is variable, should be more robust. > > >> Is there a common way of tracking / documenting dependencies of such > scripts? > > > > You mean Python packages used by the script? Not directly. > > ptxdist-cyclonedx-bom master > ❯ cat requirements.txt > pyyaml==6.0.0 > lxml==4.9.2 Yes, essentially, our current implementation needs Python3 and the cyclonedx-bom Python library. From what I understand, this could probably be set up using pipenv or something similar? > > > Is the SBOM something that should be created for an image or for the > > BSP as a whole? > > It should be per image imho. Because depending on configuration, layers, > collectionconfig, content of images differs a lot. Starting from the > image, should generate data that is 100% consistent. Not much of a > difference to licence reports, I think. I'm not 100% confident in the ptxdist terminology relating to image or bsp here, but the SBOM should ideally be created based on what ends up on a target - so I guess that's the image? My current solution just takes this from the $PACKAGES variable, but not sure how that works with more advanced setups than ours. Initial patchset coming in a bit, so happy to hear about how it can be improved. > > > For the license stuff I have plans to created documents for images. > > Because those are delivered to customers, so license compliance really > > needs to be done for each image. > > And this is the same for SBOMi, as mentioned before. > > > I'm not sure what makes sense. So you could either create a global > > option and then generate the SBOM with each image. And then select > > HOST_SYSTEM_PYTHON3_* in that option. > > Or create an SBOM "image" for the whole BSP and select the options > there. > > One thing to keep in mind is that cyclonedx is one format, but not the > only one. Today it is one of the most common used. But there are > alternatives. So there might be other users who might need the sbom in > different formats. I wonder if it is easier to adapt to other formats > using the one, or other approach? > I guess this could be handled with a configuration selecting the format to use, and just some different implementations based on that? > > >> 2. To track vulnerabilities, it's necessary to track the Common > Platform > >> Enumeration (CPE) name of each package (from > >> https://nvd.nist.gov/products/cpe). > >> This will allow matching packages to CVEs. > >> My suggestion would be to add a _CPE variable to each package (built > >> from whatever other variables make sense, typically _VERSION). > > > > Makes sense to me. From what I understand, the CPE is machine readable > > and can be split into its components if needed, right? > > I think it is. Here CPE number example for package expat: > "cpe:2.3:a:attr_project:attr:2.4.47:*:*:*:*:*:*:*" > ^- PTXCONF_EXPAT_VERSION > > >> I managed to add this > >> for the fast report (extracting it to pkg_cpe in > >> rules/post/ptxd_make_world_common.make, and adding it to the report > >> in scripts/lib/ptxd_make_world_report.sh), but I wouldn't be > >> surprised if there are other places/report that need to track this > >> also for consistency? > > > > Just the fast and full reports are enough. Anything else in that > > direction is legacy anyways and should be replaced with stuff based on > > these reports anyways. > > > >> Packages that specify _CPE would then have this included in their > >> report, and there'd be no change for the packages that don't > >> specify it. > >> > >> > >> I'd be happy to get a bit of initial feedback on the approach. I'll > >> have a look at putting up some initial patches in the coming days too. > > I like it a lot. Need reviewers, testers, 1 mio Euro? We are willing to > contribute. > > Regards > Gavin Thanks! Let's see what we can put together then - I'll send in a few simple patches in a bit, assuming that I can make this git email sending work again... - Simon