epugh commented on code in PR #4690:
URL: https://github.com/apache/solr/pull/4690#discussion_r3695655957
##########
solr/packaging/build.gradle:
##########
@@ -15,20 +15,438 @@
* limitations under the License.
*/
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
+import java.security.MessageDigest
import org.apache.tools.ant.filters.ReplaceTokens
import org.apache.tools.ant.util.TeeOutputStream
+import org.cyclonedx.gradle.CyclonedxDirectTask
+import org.cyclonedx.model.Component
// This project puts together a "distribution", assembling dependencies from
// various other projects.
plugins {
id 'base'
id 'distribution'
+ // Registers the JVM attribute schema, so that the variant-aware resolution
of
+ // the CycloneDX BOM configurations below works like a Java runtime
classpath.
+ id 'jvm-ecosystem'
+}
+
+final APACHE_SNAPSHOTS_QUALIFIER =
'&repository_url=https:%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fsnapshots%2F'
Review Comment:
this block of code seems like a lot, should it go in it's own .gradle file?
There is a lot of docs about how this works in here, and I guess I don't
totally know if it's needed. It would seem like a `sbom.gradle` might be a
better home?
##########
solr/packaging/build.gradle:
##########
@@ -15,20 +15,438 @@
* limitations under the License.
*/
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
+import java.security.MessageDigest
import org.apache.tools.ant.filters.ReplaceTokens
import org.apache.tools.ant.util.TeeOutputStream
+import org.cyclonedx.gradle.CyclonedxDirectTask
+import org.cyclonedx.model.Component
// This project puts together a "distribution", assembling dependencies from
// various other projects.
plugins {
id 'base'
id 'distribution'
+ // Registers the JVM attribute schema, so that the variant-aware resolution
of
+ // the CycloneDX BOM configurations below works like a Java runtime
classpath.
+ id 'jvm-ecosystem'
+}
+
+final APACHE_SNAPSHOTS_QUALIFIER =
'&repository_url=https:%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fsnapshots%2F'
+
+// Post-processing of the SBOMs generated by the CycloneDX plugin:
+//
+// 1. Metadata:
+// The "build" lifecycle phase is recorded (this is a "Build SBOM" in
CISA's classification)
+// and this post-processing step is listed in the tools, next to the
CycloneDX plugin.
+// 2. Main component:
+// The Solr binary archive published on downloads.apache.org is identified
by a "sid" purl
+// (a draft purl type for software distributed outside package registries,
+// see https://github.com/package-url/purl-spec/issues/516) and by the Solr
CPE.
+// 3. Removed components:
+// Maven BOM/platform dependencies (purl qualifier "type=pom") and the
internal ":platform"
+// project are stripped: they are not part of the binary distribution and
the plugin has no
+// option to exclude them.
+// 4. Solr components:
+// The plugin emits invalid purls for Solr projects, built from the Gradle
project name and a
+// non-standard "project_path" qualifier (e.g.
pkg:maven/org.apache.solr/[email protected]?project_path=:solr:core).
+// They are replaced with the Maven artifactId (base.archivesName) and a
"type=jar" qualifier
+// (e.g. pkg:maven/org.apache.solr/[email protected]?type=jar).
+// The project description and the Apache-2.0 license are also added.
+// 5. Vendored JavaScript libraries:
+// The AngularJS admin UI ships third-party JavaScript files checked into
solr/webapp/web/libs.
+// Components for them are added from a curated list (version and license
read from the file
+// headers); the hash matching of step 8 proves that each file still ships
unmodified.
+// 6. JavaScript client bundle:
+// The npm packages bundled by browserify into the OpenAPI JS client
+// (server/solr-webapp/webapp/libs/solr/index.js) are nested as
subassemblies of a first-party
+// "solr-js-client" component, using the SBOM generated by cyclonedx-npm in
:solr:webapp:js-client.
+// 7. New UI bundle:
+// The Maven dependencies compiled into the wasmJs UI
(server/solr-webapp/webapp/ui) are nested
+// as subassemblies of a first-party "solr-ui" component, using the SBOM
generated in :solr:ui,
+// together with the npm package bundled by the Kotlin toolchain (see
kotlin-js-store/wasm/yarn.lock).
+// 8. Archive locations:
+// The location of each JAR and JavaScript file within the distribution is
recorded as
+// "evidence.occurrences": the directories assembled for the distribution
are scanned and
+// their files are matched to the components by SHA-256 hash.
+// 9. Hashes:
+// Only the SHA-256 hash of each component is kept: the plugin emits eight
algorithms
+// per artifact, which only adds bulk.
+def postProcessBom = { File bomFile, String edition, Map scanDirs, File
jsClientSbomFile, File uiSbomFile ->
+ def json = new JsonSlurper().parse(bomFile)
+
+ def sha256Of = { File file ->
+ def digest = MessageDigest.getInstance('SHA-256')
+ file.eachByte(8192) { buffer, length -> digest.update(buffer, 0,
length) }
+ digest.digest().encodeHex().toString()
+ }
+
+ // Copies a child SBOM's dependency graph into this one: the child root is
replaced
+ // by the given bundle ref, dropped refs are skipped and entries for refs
that
+ // already exist (the same artifact in both graphs) are merged.
+ def mergeChildGraph = { List childDeps, String childRootRef, String
bundleRef, Set droppedRefs ->
+ childDeps.each { dep ->
+ if (dep.ref in droppedRefs) {
+ return
+ }
+ def ref = dep.ref == childRootRef ? bundleRef : dep.ref
+ def dependsOn = (dep.dependsOn ?: []).findAll { !(it in
droppedRefs) }
+ .collect { it == childRootRef ? bundleRef : it }
+ def existing = json.dependencies.find { it.ref == ref }
+ if (existing == null) {
+ json.dependencies << [ref: ref, dependsOn: dependsOn]
+ } else {
+ existing.dependsOn = ((existing.dependsOn ?: []) +
dependsOn).unique()
+ }
+ }
+ }
+
+ // 1. Metadata
+ json.metadata.lifecycles = [[phase: 'build']]
+
+ // Record this post-processing step next to the CycloneDX plugin
+ if (json.metadata.tools == null) {
+ json.metadata.tools = [components: []]
+ }
+ json.metadata.tools.components << [
+ type: 'application',
+ author: 'The Apache Software Foundation',
+ name: 'solr-sbom-post-processing',
+ version: project.version,
+ description: 'Post-processing of the generated SBOM by the Solr Gradle
build (:solr:packaging)',
+ ]
+
+ // 2. Main component
+ // Old bom-ref -> new purl, applied to the dependency graph below
+ def rewrittenRefs = [:]
+
+ Map<String, Object> mainComponent = json.metadata.component
+ def mainPurl =
"pkg:sid/apache.org/solr/solr@${mainComponent.version}?edition=${edition}".toString()
+ rewrittenRefs[mainComponent.'bom-ref'] = mainPurl
+ mainComponent.remove('group')
+ mainComponent.name = 'Apache Solr binary release'
+ mainComponent.cpe =
"cpe:2.3:a:apache:solr:${mainComponent.version}:*:*:*:*:*:*:*".toString()
+ mainComponent.purl = mainPurl
+ mainComponent.'bom-ref' = mainPurl
+
+ // Gradle project path encoded in the purls the plugin generates for Solr
projects
+ def projectPathOf = { purl ->
+ def matcher = purl =~ /[?&]project_path=([^&]+)(&|$)/
+ matcher ? URLDecoder.decode(matcher.group(1), 'UTF-8') : null
+ }
+
+ // 3. Removed components
+ def removedRefs = json.components.findAll {
+ it.purl =~ /[?&]type=pom(&|$)/ || projectPathOf(it.purl) == ':platform'
+ }.collect { it.'bom-ref' } as Set
+
+ // 4. Solr components
+ Map<String, String> artifactIdByPath =
rootProject.allprojects.collectEntries {
+ [(it.path): it.base.archivesName.get()]
+ }
+ json.components = json.components.collect { Map<String, Object> component
->
+ def projectPath = projectPathOf(component.purl)
+ // Skip external components and those about to be removed
+ if (projectPath == null || component.'bom-ref' in removedRefs) {
+ return component
+ }
+ def artifactId = artifactIdByPath[projectPath]
+ def repositoryUrlQualifier = component.version.endsWith("-SNAPSHOT") ?
APACHE_SNAPSHOTS_QUALIFIER : ''
+ def purl =
"pkg:maven/${component.group}/${artifactId}@${component.version}?type=jar${repositoryUrlQualifier}".toString()
+ rewrittenRefs[component.'bom-ref'] = purl
+ component.name = artifactId
+ component.purl = purl
+ component.'bom-ref' = purl
+ component.description = rootProject.project(projectPath).description
+ component.licenses = [[license: [id: 'Apache-2.0', url:
'https://www.apache.org/licenses/LICENSE-2.0']]]
+ // Restore the field order the plugin uses for external components
+ def ordered = [:]
+ ['type', 'bom-ref', 'group', 'name', 'version', 'description',
'hashes',
+ 'licenses', 'purl', 'modified', 'properties'].each { key ->
+ if (component.containsKey(key)) {
+ ordered[key] = component[key]
+ }
+ }
+ component.forEach { key, value ->
+ if (!ordered.containsKey(key)) {
+ ordered[key] = value
+ }
+ }
+ return ordered
+ }
+
+ // Apply the removals (3) and the ref rewrites (2, 4) to the dependency
graph
+ json.components.removeAll { it.'bom-ref' in removedRefs }
+ json.dependencies?.removeAll { it.ref in removedRefs }
+ json.dependencies?.each { dep ->
+ dep.ref = rewrittenRefs.getOrDefault(dep.ref, dep.ref)
+ dep.dependsOn?.removeAll { it in removedRefs }
+ if (dep.dependsOn != null) {
+ dep.dependsOn = dep.dependsOn.collect {
rewrittenRefs.getOrDefault(it, it) }
+ }
+ }
+
+ // The UI artifacts of steps 5 to 7 all ship inside the webapp
+ def webappRef = json.components.find {
+ it.purl?.startsWith('pkg:maven/org.apache.solr/solr-webapp@')
+ }?.'bom-ref' ?: mainComponent.'bom-ref'
+ def webappDependsOn = json.dependencies.find { it.ref == webappRef
}.dependsOn
+
+ // 5. Vendored JavaScript libraries
+ // Entries without a version marker in the file get no version and no purl
+ def vendoredJsLibs = [
+ [file: 'angular.min.js', name: 'angular', version: '1.8.0', license:
'MIT'],
Review Comment:
i don't love the hard coding here... CAn we consult other files like the
licenses etc? Yes, most of these won't ever be updated at this point.. but
what happens if they are bumped?
##########
gradle/libs.versions.toml:
##########
@@ -79,6 +79,9 @@ commons-io = "2.22.0"
compose = "1.11.1"
cuvs-java = "26.06.0"
cuvs-lucene = "25.12.0"
+cyclonedx = "3.0.2"
+# @keep npm tool generating the SBOM of the OpenAPI JS client, installed by
:solr:webapp:js-client
Review Comment:
what does `@keep` mean? Do we need this comment? the -npm suffix seems
clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
