Hi Gary,
I have found part of the issue. I copied the default configuration for
cyclone DX maven plugin [1] into the statistics project. This worked.
The only difference between this config and the config in parent is:
Default:
<outputName>bom</outputName>
CP:
<outputName>${project.artifactId}-${project.version}-bom</outputName>
Unfortunately the documentation on what this value is used for is lacking
(see [2]). With the default setting the BOM is simply named bom.xml in the
target directory. Previously there had been a bom created for each module
(parent and dependencies) in the module target directory. I presume now
each of these is overwriting the previous one as they are all named
bom.xml. When a later install mojo comes along and tries to find the file
it will pick up bom.xml even if the correct file it is looking for
(identified by artifact-version was never generated).
When the bom is installed it is correctly named, e.g.
commons-statistics-distribution-1.0-SNAPSHOT-cyclonedx.xml.
However the files are different. The installed bom has dependency
information collated from other modules which are not actually
dependencies. So the aggregation is bringing in dependencies incorrectly.
This makes the BOM incorrect.
Statistics has this structure:
- commons-statistics-parent
+ commons-statistics-distribution
+ commons-statistics-regression
regression does not depend on distribution. Both child modules have
external dependencies. Unfortunately the dependencies for regression are a
subset of distribution and so it is impossible to identify the difference
between a union of them and simply all the dependencies from distribution.
The target/bom.xml for the parent and regression modules matches that
installed into the repo. The target/bom.xml and the installed distribution
module bom do not match. The installed bom adds information on the
regression module (on which it does not depend). However it does at least
create a bom and allow the install. Why the CP configuration works for the
outputName property for VFS and not Statistics I do not know.
A quick check for VFS shows that the mangling of dependencies occurs there.
For example the installed bom for commons-vfs-jackrabbit1 has a component
entry for commons-vfs2-jackrabbit2. This seems to be a mistake. I have not
delved much deeper into the generated boms. It appears that the bom for
each module is being generated with the dependencies of the entire project
reactor.
If I update the cycloneDX configuration to use:
<outputReactorProjects>false</outputReactorProjects>
Then the target/bom.xml from each module exactly matches that installed in
the maven repo. However they still include more components than the true
dependencies of the individual module. For example the statistics
distribution and regression modules include each other. All the rest of the
dependencies are identical, which is not the case for the actual modules.
So the entire set of dependencies for the reactor are still ending up in
the installed bom.
Alex
[1] https://github.com/CycloneDX/cyclonedx-maven-plugin
[2]
https://cyclonedx.github.io/cyclonedx-maven-plugin/makeAggregateBom-mojo.html#outputName
On Tue, 20 Sept 2022 at 11:39, Alex Herbert <alex.d.herb...@gmail.com>
wrote:
> Hi Gary,
>
> I tried VFS. On my mac it did not pass the unit tests:
>
> [*ERROR*] *Errors: *
>
> [*ERROR*] *
> AbstractSftpProviderTestCase$SftpProviderTestSuite>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> [*ERROR*] *
> SftpPermissionExceptionTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> [*ERROR*] *
> SftpProviderClosedExecChannelTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> [*ERROR*] *
> SftpProviderStreamProxyModeTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> I've never built this project before so I do not know if this is just a
> flaky build. FYI:
>
> *Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)*
>
> Maven home: /usr/local/apache-maven-3.6.3
>
> Java version: 11.0.12, vendor: Eclipse Foundation, runtime:
> /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
>
> Default locale: en_GB, platform encoding: UTF-8
>
> OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
>
> I tried on linux where 'mvn install' ran OK (it took ~14 minutes). Here it
> worked OK. The CycloneDX plugin creates a bom for each project module in
> every module target directory, e.g.
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-2.10.0-SNAPSHOT-bom.xml
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-project-2.10.0-SNAPSHOT-bom.xml
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-examples-2.10.0-SNAPSHOT-bom.xml
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-bom.xml
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-jackrabbit1-2.10.0-SNAPSHOT-bom.xml
>
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-distribution-2.10.0-SNAPSHOT-bom.xml
>
> When installed the local maven repository only contains:
>
> commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-cyclonedx.json
> commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-cyclonedx.xml
>
> The installed file matches
> commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-bom.xml. A quick check in the
> other modules and it is the same. The bom matching the module name matches
> the installed cylonedx file in the maven repo. So here I think the
> plugin is working correctly.
>
> I tried Commons Numbers again on linux and got the same result (an error
> installing on the first module). So this may require some work on a minimal
> multi-module project to find out what is causing the issue. Note that on
> the projects I tried (RNG, Numbers, Statistics) they all have a first
> module that does not include any dependencies. I added one with a test case
> to exercise the code using the dependency but the install error still
> occurred. All these projects have the same multi-module structure and so I
> can investigate what is different between these and VFS.
>
> Alex
>
>
> On Tue, 20 Sept 2022 at 00:52, Gary Gregory <garydgreg...@gmail.com>
> wrote:
>
>> Hi Alex,
>>
>> Thank you for the review.
>>
>> - .gitattributes: Yes let's do that for the next release. In addition,
>> there has been talk about this and recent changes around these types of
>> files on the Maven mailing list but we can and should handle these in our
>> parent POM for now.
>>
>> - CycloneDX: At the time I integrated this, I tested with Commons VFS and
>> nothing broke but it is unfortunate that the plugin does some odd things
>> in
>> a multi module project. Would report this as an issue to CycloneDX?
>>
>> In general, and in light of security issues in the software ecosystem, I
>> think that providing these metadata is important, so I am willing to go
>> through some of the growing pains but handling multi-module projects needs
>> to get fixed upstream in CycloneDX.
>>
>> Gary
>>
>>
>> On Mon, Sep 19, 2022, 17:07 Alex Herbert <alex.d.herb...@gmail.com>
>> wrote:
>>
>> > Cloned and installed locally from the git tag.
>> >
>> > I updated Commons RNG to use parent 54 and tested with:
>> >
>> > mvn clean package
>> >
>> > I had to add '.gitattributes' to a list of excluded files for the
>> > apache-rat plugin. Not a blocker but this could be moved to
>> commons-parent.
>> >
>> > The new bill of materials generated by CycloneDX is generated for all
>> > modules and appears in the target directory. But there seems to be an
>> issue
>> > with this process.
>> >
>> > I tested a release:
>> >
>> > mvn -Dcommons.release.dryRun=true -Ptest-deploy -Prelease clean verify
>> > deploy
>> >
>> > Here I get an error message from the install for the CycloneDX bom.
>> >
>> > [*ERROR*] Failed to execute goal
>> > org.apache.maven.plugins:maven-install-plugin:2.5.2:install
>> > *(default-install)* on project commons-rng-client-api: *Failed to
>> install
>> > artifact
>> > org.apache.commons:commons-rng-client-api:xml:cyclonedx:1.5-SNAPSHOT:
>> >
>> >
>> /Users/ah403/git/commons-rng/commons-rng-client-api/target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
>> > (No such file or directory)* -> *[Help 1]*
>> >
>> > The bom files are:
>> >
>> > ./target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
>> >
>> >
>> ./commons-rng-client-api/target/commons-rng-client-api-1.5-SNAPSHOT-bom.xml
>> >
>> > ./commons-rng-core/target/commons-rng-simple-1.5-SNAPSHOT-bom.xml
>> > ./commons-rng-core/target/commons-rng-core-1.5-SNAPSHOT-bom.xml
>> > ./commons-rng-core/target/commons-rng-client-api-1.5-SNAPSHOT-bom.xml
>> > ./commons-rng-core/target/commons-rng-sampling-1.5-SNAPSHOT-bom.xml
>> > ./commons-rng-core/target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
>> >
>> > ./commons-rng-simple/target/commons-rng-simple-1.5-SNAPSHOT-bom.xml
>> > ./commons-rng-simple/target/commons-rng-client-api-1.5-SNAPSHOT-bom.xml
>> > ./commons-rng-simple/target/commons-rng-sampling-1.5-SNAPSHOT-bom.xml
>> > ./commons-rng-simple/target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
>> >
>> >
>> ./commons-rng-sampling/target/commons-rng-client-api-1.5-SNAPSHOT-bom.xml
>> > ./commons-rng-sampling/target/commons-rng-sampling-1.5-SNAPSHOT-bom.xml
>> > ./commons-rng-sampling/target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
>> >
>> >
>> > For some reason the CycloneDX bom for commons-rng-parent is placed in
>> the
>> > target directory for all the child modules except
>> commons-rng-client-api.
>> > So the install fails on this module.
>> >
>> > I do not know what is different about this module. It has no
>> dependencies
>> > other than commons-rng-parent. The other modules are all dependent on
>> it.
>> > It seems to be the issue that it is the first child module.
>> >
>> > A simpler multi-module project is Commons Statistics. It only has one
>> child
>> > module. This works with CP 54 for 'mvn verify -Dspotbugs.skip
>> -Dpmd.skip'
>> > (I did not suppress 'new' bugs found by an upgraded spotbugs and a weird
>> > PMD runtime error) but fails for 'mvn install -Dspotbugs.skip
>> -Dpmd.skip'
>> > for the same error. So it seems in the first child module of the
>> > multi-module project the parent bom is not copied by cyclone DX.
>> >
>> > As a final test I tried with Commons Numbers. This again works for 'mvn
>> > verify' but not 'mvn install' with the same issue. The first child
>> module
>> > is missing the bom for the parent module. Other child modules appear to
>> > have a bom for all their dependencies.
>> >
>> > I see that you did revert the Cyclone DX version to an earlier version
>> due
>> > to issues with building the parent POM. So perhaps this is another bug
>> in
>> > CycloneDX for multi-module builds.
>> >
>> > This is not a blocker as the plugin can simply be disabled. However it
>> is
>> > not ideal as this plugin is meant to add traceability to the build and
>> > currently it does not work for multi-module projects as configured.
>> >
>> > Alex
>> >
>> >
>> > On Sun, 18 Sept 2022 at 22:39, Bruno Kinoshita <ki...@apache.org>
>> wrote:
>> >
>> > > [x] +1 Release these artifacts
>> > >
>> > > Thanks!
>> > >
>> > > On Mon, 19 Sept 2022 at 03:47, Gary Gregory <garydgreg...@gmail.com>
>> > > wrote:
>> > >
>> > > > We have fixed a few bugs and added enhancements since Apache Commons
>> > > > Parent 53 was released, so I would like to release Apache Commons
>> > > > Parent 54.
>> > > >
>> > > > Apache Commons Parent 54 RC1 is available for review here:
>> > > >
>> > https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1
>> > > > (svn revision 56878)
>> > > >
>> > > > The Git tag commons-parent-54-RC1 commit for this RC is
>> > > > efd8232f4811706ad21bc3583e32d2473256b8d8 which you can browse here:
>> > > >
>> > > >
>> > >
>> >
>> https://gitbox.apache.org/repos/asf?p=commons-parent.git;a=commit;h=efd8232f4811706ad21bc3583e32d2473256b8d8
>> > > > You may checkout this tag using:
>> > > > git clone
>> https://gitbox.apache.org/repos/asf/commons-parent.git
>> > > > --branch <
>> > https://gitbox.apache.org/repos/asf/commons-parent.git--branch
>> > > >
>> > > > commons-parent-54-RC1 commons-parent-54-RC1
>> > > >
>> > > > Maven artifacts are here:
>> > > >
>> > > >
>> > >
>> >
>> https://repository.apache.org/content/repositories/orgapachecommons-1594/org/apache/commons/commons-parent/54/
>> > > >
>> > > > These are the artifacts and their hashes:
>> > > >
>> > > > #Release SHA-512s
>> > > > #Sun Sep 18 11:32:16 EDT 2022
>> > > > Apache\ Commons\
>> > > >
>> > > >
>> > >
>> >
>> Parent-54.spdx.rdf.xml=a5ca11505acdfefabc2bff44f52566220929d3f1b4b7164c9fea0adf4fcb8c04223f5e27089698615264e89a071400a72b19ffc54516343cacbfdeffcf3a7776
>> > > >
>> > > >
>> > >
>> >
>> commons-parent-54-bom.json=ce0bf440d926a725e840459034d59cfe9f9bfc5b9131bee087ed2e80859a8064a5efb2c8abeb9997b08ad8fe693b1a8587c38721cca7ff63701e1ee1407ac17c
>> > > >
>> > > >
>> > >
>> >
>> commons-parent-54-bom.xml=2e2f29e1d26d9f5493ea83ea9707109f755fea41a16949f56438338875ee3e21c44a362d9f58c265bf43adb7a250647c463faa3275ba042eb8673686f6a29adf
>> > > >
>> > > >
>> > >
>> >
>> commons-parent-54-site.xml=735ffceca46a0574d430b4e1213a2462b9475143c0788913312b8af117eaf3b7c02a075aaf6d9b30d2560822339651cb511b838f6c9f2bced46de1fc1227c5ff
>> > > >
>> > > >
>> > >
>> >
>> commons-parent-54-src.tar.gz=7b800ea9fcb607e2e44dea906d203abdc4452872b207b4ae4229090c3e9dc471f53dea6515c487453eeb17aef833b7394ee00cb1a9edd424cfc7bb6860841e07
>> > > >
>> > > >
>> > >
>> >
>> commons-parent-54-src.zip=9b3674b54052c7b56e9f3b1fe5a8bdf6673007e2c1e9a9aff2491fefdc04554550a6725bc58fe92f3b417e1284e5a61b20004fbcf514f9df0e1ef832a56bc890
>> > > >
>> > > > I have tested this with 'mvn -V -Duser.name=$my_apache_id
>> > > > -Ddoclint=none -Prelease -Ptest-deploy clean package site deploy'
>> > > > using:
>> > > >
>> > > > Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
>> > > > Maven home: /usr/local/Cellar/maven/3.8.6/libexec
>> > > > Java version: 1.8.0_345, vendor: Homebrew, runtime:
>> > > > /usr/local/Cellar/openjdk@8
>> > > > /1.8.0+345/libexec/openjdk.jdk/Contents/Home/jre
>> > > > Default locale: en_US, platform encoding: UTF-8
>> > > > OS name: "mac os x", version: "12.6", arch: "x86_64", family: "mac"
>> > > >
>> > > > Darwin ***.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22
>> > > > 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64
>> > > >
>> > > > Details of changes since 53 are in the release notes:
>> > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1/RELEASE-NOTES.txt
>> > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1/site/changes-report.html
>> > > >
>> > > > Site:
>> > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1/site/index.html
>> > > > (note some *relative* links are broken and the 54 directories
>> are
>> > > > not yet created - these will be OK once the site is deployed.)
>> > > >
>> > > > RAT Report:
>> > > >
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1/site/rat-report.html
>> > > >
>> > > > KEYS:
>> > > > https://www.apache.org/dist/commons/KEYS
>> > > >
>> > > > Please review the release candidate and vote.
>> > > > This vote will close no sooner than 72 hours from now.
>> > > >
>> > > > [ ] +1 Release these artifacts
>> > > > [ ] +0 OK, but...
>> > > > [ ] -0 OK, but really should fix...
>> > > > [ ] -1 I oppose this release because...
>> > > >
>> > > > Thank you,
>> > > >
>> > > > Gary Gregory,
>> > > > Release Manager (using key 86fdc7e2a11262cb)
>> > > >
>> > > > For following is intended as a helper and refresher for reviewers.
>> > > >
>> > > > Validating a release candidate
>> > > > ==============================
>> > > >
>> > > > These guidelines are NOT complete.
>> > > >
>> > > > Requirements: Git, Java, Maven.
>> > > >
>> > > > You can validate a release from a release candidate (RC) tag as
>> > follows.
>> > > >
>> > > > 1) Clone and checkout the RC tag
>> > > >
>> > > > git clone https://gitbox.apache.org/repos/asf/commons-parent.git
>> > > > --branch commons-parent-54-RC1 commons-parent-54-RC1
>> > > > cd commons-parent-54-RC1
>> > > >
>> > > > 2) Check Apache licenses
>> > > >
>> > > > This step is not required if the site includes a RAT report page
>> which
>> > > > you then must check.
>> > > >
>> > > > mvn apache-rat:check
>> > > >
>> > > > 3) Build the package
>> > > >
>> > > > mvn -V clean verify
>> > > >
>> > > > You can record the Maven and Java version produced by -V in your
>> VOTE
>> > > > reply.
>> > > > To gather OS information from a command line:
>> > > > Windows: ver
>> > > > Linux: uname -a
>> > > >
>> > > > 4) Build the site for a single module project
>> > > >
>> > > > Note: Some plugins require the components to be installed instead of
>> > > > packaged.
>> > > >
>> > > > mvn site
>> > > > Check the site reports in:
>> > > > - Windows: target\site\index.html
>> > > > - Linux: target/site/index.html
>> > > >
>> > > > -the end-
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> > > > For additional commands, e-mail: dev-h...@commons.apache.org
>> > > >
>> > > >
>> > >
>> >
>>
>
