Re: Maven internal company component sharing best practices
I've toyed with idea of partially releasing libraries (and applications, for that matter), but scrapped it because managing the versions was painful. Reasoning being that most of my changes were in dependency management block in the bom parent, so all of dependents would need to bump their parent version, and in turn their own version anyway. At the time I was using versions plugin for preparing the new releases. It worked great, since I had a bom parent, and versions:set -DnextSnapshot would bump the dependencies in that block as well. The only catch is that you can't use properties for artifact versions (not that you should, in my opinion). Regarding having a single repository for "libraries": we have that right now. It suffers from that monorepository issue that I had outlined in my previous response, but we just release all the artifacts regardless if they changed or not. You can think of it as insurance that each of those libraries won't collide in dependencies. Atleast it does not move as fast as the regular applications repository. Not that I believe in single repository for libraries, due to several reasons: * Library repositories should be scoped per domain (ex. related to single external service, and/or framework) * It becomes easier to store unrelated code in a module Regardless of rants above, if you won't go out of your way to tear maven a new one, you will do fine whether it's monorepo or multi repo approach. On Tue, Apr 13, 2021 at 6:54 PM Andy Feldman wrote: > > Yes, the release plugin works fine on just one module. That's exactly what > we do, using the --projects (-pl) flag to only build the module(s) we want > to release, in combination with release:prepare release:perform. You could > probably achieve the same thing by running Maven from the submodule > directory but with our build pipelines we find it easier to configure the > flag than the working directory. > > Note that you wouldn't want to have any snapshot dependencies, even on the > parent. The parent should be versioned and published as its own artifact. > In our case we don't use an in-repo for any submodules we intend > to publish on their own; we have a company-wide parent in its own repo and > they inherit directly from that. > > I guess I don't see a reason why the parent couldn't be in the same repo, > as long as it's versioned and published on its own in the same way that > everything else is. You'd probably want one root-level pom that declares > all the modules, but you *wouldn't* want to use that pom as the > for each of the library poms. You'd instead want the parent pom to be just > one of the modules. Otherwise you'd have no way to publish it on its own > because building it would involve building all the declared modules too. I > hope that makes sense (and I hope it's right since we don't actually do it > that way!) > > > On Tue, Apr 13, 2021 at 11:50 AM David Hoffer wrote: > > > Following up on this. > > > > First, thanks for the quick and detailed responses. > > > > I have another colleague that suggested we create a single new GIT repo for > > the 'library' code that would be a multi-module Maven build, each module > > being a component. My first thought is that it does solve the > > 'releasability' of components and applications issue but does it allow > > granular versioning/releasing of the modules? > > > > In all my experience with multi-module builds they all shared the same > > version from the parent and all were released at the same time with the > > same version. However is that a Maven requirement? > > > > I see in the link below in the multi-module projects section that there is > > an autoVersionSubmodules flag. So if that is false how does this work on > > the sub-modules? E.g. it says it will prompt for the new version for each > > module but does it allow you to skip modules that didn't have any changes > > so no need to release? E.g. is the release granularity at the module > > layer? > > > > E.g. if I have 10 components in the multi-module Maven build, can I release > > just one and with a version completely independent of the others? > > > > Also just so I know...can the release plugin be run on just one module and > > not on the parent? > > > > > > https://maven.apache.org/maven-release/maven-release-plugin/examples/prepare-release.html > > > > Again just looking for best practices here regarding sharing components and > > how they can be versioned independently. > > > > Thanks, > > -Dave > > > > On Mon, Apr 12, 2021 at 12:36 PM Mantas Gridinas > > wrote: > > > > > Being in similar situation myself I strongly suggest against storing > > > the "library" component in the same place as your application is > > > stored, because you will be producing n+1 artifacts every time, where > > > n is the modules actually required to build for your application. Your > > > application will be moving faster than that library component and you > > > will be producing duplicate builds of it,
Re: Maven internal company component sharing best practices
Yes, the release plugin works fine on just one module. That's exactly what we do, using the --projects (-pl) flag to only build the module(s) we want to release, in combination with release:prepare release:perform. You could probably achieve the same thing by running Maven from the submodule directory but with our build pipelines we find it easier to configure the flag than the working directory. Note that you wouldn't want to have any snapshot dependencies, even on the parent. The parent should be versioned and published as its own artifact. In our case we don't use an in-repo for any submodules we intend to publish on their own; we have a company-wide parent in its own repo and they inherit directly from that. I guess I don't see a reason why the parent couldn't be in the same repo, as long as it's versioned and published on its own in the same way that everything else is. You'd probably want one root-level pom that declares all the modules, but you *wouldn't* want to use that pom as the for each of the library poms. You'd instead want the parent pom to be just one of the modules. Otherwise you'd have no way to publish it on its own because building it would involve building all the declared modules too. I hope that makes sense (and I hope it's right since we don't actually do it that way!) On Tue, Apr 13, 2021 at 11:50 AM David Hoffer wrote: > Following up on this. > > First, thanks for the quick and detailed responses. > > I have another colleague that suggested we create a single new GIT repo for > the 'library' code that would be a multi-module Maven build, each module > being a component. My first thought is that it does solve the > 'releasability' of components and applications issue but does it allow > granular versioning/releasing of the modules? > > In all my experience with multi-module builds they all shared the same > version from the parent and all were released at the same time with the > same version. However is that a Maven requirement? > > I see in the link below in the multi-module projects section that there is > an autoVersionSubmodules flag. So if that is false how does this work on > the sub-modules? E.g. it says it will prompt for the new version for each > module but does it allow you to skip modules that didn't have any changes > so no need to release? E.g. is the release granularity at the module > layer? > > E.g. if I have 10 components in the multi-module Maven build, can I release > just one and with a version completely independent of the others? > > Also just so I know...can the release plugin be run on just one module and > not on the parent? > > > https://maven.apache.org/maven-release/maven-release-plugin/examples/prepare-release.html > > Again just looking for best practices here regarding sharing components and > how they can be versioned independently. > > Thanks, > -Dave > > On Mon, Apr 12, 2021 at 12:36 PM Mantas Gridinas > wrote: > > > Being in similar situation myself I strongly suggest against storing > > the "library" component in the same place as your application is > > stored, because you will be producing n+1 artifacts every time, where > > n is the modules actually required to build for your application. Your > > application will be moving faster than that library component and you > > will be producing duplicate builds of it, which ultimately just > > consumes space in your artifact repository. The upside of this > > approach is you don't have to worry about transitive dependency > > conflicts. > > > > In our case we host sonatype, put everything there and reference them > > as needed. There are two downsides to this approach, though: > > dependency conflicts and bumping the dependency in dependent > > applications. > > > > Dependency conflicts happen because of several reasons: > > * developer decided to change its artifact/group id starting with some > > version > > * you're importing some dependency pom (like spring, camel, or what > > ever overarching framework provides it) > > * you require several frameworks at the same time > > > > Maven is perfectly capable of resolving those conflicts if artifacts > > actually collide (spring-boot-camel-starter:3.0.0 provides spring boot > > 2.2.1 while you're actually using 2.4.4, and since you have shorter > > path to spring 2.4.4 maven will use that). Problems arise when > > collisions happen between artifacts that contribute to same package > > (ex. ehcache changing its group id between ehcache 2 and ehcache 3) or > > depend on each other indirectly (ex. cxf switching between blocking > > and async calls depending if you have cxf-rt-transports-http-hc on > > your classpath). This is solved by using enforcer plugin with > > dependency convergence rule. > > > > Dependency bumping is a pretty tiring process. Whenever you would > > release a new library version, you would have to go and bump its > > version across all applications and/or libraries that depend on it. > > Since we're in the process of
Re: Maven internal company component sharing best practices
Following up on this. First, thanks for the quick and detailed responses. I have another colleague that suggested we create a single new GIT repo for the 'library' code that would be a multi-module Maven build, each module being a component. My first thought is that it does solve the 'releasability' of components and applications issue but does it allow granular versioning/releasing of the modules? In all my experience with multi-module builds they all shared the same version from the parent and all were released at the same time with the same version. However is that a Maven requirement? I see in the link below in the multi-module projects section that there is an autoVersionSubmodules flag. So if that is false how does this work on the sub-modules? E.g. it says it will prompt for the new version for each module but does it allow you to skip modules that didn't have any changes so no need to release? E.g. is the release granularity at the module layer? E.g. if I have 10 components in the multi-module Maven build, can I release just one and with a version completely independent of the others? Also just so I know...can the release plugin be run on just one module and not on the parent? https://maven.apache.org/maven-release/maven-release-plugin/examples/prepare-release.html Again just looking for best practices here regarding sharing components and how they can be versioned independently. Thanks, -Dave On Mon, Apr 12, 2021 at 12:36 PM Mantas Gridinas wrote: > Being in similar situation myself I strongly suggest against storing > the "library" component in the same place as your application is > stored, because you will be producing n+1 artifacts every time, where > n is the modules actually required to build for your application. Your > application will be moving faster than that library component and you > will be producing duplicate builds of it, which ultimately just > consumes space in your artifact repository. The upside of this > approach is you don't have to worry about transitive dependency > conflicts. > > In our case we host sonatype, put everything there and reference them > as needed. There are two downsides to this approach, though: > dependency conflicts and bumping the dependency in dependent > applications. > > Dependency conflicts happen because of several reasons: > * developer decided to change its artifact/group id starting with some > version > * you're importing some dependency pom (like spring, camel, or what > ever overarching framework provides it) > * you require several frameworks at the same time > > Maven is perfectly capable of resolving those conflicts if artifacts > actually collide (spring-boot-camel-starter:3.0.0 provides spring boot > 2.2.1 while you're actually using 2.4.4, and since you have shorter > path to spring 2.4.4 maven will use that). Problems arise when > collisions happen between artifacts that contribute to same package > (ex. ehcache changing its group id between ehcache 2 and ehcache 3) or > depend on each other indirectly (ex. cxf switching between blocking > and async calls depending if you have cxf-rt-transports-http-hc on > your classpath). This is solved by using enforcer plugin with > dependency convergence rule. > > Dependency bumping is a pretty tiring process. Whenever you would > release a new library version, you would have to go and bump its > version across all applications and/or libraries that depend on it. > Since we're in the process of splitting our monorepository, we were > looking into solutions to automate that. If you're on github, and use > github as nexus, you can look into dependabot. It's free to use and > has a lot of configuration options. Since we're not so lucky, we might > end up using renovate or something similar. The idea is to create a > pullrequest to dependents whenever a new version of that library is > released with the bumped version. Upside is you won't have to do > manual labor of bumping the version, while downside is having a ton of > noise if you have complicated dependency trees. > > These are issues i've encountered so far. You can start by hosting > your own nexus, uploading artifacts to it and referencing the > artifacts in the pom. Sonatype offers free solution > https://www.sonatype.com/products/repository-oss-download. Take care > of your dependency tree. > > On Mon, Apr 12, 2021 at 6:03 PM David Hoffer wrote: > > > > Hey just looking for Maven best practices here, want to be sure I'm not > > missing something. > > > > Here is our use case. > > > > We have 10 applications that have duplicate code in each, we want to > create > > component(s) for the duplicate code and not duplicate in the 10 apps/GIT > > any longer. > > > > Each app is in a separate GIT repository & each has a Maven build. > Simple > > single module build in each. > > > > To solve this I am assuming we need a new GIT repo for the duplicate code > > and a Maven build for it. That repo and it's component would be released
Re: Maven internal company component sharing best practices
Being in similar situation myself I strongly suggest against storing the "library" component in the same place as your application is stored, because you will be producing n+1 artifacts every time, where n is the modules actually required to build for your application. Your application will be moving faster than that library component and you will be producing duplicate builds of it, which ultimately just consumes space in your artifact repository. The upside of this approach is you don't have to worry about transitive dependency conflicts. In our case we host sonatype, put everything there and reference them as needed. There are two downsides to this approach, though: dependency conflicts and bumping the dependency in dependent applications. Dependency conflicts happen because of several reasons: * developer decided to change its artifact/group id starting with some version * you're importing some dependency pom (like spring, camel, or what ever overarching framework provides it) * you require several frameworks at the same time Maven is perfectly capable of resolving those conflicts if artifacts actually collide (spring-boot-camel-starter:3.0.0 provides spring boot 2.2.1 while you're actually using 2.4.4, and since you have shorter path to spring 2.4.4 maven will use that). Problems arise when collisions happen between artifacts that contribute to same package (ex. ehcache changing its group id between ehcache 2 and ehcache 3) or depend on each other indirectly (ex. cxf switching between blocking and async calls depending if you have cxf-rt-transports-http-hc on your classpath). This is solved by using enforcer plugin with dependency convergence rule. Dependency bumping is a pretty tiring process. Whenever you would release a new library version, you would have to go and bump its version across all applications and/or libraries that depend on it. Since we're in the process of splitting our monorepository, we were looking into solutions to automate that. If you're on github, and use github as nexus, you can look into dependabot. It's free to use and has a lot of configuration options. Since we're not so lucky, we might end up using renovate or something similar. The idea is to create a pullrequest to dependents whenever a new version of that library is released with the bumped version. Upside is you won't have to do manual labor of bumping the version, while downside is having a ton of noise if you have complicated dependency trees. These are issues i've encountered so far. You can start by hosting your own nexus, uploading artifacts to it and referencing the artifacts in the pom. Sonatype offers free solution https://www.sonatype.com/products/repository-oss-download. Take care of your dependency tree. On Mon, Apr 12, 2021 at 6:03 PM David Hoffer wrote: > > Hey just looking for Maven best practices here, want to be sure I'm not > missing something. > > Here is our use case. > > We have 10 applications that have duplicate code in each, we want to create > component(s) for the duplicate code and not duplicate in the 10 apps/GIT > any longer. > > Each app is in a separate GIT repository & each has a Maven build. Simple > single module build in each. > > To solve this I am assuming we need a new GIT repo for the duplicate code > and a Maven build for it. That repo and it's component would be released > and then the 10 apps would then depend on that released artifact (SNAPSHOT > until release). > > However I have someone here that is saying he wants to not create any other > GIT repo and just make child modules in one of the 10 apps for the shared > code then have the other 9 apps depend on the child module GAV. > > However with this approach we would have to 'release' App 1 to get a > release version and then update the other 9 apps to use that app's version > for the component, right? It seems it causes releasably issues for the > component as no other app can be released until App1 is released first, > correct? > > What is the normal pattern in the Maven world when a company wants to share > components between apps when the applications are in separate GIT repos and > therefore have separate Maven builds? > > -Dave - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Maven internal company component sharing best practices
On 4/12/2021 11:11 AM, Alexander Kriegisch wrote: IMO you are right, David. I would suggest your colleagues follow your idea in order to avoid the concerns you raised. You have such nice, independent single-module apps now. Why sacrifice that and pruposefully killing modularisation by doing what your colleague(s) suggested? I am sure they have their reasons for suggesting it. But if the main reason is just avoiding to create another Git repository, I would argue that where they see a quick and easy way of extracting a common component or low-hanging fruit, I see a Pyrrhic victory in the making. Sorry for being judgemental here with limited contextual knowledge, but to me the situation looks quite clear: You are right, they are wrong. Sometimes it is as simple as that. If you want to extract the component, do it right or don't do it. If you do it wrong, the whole team will regret it very soon and come to the team manager to ask for something like a "refactoring sprint" (if you happen to have a Scrum-like framework in place). Regards I agree with this as well. We have a bunch of lightweight (no more than 6 modules) end projects. Anytime we start to see commonality across multiple projects, we refactor to create a new dependency. We then just add that dependency to the projects that require it. If the dependency is ubiquitous, we can add it to our internal parent pom with dependency management. We have at least one senior developer who likes clumping projects together. This has occasionally made releases challenging. Different parties have different release schedules / requirements, and when you bundle things together you can have priority conflicts. However you can convince developers not to do this, I think is a good thing. We do build against snapshots, which is something I'm trying to discourage. It makes downstream projects potentially unstable. Also, sometimes a developer will forget to test a project before a build system release, and the release will fail due to a SNAPSHOT dependency. It hasn't happened in a while, but it is an annoying event. This type of structure makes management of competing releases easier (each release uses its required dependency). Yes, you have to release a dependency before releasing a downstream project if the dependency is needed. However, that won't impact other customer-facing downstream projects. It also makes deploying a minor release that was built to satisfy another project's dependency into production much less likely. This is a good thing. It does make tracking a large number of projects that use different minor releases of a dependency challenging. We're working on an automated system to track that so we can eventually archive old releases. We should probably have an EOL policy to facilitate that as well - add that to my list of things to do. . . . just my two cents /mde/ OpenPGP_signature Description: OpenPGP digital signature
Re: Maven internal company component sharing best practices
IMO you are right, David. I would suggest your colleagues follow your idea in order to avoid the concerns you raised. You have such nice, independent single-module apps now. Why sacrifice that and pruposefully killing modularisation by doing what your colleague(s) suggested? I am sure they have their reasons for suggesting it. But if the main reason is just avoiding to create another Git repository, I would argue that where they see a quick and easy way of extracting a common component or low-hanging fruit, I see a Pyrrhic victory in the making. Sorry for being judgemental here with limited contextual knowledge, but to me the situation looks quite clear: You are right, they are wrong. Sometimes it is as simple as that. If you want to extract the component, do it right or don't do it. If you do it wrong, the whole team will regret it very soon and come to the team manager to ask for something like a "refactoring sprint" (if you happen to have a Scrum-like framework in place). Regards -- Alexander Kriegisch https://scrum-master.de David Hoffer schrieb am 13.04.2021 01:02 (GMT +07:00): > Hey just looking for Maven best practices here, want to be sure I'm not > missing something. > > Here is our use case. > > We have 10 applications that have duplicate code in each, we want to create > component(s) for the duplicate code and not duplicate in the 10 apps/GIT > any longer. > > Each app is in a separate GIT repository & each has a Maven build. Simple > single module build in each. > > To solve this I am assuming we need a new GIT repo for the duplicate code > and a Maven build for it. That repo and it's component would be released > and then the 10 apps would then depend on that released artifact (SNAPSHOT > until release). > > However I have someone here that is saying he wants to not create any other > GIT repo and just make child modules in one of the 10 apps for the shared > code then have the other 9 apps depend on the child module GAV. > > However with this approach we would have to 'release' App 1 to get a > release version and then update the other 9 apps to use that app's version > for the component, right? It seems it causes releasably issues for the > component as no other app can be released until App1 is released first, > correct? > > What is the normal pattern in the Maven world when a company wants to share > components between apps when the applications are in separate GIT repos and > therefore have separate Maven builds? > > -Dave > - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Maven internal company component sharing best practices
Hey just looking for Maven best practices here, want to be sure I'm not missing something. Here is our use case. We have 10 applications that have duplicate code in each, we want to create component(s) for the duplicate code and not duplicate in the 10 apps/GIT any longer. Each app is in a separate GIT repository & each has a Maven build. Simple single module build in each. To solve this I am assuming we need a new GIT repo for the duplicate code and a Maven build for it. That repo and it's component would be released and then the 10 apps would then depend on that released artifact (SNAPSHOT until release). However I have someone here that is saying he wants to not create any other GIT repo and just make child modules in one of the 10 apps for the shared code then have the other 9 apps depend on the child module GAV. However with this approach we would have to 'release' App 1 to get a release version and then update the other 9 apps to use that app's version for the component, right? It seems it causes releasably issues for the component as no other app can be released until App1 is released first, correct? What is the normal pattern in the Maven world when a company wants to share components between apps when the applications are in separate GIT repos and therefore have separate Maven builds? -Dave
