Re: Structuring bigger Maven projects

[Ben Tatham]

I am happy to see this discussion, and so many responses.  We have
struggled with it for years, and never found a great solution.  Sounds like
we do what many others are doing:

- We have many shared internal libraries amount multiple products.
- We divide each logical set of libraries into their own repos, often
multi-module builds themselves.
- Each product is in its own repo as well, also often a multi-module build
depending on the complexity of the product.

We aren't a huge team, despite the number 4-6 products we have, but
basically any team can make a change to a library as needed.  We do have
Pull Requests that are visible  to the whole dev team, and we try to
maintain some consistency at the library level, but that is a separate
discussion.

We then have a policy that whenever a PR is merged into a "library", we
"release" that "library".  We just jgitflow
 for releasing
because, well obviously, we use git flow.

It is up to the product teams to update their versions for internal
libraries.  As others have mentioned, we use versions plugin for that.  We
always keep our versions in properties so we can use mvn
versions:update-properties.  We configured it to only update our "internal"
repositories by default so that we can quickly do that without worry of
bring in some 3rd party/OSS update without vetting it first.This is
done with shared corporate parent pom configuration

  com.yourcompany*:*

jetty.version,servlet-api.version,${moreExcludeProperties>

Note the extra ${moreExcludeProperties> at the end, which allows child poms
to add to the list through another property in their own pom as needed.

We also automatically build our products every night with the latest of all
the internal libraries.  While a bit delayed, at least we get a
notification if another teams' update of a library will break a separate
product - either compile or test.

In the old days of subversion, we used to actually automatically update all
the poms in our repo whenever we did a release - without a CI server.  That
was automated through a plugin that scanned all the pom's in the svn repo
for use of the released library and auto-updated and commited the new
version.  If not familiar with svn, you can picture each top-level
directory of svn being converted to a git repo (in a svn "repo" meant
something different than git, at least practically speaking).  This was
nice because the author of the library change was the "author" of all the
pom updates, and thus easy to "blame"/"praise" for a break (or fix) and
because it was an instantaneous update, so CI would run immediately on all
the dependent projects.  That was possible because with SVN you could do
commits without having to actually clone a local copy.  With git spread
across multiple repos it would require all our devs to have all git repos
checked out in a consistent fashion to make this scalable -- so not
something we've preserved when we switch to git years ago.  Alternatively,
we would have to auto-clone all the known repos in our system -- again, not
really scalable.

Food for thought -- it's great to hear so many other ways of dealing with
this problem.

-Ben





On Thu, Dec 1, 2016 at 8:57 AM Christian Schulte  wrote:

> Am 12/01/16 um 10:02 schrieb João Cabrita:
>
> [...]
>
> > Each of the applications live in a separate repository
>
> [...]
>
> > The libraries also have a repository each
>
> [...]
>
> That's due to the SCM in use. Yes. This is something to keep in mind as
> well. What can be one (big) repository using subversion, may need to be
> split into multiple (tons) of repositories when switching to e.g. git.
> Basing something on a concept introduced by the SCM in use will make it
> hard to switch to another technology later. Subversion on the server and
> git-svn locally is working great here. Commits to the subversion server
> are really clean because developers could re-arrange theire commits
> (rebase -i, squash, etc.) locally before. Depending on the subversion
> revision for anything you'll not change to another SCM technology ever
> soon (why would you?).
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>
> --
-- 
Ben Tatham
Software Architect

*Nano**metrics* *Inc.*

Ottawa * I*  Calgary  *I*  Houston  *I*  Beijing

T: +1 613 505 5065  *I*  bentat...@nanometrics.ca
 www.nanometrics.ca  *I  *www.microseismicmonitoring.com

This message is intended exclusively for the individual or entity to which
it is addressed. This communication may contain information that is
proprietary, privileged, confidential or otherwise legally exempt from
disclosure. If you are not the named addressee, or have been inadvertently
and erroneously referenced in the address line, you are not authorized to
read, print, retain, copy or disseminate this message or any pa

Re: Structuring bigger Maven projects

[Christian Schulte]

Am 12/01/16 um 10:02 schrieb João Cabrita:

[...]

> Each of the applications live in a separate repository 

[...]

> The libraries also have a repository each

[...]

That's due to the SCM in use. Yes. This is something to keep in mind as
well. What can be one (big) repository using subversion, may need to be
split into multiple (tons) of repositories when switching to e.g. git.
Basing something on a concept introduced by the SCM in use will make it
hard to switch to another technology later. Subversion on the server and
git-svn locally is working great here. Commits to the subversion server
are really clean because developers could re-arrange theire commits
(rebase -i, squash, etc.) locally before. Depending on the subversion
revision for anything you'll not change to another SCM technology ever
soon (why would you?).


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Structuring bigger Maven projects

[João Cabrita]

Florian,

the use case you're describing is similar to what I'm using at my current
job: we have 3 applications and a bunch of internal libraries that are used
by those applications and, in some cases, applications developed by other
teams in the company.

Each of the applications live in a separate repository but are mult-module
maven projects to make advancing versions in all modules easier (we use the
maven release plugin for this, which isn't all that bad).
The libraries also have a repository each and use the same release process
as applications but, of course, each has its own version.

The way we've found to ease the burden of updating is having update jobs
that run the update goals in the versions maven plugin: those jobs update
the dependency versions, run a SNAPSHOT build to verify that it still
passes and, if it does, pushes the result into an updates branch that is
then reviewed and manually merged.
We use this not only for internal dependencies but external as well and
found it to work well.
The only downside so far is that you have to maintain an exclusions list
because several projects make "fake" releases like RCs and others use
"strange" versioning schemes like "odd minor versions are always unstable",
but this can be managed in a parent POM.

Hope this helps.

João Cabrita

On 1 December 2016 at 04:53, Christian Schulte  wrote:

> Am 11/30/16 um 19:18 schrieb Florian Schätz:
> >> A library project to be
> >> shared between multiple applications each having its own release-cycle
> >> should not be part of a multi-module project used to build such an
> >> application and should be an independent project with its own
> >> release-cycle.
> >
> > While this is of course a good idea, especially when starting to create
> > new applications, the library projects will grow with the applications
> > and not independently of them. We cannot start by first investing months
> > to create the perfect will-work-for-ten-years library and only then
> > start coding the application that will, in the first months, only use
> > 10% of all these features. So, realistically, during a development
> > cycle, both the application and the library will grow... (which doesn't
> > mean that we cannot separate them)
>
> That's what I was referring to with "inheritance vs. aggregation". In a
> multi-module project you get versioning "for free" by using the
> release:prepare release:perform goals. Maintaining the build is not a
> one time task. You adjust things over time until you get things
> structured the way it fits your situation perfectly. To not be trapped
> later, you need to carefully decide the artifact coordinates in use and
> the way versions will be assigned. If a shared library has been part of
> a multi-module project together with an application, extracting it later
> should not have an impact on the rest of the build. It should carry it's
> own version from day one. It should carry it's own coordinate namespace
> from day one. Once in a multi-module project, you will soon start to
> inherit groupId and version. That can lead to having to update each and
> every application POM when extracting that library later. Starting some
> application from scatch today, keep in mind you will re-structure your
> build over time until things have been sorted out. If you know from day
> one that a specific module/project will have it's own release-cycle
> sooner or later, be prepared for that from day one.
>
> Regards,
> --
> Christian
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>
>

Re: Structuring bigger Maven projects

[Christian Schulte]

Am 11/30/16 um 19:18 schrieb Florian Schätz:
>> A library project to be
>> shared between multiple applications each having its own release-cycle
>> should not be part of a multi-module project used to build such an
>> application and should be an independent project with its own
>> release-cycle.
> 
> While this is of course a good idea, especially when starting to create 
> new applications, the library projects will grow with the applications 
> and not independently of them. We cannot start by first investing months 
> to create the perfect will-work-for-ten-years library and only then 
> start coding the application that will, in the first months, only use 
> 10% of all these features. So, realistically, during a development 
> cycle, both the application and the library will grow... (which doesn't 
> mean that we cannot separate them)

That's what I was referring to with "inheritance vs. aggregation". In a
multi-module project you get versioning "for free" by using the
release:prepare release:perform goals. Maintaining the build is not a
one time task. You adjust things over time until you get things
structured the way it fits your situation perfectly. To not be trapped
later, you need to carefully decide the artifact coordinates in use and
the way versions will be assigned. If a shared library has been part of
a multi-module project together with an application, extracting it later
should not have an impact on the rest of the build. It should carry it's
own version from day one. It should carry it's own coordinate namespace
from day one. Once in a multi-module project, you will soon start to
inherit groupId and version. That can lead to having to update each and
every application POM when extracting that library later. Starting some
application from scatch today, keep in mind you will re-structure your
build over time until things have been sorted out. If you know from day
one that a specific module/project will have it's own release-cycle
sooner or later, be prepared for that from day one.

Regards,
-- 
Christian


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Structuring bigger Maven projects

[John Patrick]

I've worked on several large 'enterprise' projects, several could have
been 4-6 projects in their own right so maybe 12 ear's, one had 120
pom files. Lucky it got refactored and split into I think 40 repos.

I would suggest;
) branching, using git flow method.

) jgit-flow plugin (https://bitbucket.org/atlassian/jgit-flow), not
maven-release-plugin

) deployment, now only using Ansible as finding easier to use than chef/puppet

) versioning, using
http://www.mojohaus.org/build-helper-maven-plugin/parse-version-mojo.html
have a release profile in the companies parent pom

) for code quality, using atlassian stash which is forcing 1
successful build and 1 approver before for code to be merged into
develop, successful build also means code, test, findbugs etc.

) cim, still using jenkins v1, not upgraded to v2 yet. have a mix of
jobs, most deployable artifacts have a pipelines checking develop and
waiting for 15 mins before kicking off a build to allow multiple
merges, then automatically building, deploying and configuring using
ansible to the development environment, running qa step(s) either
cucumber or selenium, if they all pass then automatically creating a
release candidate and closing the release branch and merging into
master

) 1 git repo for each deployable artifact
) git repo for shared code/framework, logically group so not all in 1
repo but not all with their own repo.

) each logical cucumber project was it's own git repo, so if it
changed it was easier to automatically trigger it's execution

) jenkins or what every cim you pick, use multiple slaves to run the
tasks, yes you could get everything running on the master jenkins node
but it will quickly become overload.

) this is how I do release, with the mvn -Pdo-release -N validate, it
will bump the version numbers create the release branch, automatically
finish it, tag it, so develop branch always see -SNAPSHOT's and master
never sees -SNAPSHOT's http://pastebin.com/DbAGACk7


John


On 30 November 2016 at 18:44, Karl Heinz Marbaise  wrote:
> Hi,
>
>
>
> On 30/11/16 19:18, Florian Schätz wrote:
>>>
>>> A library project to be
>>> shared between multiple applications each having its own release-cycle
>>> should not be part of a multi-module project used to build such an
>>> application and should be an independent project with its own
>>> release-cycle.
>>
>>
>> While this is of course a good idea, especially when starting to create
>> new applications, the library projects will grow with the applications
>> and not independently of them. We cannot start by first investing months
>> to create the perfect will-work-for-ten-years library and only then
>> start coding the application that will, in the first months, only use
>> 10% of all these features. So, realistically, during a development
>> cycle, both the application and the library will grow... (which doesn't
>> mean that we cannot separate them)
>>
>> Which leads me, for example, to the problem that I still want to
>> automate as much as possible. I would like, for example, to click one
>> button in my build server, perhaps enter some parameters and get a new
>> release candidate of the libraries from the current release branch,
>> update the dependency version of the application to this rc version,
>> make the application an rc (from the application's release branch),
>> install them both into the repository, tag the current status on git and
>> deploy the rc application onto the server.
>>
>> Especially for bigger projects, I want to keep the amount of manual work
>> needed as small as possible. As much building should be done
>> automatically, if possible. And I would like not having to code a
>> jenkins or maven plugin for that purpose (but of course, I would, if
>> needed, no problem there).
>
>
>
> You can simply use maven-release-plugin (in Maven itself) which you can used
> to  fully automatically create releases of your project...This will need
> only a plugin in Jenkins which handles all those stuff...correctly
> configured things like scm entries in your pom file tagging in Git will be
> done as well
>
> Sometimes you might need to improve that using the versions-maven-plugin in
> combination with some pipeline steps in Jenkins...which prevents some
> drawbacks of the maven-release-plugin
>
>
> May be you need to think about using the Maven integration in Jenkins or
> using freestyle projects or better start using pipelines ...(I often
> observed performance drawbacks in relationship with Maven integration in
> Jenkins)...
>
>
> https://wiki.jenkins-ci.org/display/JENKINS/M2+Release+Plugin
> https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin
>
>
> Kind regards
> Karl Heinz Marbaise
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>

-
To unsubsc

Re: Structuring bigger Maven projects

[Benson Margulies]

An alternative, which I never fully explored, is the CI/CD approach.
Don't use the release plugin. Use -D, or an extension, to set a
unique, non-snapshot, version for each build, and then use version
ranges for the dependencies. I built an experimental extension at my
last job to automatically set a rev property from the current git
commit, but it wasn't entirely satisfactory.


On Wed, Nov 30, 2016 at 1:44 PM, Karl Heinz Marbaise  wrote:
> Hi,
>
>
>
> On 30/11/16 19:18, Florian Schätz wrote:
>>>
>>> A library project to be
>>> shared between multiple applications each having its own release-cycle
>>> should not be part of a multi-module project used to build such an
>>> application and should be an independent project with its own
>>> release-cycle.
>>
>>
>> While this is of course a good idea, especially when starting to create
>> new applications, the library projects will grow with the applications
>> and not independently of them. We cannot start by first investing months
>> to create the perfect will-work-for-ten-years library and only then
>> start coding the application that will, in the first months, only use
>> 10% of all these features. So, realistically, during a development
>> cycle, both the application and the library will grow... (which doesn't
>> mean that we cannot separate them)
>>
>> Which leads me, for example, to the problem that I still want to
>> automate as much as possible. I would like, for example, to click one
>> button in my build server, perhaps enter some parameters and get a new
>> release candidate of the libraries from the current release branch,
>> update the dependency version of the application to this rc version,
>> make the application an rc (from the application's release branch),
>> install them both into the repository, tag the current status on git and
>> deploy the rc application onto the server.
>>
>> Especially for bigger projects, I want to keep the amount of manual work
>> needed as small as possible. As much building should be done
>> automatically, if possible. And I would like not having to code a
>> jenkins or maven plugin for that purpose (but of course, I would, if
>> needed, no problem there).
>
>
>
> You can simply use maven-release-plugin (in Maven itself) which you can used
> to  fully automatically create releases of your project...This will need
> only a plugin in Jenkins which handles all those stuff...correctly
> configured things like scm entries in your pom file tagging in Git will be
> done as well
>
> Sometimes you might need to improve that using the versions-maven-plugin in
> combination with some pipeline steps in Jenkins...which prevents some
> drawbacks of the maven-release-plugin
>
>
> May be you need to think about using the Maven integration in Jenkins or
> using freestyle projects or better start using pipelines ...(I often
> observed performance drawbacks in relationship with Maven integration in
> Jenkins)...
>
>
> https://wiki.jenkins-ci.org/display/JENKINS/M2+Release+Plugin
> https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin
>
>
> Kind regards
> Karl Heinz Marbaise
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Structuring bigger Maven projects

[Karl Heinz Marbaise]

Hi,


On 30/11/16 19:18, Florian Schätz wrote:

A library project to be
shared between multiple applications each having its own release-cycle
should not be part of a multi-module project used to build such an
application and should be an independent project with its own
release-cycle.


While this is of course a good idea, especially when starting to create
new applications, the library projects will grow with the applications
and not independently of them. We cannot start by first investing months
to create the perfect will-work-for-ten-years library and only then
start coding the application that will, in the first months, only use
10% of all these features. So, realistically, during a development
cycle, both the application and the library will grow... (which doesn't
mean that we cannot separate them)

Which leads me, for example, to the problem that I still want to
automate as much as possible. I would like, for example, to click one
button in my build server, perhaps enter some parameters and get a new
release candidate of the libraries from the current release branch,
update the dependency version of the application to this rc version,
make the application an rc (from the application's release branch),
install them both into the repository, tag the current status on git and
deploy the rc application onto the server.

Especially for bigger projects, I want to keep the amount of manual work
needed as small as possible. As much building should be done
automatically, if possible. And I would like not having to code a
jenkins or maven plugin for that purpose (but of course, I would, if
needed, no problem there).



You can simply use maven-release-plugin (in Maven itself) which you can 
used to  fully automatically create releases of your project...This will 
need only a plugin in Jenkins which handles all those stuff...correctly 
configured things like scm entries in your pom file tagging in Git will 
be done as well


Sometimes you might need to improve that using the versions-maven-plugin 
in combination with some pipeline steps in Jenkins...which prevents some 
drawbacks of the maven-release-plugin



May be you need to think about using the Maven integration in Jenkins or 
using freestyle projects or better start using pipelines ...(I often 
observed performance drawbacks in relationship with Maven integration in 
Jenkins)...



https://wiki.jenkins-ci.org/display/JENKINS/M2+Release+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin


Kind regards
Karl Heinz Marbaise



-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Structuring bigger Maven projects

[Florian Schätz]

A library project to be
shared between multiple applications each having its own release-cycle
should not be part of a multi-module project used to build such an
application and should be an independent project with its own
release-cycle.


While this is of course a good idea, especially when starting to create 
new applications, the library projects will grow with the applications 
and not independently of them. We cannot start by first investing months 
to create the perfect will-work-for-ten-years library and only then 
start coding the application that will, in the first months, only use 
10% of all these features. So, realistically, during a development 
cycle, both the application and the library will grow... (which doesn't 
mean that we cannot separate them)


Which leads me, for example, to the problem that I still want to 
automate as much as possible. I would like, for example, to click one 
button in my build server, perhaps enter some parameters and get a new 
release candidate of the libraries from the current release branch, 
update the dependency version of the application to this rc version, 
make the application an rc (from the application's release branch), 
install them both into the repository, tag the current status on git and 
deploy the rc application onto the server.


Especially for bigger projects, I want to keep the amount of manual work 
needed as small as possible. As much building should be done 
automatically, if possible. And I would like not having to code a 
jenkins or maven plugin for that purpose (but of course, I would, if 
needed, no problem there).


Regards,

Flo

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Structuring bigger Maven projects

[Christian Schulte]

Am 30.11.2016 um 15:06 schrieb Florian Schätz:
> Deployment should be automatic, not requiring many steps, but obviously 
> it gets a little bit complicated since, for example, when we want to 
> release an RC1, we would also (preferably) automatically release an RC1 
> of the shared code modules that were modified, too. If they weren't 
> modified, we would preferably not release new versions of the shared 
> code, but that point if debatable.

A rule of thumb working well for many of us is that only things to be
released together are candidates for multi-module projects. Things to be
released independently should be independent projects. That does not
mean you cannot inherit from the same parent. A library project to be
shared between multiple applications each having its own release-cycle
should not be part of a multi-module project used to build such an
application and should be an independent project with its own
release-cycle. It's important to understand how to make use of
inheritance and aggregation [1]. You can also use the archetype plugin
to generate example/template projects by executing 'mvn archetype:generate'.



Regards,
-- 
Christian


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Structuring bigger Maven projects

[Florian Schätz]

Hello Oliver (and everyone else),


can you tell us who do you define a "bigger project"?


Well, bigger project in this case would be starting with two web 
applications, with some shared code (thus best put into separate modules 
that get developed along the main applications). More applications, also 
sharing code with the other applications, are likely in the future.


All applications get deployed to tomcat servers (some applications might 
share the same server, others might have different servers - and of 
course, there are test servers and pre-release servers to consider as 
well).


We are currently a SCRUM team, so normally only one application will be 
actively developed in one development cycle ("sprint"), but after each 
cycle , we want to release a new version of the currently developed 
application and, if they were modified, also new versions of the shared 
code modules.


Deployment should be automatic, not requiring many steps, but obviously 
it gets a little bit complicated since, for example, when we want to 
release an RC1, we would also (preferably) automatically release an RC1 
of the shared code modules that were modified, too. If they weren't 
modified, we would preferably not release new versions of the shared 
code, but that point if debatable.


The whole thing is on GIT, currently in one repository, but of course, 
there are arguments for using separate repositories for the applications 
and the shared code. But my question aims more in the Maven direction 
(obviously), because there seem to be a lot of articles about the 
basics, but not many about more complex topics like handling more 
complex projects... (Or I am simply missing them).


Regards,

Flo

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Structuring bigger Maven projects

[Oliver B. Fischer]

Hi Flo,

can you tell us who do you define a "bigger project"?

Oliver


Am 30.11.16 um 08:27 schrieb Florian Schaetz:

Hello,

are there some good guides about structuring bigger maven projects 
(including shared code, easy deployment & versioning, etc.)? There's 
much about the basic stuff, default folders, etc. but somehow, I seem 
to keep missing the articles (and books) about the "big picture" for 
bigger projects...


For example, I am having slight problems deciding between one parent 
project with many modules (and how to version the modules differently 
and automatically)... I guess there are many considerations like this, 
so has anyone good literature on the topic?


Regards,

Flo

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org



--
N Oliver B. Fischer
A Schönhauser Allee 64, 10437 Berlin, Deutschland/Germany
P +49 30 44793251
M +49 178 7903538
E o.b.fisc...@swe-blog.net
S oliver.b.fischer
J oliver.b.fisc...@jabber.org
X http://xing.to/obf


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org