Re: Maven Reactor and plugin "local" dependencies

[Tamás Cservenák]

Howdy,

On Fri, May 6, 2022 at 4:01 PM Thomas Broyer  wrote:

> Fwiw, my "daily routine" with That Other Build Tool is to just ask it to
> build and let it handle everything for me: never need to clean (through git
> clean or the clean task), never need to nuke my local cache because it
> really only includes remote artifacts and I never need to use SNAPSHOTs
> (again, because everything can be built in the same invocation without the
> need to split things in separate "projects" –like you suggested for the
> extension or even annotation processor–, so I never need to 'install'
> things locally, or deploy snapshots to a repository), and never need to do
> "quick-builds" either. I ask it to build, it builds what it needs to build,
> everything Just Works™.
> It also has accurate incremental builds, and even a (local or remote, I
> only use local) build cache so things don't necessarily have to be rebuilt
> when I switch between branches.
> This is what I expect from any build tool, and we even start to see such
> things coming to Node/NPM!
>

All the same thing I'd expect from products backed by commercials companies
too! And I find it appealing that some companies keep comparing their
product to ASF Maven :)

But joking aside, I did not say a word about my "daily routine" being "best
practice", or that it's something like "that's how things should be done"
with Maven, I just told you what my own practice is.

And my suggestion was more about logically separating things, for
extensions is a must (technical reasons), while for "annotation case" is as
you correctly see: dependencies from plugin configurations are "not
visible" for Maven project sorter, true. Something that probably needs
improvement. But I still claim the opposite side: you tend to
build/release your annotation always (I believe your CI does full build,
what you build on your workstation remains on your workstation), while I
tend to spend cpu cycles on a when-needed basis: build once, deploy, reuse.
And both, my workstation and CI will just "pull" the dependency, not
"figuring out" if it needs to be built (on WS) or just build it always (on
CI).


HTH
T

Re: Maven Reactor and plugin "local" dependencies

[Thomas Broyer]

On Fri, May 6, 2022 at 12:29 PM Tamás Cservenák  wrote:

> Howdy,
>
> no, not my memory, I rely on my "daily routine": I use Maven Repository
> Manager, I regularly nuke my local repo, regularly execute `git clean -fdx`
> on my checkout, and on checkout/branch change regularly do "quick-build"
> `mvn clean install` (usually `mvn clean install -Dtest=void`) as very first
> step. Have to add, I tend to work on projects that finish these "quick
> builds" in a matter of minutes at most, not hours :) This "process" works
> for me just fine.
>

Fwiw, my "daily routine" with That Other Build Tool is to just ask it to
build and let it handle everything for me: never need to clean (through git
clean or the clean task), never need to nuke my local cache because it
really only includes remote artifacts and I never need to use SNAPSHOTs
(again, because everything can be built in the same invocation without the
need to split things in separate "projects" –like you suggested for the
extension or even annotation processor–, so I never need to 'install'
things locally, or deploy snapshots to a repository), and never need to do
"quick-builds" either. I ask it to build, it builds what it needs to build,
everything Just Works™.
It also has accurate incremental builds, and even a (local or remote, I
only use local) build cache so things don't necessarily have to be rebuilt
when I switch between branches.
This is what I expect from any build tool, and we even start to see such
things coming to Node/NPM!


> But, now that you bring up topic of "branched development", something I
> always hated for very same reason you bring up, I think you will be happy
> to see upcoming changes:
>
>
> https://github.com/apache/maven-resolver/blob/master/src/site/markdown/local-repository.md
>
> (unreleased yet, we plan this for Resolver 1.8.1 and Maven 3.9.0).
>
> All in all I think we both agree, but still: Maven as a project is big
> (huge actually), and volunteers working on it have hard time to catch up,
>

Indeed it's a shame that the project is so underfunded given the many
wealthy companies relying on it!
For my part, I try to avoid Maven if I can, except possibly for very simple
builds. I still have to work with Maven in some projects, I maintain some
Maven plugin and archetypes, and have to deal with Maven as a consumer of
libraries I contribute to (lately having to make tests with artifact
relocation and BOMs)

Am not saying either "maven is the best", but I am convinced "maven can be
> better", MUCH better and we will do it. As a matter of fact, we are already
> doing it, step by step, baby steps :)
>

Yes, and Maven 4 has some good things coming!

-- 
Thomas Broyer
/tɔ.ma.bʁwa.je/ 

Re: Maven Reactor and plugin "local" dependencies

[Thomas Broyer]

On Fri, May 6, 2022 at 1:24 PM Tamás Cservenák  wrote:

> Hm, I need to look more into MCOMPILER-496, as I found a m-compiler-p IT
> that does exactly what Thomas described:
>
> https://github.com/apache/maven-compiler-plugin/tree/master/src/it/MCOMPILER-203-processorpath
>
> 3 module multi module build:
> - an annotation processor
> - a verifier plugin
> - a "user" module that uses those above two within same build
>
> and it works as expected so more feedback or reproducer needed here as
> well.
>

Yes, because annotation-processor comes before
annotation-user in the aggregator POM (it also runs the
whole build, rather than '-pl annotation-user -am', using that you should
see that you need to explicitly add annotation-processor to the project
list on the command line)

-- 
Thomas Broyer
/tɔ.ma.bʁwa.je/ 

Re: Maven Reactor and plugin "local" dependencies

[Thomas Broyer]

On Fri, May 6, 2022 at 12:06 PM Tamás Cservenák  wrote:

> Howdy,
>
> a lot to unravel here...
>
> But let me start first with the obvious: I believe you are quite
> experienced, and I'd bet you did not start with Maven3, but most probably
> Maven2 or even Maven1 :) The reason why I mention this, is that you use
> terms from the "Maven2 era", like "reactor". Maven2 had the well known
> "maven-reactor-plugin", that ceased to exist in Maven3. IMHO, let's clear
> up the terms as otherwise we will just accumulate more misunderstandings.
>

This is far back but I believe I started with Maven 3.0 in late 2010, as
I've actually never used the maven-reactor-plugin, but the term "reactor"
is still in use with Maven 4 so…
https://maven.apache.org/guides/mini/guide-multiple-modules-4.html


> For start, let's take Maven Extensions (core or build, does not matter).
>
> IMHO, having them both (the tool you build with, and the source you build)
> in one multi module project is IMHO wrong. That would be like a carpenter
> keeping his nails and other tools along with timber in the same
> box/storage, which I believe is not true. There are two kinds of extensions
> in Maven: "core extension" and "build extension". Yes, both of them need to
> be loaded ahead of the build. So yes, in case you keep your nails and
> timber in the same box, you need to "bootstrap" things somehow (like
> installing extension), as otherwise Maven will be unable to load up your
> build in the first place.
>
> But, let me ask this: how often your tooling changes compared to the
> project? In other words, are you sure your extension you use to build with,
> and the project itself, are changing at the same pace? Are you sure they
> should share the same "release lifecycle"?
>
> My typical example here (not extension, but similar intent): if we examine
> Maven build, look at maven-model module. With every (even dot) release of
> Maven, we generate, compile, package etc the maven-model module, while the
> Maven model did not change since Maven 3.0.0 (or so, did not fact check,
> but the point is there: very rarely changes). IMHO, the Maven project
> should simply "move out" the maven-model from Maven build, as it does not
> belong there: they do not share any release lifecycle at all.
>
> If we get back to extensions kept in the same build where modules are built
> as well, I wonder, how often does your extension change? Does it really
> change with every dot release of your project? If so, then something else
> may be improved in tooling (like missing some abstraction and/or
> separation). If not, it is simply in the "wrong place": why do you
> build/package/deploy it every time your project changes?
>
> But the same thing can be applied to examples like in this thread, for
> example checkstyle configuration: is this the ONLY project in your company?
> If not, is this the ONLY project having this checkstyle applied (and each
> project has a different one?). Maven is all about "sharing" :)
>
> Your next example of an annotation processor falls under the same scrutiny
> as above: does it really change with every release?


Let me take the problem from the other side: why should I introduce a
separate release cycle for something that'll never be released on its own?
If/when I use annotations and an annotation processor from the same
aggregator, they're generally only some tooling to build the final "thing"
(an application most of the time, as an executable JAR or a WAR). I don't
(and shouldn't) really care whether/when they change, they're part of a
single "project".
You're suggesting that I should manage those "helpers" as a separate
project with its own release cycle, and deploy the artifacts to a repo for
consumption by the "real" project.

What about cases where you want to test-drive a change with a "real"
project? It could (should?) be as easy a creating a new aggregator POM
referencing both the "project to be tested" and the "real project where you
want to test it" (or directly add the "project to be tested" as a new,
temporary, submodule), and change the version of the dependencies in the
"real" project; run the build, it can now resolve the dependencies from the
reactor and you can iterate as you find bugs, without having to 'mvn
install' in between every time you make a change.
I admit this is a bit "stretched" of an example, and one should actually
iterate with an integration test, and/or possibly run the "real" project,
temporarily, as an integration test by means of the maven-invoker-plugin
(which will automate the 'mvn install' dance for you). It makes it a bit
harder to debug things though.


> But that said, and
> reading your mail and in parallel reading m-compiler-p sources, I already
> see a bug in m-compiler-p and how it resolves annotation processors: if you
> look, it ends up in maven-compat (Maven2 compatibility layer) and does the
> work in "Maven2 way", while you are right, this should work just fine out
> of the box. Filed 

Re: Maven Reactor and plugin "local" dependencies

[Tamás Cservenák]

Hm, I need to look more into MCOMPILER-496, as I found a m-compiler-p IT
that does exactly what Thomas described:
https://github.com/apache/maven-compiler-plugin/tree/master/src/it/MCOMPILER-203-processorpath

3 module multi module build:
- an annotation processor
- a verifier plugin
- a "user" module that uses those above two within same build

and it works as expected so more feedback or reproducer needed here as
well.


T

On Fri, May 6, 2022 at 12:05 PM Tamás Cservenák  wrote:

> Howdy,
>
> a lot to unravel here...
>
> But let me start first with the obvious: I believe you are quite
> experienced, and I'd bet you did not start with Maven3, but most probably
> Maven2 or even Maven1 :) The reason why I mention this, is that you use
> terms from the "Maven2 era", like "reactor". Maven2 had the well known
> "maven-reactor-plugin", that ceased to exist in Maven3. IMHO, let's clear
> up the terms as otherwise we will just accumulate more misunderstandings.
>
> For start, let's take Maven Extensions (core or build, does not matter).
>
> IMHO, having them both (the tool you build with, and the source you build)
> in one multi module project is IMHO wrong. That would be like a carpenter
> keeping his nails and other tools along with timber in the same
> box/storage, which I believe is not true. There are two kinds of extensions
> in Maven: "core extension" and "build extension". Yes, both of them need to
> be loaded ahead of the build. So yes, in case you keep your nails and
> timber in the same box, you need to "bootstrap" things somehow (like
> installing extension), as otherwise Maven will be unable to load up your
> build in the first place.
>
> But, let me ask this: how often your tooling changes compared to the
> project? In other words, are you sure your extension you use to build with,
> and the project itself, are changing at the same pace? Are you sure they
> should share the same "release lifecycle"?
>
> My typical example here (not extension, but similar intent): if we examine
> Maven build, look at maven-model module. With every (even dot) release of
> Maven, we generate, compile, package etc the maven-model module, while the
> Maven model did not change since Maven 3.0.0 (or so, did not fact check,
> but the point is there: very rarely changes). IMHO, the Maven project
> should simply "move out" the maven-model from Maven build, as it does not
> belong there: they do not share any release lifecycle at all.
>
> If we get back to extensions kept in the same build where modules are
> built as well, I wonder, how often does your extension change? Does it
> really change with every dot release of your project? If so, then something
> else may be improved in tooling (like missing some abstraction and/or
> separation). If not, it is simply in the "wrong place": why do you
> build/package/deploy it every time your project changes?
>
> But the same thing can be applied to examples like in this thread, for
> example checkstyle configuration: is this the ONLY project in your company?
> If not, is this the ONLY project having this checkstyle applied (and each
> project has a different one?). Maven is all about "sharing" :)
>
> Your next example of an annotation processor falls under the same scrutiny
> as above: does it really change with every release? But that said, and
> reading your mail and in parallel reading m-compiler-p sources, I already
> see a bug in m-compiler-p and how it resolves annotation processors: if you
> look, it ends up in maven-compat (Maven2 compatibility layer) and does the
> work in "Maven2 way", while you are right, this should work just fine out
> of the box. Filed issue for this bug:
>
> https://issues.apache.org/jira/browse/MCOMPILER-496
>
> So, your 2nd example (and 2nd paragraph) about the annotation processor is
> IMHO a m-compiler-p bug, and shall be fixed.
>
> And your 3rd paragraph is a bit too vague for me. Could you, like Florent
> did, create an example project that shows clearly what you want, and shows
> what Maven does not as expected?
>
>
> HTH
> Tamas
>
>
>
>
>
> [1] http://takari.io/book/91-maven-classloading.html
>
> On Fri, May 6, 2022 at 10:40 AM Thomas Broyer  wrote:
>
>>
>> Well, one limitation is with extensions: to use a project providing an
>> extension (custom lifecycle, whatever), you have to install it first. You
>> could argue that the project would otherwise be broken topologically (and
>> I
>> agree!) but there's really no way (or did I miss some new feature?) to do
>> such things without installing the extension first (with Gradle, in
>> addition to the buildSrc, you can use a "composite build" where one build
>> provides plugins for the other, including "settings plugins" –where the
>> settings.gradle file is also where you declare the included build–; Pants
>> allows you to have in-repo plugins, it's a no-brainer with Bazel too as
>> it's just how things work, I can't really tell for SBT but you could at
>> least put plugins in a project/ and 

Re: Maven Reactor and plugin "local" dependencies

[Tamás Cservenák]

Howdy,

no, not my memory, I rely on my "daily routine": I use Maven Repository
Manager, I regularly nuke my local repo, regularly execute `git clean -fdx`
on my checkout, and on checkout/branch change regularly do "quick-build"
`mvn clean install` (usually `mvn clean install -Dtest=void`) as very first
step. Have to add, I tend to work on projects that finish these "quick
builds" in a matter of minutes at most, not hours :) This "process" works
for me just fine.

But, now that you bring up topic of "branched development", something I
always hated for very same reason you bring up, I think you will be happy
to see upcoming changes:

https://github.com/apache/maven-resolver/blob/master/src/site/markdown/local-repository.md

(unreleased yet, we plan this for Resolver 1.8.1 and Maven 3.9.0).

All in all I think we both agree, but still: Maven as a project is big
(huge actually), and volunteers working on it have hard time to catch up,
like that compiler plugin bug, so as ecosystem, as a whole is very resource
demanding to "keep up", and I bet (in a six pack) that there are a LOT MORE
places where legacy APIs are used from plugins, also missing "workspace
resolution" the very same way compiler misses it. Please help us, by
reporting these (ideally with a reproducer).

Am not saying either "maven is the best", but I am convinced "maven can be
better", MUCH better and we will do it. As a matter of fact, we are already
doing it, step by step, baby steps :)

HTH
T

On Fri, May 6, 2022 at 11:00 AM Thomas Broyer  wrote:

>
> So you rely on your memory (when was the last time I ran 'mvn install'? did
> I switch branch or changed the code since then?) and you end up running
> 'mvn install' all the time, "just in case".
> (checking the sha means that target/XXX.jar is up-to-date with the code
> too, i.e. you ran 'mvn package', which brings use back to relying on your
> memory)
>
> Also, if you 'git pull' or 'git switch', how can you tell whether you
> actually need to rebuild (and re-install) that piece of code? You have to
> look at the changed files from the previous state and rely on assumptions
> about whether they'll change that thing that you have to 'mvn install' to
> make your build work; or I bet you'll actually just blindly 'mvn install'
> every time.
>
> See my other answer (to your other mail), there ARE cases (admittedly few
> of them, but still) where you DO have to rely on 'mvn install'.
>
> Note: I'm not here to debate about whether Maven is "good" or "good
> enough", better than other tools or not (I made my opinion years ago, I'm
> not asking you to change my mind). Maven is a good tool and we owe it many
> things, it really helped make the whole Java (and more) development
> experience better and inspired many other tools; but it DOES have
> limitations, and people have to be aware of them. And I'll stand by what I
> said: 'mvn install' is an anti-pattern. Even using one project from another
> (a lib you're developing, from some sample project using it) should be
> possible without 'mvn install', through an aggregator build (this is mostly
> the case, but there are exceptions). In other words, I should be able to
> 'cd' into a project and 'mvn verify' it and it should Just Work™ without
> relying on some "ambient" state such as "when did I last 'mvn install' that
> thing? am I actually using the 'correct version' of it?".
>
> --
> Thomas Broyer
> /tɔ.ma.bʁwa.je/  <
> http://xn--nna.ma.xn--bwa-xxb.je/>
>

Re: Maven Reactor and plugin "local" dependencies

[Tamás Cservenák]

Howdy,

a lot to unravel here...

But let me start first with the obvious: I believe you are quite
experienced, and I'd bet you did not start with Maven3, but most probably
Maven2 or even Maven1 :) The reason why I mention this, is that you use
terms from the "Maven2 era", like "reactor". Maven2 had the well known
"maven-reactor-plugin", that ceased to exist in Maven3. IMHO, let's clear
up the terms as otherwise we will just accumulate more misunderstandings.

For start, let's take Maven Extensions (core or build, does not matter).

IMHO, having them both (the tool you build with, and the source you build)
in one multi module project is IMHO wrong. That would be like a carpenter
keeping his nails and other tools along with timber in the same
box/storage, which I believe is not true. There are two kinds of extensions
in Maven: "core extension" and "build extension". Yes, both of them need to
be loaded ahead of the build. So yes, in case you keep your nails and
timber in the same box, you need to "bootstrap" things somehow (like
installing extension), as otherwise Maven will be unable to load up your
build in the first place.

But, let me ask this: how often your tooling changes compared to the
project? In other words, are you sure your extension you use to build with,
and the project itself, are changing at the same pace? Are you sure they
should share the same "release lifecycle"?

My typical example here (not extension, but similar intent): if we examine
Maven build, look at maven-model module. With every (even dot) release of
Maven, we generate, compile, package etc the maven-model module, while the
Maven model did not change since Maven 3.0.0 (or so, did not fact check,
but the point is there: very rarely changes). IMHO, the Maven project
should simply "move out" the maven-model from Maven build, as it does not
belong there: they do not share any release lifecycle at all.

If we get back to extensions kept in the same build where modules are built
as well, I wonder, how often does your extension change? Does it really
change with every dot release of your project? If so, then something else
may be improved in tooling (like missing some abstraction and/or
separation). If not, it is simply in the "wrong place": why do you
build/package/deploy it every time your project changes?

But the same thing can be applied to examples like in this thread, for
example checkstyle configuration: is this the ONLY project in your company?
If not, is this the ONLY project having this checkstyle applied (and each
project has a different one?). Maven is all about "sharing" :)

Your next example of an annotation processor falls under the same scrutiny
as above: does it really change with every release? But that said, and
reading your mail and in parallel reading m-compiler-p sources, I already
see a bug in m-compiler-p and how it resolves annotation processors: if you
look, it ends up in maven-compat (Maven2 compatibility layer) and does the
work in "Maven2 way", while you are right, this should work just fine out
of the box. Filed issue for this bug:

https://issues.apache.org/jira/browse/MCOMPILER-496

So, your 2nd example (and 2nd paragraph) about the annotation processor is
IMHO a m-compiler-p bug, and shall be fixed.

And your 3rd paragraph is a bit too vague for me. Could you, like Florent
did, create an example project that shows clearly what you want, and shows
what Maven does not as expected?


HTH
Tamas





[1] http://takari.io/book/91-maven-classloading.html

On Fri, May 6, 2022 at 10:40 AM Thomas Broyer  wrote:

>
> Well, one limitation is with extensions: to use a project providing an
> extension (custom lifecycle, whatever), you have to install it first. You
> could argue that the project would otherwise be broken topologically (and I
> agree!) but there's really no way (or did I miss some new feature?) to do
> such things without installing the extension first (with Gradle, in
> addition to the buildSrc, you can use a "composite build" where one build
> provides plugins for the other, including "settings plugins" –where the
> settings.gradle file is also where you declare the included build–; Pants
> allows you to have in-repo plugins, it's a no-brainer with Bazel too as
> it's just how things work, I can't really tell for SBT but you could at
> least put plugins in a project/ and possibly symlink to another project to
> mimic an "included build" like Gradle)
>
> Other setups seem to work "by accident"; e.g. a project with 3 submodules:
> annotations, processor, whatever; where 'whatever' has a dependency on
> 'annotations' and uses 'processor' in
> maven-compiler-plugin's , the order of the
>  in the aggregator POM is crucial to make it work ('processor' has
> to come before 'whatever') *and* you'll have to explicitly include
> 'processor' in your project list: 'mvn -pl whatever -am verify' won't build
> 'processor', you have to use 'mvn -pl processor,whatever -am verify'. I
> would infer Maven's behavior 

Re: Maven Reactor and plugin "local" dependencies

[Thomas Broyer]

On Thu, May 5, 2022 at 10:22 PM Tamás Cservenák  wrote:

> IMHO the antipattern here is the idea that "mvn install" is bad.
>
> But let's take it step by step. For simplicity sake, let's ignore remote
> deploys/downloads for now.
> Also, let's assume a developer workstation for the same reason
> (simplicity).
>
> Q: which one of your commit is your snapshot in your local repository from?
> A: the last one you `mvn install`-ed?
>
> Q: how can you tell?
> A: Your build should be reproducible (
> https://maven.apache.org/guides/mini/guide-reproducible-builds.html) so
> you
> can sha target/XXX.jar and local repo? But if unsure, just `mvn install` :D
>

So you rely on your memory (when was the last time I ran 'mvn install'? did
I switch branch or changed the code since then?) and you end up running
'mvn install' all the time, "just in case".
(checking the sha means that target/XXX.jar is up-to-date with the code
too, i.e. you ran 'mvn package', which brings use back to relying on your
memory)

Also, if you 'git pull' or 'git switch', how can you tell whether you
actually need to rebuild (and re-install) that piece of code? You have to
look at the changed files from the previous state and rely on assumptions
about whether they'll change that thing that you have to 'mvn install' to
make your build work; or I bet you'll actually just blindly 'mvn install'
every time.


> Maven does not require anything to install, or at least, am not getting
> what you are saying.
>

See my other answer (to your other mail), there ARE cases (admittedly few
of them, but still) where you DO have to rely on 'mvn install'.

Note: I'm not here to debate about whether Maven is "good" or "good
enough", better than other tools or not (I made my opinion years ago, I'm
not asking you to change my mind). Maven is a good tool and we owe it many
things, it really helped make the whole Java (and more) development
experience better and inspired many other tools; but it DOES have
limitations, and people have to be aware of them. And I'll stand by what I
said: 'mvn install' is an anti-pattern. Even using one project from another
(a lib you're developing, from some sample project using it) should be
possible without 'mvn install', through an aggregator build (this is mostly
the case, but there are exceptions). In other words, I should be able to
'cd' into a project and 'mvn verify' it and it should Just Work™ without
relying on some "ambient" state such as "when did I last 'mvn install' that
thing? am I actually using the 'correct version' of it?".

-- 
Thomas Broyer
/tɔ.ma.bʁwa.je/ 

Re: Maven Reactor and plugin "local" dependencies

[Thomas Broyer]

On Thu, May 5, 2022 at 10:16 PM Tamás Cservenák  wrote:

> Hm,
>
> By reading this blog entry from 2012 (!), I see it summarizes nicely at the
> end:
> * You may be using some broken plugins that you will need to fix, or
> replace.
> * You may be battling a circular dependency hell (which was why you used
> dependency:copy in the first place) which indicates a bad build smell that
> you should eradicate.
> * You may not have the time… well think of all the time you are wasting
> running "mvn install" when you could get away without going so far along
> the lifecycle.
>
> Can you point me to some "reproducer" project (of ANY of these issues
> mentioned in this 10 year old blog), that
> * will reproduce this with modern Maven, think 3.8.x
> * is NOT broken topologically (cycles), is not using broken (by using
> broken plugins) and developers does have time?


Well, one limitation is with extensions: to use a project providing an
extension (custom lifecycle, whatever), you have to install it first. You
could argue that the project would otherwise be broken topologically (and I
agree!) but there's really no way (or did I miss some new feature?) to do
such things without installing the extension first (with Gradle, in
addition to the buildSrc, you can use a "composite build" where one build
provides plugins for the other, including "settings plugins" –where the
settings.gradle file is also where you declare the included build–; Pants
allows you to have in-repo plugins, it's a no-brainer with Bazel too as
it's just how things work, I can't really tell for SBT but you could at
least put plugins in a project/ and possibly symlink to another project to
mimic an "included build" like Gradle)

Other setups seem to work "by accident"; e.g. a project with 3 submodules:
annotations, processor, whatever; where 'whatever' has a dependency on
'annotations' and uses 'processor' in
maven-compiler-plugin's , the order of the
 in the aggregator POM is crucial to make it work ('processor' has
to come before 'whatever') *and* you'll have to explicitly include
'processor' in your project list: 'mvn -pl whatever -am verify' won't build
'processor', you have to use 'mvn -pl processor,whatever -am verify'. I
would infer Maven's behavior for reactor ordering as taking the 
declaration order by default, when there's no dependency between modules,
and then the maven-compiler-plugin happens to be able to resolve
'processor' from the reactor build; change the order  are declared
and *kaboom*; in the end it does work only "by accident" (or "by chance" if
you prefer).
I wouldn't say the maven-compiler-plugin is "broken", it's just another
Maven limitation: it has no way of knowing that the
 actually lists dependencies that could come from
the reactor build and that it would take them into account for determining
which modules to build and in which order.

I don't have an actual use case at hand, this is all theoretical based on
my knowledge of how Maven works so correct me if I'm wrong, but using
custom lifecycles, and due to how build lifecycles work in Maven, running a
goal "in a project" (it's never actually "in a project", it's always run in
all projects of the reactor build, and the plugin itself has to detect or
be configured to know in which module to actually run or skip) might only
partially "build" the depended-on modules in such a way that they won't
work when the plugin actually need them.
Let's take a very hypothetical example, a plugin goal that scaffolds things
in a project, so must not trigger the project's lifecycle as that would
possibly break and we don't care (so either the plugin has no @execute, or
it executes an "early" phase such as 'initialize', or maybe
'process-classes' if it scaffolds tests and somehow needs the project's
main code to do that), but will use some dependency to determine what/how
to do its job (so has a plugin dependency, let's say on another module of
the reactor build). If we do 'mvn -pl target-project -am
the-plugin:scaffold -Dwhatever', then the 'scaffolding-config' submodule
declared as a plugin dependency of 'the-plugin' in project 'target-project'
won't be built besides what the plugin's goal @execute declares. So you'd
actually have to first 'mvn install -pl scaffolding-config' and then 'mvn
-pl target-project the-plugin:scaffold -Dwhatever' (note that I haven't
included -am here).
You could argue this is somehow "broken", I wouldn't disagree, but then
don't argue that Maven itself is not somehow "broken by design" or at a
minimum "limited" in its capabilities: there are cases where using "mvn
install" would be *required* to make things work. Either that or not using
Maven for those things, and rather rely on third-party tooling outside
Maven.

-- 
Thomas Broyer
/tɔ.ma.bʁwa.je/ 

Re: Maven Reactor and plugin "local" dependencies

[Tamás Cservenák]

IMHO the antipattern here is the idea that "mvn install" is bad.

But let's take it step by step. For simplicity sake, let's ignore remote
deploys/downloads for now.
Also, let's assume a developer workstation for the same reason (simplicity).

Q: which one of your commit is your snapshot in your local repository from?
A: the last one you `mvn install`-ed?

Q: how can you tell?
A: Your build should be reproducible (
https://maven.apache.org/guides/mini/guide-reproducible-builds.html) so you
can sha target/XXX.jar and local repo? But if unsure, just `mvn install` :D

Maven does not require anything to install, or at least, am not getting
what you are saying.
OTOH, you DO need to install, if you plan to deploy (but, we can exclude
this part, as we agreed, let's stick to developer workstation case, and
developer do not deploy, CI is).

HTH
T

On Thu, May 5, 2022 at 9:58 PM Thomas Broyer  wrote:

> Because it is an antipattern?
> https://www.cloudbees.com/blog/maven-and-install-hack
>
> If you do rely on "mvn install", which one of your commit is your snapshot
> in your local repository from? How can you tell? And it gets worse if you
> deploy snapshots to a repository, as now it could also be from someone
> else's commit (you might not have pulled from your git repo, but the
> snapshot comes from the origin/master that someone else pushed)
>
> Maven is among the only (modern) build tools requiring you to "install"
> part of your project to some widely shared place (shared with all you local
> projects) to use it from within itself!
>
>
> Le jeu. 5 mai 2022 à 21:22, Tamás Cservenák  a écrit
> :
>
> > Florent,
> >
> > Not directly related to your question, but reading your mail caught
> > my attention one sentence.
> > Can you shed some light on why "... but I'm trying to get rid of the need
> > for `mvn install` entirely."?
> >
> > Thanks
> > Tamas
> >
> > On Thu, May 5, 2022 at 3:11 PM Florent Biville <
> florent.bivi...@gmail.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > TL;DR? https://github.com/fbiville/mvn-plugin-dependency-reactor-issue
> > >
> > > I recently noticed that, within the big multi-module project of my
> > company,
> > > a specific module is listed as a dependency to plugins such as Maven
> > > Checkstyle plugin or com.mycila:license-maven-plugin.
> > >
> > > The problem occurs when I try one of these plugins against another
> > specific
> > > module.
> > > Maven CLI will basically complain that it cannot resolve the module
> > needed
> > > by the plugin and fail, even if I explicitly list that module in the
> > > project list.
> > >
> > > That occurs both with Maven 3 and 4.
> > >
> > > The obvious workaround is to install the required module first and then
> > run
> > > the plugin but I'm trying to get rid of the need for `mvn install`
> > > entirely.
> > >
> > > Is there a particular reason plugin dependencies are not considered by
> > > Maven Reactor?
> > > Is module-as-plugin-dependency a Maven antipattern?
> > >
> > > Thanks a lot for your help!
> > > Florent
> > >
> >
>

Re: Maven Reactor and plugin "local" dependencies

[Tamás Cservenák]

Hm,

By reading this blog entry from 2012 (!), I see it summarizes nicely at the
end:
* You may be using some broken plugins that you will need to fix, or
replace.
* You may be battling a circular dependency hell (which was why you used
dependency:copy in the first place) which indicates a bad build smell that
you should eradicate.
* You may not have the time… well think of all the time you are wasting
running "mvn install" when you could get away without going so far along
the lifecycle.

Can you point me to some "reproducer" project (of ANY of these issues
mentioned in this 10 year old blog), that
* will reproduce this with modern Maven, think 3.8.x
* is NOT broken topologically (cycles), is not using broken (by using
broken plugins) and developers does have time?

Thanks
T

On Thu, May 5, 2022 at 9:58 PM Thomas Broyer  wrote:

> Because it is an antipattern?
> https://www.cloudbees.com/blog/maven-and-install-hack
>
> If you do rely on "mvn install", which one of your commit is your snapshot
> in your local repository from? How can you tell? And it gets worse if you
> deploy snapshots to a repository, as now it could also be from someone
> else's commit (you might not have pulled from your git repo, but the
> snapshot comes from the origin/master that someone else pushed)
>
> Maven is among the only (modern) build tools requiring you to "install"
> part of your project to some widely shared place (shared with all you local
> projects) to use it from within itself!
>
>
> Le jeu. 5 mai 2022 à 21:22, Tamás Cservenák  a écrit
> :
>
> > Florent,
> >
> > Not directly related to your question, but reading your mail caught
> > my attention one sentence.
> > Can you shed some light on why "... but I'm trying to get rid of the need
> > for `mvn install` entirely."?
> >
> > Thanks
> > Tamas
> >
> > On Thu, May 5, 2022 at 3:11 PM Florent Biville <
> florent.bivi...@gmail.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > TL;DR? https://github.com/fbiville/mvn-plugin-dependency-reactor-issue
> > >
> > > I recently noticed that, within the big multi-module project of my
> > company,
> > > a specific module is listed as a dependency to plugins such as Maven
> > > Checkstyle plugin or com.mycila:license-maven-plugin.
> > >
> > > The problem occurs when I try one of these plugins against another
> > specific
> > > module.
> > > Maven CLI will basically complain that it cannot resolve the module
> > needed
> > > by the plugin and fail, even if I explicitly list that module in the
> > > project list.
> > >
> > > That occurs both with Maven 3 and 4.
> > >
> > > The obvious workaround is to install the required module first and then
> > run
> > > the plugin but I'm trying to get rid of the need for `mvn install`
> > > entirely.
> > >
> > > Is there a particular reason plugin dependencies are not considered by
> > > Maven Reactor?
> > > Is module-as-plugin-dependency a Maven antipattern?
> > >
> > > Thanks a lot for your help!
> > > Florent
> > >
> >
>

Re: Maven Reactor and plugin "local" dependencies

[Tamás Cservenák]

Florent,

created response on reproducer
https://github.com/fbiville/mvn-plugin-dependency-reactor-issue/issues/1

HTH
T

On Thu, May 5, 2022 at 3:11 PM Florent Biville 
wrote:

> Hello everyone,
>
> TL;DR? https://github.com/fbiville/mvn-plugin-dependency-reactor-issue
>
> I recently noticed that, within the big multi-module project of my company,
> a specific module is listed as a dependency to plugins such as Maven
> Checkstyle plugin or com.mycila:license-maven-plugin.
>
> The problem occurs when I try one of these plugins against another specific
> module.
> Maven CLI will basically complain that it cannot resolve the module needed
> by the plugin and fail, even if I explicitly list that module in the
> project list.
>
> That occurs both with Maven 3 and 4.
>
> The obvious workaround is to install the required module first and then run
> the plugin but I'm trying to get rid of the need for `mvn install`
> entirely.
>
> Is there a particular reason plugin dependencies are not considered by
> Maven Reactor?
> Is module-as-plugin-dependency a Maven antipattern?
>
> Thanks a lot for your help!
> Florent
>

Re: Maven Reactor and plugin "local" dependencies

[Thomas Broyer]

Because it is an antipattern?
https://www.cloudbees.com/blog/maven-and-install-hack

If you do rely on "mvn install", which one of your commit is your snapshot
in your local repository from? How can you tell? And it gets worse if you
deploy snapshots to a repository, as now it could also be from someone
else's commit (you might not have pulled from your git repo, but the
snapshot comes from the origin/master that someone else pushed)

Maven is among the only (modern) build tools requiring you to "install"
part of your project to some widely shared place (shared with all you local
projects) to use it from within itself!


Le jeu. 5 mai 2022 à 21:22, Tamás Cservenák  a écrit :

> Florent,
>
> Not directly related to your question, but reading your mail caught
> my attention one sentence.
> Can you shed some light on why "... but I'm trying to get rid of the need
> for `mvn install` entirely."?
>
> Thanks
> Tamas
>
> On Thu, May 5, 2022 at 3:11 PM Florent Biville 
> wrote:
>
> > Hello everyone,
> >
> > TL;DR? https://github.com/fbiville/mvn-plugin-dependency-reactor-issue
> >
> > I recently noticed that, within the big multi-module project of my
> company,
> > a specific module is listed as a dependency to plugins such as Maven
> > Checkstyle plugin or com.mycila:license-maven-plugin.
> >
> > The problem occurs when I try one of these plugins against another
> specific
> > module.
> > Maven CLI will basically complain that it cannot resolve the module
> needed
> > by the plugin and fail, even if I explicitly list that module in the
> > project list.
> >
> > That occurs both with Maven 3 and 4.
> >
> > The obvious workaround is to install the required module first and then
> run
> > the plugin but I'm trying to get rid of the need for `mvn install`
> > entirely.
> >
> > Is there a particular reason plugin dependencies are not considered by
> > Maven Reactor?
> > Is module-as-plugin-dependency a Maven antipattern?
> >
> > Thanks a lot for your help!
> > Florent
> >
>

Re: Maven Reactor and plugin "local" dependencies

[Tamás Cservenák]

Florent,

Not directly related to your question, but reading your mail caught
my attention one sentence.
Can you shed some light on why "... but I'm trying to get rid of the need
for `mvn install` entirely."?

Thanks
Tamas

On Thu, May 5, 2022 at 3:11 PM Florent Biville 
wrote:

> Hello everyone,
>
> TL;DR? https://github.com/fbiville/mvn-plugin-dependency-reactor-issue
>
> I recently noticed that, within the big multi-module project of my company,
> a specific module is listed as a dependency to plugins such as Maven
> Checkstyle plugin or com.mycila:license-maven-plugin.
>
> The problem occurs when I try one of these plugins against another specific
> module.
> Maven CLI will basically complain that it cannot resolve the module needed
> by the plugin and fail, even if I explicitly list that module in the
> project list.
>
> That occurs both with Maven 3 and 4.
>
> The obvious workaround is to install the required module first and then run
> the plugin but I'm trying to get rid of the need for `mvn install`
> entirely.
>
> Is there a particular reason plugin dependencies are not considered by
> Maven Reactor?
> Is module-as-plugin-dependency a Maven antipattern?
>
> Thanks a lot for your help!
> Florent
>

Re: Maven Reactor and plugin "local" dependencies

[Francois Marot]

Hello Florent,

I have had this problem a few years ago, and from what I can remember it is
the expected behavior. Or at least this is the way it is right now.
Sorry, It is from memory and I do not have any link to share nor
explanation about it, but at least you know you are not alone !

Regards

François



Le jeu. 5 mai 2022 à 15:11, Florent Biville  a
écrit :

> Hello everyone,
>
> TL;DR? https://github.com/fbiville/mvn-plugin-dependency-reactor-issue
>
> I recently noticed that, within the big multi-module project of my company,
> a specific module is listed as a dependency to plugins such as Maven
> Checkstyle plugin or com.mycila:license-maven-plugin.
>
> The problem occurs when I try one of these plugins against another specific
> module.
> Maven CLI will basically complain that it cannot resolve the module needed
> by the plugin and fail, even if I explicitly list that module in the
> project list.
>
> That occurs both with Maven 3 and 4.
>
> The obvious workaround is to install the required module first and then run
> the plugin but I'm trying to get rid of the need for `mvn install`
> entirely.
>
> Is there a particular reason plugin dependencies are not considered by
> Maven Reactor?
> Is module-as-plugin-dependency a Maven antipattern?
>
> Thanks a lot for your help!
> Florent
>