Re: Quicker local maven builds article

[Nathan Fisher]

Those are some great resources thanks for sharing!

On Tue, Oct 22, 2019 at 16:08, Jim N  wrote:

> how to maven plugin:
>
> TLDR
>
> https://github.com/0xCopy/prautobeans/tree/master/prautobeans-maven-plugin
> is
> how a lazy man makes a hard problem as simple as possible.
>
>
> pom boilerplate from this line until EOF:
>
> https://github.com/0xCopy/prautobeans/blob/master/prautobeans-maven-plugin/pom.xml#L24
>
>
> plugin annotation:
>
> https://github.com/0xCopy/prautobeans/blob/master/prautobeans-maven-plugin/src/main/java/prauto/PrautoGen.java#L40
>
>
> parameter annotation:
>
> https://github.com/0xCopy/prautobeans/blob/master/prautobeans-maven-plugin/src/main/java/prauto/PrautoGen.java#L47
>
>
> intellij will pop this one in for you:
>
> https://github.com/0xCopy/prautobeans/blob/1ab26409cf9784f2ec579b631ced39d811689317/prautobeans-maven-plugin/src/main/java/prauto/PrautoGen.java#L163
>
> *maybe* relevant to a build queue, the files visitor:
>
> https://github.com/0xCopy/prautobeans/blob/master/prautobeans-maven-plugin/src/main/java/prauto/PrautoGen.java#L176
>
>
>
>
>
> On Wed, Oct 23, 2019 at 1:46 AM Jim N  wrote:
>
> > I'm in Indonesia, and reddit is censored here :-) even though it's as
> > prevalent as youtube in technical value second only to SO.
> >
> > |Python3 proves to be a good prototyping tool. Maven's way is plugins,
> and
> > |my creation/publish elapsed time would be 10x greater that way as I'd
> > have
> > |to learn it first. I think I'm faster with Java solutions than Python
> > ones
> > |generally, but not when I'm close to Bash. I could have done this in
> > Bash,
> > |but it'd have been six hours to make to this level of polish instead of
> > the
> > |three that it was.
> >
> > amen.  i really, really hate build tool dualism.   maven's success in
> > eradicating jelly, and all conditionals and branching options was the
> > impetus that made ORM's viable and transferable project nkowledge
> probably
> > for the first time.
> >
> > jdk nio Files.* is better than what was on hand when maven plugins were
> > defined.  I don't know  how you solved build resolution but .. the maven
> > plugin boilerplate is as few as two annotations and reading some visitor
> > javadocs.  for god sakes why can't gradle just emulate a maven
> build-daemon
> > instead of deprecating the object model at every chance?
> >
> > Reactor architecture is ancient, and last i checked, the maven plugin
> list
> > is a ghost town barely active enough to review and retire some of the
> first
> > codehaus (your old stomping ground) ported plugins
> >
> > python is great, while you're on the machine you wrote it on.  once you
> > import a package, it's a less reliable tool than a  maven plugin, at a
> > cheap cost to prototype.
> >
> > polyglot-maven  might be
> what's
> > next,  enabling type-optional languages which incidentally brings inline
> > imperative syntaxes for free.  is it time to look the other way on
> > imperative languages in maven project object models?
> >
> >
> > On Wed, Oct 23, 2019 at 12:26 AM Paul Hammant  wrote:
> >
> >> Thanks for the links and words of encouragement, gents. I'll update the
> >> blog entry accordingly.
> >>
> >> I'm not a Maven committer (though i am an ASF member). I harang the
> >> committers on rotating topics over the years, and at some point they'll
> >> implement something similar to this if they want to. I love open source
> >> where the 'upstream' team has a policy of patch consumption if they
> can't
> >> state strong reasons for not doing so. And that's not Apache's policy.
> >>
> >> Python3 proves to be a good prototyping tool. Maven's way is plugins,
> and
> >> my creation/publish elapsed time would be 10x greater that way as I'd
> have
> >> to learn it first. I think I'm faster with Java solutions than Python
> ones
> >> generally, but not when I'm close to Bash. I could have done this in
> Bash,
> >> but it'd have been six hours to make to this level of polish instead of
> >> the
> >> three that it was.
> >>
> >> Perfect world for me would be:
> >>
> >> mvn -f buildThis.txt
> >>
> >> Where buildThis.txt was:
> >>
> >>   compile:
> >>  foo, bar
> >>   test:
> >>  foo, bar, baz
> >>
> >> That'd allow one invocation of Java, rather than two as I have it.
> >>
> >> On the Maven sub-reddit, we've 40 new subscribers now as of this post
> :) I
> >> love Reddit because of threading, in the same way I loved NNTP 20 years
> >> ago.
> >>
> >
>
-- 
Nathan Fisher
 w: http://junctionbox.ca/

Re: Quicker local maven builds article

[Nathan Fisher]

On Tue, Oct 22, 2019 at 19:02, Mark H. Wood  wrote:

> On Tue, Oct 22, 2019 at 09:34:46PM +0100, Paul Hammant wrote:
> > Maven's XML is too element normal for my liking.  Attributes where
> > appropruate would be good.
>
> Yes.  But changing now would be quite a wrench, even with tools to
> translate the old dialect to the new.  A lot of other code also
> understands POM documents nowadays.


Feels like it’s something that could be achieved with a bump in the pom
version to support the changes. As an interchange format in maven repos i
think you’d still want the maximally supported current version but for dev
it would be a nice option.


> > Also deps and GAVs needs a shakeup, IMO:
> >
> > 
> >   
> >  com.thoughtworks.xstream:xsteam:1.4.3
> >   
> >   
> > org.junit:junit:4.12
> > org.seleniumhq.selenium:selenium-java:3.14159
> >   
>
> Please, no.   version='4.12'/>
> perhaps.
>
> > Not being able to grep for specific GAVs is a critical flaw.
>
> Wrong tool.  grep is for text, not structured documents.  Try XQuery.


Personally I prefer the tool at hand even if not perfect. If I’m diagnosing
a problem on CI I would prefer not to be distracted by rabbit holes. Its
rare to work directly with XML these days so it adds both the burden of
getting the tool and remembering the syntax to extract what you want. The
compact notation would easily work with grep and is already employed by a
number of other build tools (gradle, buildr, bazel). It’s still the
coordinate but the component parts aren’t as granular.


> But this is wandering away from "how can we avoid rebuilding unchanged
> stuff?"


>
> --
> Mark H. Wood
> Lead Technology Analyst
>
> University Library
> Indiana University - Purdue University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> www.ulib.iupui.edu
>
-- 
Nathan Fisher
 w: http://junctionbox.ca/

Re: Quicker local maven builds article

[Mark H. Wood]

On Tue, Oct 22, 2019 at 09:34:46PM +0100, Paul Hammant wrote:
> Maven's XML is too element normal for my liking.  Attributes where
> appropruate would be good.

Yes.  But changing now would be quite a wrench, even with tools to
translate the old dialect to the new.  A lot of other code also
understands POM documents nowadays.

> Also deps and GAVs needs a shakeup, IMO:
> 
> 
>   
>  com.thoughtworks.xstream:xsteam:1.4.3
>   
>   
> org.junit:junit:4.12
> org.seleniumhq.selenium:selenium-java:3.14159
>   

Please, no.  
perhaps.

> Not being able to grep for specific GAVs is a critical flaw.

Wrong tool.  grep is for text, not structured documents.  Try XQuery.



But this is wandering away from "how can we avoid rebuilding unchanged
stuff?"

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu


signature.asc
Description: PGP signature

Re: Quicker local maven builds article

[Paul Hammant]

First time I met Jason was at a Codehaus party in Amsterdam in 2003. I miss
that portal.

Maven's XML is too element normal for my liking.  Attributes where
appropruate would be good. Also deps and GAVs needs a shakeup, IMO:


  
 com.thoughtworks.xstream:xsteam:1.4.3
  
  
org.junit:junit:4.12
org.seleniumhq.selenium:selenium-java:3.14159
  

Not being able to grep for specific GAVs is a critical flaw.

So I've played with Takari and I like it. I'm able to make super fast
builds without it though. The time-pit is all categories of integration
test. That Maven's recursion fu is slower than Gradles isn't where time is
lost for most *corporate* builds.

- Paul

Re: Alternate Maven dependency upgrade opportunities report

[Jim N]

https://www.mojohaus.org/versions-maven-plugin

 ?

On Fri, Jul 27, 2018 at 4:46 PM Paul Hammant  wrote:

> If Maven-central has some upgrades for one of your project's dependencies,
> this little Python script can quickly* tell you.
>
> https://github.com/paul-hammant/analyze-mvn-deps
>
> Key feature:
>
> In some cases, the script may suggest two or more alternate upgrade
> suggestions. For example (as of July 27th), a Maven project depending on
> the jar for Groovy v2.5.0 would see *three alternate suggested upgrades*:
> 3.0.0-alpha-3, 2.6.0-alpha-4, or 2.5.1.
>
> * "quickly" may be minutes if the multi-module project is big enough.
>
> --
> Paul Hammant DevOps  Let me give your
> enterprise a step by step plan to migrate from GitFlow (or worse) to
> high-throughput CD on the essential DevOps foundation: Trunk-Based
> Development.
>

Re: Quicker local maven builds article

[Jim N]

how to maven plugin:

TLDR

https://github.com/0xCopy/prautobeans/tree/master/prautobeans-maven-plugin is
how a lazy man makes a hard problem as simple as possible.


pom boilerplate from this line until EOF:
https://github.com/0xCopy/prautobeans/blob/master/prautobeans-maven-plugin/pom.xml#L24


plugin annotation:
https://github.com/0xCopy/prautobeans/blob/master/prautobeans-maven-plugin/src/main/java/prauto/PrautoGen.java#L40


parameter annotation:
https://github.com/0xCopy/prautobeans/blob/master/prautobeans-maven-plugin/src/main/java/prauto/PrautoGen.java#L47


intellij will pop this one in for you:
https://github.com/0xCopy/prautobeans/blob/1ab26409cf9784f2ec579b631ced39d811689317/prautobeans-maven-plugin/src/main/java/prauto/PrautoGen.java#L163

*maybe* relevant to a build queue, the files visitor:
https://github.com/0xCopy/prautobeans/blob/master/prautobeans-maven-plugin/src/main/java/prauto/PrautoGen.java#L176





On Wed, Oct 23, 2019 at 1:46 AM Jim N  wrote:

> I'm in Indonesia, and reddit is censored here :-) even though it's as
> prevalent as youtube in technical value second only to SO.
>
> |Python3 proves to be a good prototyping tool. Maven's way is plugins, and
> |my creation/publish elapsed time would be 10x greater that way as I'd
> have
> |to learn it first. I think I'm faster with Java solutions than Python
> ones
> |generally, but not when I'm close to Bash. I could have done this in
> Bash,
> |but it'd have been six hours to make to this level of polish instead of
> the
> |three that it was.
>
> amen.  i really, really hate build tool dualism.   maven's success in
> eradicating jelly, and all conditionals and branching options was the
> impetus that made ORM's viable and transferable project nkowledge probably
> for the first time.
>
> jdk nio Files.* is better than what was on hand when maven plugins were
> defined.  I don't know  how you solved build resolution but .. the maven
> plugin boilerplate is as few as two annotations and reading some visitor
> javadocs.  for god sakes why can't gradle just emulate a maven build-daemon
> instead of deprecating the object model at every chance?
>
> Reactor architecture is ancient, and last i checked, the maven plugin list
> is a ghost town barely active enough to review and retire some of the first
> codehaus (your old stomping ground) ported plugins
>
> python is great, while you're on the machine you wrote it on.  once you
> import a package, it's a less reliable tool than a  maven plugin, at a
> cheap cost to prototype.
>
> polyglot-maven  might be what's
> next,  enabling type-optional languages which incidentally brings inline
> imperative syntaxes for free.  is it time to look the other way on
> imperative languages in maven project object models?
>
>
> On Wed, Oct 23, 2019 at 12:26 AM Paul Hammant  wrote:
>
>> Thanks for the links and words of encouragement, gents. I'll update the
>> blog entry accordingly.
>>
>> I'm not a Maven committer (though i am an ASF member). I harang the
>> committers on rotating topics over the years, and at some point they'll
>> implement something similar to this if they want to. I love open source
>> where the 'upstream' team has a policy of patch consumption if they can't
>> state strong reasons for not doing so. And that's not Apache's policy.
>>
>> Python3 proves to be a good prototyping tool. Maven's way is plugins, and
>> my creation/publish elapsed time would be 10x greater that way as I'd have
>> to learn it first. I think I'm faster with Java solutions than Python ones
>> generally, but not when I'm close to Bash. I could have done this in Bash,
>> but it'd have been six hours to make to this level of polish instead of
>> the
>> three that it was.
>>
>> Perfect world for me would be:
>>
>> mvn -f buildThis.txt
>>
>> Where buildThis.txt was:
>>
>>   compile:
>>  foo, bar
>>   test:
>>  foo, bar, baz
>>
>> That'd allow one invocation of Java, rather than two as I have it.
>>
>> On the Maven sub-reddit, we've 40 new subscribers now as of this post :) I
>> love Reddit because of threading, in the same way I loved NNTP 20 years
>> ago.
>>
>

Re: Quicker local maven builds article

[Jim N]

I'm in Indonesia, and reddit is censored here :-) even though it's as
prevalent as youtube in technical value second only to SO.

|Python3 proves to be a good prototyping tool. Maven's way is plugins, and
|my creation/publish elapsed time would be 10x greater that way as I'd have
|to learn it first. I think I'm faster with Java solutions than Python ones
|generally, but not when I'm close to Bash. I could have done this in Bash,
|but it'd have been six hours to make to this level of polish instead of the
|three that it was.

amen.  i really, really hate build tool dualism.   maven's success in
eradicating jelly, and all conditionals and branching options was the
impetus that made ORM's viable and transferable project nkowledge probably
for the first time.

jdk nio Files.* is better than what was on hand when maven plugins were
defined.  I don't know  how you solved build resolution but .. the maven
plugin boilerplate is as few as two annotations and reading some visitor
javadocs.  for god sakes why can't gradle just emulate a maven build-daemon
instead of deprecating the object model at every chance?

Reactor architecture is ancient, and last i checked, the maven plugin list
is a ghost town barely active enough to review and retire some of the first
codehaus (your old stomping ground) ported plugins

python is great, while you're on the machine you wrote it on.  once you
import a package, it's a less reliable tool than a  maven plugin, at a
cheap cost to prototype.

polyglot-maven  might be what's
next,  enabling type-optional languages which incidentally brings inline
imperative syntaxes for free.  is it time to look the other way on
imperative languages in maven project object models?


On Wed, Oct 23, 2019 at 12:26 AM Paul Hammant  wrote:

> Thanks for the links and words of encouragement, gents. I'll update the
> blog entry accordingly.
>
> I'm not a Maven committer (though i am an ASF member). I harang the
> committers on rotating topics over the years, and at some point they'll
> implement something similar to this if they want to. I love open source
> where the 'upstream' team has a policy of patch consumption if they can't
> state strong reasons for not doing so. And that's not Apache's policy.
>
> Python3 proves to be a good prototyping tool. Maven's way is plugins, and
> my creation/publish elapsed time would be 10x greater that way as I'd have
> to learn it first. I think I'm faster with Java solutions than Python ones
> generally, but not when I'm close to Bash. I could have done this in Bash,
> but it'd have been six hours to make to this level of polish instead of the
> three that it was.
>
> Perfect world for me would be:
>
> mvn -f buildThis.txt
>
> Where buildThis.txt was:
>
>   compile:
>  foo, bar
>   test:
>  foo, bar, baz
>
> That'd allow one invocation of Java, rather than two as I have it.
>
> On the Maven sub-reddit, we've 40 new subscribers now as of this post :) I
> love Reddit because of threading, in the same way I loved NNTP 20 years
> ago.
>

Re: Quicker local maven builds article

[Paul Hammant]

Thanks for the links and words of encouragement, gents. I'll update the
blog entry accordingly.

I'm not a Maven committer (though i am an ASF member). I harang the
committers on rotating topics over the years, and at some point they'll
implement something similar to this if they want to. I love open source
where the 'upstream' team has a policy of patch consumption if they can't
state strong reasons for not doing so. And that's not Apache's policy.

Python3 proves to be a good prototyping tool. Maven's way is plugins, and
my creation/publish elapsed time would be 10x greater that way as I'd have
to learn it first. I think I'm faster with Java solutions than Python ones
generally, but not when I'm close to Bash. I could have done this in Bash,
but it'd have been six hours to make to this level of polish instead of the
three that it was.

Perfect world for me would be:

mvn -f buildThis.txt

Where buildThis.txt was:

  compile:
 foo, bar
  test:
 foo, bar, baz

That'd allow one invocation of Java, rather than two as I have it.

On the Maven sub-reddit, we've 40 new subscribers now as of this post :) I
love Reddit because of threading, in the same way I loved NNTP 20 years ago.

Re: Quicker local maven builds article

[Martin Todorov]

Hi Paul,

Thanks for sharing this!

Wouldn't it be more useful, though, if you tried to implement this as a
feature of Maven itself? I'm not known to be the biggest fan of Gradle, but
Gradle does know what needs to be rebuilt upon a change and will build
incrementally.

I'm really impressed how the Gradle folks are contributing fixes to Maven
that will make a huge impact when calculating the build order for large
projects (and so on)! Despite all the great ideas that they have, I'm still
more of a fan of how Maven does things. However, I think that it's about
time Maven becomes more sophisticated in regards to incremental builds and
figuring out what sets of tests are required to be executed, when a certain
change has been made.

Scripting things outside of Maven and using them externally, is perhaps an
option, but it's not following Maven's own philosophy of being "portable".
The books on Maven have all been preaching this for quite a long time and
moving away from this principle will, in my opinion, have an adverse effect.

Is it not perhaps time that these issues and ideas get collected and an
action plan gets drafted on how to implement them as built-in features?

Kind regards,

Martin




On Tue, Oct 22, 2019 at 3:53 PM Nathan Fisher 
wrote:

> Hi Paul,
>
> Looks great! Thanks for sharing.
>
> I've been wondering how difficult it is to make Maven behave more like
> Buck/Bazel (e.g. better build caching, employ a DAG, remote
> execution/caching, etc). There's obviously a whole lot of gaps in the
> current Maven process that would need to be closed (e.g. banning circular
> deps) but would be nice if the rich ecosystem of tooling around maven could
> still be employed insitui.
>
> I've heard the folks at Gradle have contributed an early contribution to
> remote caches but haven't had the opportunity to dig into it much.
>
> To be honest though my initial wanderings into Maven's dependency graph
> resolution left me wanting. There appears to be a few different
> implementations in the Maven Github org to build the graph. Even in maven
> itself there appears to be the M2 fallback and the newer M3 approach which
> change based on context. Some others seem to be experiments but are unused
> in Maven itself, it's all a little messy coming from the outside.
>
> From the mailing list sounds like it's a small band of developers that are
> seriously understaffed with a large tract of land to cover.
>
> Cheers,
> Nathan
>
> On Mon, Oct 21, 2019 at 1:50 PM Jason Young 
> wrote:
>
> > This is definitely a helpful thing to be able to do. I wish Maven would
> > make some effort to keep up with what jobs it's already done and doesn't
> > need to repeat vs. what it has left to do. This is a fundamentally
> > automatable job that is very difficult for people to do correctly, so
> > everyone just `mvn clean install` on the root project every time, or
> avoids
> > using Maven from the command line altogether.
> >
> > Similar tool: https://github.com/vackosar/gitflow-incremental-builder -
> > works well overall for my team's workflow, thought I don't think devs
> have
> > played with it; it's only used in CI for certain jobs right now.
> >
> > Note that some plugins at some versions don't work correctly if you skip
> > some projects, e.g. we mitigate
> > https://issues.apache.org/jira/browse/MENFORCER-306 by ensuring the root
> > pom is in the build and disable maven-enforcer-plugin by default on dev
> > machines; another way to avoid the problem is to use any previous version
> > of enforcer (we need the latest because of
> > https://issues.apache.org/jira/browse/MENFORCER-268).
> >
> > (Replying here as I'm not a Reddit user - sorry!)
> >
> > On Sun, Oct 20, 2019 at 11:44 AM Paul Hammant  wrote:
> >
> > >
> >
> https://www.reddit.com/r/Maven/comments/dklz1e/quicker_local_maven_builds/
> > >
> > > ^ A small script to help you have quicker Maven invocations for changes
> > in
> > > your (Git) checkout on your dev workstation.
> > >
> > > Also, a reminder that there is a sub-reddit for Maven now -
> > > https://www.reddit.com/r/Maven/
> > >
> > > - Paul
> > >
> >
>
>
> --
> Nathan Fisher
>  w: http://junctionbox.ca/
>

Re: Quicker local maven builds article

[Nathan Fisher]

Hi Paul,

Looks great! Thanks for sharing.

I've been wondering how difficult it is to make Maven behave more like
Buck/Bazel (e.g. better build caching, employ a DAG, remote
execution/caching, etc). There's obviously a whole lot of gaps in the
current Maven process that would need to be closed (e.g. banning circular
deps) but would be nice if the rich ecosystem of tooling around maven could
still be employed insitui.

I've heard the folks at Gradle have contributed an early contribution to
remote caches but haven't had the opportunity to dig into it much.

To be honest though my initial wanderings into Maven's dependency graph
resolution left me wanting. There appears to be a few different
implementations in the Maven Github org to build the graph. Even in maven
itself there appears to be the M2 fallback and the newer M3 approach which
change based on context. Some others seem to be experiments but are unused
in Maven itself, it's all a little messy coming from the outside.

>From the mailing list sounds like it's a small band of developers that are
seriously understaffed with a large tract of land to cover.

Cheers,
Nathan

On Mon, Oct 21, 2019 at 1:50 PM Jason Young 
wrote:

> This is definitely a helpful thing to be able to do. I wish Maven would
> make some effort to keep up with what jobs it's already done and doesn't
> need to repeat vs. what it has left to do. This is a fundamentally
> automatable job that is very difficult for people to do correctly, so
> everyone just `mvn clean install` on the root project every time, or avoids
> using Maven from the command line altogether.
>
> Similar tool: https://github.com/vackosar/gitflow-incremental-builder -
> works well overall for my team's workflow, thought I don't think devs have
> played with it; it's only used in CI for certain jobs right now.
>
> Note that some plugins at some versions don't work correctly if you skip
> some projects, e.g. we mitigate
> https://issues.apache.org/jira/browse/MENFORCER-306 by ensuring the root
> pom is in the build and disable maven-enforcer-plugin by default on dev
> machines; another way to avoid the problem is to use any previous version
> of enforcer (we need the latest because of
> https://issues.apache.org/jira/browse/MENFORCER-268).
>
> (Replying here as I'm not a Reddit user - sorry!)
>
> On Sun, Oct 20, 2019 at 11:44 AM Paul Hammant  wrote:
>
> >
> https://www.reddit.com/r/Maven/comments/dklz1e/quicker_local_maven_builds/
> >
> > ^ A small script to help you have quicker Maven invocations for changes
> in
> > your (Git) checkout on your dev workstation.
> >
> > Also, a reminder that there is a sub-reddit for Maven now -
> > https://www.reddit.com/r/Maven/
> >
> > - Paul
> >
>


-- 
Nathan Fisher
 w: http://junctionbox.ca/