Re: Speeding up Maven

[Paul Hammant]

Can you yourself try another dissimilar machine?

Re: Speeding up Maven

[Mark Raynsford]

On 2018-04-13T16:29:27 -0400
Paul Hammant  wrote:

> Mark,
> 
> Assuming a pre-filled Maven local cache, I've a 16 second build that's only
> longer when I run a screen recorder in order to make the video show here -
> https://paulhammant.com/2017/02/05/a-16-second-java-webapp-build-including-webdriver-tests/
> 
> That one runs three WebDriver tests too after the unit and service tests -
> meaning (implicitly) it's testing JavaScript as well as Java in that time.
> 
> In order to make it fast I've abandoned the fail-safe plugin and do all
> parts of the test-pyramid in surefire (see the pom).

'Ello.

So you're saying it's my problem? ;)

I'm also assuming a pre-filled Maven local cache, but this example
project (with no dependencies) takes about six seconds to build with
(usually) about ten seconds of Maven churning before the build even
begins. This is for a build that compiles one tiny file, produces a
jar, and does nothing else.

I assumed that it was happening for everyone, but apparently it isn't.

-- 
Mark Raynsford | http://www.io7m.com



pgp0AU5kw2b8y.pgp
Description: OpenPGP digital signature

Re: Speeding up Maven

[Mark Raynsford]

On 2018-04-13T21:48:58 +0200
Karl Heinz Marbaise  wrote:

> Hi Mark,

'Ello.

> On 13/04/18 21:20, Mark Raynsford wrote:
> 
> First it would be good to know which Maven version are you using? And on 
> which JDK you are running?

Hah, I purposefully left out details in the hope that it wouldn't be my
problem specifically. From the one second build you've described, it
appears that perhaps it might be.

My system:

Linux copperhead.int.arc7.info 4.15.14-1-ARCH #1 SMP PREEMPT Wed Mar 28
17:34:29 UTC 2018 x86_64 GNU/Linux

Intel(R) Core(TM) i5-2410M CPU @ 2.30GHz, 8GB ram.

openjdk version "9.0.4"
OpenJDK Runtime Environment (build 9.0.4+11)
OpenJDK 64-Bit Server VM (build 9.0.4+11, mixed mode)

Apache Maven 3.5.2 (NON-CANONICAL_2017-10-25T13:09:52+03:00_root;
2017-10-25T10:09:52Z) Maven home: /opt/maven
Java version: 9.0.4, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-9-openjdk
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux", version: "4.15.14-1-arch", arch: "amd64", family:
"unix"

> Maybe I misunderstand you ...are you running it from plain command line 
> or from inside your IDE ?

The command line in the log is:

  mvn -Dmaven.tests.skip=true -DskipTests=true package

Although in the IDE in real projects, I'm skipping more plugin
executions than that.

> 
> What are you memory settings? How is your JDK configured ? Which command 
> line options ? (MAVEN_OPTS?):??

I'm both running it from the command line and from the IDE. My point
was that 16 seconds for a command line build is tolerable (although 
still pretty damn high for a project that only compiles one Java file 
and doesn't do anything else), but running the same build from within 
an IDE is painful (because in the IDE, I'm hoping for instant gratification
when I click "Run..." and I don't care about producing a full build
that someone else will use).

$ env | grep MAVEN
MAVEN_OPTS=-Xmx512m

My JDK is essentially unconfigured. Whatever the default settings are
for Linux AMD64, OpenJDK 9.

> Depending on your JVM and the configuration that can be quite large 
> different time on startup...there are several configurations which can 
> be used to improve that...

I'm open to suggestions. I'm fairly confident that it's not JVM startup
time that's the issue. For example, if I add the -X flag to Maven, you
can see that control reaches whatever passes for Maven's main() function
in roughly one second:

2018-04-13 21:00:01.455570500 
2018-04-13 21:00:02.740905500 Apache Maven 3.5.2 
(NON-CANONICAL_2017-10-25T13:09:52+03:00_root; 2017-10-25T10:09:52Z)
2018-04-13 21:00:02.741034500 Maven home: /opt/maven
2018-04-13 21:00:02.741192500 Java version: 9.0.4, vendor: Oracle Corporation
2018-04-13 21:00:02.741333500 Java home: /usr/lib/jvm/java-9-openjdk
2018-04-13 21:00:02.741454500 Default locale: en_GB, platform encoding: UTF-8
2018-04-13 21:00:02.741829500 OS name: "linux", version: "4.15.14-1-arch", 
arch: "amd64", family: "unix"
2018-04-13 21:00:06.841656500 [DEBUG] Created new class realm maven.api
2018-04-13 21:00:06.842724500 [DEBUG] Importing foreign packages into class 
realm maven.api
2018-04-13 21:00:06.844680500 [DEBUG]   Imported: javax.annotation.* < 
plexus.core
2018-04-13 21:00:06.847413500 [DEBUG]   Imported: javax.enterprise.inject.* < 
plexus.core
2018-04-13 21:00:06.847911500 [DEBUG]   Imported: javax.enterprise.util.* < 
plexus.core
2018-04-13 21:00:06.848473500 [DEBUG]   Imported: javax.inject.* < plexus.core
2018-04-13 21:00:06.849076500 [DEBUG]   Imported: org.apache.maven.* < 
plexus.core
2018-04-13 21:00:06.849575500 [DEBUG]   Imported: org.apache.maven.artifact < 
plexus.core
2018-04-13 21:00:06.850103500 [DEBUG]   Imported: org.apache.maven.classrealm < 
plexus.core
2018-04-13 21:00:06.850619500 [DEBUG]   Imported: org.apache.maven.cli < 
plexus.core
2018-04-13 21:00:06.851178500 [DEBUG]   Imported: 
org.apache.maven.configuration < plexus.core
2018-04-13 21:00:06.851678500 [DEBUG]   Imported: org.apache.maven.exception < 
plexus.core
2018-04-13 21:00:06.852183500 [DEBUG]   Imported: org.apache.maven.execution < 
plexus.core
2018-04-13 21:00:06.852685500 [DEBUG]   Imported: 
org.apache.maven.execution.scope < plexus.core
2018-04-13 21:00:06.853183500 [DEBUG]   Imported: org.apache.maven.lifecycle < 
plexus.core
2018-04-13 21:00:06.853653500 [DEBUG]   Imported: org.apache.maven.model < 
plexus.core
2018-04-13 21:00:06.854292500 [DEBUG]   Imported: org.apache.maven.monitor < 
plexus.core
2018-04-13 21:00:06.854777500 [DEBUG]   Imported: org.apache.maven.plugin < 
plexus.core
2018-04-13 21:00:06.855287500 [DEBUG]   Imported: org.apache.maven.profiles < 
plexus.core
2018-04-13 21:00:06.855786500 [DEBUG]   Imported: org.apache.maven.project < 
plexus.core
2018-04-13 21:00:06.856284500 [DEBUG]   Imported: org.apache.maven.reporting < 
plexus.core
2018-04-13 21:00:06.856757500 [DEBUG]   Imported: org.apache.maven.repository < 
plexus.core
2018-04-13 21:00:06.857291500 [DEBUG]   Imported: 

Re: Speeding up Maven

[Paul Hammant]

Mark,

Assuming a pre-filled Maven local cache, I've a 16 second build that's only
longer when I run a screen recorder in order to make the video show here -
https://paulhammant.com/2017/02/05/a-16-second-java-webapp-build-including-webdriver-tests/

That one runs three WebDriver tests too after the unit and service tests -
meaning (implicitly) it's testing JavaScript as well as Java in that time.

In order to make it fast I've abandoned the fail-safe plugin and do all
parts of the test-pyramid in surefire (see the pom).

- Paul

On Fri, Apr 13, 2018 at 3:20 PM, Mark Raynsford <
org.apache.maven.u...@io7m.com> wrote:

> Hello.
>
> Is there any way I might speed up Maven? I've been using it for years
> and, for the first time, I'm actually executing it from an IDE rather
> than using the IDE's built-in compiler. It seems like there's a rather
> large and apparently fixed amount of overhead per invocation. Take the
> following "could not be simpler" project:
>
>   https://github.com/io7m/maven-minimal-20180413
>
> I piped the output through tai64n [1] to give a better indication of
> exactly how long everything is taking. That's a project that compiles a
> single Java file and produces a jar. It doesn't run any tests, it
> doesn't include any resources. 16 seconds total execution [2]!
>
> Now for an infrequent command-line build, that's not *too* obnoxious.
> For a "click a button in the IDE because I want to try this bit of code
> now" build, that's a world of suffering.
>
> It seems like Maven first spends ten entire seconds doing not very
> much. That cannot be attributed to JVM startup time (Hello World on my
> system takes about 100ms to reach user code, and even the largest Java
> application I still takes less than a second to reach user code after
> all the classes have loaded).
>
> Is there anything I can do to cut this down? If not, is this something
> that's intended to be addressed in the near future?
>
> [1] http://cr.yp.to/daemontools/tai64n.html
> [2] The Maven output claims 6 seconds, but that doesn't include the
> time it took for the build to start.
>
> --
> Mark Raynsford | http://www.io7m.com
>
>


-- 
Paul Hammant DevOps <https://devops.paulhammant.com> Let me give your
enterprise a step by step plan to get out of the hell of crazy branching
models (ClearCase maybe?) and into the world of high-throughput CD on
DevOps foundations.

Re: [RESULT] [VOTE] Release Apache Maven Clean Plugin version 3.1.0

[Karl Heinz Marbaise]

Hi,

On 13/04/18 17:51, Karl Heinz Marbaise wrote:

Hi,

The vote has passed with the following result:

+1 : Tibor Digana, Olivier Lamy, Karl Heinz Marbaise


Sorry I missed Hervé Boutemy with it's vote...



PMC quorum: reached.

I will promote the artifacts to the central repo.

Kind regards
Karl Heinz Marbaise



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Speeding up Maven

[Karl Heinz Marbaise]

Hi Mark,

I simply missed this project:

https://github.com/khmarbaise/maven-buildtime-profiler

Kind regards
Karl Heinz Marbaise


On 13/04/18 21:20, Mark Raynsford wrote:

Hello.

Is there any way I might speed up Maven? I've been using it for years
and, for the first time, I'm actually executing it from an IDE rather
than using the IDE's built-in compiler. It seems like there's a rather
large and apparently fixed amount of overhead per invocation. Take the
following "could not be simpler" project:

   https://github.com/io7m/maven-minimal-20180413

I piped the output through tai64n [1] to give a better indication of
exactly how long everything is taking. That's a project that compiles a
single Java file and produces a jar. It doesn't run any tests, it
doesn't include any resources. 16 seconds total execution [2]!

Now for an infrequent command-line build, that's not *too* obnoxious.
For a "click a button in the IDE because I want to try this bit of code
now" build, that's a world of suffering.

It seems like Maven first spends ten entire seconds doing not very
much. That cannot be attributed to JVM startup time (Hello World on my
system takes about 100ms to reach user code, and even the largest Java
application I still takes less than a second to reach user code after
all the classes have loaded).

Is there anything I can do to cut this down? If not, is this something
that's intended to be addressed in the near future?

[1] http://cr.yp.to/daemontools/tai64n.html
[2] The Maven output claims 6 seconds, but that doesn't include the
 time it took for the build to start.



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Speeding up Maven

[Karl Heinz Marbaise]

Hi Mark,


On 13/04/18 21:20, Mark Raynsford wrote:

First it would be good to know which Maven version are you using? And on 
which JDK you are running?




Hello.

Is there any way I might speed up Maven? I've been using it for years
and, for the first time, I'm actually executing it from an IDE rather
than using the IDE's built-in compiler. It seems like there's a rather
large and apparently fixed amount of overhead per invocation. Take the
following "could not be simpler" project:

   https://github.com/io7m/maven-minimal-20180413



Maybe I misunderstand you ...are you running it from plain command line 
or from inside your IDE ?


What are you memory settings? How is your JDK configured ? Which command 
line options ? (MAVEN_OPTS?):??






I piped the output through tai64n [1] to give a better indication of
exactly how long everything is taking. That's a project that compiles a
single Java file and produces a jar. It doesn't run any tests, it
doesn't include any resources. 16 seconds total execution [2]!



Unfortunately I don't have tai64n on my Mac but based on simple tests 
with your project I can run the whole project in about 1 Second for the 
whole build...





Now for an infrequent command-line build, that's not *too* obnoxious.
For a "click a button in the IDE because I want to try this bit of code
now" build, that's a world of suffering.

It seems like Maven first spends ten entire seconds doing not very
much. That cannot be attributed to JVM startup time (Hello World on my
system takes about 100ms to reach user code, and even the largest Java
application I still takes less than a second to reach user code after
all the classes have loaded).


Depending on your JVM and the configuration that can be quite large 
different time on startup...there are several configurations which can 
be used to improve that...




Is there anything I can do to cut this down? If not, is this something
that's intended to be addressed in the near future?


I'm doing from time to time analyzes based on performance / Memory...but 
I'm interested to do more here...





[1] http://cr.yp.to/daemontools/tai64n.html
[2] The Maven output claims 6 seconds, but that doesn't include the
 time it took for the build to start.




Kind regards
Karl Heinz Marbaise

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Speeding up Maven

[Mark Raynsford]

Hello.

Is there any way I might speed up Maven? I've been using it for years
and, for the first time, I'm actually executing it from an IDE rather
than using the IDE's built-in compiler. It seems like there's a rather
large and apparently fixed amount of overhead per invocation. Take the
following "could not be simpler" project:

  https://github.com/io7m/maven-minimal-20180413

I piped the output through tai64n [1] to give a better indication of
exactly how long everything is taking. That's a project that compiles a
single Java file and produces a jar. It doesn't run any tests, it
doesn't include any resources. 16 seconds total execution [2]!

Now for an infrequent command-line build, that's not *too* obnoxious.
For a "click a button in the IDE because I want to try this bit of code
now" build, that's a world of suffering.

It seems like Maven first spends ten entire seconds doing not very
much. That cannot be attributed to JVM startup time (Hello World on my
system takes about 100ms to reach user code, and even the largest Java
application I still takes less than a second to reach user code after
all the classes have loaded).

Is there anything I can do to cut this down? If not, is this something
that's intended to be addressed in the near future?

[1] http://cr.yp.to/daemontools/tai64n.html
[2] The Maven output claims 6 seconds, but that doesn't include the
time it took for the build to start.

-- 
Mark Raynsford | http://www.io7m.com



pgpfIHio6nPxi.pgp
Description: OpenPGP digital signature

Re: [IMPORTANT CHANGE] building and publishing our website from Jenkins and Git instead of CMS

[Karl Heinz Marbaise]

Hi Hervé,

On 12/04/18 08:17, Hervé BOUTEMY wrote:

Hi,

I'm working with infra for a good time on Git migration for site source, which
will de-facto mean that we drop Apache CMS (that only supports svn for
source): see INFRA-16088 [1]

Today, we got a first visible result: Jenkins was able to build and publish
html output of the Maven main website to svn [2]

We're now ready to switch to the new Jenkins build, then drop CMS: any
objection to do it?

The plan will be then to switch source to GitBox, to benefit from immediate
commit detection (instead of svn hourly polling): at this step, you'll be able
to do easy site source updates through GitHub, with immediate publish. I don't
see the value of adding a staging step, but if someone wants staging, it will
be a matter of Jenkinsfile tweaking.

In the temporary time with Jenkins polling on source from svn , either you'll
wait for less than 1 hour or you'll manually launch the Jenkins job

any question?


Jenkins will build the Site (as done before 
ci.apache.org/builders/maven-site-staging) and deploys it to the site 
(maven.staging.apache.org?)...but it's gets the source of the site from 
svn at the moment ?.


If I wan't to publish it to live I have to go via 
cms.apache.org/maven/publish as before (for now)...




Kind regards
Karl Heinz Marbaise

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: [IMPORTANT CHANGE] building and publishing our website from Jenkins and Git instead of CMS

[Karl Heinz Marbaise]

Hi,

On 12/04/18 08:17, Hervé BOUTEMY wrote:

Hi,

I'm working with infra for a good time on Git migration for site source, which
will de-facto mean that we drop Apache CMS (that only supports svn for
source): see INFRA-16088 [1]

Today, we got a first visible result: Jenkins was able to build and publish
html output of the Maven main website to svn [2]

We're now ready to switch to the new Jenkins build, then drop CMS: any
objection to do it?


No please go ahead...

+1 +1 +1...

Kind regards
Karl Heinz Marbaise


The plan will be then to switch source to GitBox, to benefit from immediate
commit detection (instead of svn hourly polling): at this step, you'll be able
to do easy site source updates through GitHub, with immediate publish. I don't
see the value of adding a staging step, but if someone wants staging, it will
be a matter of Jenkinsfile tweaking.

In the temporary time with Jenkins polling on source from svn , either you'll
wait for less than 1 hour or you'll manually launch the Jenkins job

any question?

Regards,

Hervé


[1] https://issues.apache.org/jira/browse/INFRA-16088

[2] https://builds.apache.org/view/M-R/view/Maven/job/maven-site/


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

[RESULT] [VOTE] Release Apache Maven Clean Plugin version 3.1.0

[Karl Heinz Marbaise]

Hi,

The vote has passed with the following result:

+1 : Tibor Digana, Olivier Lamy, Karl Heinz Marbaise

PMC quorum: reached.

I will promote the artifacts to the central repo.

Kind regards
Karl Heinz Marbaise

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Help with testing on OS/X 10

[Tibor Digana]

@Paul
I am not a Mac user. I only know from what the user has specified in the
Jira issue.
Feel free to participate in the Jira ticket.

On Fri, Apr 13, 2018 at 2:04 PM, Paul Hammant  wrote:

> OS/X == OS X (former OS name) == macOS (current OS name) ?
>
> 10 == 10.10.x ?
>



-- 
Cheers
Tibor

Re: Help with testing on OS/X 10

[Tibor Digana]

Hi Olivier,

The Dockerfile is in the jira comment too. Only these two lines of code in
Dockerfile:

FROM maven:3.5.3-jdk-8-alpine
COPY ./. /



On Fri, Apr 13, 2018 at 2:13 PM, Olivier Lamy  wrote:

> Hi
> Happy to help,
> But your comment says
>
> sudo docker build --no-cache -t mymvn:1 -f ./Dockerfile .
>
> where is this Dockerfile?
>
> On Fri, 13 Apr 2018 at 22:04, Paul Hammant  wrote:
>
> > OS/X == OS X (former OS name) == macOS (current OS name) ?
> >
> > 10 == 10.10.x ?
> >
>
>
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>



-- 
Cheers
Tibor

Re: Help with testing on OS/X 10

[Igor Fedorenko]

To make sure I understand, you need somebody to run that test in a Docker 
container running on OS/X host, did I get it right? The reason I ask, Docker 
runs containers in a linux virtual machine on OS/X hosts, which is quite 
different environment compared to what I _think_ the bug reporter is doing, 
i.e. running the build directly on OS/X.

FWIW, I could not reproduce the problem using your test class running directly 
on OS/X 10.13.4. I suggest you ask the reporter to provide small standalone 
example project that fails on their system. From my experience this often 
uncovers important configuration details missing from the original report.

-- 
Regards,
Igor

On Fri, Apr 13, 2018, at 4:32 AM, Tibor Digana wrote:
> Hi All,
> 
> I want to ask you to help us with testing one issue which can be reproduced
> only on OS/X 10.
> I tried to run a small test on Ubuntu 17 x86_64 but I could not reproduce
> an issue of surefire plugin.
> 
> Here is a comment in Jira issue. I described there steps I run Docker (CE
> 18.03) container with particular test. If you have a little time and you
> want to improve quality, pls have a look and try to run this test:
> 
> https://issues.apache.org/jira/browse/SUREFIRE-1502?focusedCommentId=16434550=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16434550
> 
> 
> Thx
> Tibor

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Help with testing on OS/X 10

[Olivier Lamy]

Hi
Happy to help,
But your comment says

sudo docker build --no-cache -t mymvn:1 -f ./Dockerfile .

where is this Dockerfile?

On Fri, 13 Apr 2018 at 22:04, Paul Hammant  wrote:

> OS/X == OS X (former OS name) == macOS (current OS name) ?
>
> 10 == 10.10.x ?
>


-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

Re: Help with testing on OS/X 10

[Paul Hammant]

OS/X == OS X (former OS name) == macOS (current OS name) ?

10 == 10.10.x ?

Help with testing on OS/X 10

[Tibor Digana]

Hi All,

I want to ask you to help us with testing one issue which can be reproduced
only on OS/X 10.
I tried to run a small test on Ubuntu 17 x86_64 but I could not reproduce
an issue of surefire plugin.

Here is a comment in Jira issue. I described there steps I run Docker (CE
18.03) container with particular test. If you have a little time and you
want to improve quality, pls have a look and try to run this test:

https://issues.apache.org/jira/browse/SUREFIRE-1502?focusedCommentId=16434550=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16434550


Thx
Tibor