Re: versioning by hashes to speedup multi-module build (a'la nix package manager)

[Jason Young]

Good questions. First of all, this plugin is CI-agnostic, but it does
require the project to exist in a `git` repository, whether that is in CI
or on your machine. Check the github page I linked to for more instructions
on how it determines what projects in a reactor are considered "changed"
and need to be built versus which are not changed and will be omitted from
the reactor.

In every Maven build, every dependency is checked this way:

   1. If it is a project in the reactor, use the artifact of that project.
   2. Otherwise, if the artifact is in the local repo, is that artifact.
   3. Last resort: Download from the remote repository.

There are some other rules omitted above, e.g. when to download a fresh
SNAPSHOT artifact based on your chosen snapshot policy, etc., but that's
the gist of it: Maven will obtain the artifact if it is not present, no
further configuration needed.

E.g. let's say you have one project that names 2 other projects A and B as
submodules, and A depends on B. If you run `mvn install -pl A` (NOT SURE
about that syntax), then Maven will look for B.jar from your local repo,
and resort to checking your remote repo (e.g. Maven Central) if it's not
there. But if you omit the `-pl A` part, Maven will build B, then build A
using B.jar.

Essentially, the plugin I linked to determines the project list based on
what has changed and what has not. Maven then decides whether to use
B/target/B.jar, ~/.m2/repository/.../B.jar, or to look to Maven Central.

HTH.

On Tue, Feb 4, 2020 at 4:08 PM Anton Vodonosov  wrote:

>
>
> 04.02.2020, 23:32, "Jason Young" :
> >
> > Not what you're looking for, but maybe useful: We use one plugin that
> will
> > skip whole projects that have not changed WRT a given Git branch:
> > https://github.com/vackosar/gitflow-incremental-builder. With careful
> > configuration, this is an effective shortcut without sacrificing
> > repeatability.
>
> How do you use it? I mean if you need to start full system,
> (locally or deploying it to a qa server),
> but the plugin has only built changed modules,
> how do you download the rest?
>
> Do you use this plugin in CI?
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>
>

--

Re: versioning by hashes to speedup multi-module build (a'la nix package manager)

[Jason Young]

It seems Maven itself never omits re-doing anything except for downloading
artifacts from a remote repository. The command you give to Maven and the
configuration of your projects dictates what it will do, no matter what
happened in the previous build. You _can_ omit projects in a multi-module
project if by manually specifying what projects to run:
https://blog.sonatype.com/2009/10/maven-tips-and-tricks-advanced-reactor-options/

Not what you're looking for, but maybe useful: We use one plugin that will
skip whole projects that have not changed WRT a given Git branch:
https://github.com/vackosar/gitflow-incremental-builder. With careful
configuration, this is an effective shortcut without sacrificing
repeatability.

Gradle advertises as a feature that it will not rebuild if rebuilding is
not required, or something to that effect. I assume some configuration
required sometimes.

On Tue, Feb 4, 2020 at 1:57 PM Anton Vodonosov  wrote:

> Ha, only after completing the script (even though a slow one)
> I discovered that maven rebuilds modules even if
> an artifact of the same version already exists in artifact
> repository.
>
> I hoped maven, in case a non -SNAPSHOT artifact
> found in an artifact repository will just use it
> and won't build the same version of a module again.
>
> Is there a way to tell maven to do so?
>
> Best regards,
> - Anton
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>
>

--

Re: Using Maven repos from https endpoint in sbt(scala) projects

[Jason Young]

I believe this is the info you need:
https://www.alphabot.com/security/blog/2020/java/Your-Java-builds-might-break-starting-January-13th.html

On Thu, Jan 16, 2020 at 2:47 PM Ankur Shukla 
wrote:

> Hi
>
> I have a sbt project, the builds for which started failing today because
> the maven repos seem to have been moved to https endpoint. I’m not able to
> find how I can correctly get my project to pull maven dependencies from
> https endpoint. I raised a detailed question on stackoverflow (
>
> https://stackoverflow.com/questions/59763435/access-maven-repo-over-https-in-sbt
> )
> Can someone please help me with this.
>
> --
> Thanks & Regards
> Ankur Shukla
> Software Engineer, MindTickle
> (P) +91- 9764795832
>


--

Re: Building aggregate module using -pl option

[Jason Young]

Does the Maven documentation actually say anywhere that the
project-submodule relationship and the parent-child relationship should be
in any way related? I don't see any reason that one should imply the other;
however, I gave up long ago on trying to have children that aren't
submodules of their parents because some plugins would not work as
documented in that case.

On Fri, Dec 27, 2019 at 5:59 AM Kypros Chrysanthou 
wrote:

> Hi,
>
> I do agree that this is an awkward setup (that not all sibling modules are
> pointing to the same parent).
> I also agree that this might be a bug or missed use-case.
>
> What I was trying to do is create a Jenkins job that will only build
> affected modules (which will be passed to -pl).
> It could happen that these modules are aggregators (why not all sub-module
> projects might define the aggregator as parent,
> is an issue to be investigated separately).
>
> I would expect to have a way (an explicit option, not necessarily the
> default functionality) to build the sub-modules as well.
> Basically a way to configure "mvn  -pl ." to be equivalent to "mvn
> ".
>
> Regards,
> Kypros Chrysanthou
>
>
> On Fri, Dec 27, 2019 at 12:30 PM Robert Scholte 
> wrote:
>
> > Hi,
> >
> > apart from the awkward project setup (moduleC not pointing to the same
> > parent), it looks like a potential bug to me.
> > What Maven does is looking for the dependencies of the listed projects.
> > You're pointing to the root pom, which has only a dependency on junit,
> not
> > on any of the multimodule projects.
> > Next it will look more modules that use any of the listed projects.
> > ModuleA and ModuleB have a parent pointing to the root.
> >
> > The missing step seems to be that their dependencies should be respected
> > too.
> >
> > In general you would use -pl to point to an artifact providing project,
> > not an aggregator, but it might be useful.
> > So -pl ModuleA or -pl ModuleB or -pl ModuleC all work as expected.
> >
> > thanks,
> > Robert
> > On 27-12-2019 08:25:42, Kypros Chrysanthou  wrote:
> > Hello,
> >
> > I'm sorry for double-posting.
> > I had created the other issue through the "Start new thread" option in
> the
> > Pony Mail! page and I thought it got dropped somehow.
> >
> > If you would be so kind as to look into this, please use the thread
> > titled "Building
> > multi-module project using -pl":
> >
> >
> https://lists.apache.org/thread.html/5590fe115ea792f86c114744efa7f1edf1b2988d0f231249098b95f4%40%3Cusers.maven.apache.org%3E
> >
> > Kind regards,
> > Kypros Chrysanthou
> >
> >
> > On Thu, Dec 26, 2019 at 6:36 PM Kypros Chrysanthou
> > wrote:
> >
> > > Hello,
> > >
> > > I am trying to build an a aggregate module using the -pl option but I
> > > don't seem to be able to configure the build to pick up the sub-modules
> > > when they are not child projects.
> > >
> > > When a sub-module is a child project it can be picked up using the -amd
> > > option.
> > >
> > > I created a dummy project for this at
> > > https://github.com/chkypros/aggregation
> > >
> > > My assumption is that the sub-modules should have been picked up when
> > > using the -am option but this doesn't seem to be the case.
> > >
> > > Any advice/thought about why this is so or what I'm thinking wrong
> would
> > > be greatly appreciated.
> > >
> > > Kind regards,
> > > Kypros
> > >
> > > -
> > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> > > For additional commands, e-mail: users-h...@maven.apache.org
> > >
> > >
> >
>


--

Re: mvn versions:display-plugin-updates says I can update from 3 to 2

[Jason Young]

Thanks! I finally tried looking at this. This works for me.

Side note: We use the enforcer plugin only when we enable a certain profile
because of MENFORCER-306. We can't downgrade before that defect because
of MENFORCER-268. 306 is fixed in 3.0.0-M3, which is not in Maven Central
yet. I forgot to enable the profile.

On Sat, Sep 7, 2019 at 4:37 AM John Patrick  wrote:

> Technically you didn't do anything wrong...
> If you don't configure what version of java and maven your project
> requires, then versions will tell you about all options available.
>
> So that is saying if you want users to build using maven 2.0.1 then
> the maven-war-plugin needs to be 2.0.2.
>
> If you add something like below which says I can run on maven 3.6.0 or
> newer, I can also only run on java 1.8, or java 11 or newer, e.g. not
> java 9 and not java 10. Then when you re-run versions older versions
> of maven won't be shown. Also check your using versions 2.7
> 
>   org.apache.maven.plugins
>   maven-enforcer-plugin
>   3.0.0-M2
>   
> 
>   enforce-requirements
>   
> enforce
>   
>   
> 
>   
> [1.8,9),[11,)
>   
>   
> [3.6.0,)
>   
>     
>   
> 
>   
> 
>
> Hope that helps.
>
> John
>
> On Fri, 6 Sep 2019 at 16:37, Jason Young 
> wrote:
> >
> > Did I do something wrong?
> >
> > $ mvn versions:display-plugin-updates
> > ...
> > [INFO] Require Maven 2.0.1 to use the following plugin updates:
> > [INFO]   maven-war-plugin ... 3.2.3 ->
> 2.0.2
> >
> > --
> >
> > Jason Young
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>
>

Re: Proper way to build a Maven repository without Internet access

[Jason Young]

Sean, last time I checked, Maven does not handle concurrent Maven processes
using the same local repository--the kind that the Maven client sets up.

What does work that I know of is using a company-wide repo like Nexus or
Artifactory. IME, issues with maven-metadata.xml, etc. do not come up in
this configuration--I don't ever think about it myself. Each client will
still have its own local repository, but will go to Nexus or Artifactory
for their artifacts. Optionally, you can let Nexus or Artifactory fetch
(once) all the third-party artifacts you need from Maven Central. With your
CI server and all devs using the organization's central artifact repo,
everyone is using the same artifacts as each other all the time.

More specifically, `group:project:1.2.3` is conventionally just one
artifact, not possibly a couple of different artifacts, as enforced by
artifact repos like Nexus, Artifactory, and Maven Central. Once someone
uploads a non-SNAPSHOT artifact to your company's artifact repository, it
never changes (assuming you configure the repository according to
convention). Certain things you can add on to the version like `-SNAPSHOT`
indicate mutability.

Someone please correct any misconceptions / outdated info on my part.

On Mon, Nov 11, 2019 at 2:59 PM Sean Horan  wrote:

> The developers can use whatever repos on the Internet they feel is
> appropriate, but it is up to the team lead to sync that up to the build
> server which does not have access to the public Internet.
>
> Syncing the local repository that exists now on the build server with
> Artifactory is where I'm blocked because maven is looking for
> maven-metadata.xml files that I don't have and I don't understand the
> difference between maven-metadata.xml, maven-metadata-central.xml, and
> maven-metadata-mycompanyname.xml that are in the folder.
>
> Sean
>
> On Sun, Nov 10, 2019 at 11:31 PM Robert Scholte 
> wrote:
>
> > If you don't have a repository manager in place to serve your developers
> > (that is, bound to the internet), then I consider that as a huge risk.
> > Developers will find other ways to get their libraries, which means you
> > can't know anymore if they came from a valid and secure location.
> > A repository manager should be the only place that acts as the single
> > point of trust. Only this way you can control all incoming (and
> optionally
> > outgoing) traffic.
> >
> > Robert
> > On 8-11-2019 00:22:28, Sean Horan  wrote:
> > Hi all,
> >
> > I am tasked with ensuring that the Maven build process of a large
> > government/enterprise-class system does not reach out to the Internet.
> Our
> > Jenkins server's local maven repository has 10,000 POMs. There are many
> > individual builds that are specific to our product and what we customize
> > for government clients.
> >
> > I have a lot of devops experience but practically no experience with
> Maven
> > and Java beyond struggling to set this up.
> >
> > We are using Artifactory and I'm not sure whether a generic or
> > Maven-specific repository is suitable for this project.
> >
> > As I'm trying to understand it, I am using curl in a find/curl loop
> adapted
> > from
> >
> >
> https://github.com/jfrog/project-examples/blob/master/bash-example/deploy-folder-by-checksum.sh
> >
> > to traverse the ~/.m2/repository on our existing Jenkins server and HTTP
> > PUT it over to Artifactory. This script would be hardened and sent to
> > internal customers to sync as part of the development process.
> >
> > The problem I am seeing is that the build process is looking for
> > maven-metadata.xml which does not exist on our server. We do have
> > -companyname and -central XML files for eg, the maven-source-plugin that
> > are slightly different.
> >
> > I have the sense that my approach to this is off and I'm in over my head
> so
> > I could use some help.
> >
> > Any pointers in the right direction would be more than welcome.
> >
> > We are using Maven 3.3.9 and JDK8 on Centos 7 and cannot upgrade at this
> > time.
> >
> > Sean Horan
> >
>


-- 
Jason Young
Systems Engineer | TEVERA
[image: Phone] 715 245 8000 x7609
[image: Mobile] 715 781 0845
[image: Web] tevera.com
Confidentiality Notice: This message is intended for the sole use of the
individual and entity to which it is addressed, and may contain information
that is privileged, confidential and exempt from disclosure under
applicable law. Any unauthorized review, use, disclosure or distribution of
this email message, including any attachment, is prohibited. If you are not
the intended recipient, please advise the sender by reply email and destroy
all copies of the original message.

Re: Using Maven outside of L/M/WAMP environment

[Jason Young]

These are good questions.

Two concepts to keep separate at all times are building the application and
running the application. Tools and environments used for one are not
necessarily present in the other. For example, Maven is used to make an
application, and a MySql DB is usually used by a running application.

OTOH, to simplify things, there are some entities you will use at both
times. For example, a Java .jar file contains both instructions which the
java executable can convert to machine code, and metadata that programs
like Maven can read to know how to fit them into a larger program. Also,
whatever is needed at runtime is _useful_ at development time for testing
the application, as the computer can run the code faster and more
accurately than any human can read the code and imagine what will happen
when it runs.

A MySql database usually serves the running application. Typically, Java
programs access relational databases (typical SQL databases, but not all
databases) with a JDBC driver. MySql is one such relational SQL database,
and there are MySql JDBC drivers.

Maven is not related to MySql in any specific way, but is a tool for
building an application (and need not be present when _running_ an
application or connecting to a database). There are JDBC drivers, including
MySql drivers, conveniently available in Maven Central (which you can
search at https://search.maven.org/). I imagine the example you saw
probably had a `` section in a `pom.xml` file that included a
MySql driver as a dependency--at least that's a typical way to tell Maven
that our program needs or _depends on_ that mysql driver.

So with that dependency declared, our program can now use that mysql driver
and successfully build--specifically, javac can compile the source files
that reference the classes and interfaces in the MySql driver, and make a
.jar or .war based on it. Also, with a few further steps I won't go into
now, you can run that .jar or .war, and the running program will try to
connect to a database as you dictate in your program.

HTH

On Wed, Oct 30, 2019 at 4:27 PM New_Tech  wrote:

> I am a new programmer and have started the “on paper” design of my first
> project.  While reading a tutorial on linking a Java program to a MySQL DB,
> I was introduced to Maven as part of the conduit between Java and the DB. I
> have a couple of questions about this.
>
> 1. Is Maven an absolute requirement to allow communication between those
> two
> entities?
>
> 2. Can Maven be used outside of a *AMP environment? I may be
> misinterpreting
> the use of Maven in this way but with it being a part of the Apache
> Project,
> I’m assuming a *AMP server is required to create a system with Maven as a
> part of it.
>
> If this is the wrong place for this post please direct me to the correct
> area. Any helpful resources to understand Maven’s purpose and necessity are
> much appreciated.
>
> Thank you
>
>
>
> --
> Sent from: http://maven.40175.n5.nabble.com/Maven-Users-f40176.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>
>

--

Re: Quicker local maven builds article

[Jason Young]

This is definitely a helpful thing to be able to do. I wish Maven would
make some effort to keep up with what jobs it's already done and doesn't
need to repeat vs. what it has left to do. This is a fundamentally
automatable job that is very difficult for people to do correctly, so
everyone just `mvn clean install` on the root project every time, or avoids
using Maven from the command line altogether.

Similar tool: https://github.com/vackosar/gitflow-incremental-builder -
works well overall for my team's workflow, thought I don't think devs have
played with it; it's only used in CI for certain jobs right now.

Note that some plugins at some versions don't work correctly if you skip
some projects, e.g. we mitigate
https://issues.apache.org/jira/browse/MENFORCER-306 by ensuring the root
pom is in the build and disable maven-enforcer-plugin by default on dev
machines; another way to avoid the problem is to use any previous version
of enforcer (we need the latest because of
https://issues.apache.org/jira/browse/MENFORCER-268).

(Replying here as I'm not a Reddit user - sorry!)

On Sun, Oct 20, 2019 at 11:44 AM Paul Hammant  wrote:

> https://www.reddit.com/r/Maven/comments/dklz1e/quicker_local_maven_builds/
>
> ^ A small script to help you have quicker Maven invocations for changes in
> your (Git) checkout on your dev workstation.
>
> Also, a reminder that there is a sub-reddit for Maven now -
> https://www.reddit.com/r/Maven/
>
> - Paul
>

Re: Same snapshot deploy number for entire build - possible

[Jason Young]

Dan,

Thank you for clarifying the problem more. I think I understand better now.

It sounds like there are really 2 problems:
1. QA needs projects built from the same revision.
2. When the projects are built from the same revision, no one can verify
that easily because the build numbers are different.

Is that correct?

For the first problem, I think Tibor's example of using a CI pipeline to
use the artifacts only if the whole reactor succeeds is what will work for
most people (it's what my team does, in effect, just with a different CI
tool).

On Sun, Sep 15, 2019 at 5:35 PM Dan Tran  wrote:

> Please keep in mind QA is not a maven user.  All download/deployment and
> testing are mostly automated, however, when come to reporting they have to
> use something concrete which is the exact snapshot downloaded for them
>
> They also come to accept that each artifact has its own buildNo/snapshotNo,
> but the same conversation keeps repeating about version inconsistency.  (
> if you add Jenkins build number it does get worse).  My take here is to
> find a way to unify the snapshotNum for the entire build, then the ongoing
> problem will be fixed
>
> Speaking of implementation, I dig a litter deeper by debugging thru maven
> deploy, but my IDE (eclipse) step debugger is out of syn ( maven deploy
> 2.8.2 --> artifact-transfier-->maven-resolver)
> so I dont have much luck getting more understanding of the code.  Basically
> I am looking for the exact place where Maven calculate the next snapshot
> number
>
> Thanks
>
> -D
>
>
>
> On Sun, Sep 15, 2019 at 1:57 PM Tibor Digana 
> wrote:
>
> > Hi Francois and Dan,
> >
> > I understood it the same way as Francois mentioned. Not sure if NN in the
> > format "artifactId-version-timestamp-NN" is a bug. Who cares is probably
> > someone who downloads the artifact manually, maybe the QA.
> > Also downloading the artifacts from Nexus never was so trivial for QA and
> > development team.
> > There's Nexus Proffessional and the Staging Repos but I found it useful
> > only in OSS developers.
> >
> > What was found acceptable was the solution where the QA does not traverse
> > the path of groupId/artifactId in the repo, and it was a simple button in
> > GUI and automatic deployment. Something where nobody downloads the
> artifact
> > directly.
> > My colleagues like integrating release candidates and not the Snapsthots
> > because Snapshots are very volatile and the RCs are positively deployed
> > ones or two times a month - rarely.
> >
> > If the Snapshots must be downloaded every day then the question is why
> such
> > processes exist in the company or what's wrong with the project
> structures,
> > and what is missing in the automatic deployment.
> >
> > Cheers
> > Tibor17
> >
> >
> >
> > On Sun, Sep 15, 2019 at 10:15 PM Francois Marot <
> francois.ma...@gmail.com>
> > wrote:
> >
> > > I'm sorry to insist but nevertheless I insist ;). I may have
> > misunderstood
> > > the problem but as I understand it, the whole problem can be sumed up
> by:
> > > " the fact that the repository adds a number at the end of the deployed
> > > files is a problem because not all artifacts deployed from the same
> > reactor
> > > may not get the same suffix/number if some previous build failed half
> > way".
> > > I still do not see how it is a problem as it is only a problem if one
> > wants
> > > to get each files manually from the filesystem. In this case he does
> not
> > > know the *right* number for each artifact.
> > > But if the QA want to download all the artifacts belonging to the
> latest
> > > build, all he has to do is create a pom.xml referencing all those
> > artifact,
> > > use a variable as the version number and use the dependency plugin (
> > >
> > >
> >
> https://maven.apache.org/plugins/maven-dependency-plugin/copy-dependencies-mojo.html
> > > ) to retrieve the artifacts locally.
> > >
> > > This way, what you see as a problem is hust an internal implementation
> > > detail of Maven/repository that you can circumvent easily.
> > >
> > > Again, I may have misunderstood so please excuse me if I'm talking
> > nonsense
> > > but I thought it could help.
> > >
> > > Regards
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *- - - - -François Marot06 50 91 96 38*
> > >
> > >
> > >
> > > On Thu, 12 Sep 2019 at 04:07, Dan Tran  wrote:
> > >
> > > > Hello Maven Users and Development Team
> > > >
> > > > Currently, artifact deployed as snapshot at Maven repository has the
> > > > following format
> > > >
> > > >  artifactId-version-timestamp-NN
> > > >
> > > > where NN auto-incremented at each maven module and the number varies
> > > >
> > > > Is there a way to use same snapshot NN for the entire multi-module
> > maven
> > > > build?
> > > >
> > > > If I have to implement a solution, would it be as an extension or I
> > have
> > > to
> > > > tinker with maven-deploy-plugin?
> > > >
> > > > Very appreciated any advice
> > > >
> > > > -D
> > > >
> > >
> >
>


--

Re: Same snapshot deploy number for entire build - possible

[Jason Young]

Dan,

Why are people asking about anything after `SNAPSHOT` in the version
number, i.e. why do they care? Are discrepancies causing build failures
somehow?

My team's projects depend on each other with `SNAPSHOT` versions but
without specifying a build number or timestamp, e.g.
`1.2.3-SNAPSHOT`. This, combined with a policy to always
get the latest SNAPSHOT (just like the `-U` argument only applied
implicitly), makes every build use the latest SNAPSHOT of any dependencies
it does not build from source. Nobody ever asks me about timestamps or
build numbers that e.g. Nexus keeps up with.

Regarding the policy or the -U argument, IMO these are necessary for
correct builds because the default policy, last I checked, is to download
the latest SNAPSHOT only if Maven hasn't checked in the last 24 hours,
which I find bewildering.

Again, you can define the version _once for all projects_ in a parent
pom.xml. Let us know if that makes sense or if it is not an option for some
reason.

HTH. Sorry if I still don't understand the problem.

On Fri, Sep 13, 2019 at 6:22 PM Dan Tran  wrote:

> @ Tamás Cservenák you read my mind thanks
>
> The problem I am facing is, out of a few hundred artifacts deployed every
> build.
> There is a handful of artifacts that landed on QA side. By having the same
> 'buildNo/snapshotNo'
>  ( ie artifactId-version-timestamp-buildNo.xx)  it is much easier to
> communicate,
> otherwise, it is a constant problem explaining why they are not the same
>
> This is an ongoing problem at my workplace.
>
> deployAtEnd never work for us
> https://issues.apache.org/jira/projects/MDEPLOY especially this one
> https://issues.apache.org/jira/projects/MDEPLOY/issues/MDEPLOY-226
>
> We can switch to release build but that is not an option since we would run
> out  of storage quickly and have to purge ourselves
>
> The question here what is the best way to implement an option asking
> maven-deploy-plugin to use the same provided 'buildNo' ? possible?
>
> Thanks
>
> -D
>
>
>
>
>
>
>
>
>
> On Fri, Sep 13, 2019 at 1:53 PM Francois Marot 
> wrote:
>
> > OK, I get the "problem" but in fact I do not think this is a problem.
> > How those files (with the number suffix) are named, whether you deploy
> (to
> > a remote repo) or install (to a local folder), is just an implementation
> > detail. You should never end up accessing those files by their name.
> > Let me explain: if you want to get all the 3 latest artifacts, whatever
> the
> > file name and the timestamp version, you can just create a pom having
> those
> > 3 artifacts declared as dependencies and use maven dependency plugin (
> >
> https://maven.apache.org/plugins/maven-dependency-plugin/plugin-info.html
> > )
> > to copy those artifacts in a directory. Use the "-U" flag to make sure to
> > get the latest version. This way, maven will care about the file names,
> but
> > you will not.
> >
> > I hope I can help
> >
> >
> >
> >
> >
> > On Fri, 13 Sep 2019 at 22:42, Tamás Cservenák 
> wrote:
> >
> > > Howdy,
> > >
> > > so what Dan is asking for is I think the following thing:
> > > On multi module snapshot deploy, the last bit of snapshot timestamped
> > > version is "buildNo".
> > >
> > > Consider following scenario: you have a 3 module reactor build:
> > > i) you deploy first time: the buildNo is -1
> > > ii) you deploy second time, and 2nd module UT fails: result in repo is
> > 1st
> > > module has -2, while the rest, as module2 failed, is not deployed
> > > iii) you deploy third time: the build No is -3 for 1st module, while it
> > is
> > > -2 for the rest.
> > >
> > > All sounds great ("as it should be"), but buildNo has "fall apart". 3rd
> > > deployment will have snapshots that will have their version like
> > > 1.0-mmdd.hhmmss-3 while the rest version is 1.0-mmdd.hhmmss-2.
> > >
> > > Determining buildNo is possible only from maven metadata.xml, that may
> be
> > > overkill. Nothing logically "groups" the two version above, even if
> they
> > > were deployed at same time, by the same build.
> > >
> > > Deploy at end may help here, yes, but personally, I dislike the fact
> how
> > > buildNo is calculated (md get, buildNo++ jar put,, modified md put).
> > >
> > > Also, this topic seems to me, like somewhat related to the overall
> issue
> > of
> > > https://issues.apache.org/jira/browse/MNG-6581
> > >
> > > Thanks,
> > > T
> > >
> > >
> > >
> > > On Fri, Sep 13, 2019 at 8:32 PM Francois Marot <
> francois.ma...@gmail.com
> > >
> > > wrote:
> > >
> > > >  can you tell us a little bit more about your problem please ?
> > > > You say "Currently, artifact deployed as snapshot at Maven repository
> > has
> > > > the following format: artifactId-version-timestamp-NN ". Do youmean
> the
> > > > filename in the repo ? If so, why is it a problem for you ?
> > > > Share more with us and I think we'll suggest solution more adapted
> for
> > > your
> > > > use case  ;)
> > > >
> > > > Regards,
> > > > Francois
> > > >
> > > >
> > > >
> > > > On Thu, 12 

Re: Same snapshot deploy number for entire build - possible

[Jason Young]

You never established that you're talking about the projects' versions, as
in the part between the  tags. Is that what you're talking about?
If so, did you say the artifactId is part of the version? Can you take
artifactId out of the version and make all projects in your reactor have
the same version, not just the same incremented part?

You can ensure that the whole version is the same throughout the reactor
most easily by:
1. making a parent pom for all projects to inherit from to define things
common to all projects, including the version.
2. inheriting from that pom from all modules.
3. Don't define  in any modules inheriting from that parent.
4. Adjust your version-incrementing script or Maven plugin accordingly.

NOTE that IME not all plugins, including official ones, understand the
difference between a parent pom and a multi-module pom, so to ensure
compatibility, please _make the multi-module pom also the parent of its
modules_.

HTH.

On Fri, Sep 13, 2019 at 5:50 AM Thomas Broyer  wrote:

> Maybe you'd want to deployAtEnd?
>
> https://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html#deployAtEnd
>
> On Fri, Sep 13, 2019 at 12:03 PM Dan Tran  wrote:
>
> > Looks like I need to clear thing a little bit
> >
> > Assume I have a reactor of few hundreds of maven modules and my CI
> > build deploys snapshots, first few good builds, each module deployed to
> > maven repository have same snapshot number
> >
> > Once we encounter a build failure at a module,  the snapshot number
> begins
> > to vary and over time, all modules no longer have the same snapshot
> number
> > anymore
> >
> > To be able to determine a consistent snapshot number at each build,  I
> > would like to have an ability the tell deploy maven plugin to use an
> > assigned number for all modules.
> >
> > Possible?
> >
> > Thanks
> >
> > -D
> >
> >
> >
> >
> >
> >
> > On Thu, Sep 12, 2019 at 11:31 PM Enrico Olivelli 
> > wrote:
> >
> > > Dan,
> > > Are you running a single 'mvn deploy' or do you have multiple runs?
> > > I have never seen weird behaviours in multi module projects
> > >
> > > Cheers
> > > Enrico
> > >
> > > Il ven 13 set 2019, 08:19 Dan Tran  ha scritto:
> > >
> > > > Hello, Maven dev
> > > >
> > > > any suggestion/thoughts on this issue are very much appreciated
> > > >
> > > > Regards
> > > >
> > > > -D
> > > >
> > > > On Wed, Sep 11, 2019 at 7:06 PM Dan Tran  wrote:
> > > >
> > > > > Hello Maven Users and Development Team
> > > > >
> > > > > Currently, artifact deployed as snapshot at Maven repository has
> the
> > > > > following format
> > > > >
> > > > >  artifactId-version-timestamp-NN
> > > > >
> > > > > where NN auto-incremented at each maven module and the number
> varies
> > > > >
> > > > > Is there a way to use same snapshot NN for the entire multi-module
> > > maven
> > > > > build?
> > > > >
> > > > > If I have to implement a solution, would it be as an extension or I
> > > have
> > > > > to tinker with maven-deploy-plugin?
> > > > >
> > > > > Very appreciated any advice
> > > > >
> > > > > -D
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Thomas Broyer
> /tɔ.ma.bʁwa.je/  <
> http://xn--nna.ma.xn--bwa-xxb.je/>
>


--

mvn versions:display-plugin-updates says I can update from 3 to 2

[Jason Young]

Did I do something wrong?

$ mvn versions:display-plugin-updates
...
[INFO] Require Maven 2.0.1 to use the following plugin updates:
[INFO]   maven-war-plugin ... 3.2.3 -> 2.0.2

-- 

Jason Young

Re: Problem with using Parent POMs

[Jason Young]

One thing to remember that makes Maven make sense is this: As you would
expect, Maven doesn't scan your whole SSD for pom.xml files or jars, but
rather looks in the _current reactor_ (look up multi-module projects for
details), then in your private repo, and finally in whatever external repo
you have configured (e.g. maybe your company's Nexus repo, or Maven Central
by default). It looks up dependency artifacts by group id, artifact id,
version, and a few other things (not always present because they have
default values). Just because a project is present on your filesystem as
part of a multi-project doesn't mean it will be part of the build; you have
to tell Maven what you want.

As a convenience, you may optionally provide a parent pom filepath,
relative to the path of the child pom, to tell Maven to go ahead and read
that file when building the child instead of looking to the repo, etc., but
the parent's GAV coordinates are required in case that file isn't there,
i.e. you're always allowed to use means outside of Maven to _inject_
dependencies of various kinds: so-called "dependencies", plugin
dependencies, and parents. (But no one does that because changing files to
inject dependencies is a pain.)

Beware that the parent-child relationship is one concept, but the
multi-module / reactor idea is something entirely different, even though
many Maven users and even some official plugins conflate the two. It is
highly conventional therefore to make a multi-module project also the
parent of its modules (though it's also conventional to have a parent pom
without modules).

On Wed, Sep 4, 2019 at 9:02 AM Nelligan, Steven M 
wrote:

>
>
> This information and a little tweaking, I got it to work.
>
>
>
> Thank you for your help.
>
>
>
> *Steven M Nelligan*
> *APPLICATION DEVELOPER*
>
>
> Facilities and Services
>
> Information Technology Services
> University of Illinois at Urbana-Champaign
> Facilities and Services
> 1501 S. Oak Street | M/C 800
> Champaign, IL 61820
> 217.244.8097 | *other*: 217.244.8097 | snell...@illinois.edu
> www.fs.illinois.edu
>
> [image: facebook]
> [image:
> twitter] [image: instagram]
> [image: youtube]
> 
>
> [image: https://webtools.illinois.edu/webservices/js/ds/signature_logo.png]
> 
>
> *Under the Illinois Freedom of Information Act any written communication
> to or from university employees regarding university business is a public
> record and may be subject to public disclosure.*
>
>
>
> *From:* Andy Feldman 
> *Sent:* Tuesday, September 3, 2019 11:05 PM
> *To:* Maven Users List 
> *Subject:* Re: Problem with using Parent POMs
>
>
>
> If I interpreted this setup correctly, the easiest solution would be to
> rename ParentPom.xml to just Pom.xml, and change the version of the
> dependency from 0.0.1-20190903.165620-1 to 0.0.1-SNAPSHOT, to match what's
> in the parent pom. That should be sufficient, since the parent folder is
> the default location to check for the parent pom, and Pom.xml is the
> default filename. Alternatively, I would expect it to work if you specified
> ../ParentPom.xml.
>
>
>
> This is covered in the documentation for Project Inheritance here:
> https://maven.apache.org/guides/introduction/introduction-to-the-pom.html#Project_Inheritance
>
>
>
> Where did you get the version 0.0.1-20190903.165620-1 from? Is the parent
> artifact already published to a Maven repo? In that case, you would instead
> want , which tells it to look in a repo for the parent.
>
>
>
> Hope that helps,
>
> --
>
> Andy Feldman
>
>
>
> On Tue, Sep 3, 2019 at 4:42 PM Nelligan, Steven M 
> wrote:
>
>
>
> I am trying implement a parent POM in my project tree.
>
> Project Tree:
>
> C:/
>
> Dev/
>
> Projects/
>
>ParentPom.xml
>
>
> Dor-enterprise/ this level>
>
>Pom.xml
>
>Src/
>
>
>
>
>
> Attached is the parentPom, pom, and the log of the execution when I run
> from Dor-enterprise
>
>
>
> C:\dev\Projects\dor-enterprise>mvn -PvDev CLEAN INSTALL SITE -e -X
>
>
>
> I have never tried to use parent pom’s before and may be trying to use
> them incorrectly.
>
>
>
> Any help would be apprecrated,
>
> *Steven M Nelligan
> *
> *APPLICATION DEVELOPER*
>
>
> Facilities and Services
>
> Information Technology Services
> University of Illinois at Urbana-Champaign
> Facilities and Services
> 1501 S. Oak Street | M/C 800
> Champaign, IL 61820
> 217.244.8097 | *other*: 217.244.8097 | snell...@illinois.edu
> www.fs.illinois.edu
>
> [image: facebook]
> [image:
> twitter] [image: instagram]
> 

Re: Maven Eclipse error

[Jason Young]

AFAIK there is no fix to prevent eclipse from failing to download files
sometimes--seems to do this regularly with big download batches.

>From the command line, `cd  && mvn
dependency:go-offline`

Right-click your project in eclipse -> Maven -> Update
Update it


On Fri, Aug 2, 2019 at 3:01 PM Lakshmikanth Manubolu(UST,IN)
 wrote:

> Hi Team,
>
>
>
> I’m trying to create a maven java project and while configuring the
> pom.xml file I could able to see the below errors:
>
> Please try to help me to resolve the issues.
>
>
>
> Error1:
>
> Failure to transfer
> org.apache.maven.plugins:maven-resources-plugin:pom:2.6 from
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not
>
>  be reattempted until the update interval of central has elapsed or
> updates are forced. Original error: Could not transfer artifact
> org.apache.maven.plugins:maven-resources-
>
> plugin:pom:2.6 from/to central (https://repo.maven.apache.org/maven2):
> connect timed out
>
>
>
> Error2:
>
> Multiple annotations found at this line:
>
> - Plugin execution not covered by lifecycle configuration:
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (execution:
> default-testCompile, phase: test-
>
> compile)
>
> - Plugin execution not covered by lifecycle configuration:
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (execution:
> default-compile, phase: compile)
>
> - CoreException: Could not calculate build plan: Plugin
> org.apache.maven.plugins:maven-compiler-plugin:3.1 or one of its
> dependencies could not be resolved: Failed to
>
>  read artifact descriptor for
> org.apache.maven.plugins:maven-compiler-plugin:jar:3.1:
> ArtifactResolutionException: Failure to transfer
> org.apache.maven.plugins:maven-compiler-
>
> plugin:pom:3.1 from https://repo.maven.apache.org/maven2
> was cached in the local repository, resolution will not be reattempted
> until the update interval of central has elapsed
>
>  or updates are forced. Original error: Could not transfer
> artifact org.apache.maven.plugins:maven-compiler-plugin:pom:3.1 from/to
> central (https://repo.maven.apache.org/
>
> maven2): connect timed out
>
>
>
> Attached is the pom.xml file.
>
>
>
> Thanks,
>
> Lakshmikanth Manubolu
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org

Re: Maven question - how to pull code from bitbucket repository as dependency

[Jason Young]

On Wed, Jun 12, 2019 at 1:04 AM Anders Hammar  wrote:

> Having a dependency to some other scm repo is not a good approach. If that
> changes your build could fail all of a sudden and you want consistency.
>

This is not true. The "big 3" VCSes each allow you to checkout a specific
tag or revision if you like. If both projects (the dependent and the
dependency) use the same VCS, then, at least in the cases of Git and SVN,
you have a VCS-specific feature, e.g. Git subrepos or SVN externals, to
work with the dependency's repo pretty seamlessly (though admittedly their
use is controversial, but let's not go on a tangent about it :) ).


> Your Maven project should typically be self contained. Any dependency
> should be a Maven artifact (that is immutable).
>

It is certainly preferable to make one Maven project depend on another when
possible, but keep in mind there are by necessity other kinds of
dependencies too: The JDK, system-level libraries not provided by some
JVMs, a SQL database, etc. You _can_ mavenize a lot of dependencies (like a
SQL database project) and gain Maven's dependency management, but that is
not invariably worth the cost of implementing and maintaining that
mavenization.

/Anders
>
> On Wed, Jun 12, 2019 at 7:50 AM Rajesh Deshpande <
> rajesh.deshpa...@gmail.com>
> wrote:
>
> > Hello,
> > I have a maven project that builds Java jar files. I wan to run a python
> > script as one of the goals in the verify phase. I am planning to use the
> > ant run plugin for running the python script. The python script is stored
> > in a separate repository that I would like to copy to my project root
> > folder using maven. Is this possible? What's the best way to approach
> this?
> >
> > Thanks!
> >
>

Re: Maven local repo in a common global directory for multiple parallel execution

[Jason Young]

Bernd, I believe the OP is talking about using the -T argument to build
projects concurrently, not running concurrent Jenkins builds. However, I
would still absolutely use per-executor repos before having more than one
concurrent Maven build, whether its the same job or different Maven jobs.

On Mon, May 27, 2019 at 5:41 AM Bernd Eckenfels 
wrote:

> Hello,
>
> It is the other way around, there might be situations where one job relies
> on the installed artifacts from the other, in this case you would need a
> shared Repo. It is however bad style. In all other cases you can use a
> executor- or even workspace-local local repository if you have enough space.
>
> The advantage of avoiding concurrent access is that you get less
> influences from one job to the other (especially if you work with snapshots
> or multiple different upstream artifacts with same version in different
> Repos) and that there is no concurrent downloads (which in itself sometimes
> leads to checksum errors).
>
> Gruss
> Bernd
> --
> http://bernd.eckenfels.net
>
> 
> Von: Debraj Manna 
> Gesendet: Montag, Mai 27, 2019 11:29 AM
> An: Maven Users List
> Betreff: Re: Maven local repo in a common global directory for multiple
> parallel execution
>
> Yes disk required will be more .
>
> I don't have a requirement where one of my projects requires artifact
> created by another. So I can use a common local repo for all parallel
> executions?
>
> On Mon, May 27, 2019 at 11:34 AM Golan, Yaron 
> wrote:
>
> > Please remember that you need much bigger disk space as you are
> > duplicating binaries.
> >
> >
> > -Original Message-
> > From: Debraj Manna 
> > Sent: Sunday, May 26, 2019 4:49 PM
> > To: Maven Users List 
> > Subject: Maven local repo in a common global directory for multiple
> > parallel execution
> >
> > I have a machine in which multiple parallel maven execution happen . Each
> > execution executes the below command in a seperate workspace directory
> >
> > mvn -f main/pom.xml clean package -DskipTests -T 6
> >
> > Can someone let me know should I use a seperate maven local repo path (
> > -Dmaven.repo.local=$MAVEN_REPO) for each execution or I can use a common
> > .m2 directory for all parallel runs? What is the best practice?
> >
> >
> > - Maven Version 3.5
> > - Java 8
> >
>

Re: Maven adjusts transitive compile-scope dependency to test-scope

[Jason Young]

I think I understand now. You're saying MyProject requires X for testing
and Y for compiling, but Y also requires X for compiling, thus X is a
transitive compile-time (or at least runtime) dependency of MyProject,
incidentally (not directly). Maven is not interpreting your input that way.

You mean to tell Maven that MyProject requires X _in addition to_ other
dependencies specified, but Maven interprets that as _overriding_ or
_replacing_ the transitive dependency. If I understand correctly, your
original post was saying that seems worng or at least needlessly opens the
door to mistakes which a different design could prevent. E.g. if Maven
required you to use an exclusion to remove a dependency instead of allowing
you to quietly override a dependency with a weaker scope, then you could
_just_ specify what a project needs directly, and Maven would figure out
the details or fail the build if what you ask for has an ambiguity with no
safe, meaningful default resolution. Hope that makes sense.

On Wed, May 15, 2019 at 12:42 AM Simon Taddiken 
wrote:

> In my case, the pom of Y explicitly declares X with scope compile. The
> dependency tree should look something like this:
>
> MyProject <- this is a standalone application, that will not be depended on
> by s/o else
> |- X:test
> |- Y:compile
>\- X:compile
>
> So for MyProject, X is explicitly declared test but X also comes in
> transitively via Y as compile. I do understand that declarations higher in
> the tree are given precedence when it comes to dependency resolution but in
> this case it simply breaks runtime code.
> Of course I did not want to ship MyProject including X when I used X only
> for tests. But now that Y also requires X I need MyProject to ship with X
> in order to work correctly.
>
> I was just wondering whether there is some mechanism like a warning or a
> flag to break the build in such cases, so that they do not go unnoticed.
>
>
> Am Di., 14. Mai 2019 um 22:22 Uhr schrieb Jason Young <
> jason.yo...@procentive.com>:
>
> > Did you declare that Y depends on X at all (via Y's pom.xml) or did it
> > figure that out on its own (via transitive dependencies)?
> >
> > Test scope means it's for testing that one project, thus it's not
> > transitive, e.g. I don't need JUnit for my tests just because
> > SomeAwesomeProject uses JUnit for its tests, and I don't want to ship
> JUnit
> > in my project.
> >
> > On Tue, May 14, 2019 at 2:40 PM Simon Taddiken  >
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I've encountered the following behavior and I'm not quite sure whether
> it
> > > is desirable.
> > > In my project, I have declared a dependency *X* with scope *test*. I
> then
> > > updated the version of a 3rd party dependency *Y*. In its new version,
> > *Y*
> > > suddenly requires the aforementioned dependency *X* as a *compile
> *scoped
> > > dependency.
> > >
> > > In this scenario, maven resolves the scope of *X* to be *test,
> *although
> > it
> > > is now required by *Y* during runtime, causing ClassNotFoundExceptions.
> > By
> > > the very nature of this behavior those mistakes can't even be detected
> in
> > > unit tests because *X* is available on test classpath.
> > >
> > > I have just found out that this behavior seems to be intended (
> > >
> > >
> >
> https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
> > > )
> > > but I can't really come up with a rationale for this design. Managing
> any
> > > transitive compile scoped dependency down to test scope will almost
> > > certainly cause ClassNotFoundExceptions during runtime.
> > >
> > > Thoughts?
> > >
> >

Re: Maven adjusts transitive compile-scope dependency to test-scope

[Jason Young]

Did you declare that Y depends on X at all (via Y's pom.xml) or did it
figure that out on its own (via transitive dependencies)?

Test scope means it's for testing that one project, thus it's not
transitive, e.g. I don't need JUnit for my tests just because
SomeAwesomeProject uses JUnit for its tests, and I don't want to ship JUnit
in my project.

On Tue, May 14, 2019 at 2:40 PM Simon Taddiken 
wrote:

> Hi everyone,
>
> I've encountered the following behavior and I'm not quite sure whether it
> is desirable.
> In my project, I have declared a dependency *X* with scope *test*. I then
> updated the version of a 3rd party dependency *Y*. In its new version, *Y*
> suddenly requires the aforementioned dependency *X* as a *compile *scoped
> dependency.
>
> In this scenario, maven resolves the scope of *X* to be *test, *although it
> is now required by *Y* during runtime, causing ClassNotFoundExceptions. By
> the very nature of this behavior those mistakes can't even be detected in
> unit tests because *X* is available on test classpath.
>
> I have just found out that this behavior seems to be intended (
>
> https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
> )
> but I can't really come up with a rationale for this design. Managing any
> transitive compile scoped dependency down to test scope will almost
> certainly cause ClassNotFoundExceptions during runtime.
>
> Thoughts?
>

Re: Concurrency issue while running Maven on Jenkins host

[Jason Young]

IME, the default settings for plugins in Jenkins are sometimes objectively
incorrect. One example is the setting Karl referred to. More specifically,
if you are using the Maven plugin for Jenkins, got to your job config ->
Build -> Advanced -> Use private Maven repository. The default setting of
using one repo for all jobs is only correct if you never run more than one
Maven job at a time. Instead, set it to "Local to the executor" to avoid
concurrent access and maximize cache reuse, or "Local to the workspace" for
more isolation. (My Jenkins installation might be old.)

On Thu, May 9, 2019 at 9:33 AM Karl Heinz Marbaise 
wrote:

> Hi,
>
> On 09.05.19 16:09, Frizz wrote:
> >   I regularly suffer from corrupted maven-metadata-local.xml files on my
> > Jenkins host in directory /home/jenkins/.m2/.../some-project/
>
>
> I suppose you are using the local cache for all your jobs?
>
> If so this is the problem. The cache is and was intended for running a
> single build on it...If you run on a CI solution like Jenkins you should
> use the cache per job ...
>
> Kind regards
> Karl Heinz Marbaise
>
>
> >
> > E.g. extra lines added to the end of the maven-metadata-local.xml file
> like
> > this:
> > ...
> >
> > 
> > astUpdated>
> >
> > 
> >
> > Do I suffer from concurrency issues? Like the one described here (created
> > 2007, but still unresolved):
> https://issues.apache.org/jira/browse/MNG-2802
> >
> > What could I do to mitigate the issue?
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>
>

Re: Importing pom in dependency management

[Jason Young]

   - A multi-module project may contain other projects as submodules, so
   it's a _composite project_ consisting of itself non-recursively and of its
   submodules. That is, if I run `mvn install` on BigProject, then Maven will
   start a _reactor build_ that runs `mvn install` on BigProject itself
   non-recursively, and also runs `mvn install` on each sub-module
   _recursively_ (that is, also running their submodules, if any).
   - A project may have a parent from which it inherits stuff.

These are 2 separate relationships, or at least they used to be in previous
versions of Maven. That is, there can be a parent-child relationship
without a submodule relationship (consider that the official super project
does not contain your project as a sub-module), and a multi-module project
need not be the parent of its sub-project. A project could even declare its
own parent as a sub-module IIRC.

This distinction would be awfully useful in some cases, e.g. if you want to
change parents or sub-modules dynamically with a separate repo. However,
some plugins, even in the`org.apache.maven.plugins`groupId, _implicitly
assume_ that every sub-module will declare the multi-module project which
it is part of as a parent, and I would expect things to break if you
venture far from common Maven tutorials.

On Sat, Apr 27, 2019 at 9:20 PM Shashank Tiwari <
anddowncamether...@gmail.com> wrote:

> Okay , I understood that 1st point is for sub module and 2nd is for child
> module. But don;t sub module build before the parent ? In that case , won't
> maven be able to locate the pom as in 1st point ?
> Also, there is a 3rd point , which I need help in understanding : (from the
> same doc)
>
>
>
>- When referring to artifacts whose poms have transitive dependencies
>the project will need to specify versions of those artifacts as managed
>dependencies. Not doing so will result in a build failure since the
>artifact may not have a version specified. (This should be considered a
>best practice in any case as it keeps the versions of artifacts from
>changing from one build to the next).
>
> What does this mean?
>
> On Sun, Apr 28, 2019 at 7:32 AM Shashank Tiwari <
> anddowncamether...@gmail.com> wrote:
>
> > Reading from the doc
> > <
> https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
> >
> >  ,
> >
> >
> >1.
> >
> >Do not attempt to import a pom that is defined in a submodule of the
> >current pom. Attempting to do that will result in the build failing
> since
> >it won't be able to locate the pom.
> >2.
> >
> >Never declare the pom importing a pom as the parent (or grandparent,
> >etc) of the target pom. There is no way to resolve the circularity
> and an
> >exception will be thrown.
> >
> > What is difference between the 2 points ? Are not they same ? Also, in
> > point 1 , why would maven not be able to locate pom ?
> >
>

Re: Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

Thanks.

I finally took a little time to look at why the `ps` command would not
work. For our project, we cannot use a default Maven image, so I made a
custom one based on alpine. One piece of software the Maven alpine image
has is `pcprops`; adding that fixes the problem:

$ docker run -it alpine sh
> / # apk update
> ...
> OK: 8445 distinct packages available
> / # apk upgrade
> ...
> OK: 4 MiB in 13 packages
> / # ps -o etime= -p 1
> ps: unrecognized option: p
> BusyBox v1.26.2 (2018-05-30 13:51:20 GMT) multi-call binary.
> Usage: ps [-o COL1,COL2=HEADER]
> Show list of processes
> -o COL1,COL2=HEADER Select columns for display
> / # apk add procps
> ...
> OK: 12 MiB in 19 packages
> / # ps -o etime= -p 1
>   05:21


Hope that helps anyone else using a custom-built alpine-based Maven image.

I'm not sure if I'll have time to experiment much with Surefire changes in
the near future; my focus at the moment is figuring out all the reasons our
test suite sometimes takes 5 or more times as long as normal (often also
causes Surefire to kill the child process, but even if it didn't, we have a
problem!).

On Sun, Apr 7, 2019 at 2:04 AM Tibor Digana  wrote:

> Hi Jason,
>
> I am sorry I haven't reply.
> Excellent analysis!
>
> Can you make an experiment and use pingJob on a "fixed delay" instead of a
> "fixed rate"?
> If you want to make more, feel free.
>
> We should mabe think about how we would add more intelligence in these
> timers in order to detect the "blackout".
>
> Cheers
> Tibor
>
>
>
>
> On Fri, Mar 22, 2019 at 9:53 PM Jason Young 
> wrote:
>
> > Notice:
> > * an over 3-minute gap between the first two entries above--neither the
> > command reader thread nor the pingJob had a chance to run.
> > * the command reader thread and the ping job executor evidently get a
> > chance to run from 19:18:02 to 19:19:19. They ran with increased
> frequency
> > during that time to "catch up": The command reader thread is programmed
> to
> > read from stdin until there's nothing left to read, and the pingScheduler
> > runs the pingJob at a "fixed" rate.
> > * Another 3-minute gap and another attempt to catch up at 19:22:28. This
> > time though, the command reader thread evidently did not get a turn in
> > time.
> >
> > I did not bother to log what threads the ping job was running under; I
> can
> > do that later. I also did not add any logging on the Maven process side;
> I
> > can do that as well.
> >
> > This is on a VM with unthrottled access to 4 processor cores and 8GB
> RAM. I
> > think the code under test is CPU- and RAM-intensive. There probably were
> GC
> > pauses, but I don't have proper logging for that in place.
> >
> > Note that this is all very normal behavior: Both the command reader and
> the
> > ping job's thread (run by the pingScheduler) are instructed to run until
> > their work is done, not to sleep or yield. For high throughput, an OS
> will
> > favor continuing to run a thread for a while once it is started over
> > switching tasks, so it's normal both that each thread would run for a
> while
> > after a pause, and (by extension) that each would have long pauses as
> other
> > threads had to run, too. Because of the long gaps between runs of the
> > command reader thread, my previous advice (schedule the pingJob on a
> "fixed
> > delay" instead of a "fixed rate") would not have saved the build in this
> > case (though "fixed delay" is more correct and would save the build in
> > other possible cases). Our code base has some unit tests that rely on
> > "normal case" timing, but they now have the @Ignore attribute because
> they
> > would rarely, but too often, fail. For most teams and projects, unit
> tests
> > have to be more reliable than that.
> >
> > So I think we need a solution that does not rely on preemptive scheduling
> > from the OS or on thread scheduling, but which ensures that commnads have
> > been read before running code depending on commands. There are multiple
> > ways to do that; maybe I can find time to write a solution later.
> >
> >
> > On Fri, Mar 22, 2019 at 3:46 PM Jason Young 
> > wrote:
> >
> > > Thanks Tibor. I'll use `encodeAndWriteToOutput` later. For now, I
> hacked
> > > together some temporary changes to ForkedBooter so that it "logs" (just
> > > `System.out.println`) the current time every time:
> > >
> > >- the command reader thread reads a NOOP via stdin,
> > >- it starts the ping

Re: Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

Notice:
* an over 3-minute gap between the first two entries above--neither the
command reader thread nor the pingJob had a chance to run.
* the command reader thread and the ping job executor evidently get a
chance to run from 19:18:02 to 19:19:19. They ran with increased frequency
during that time to "catch up": The command reader thread is programmed to
read from stdin until there's nothing left to read, and the pingScheduler
runs the pingJob at a "fixed" rate.
* Another 3-minute gap and another attempt to catch up at 19:22:28. This
time though, the command reader thread evidently did not get a turn in time.

I did not bother to log what threads the ping job was running under; I can
do that later. I also did not add any logging on the Maven process side; I
can do that as well.

This is on a VM with unthrottled access to 4 processor cores and 8GB RAM. I
think the code under test is CPU- and RAM-intensive. There probably were GC
pauses, but I don't have proper logging for that in place.

Note that this is all very normal behavior: Both the command reader and the
ping job's thread (run by the pingScheduler) are instructed to run until
their work is done, not to sleep or yield. For high throughput, an OS will
favor continuing to run a thread for a while once it is started over
switching tasks, so it's normal both that each thread would run for a while
after a pause, and (by extension) that each would have long pauses as other
threads had to run, too. Because of the long gaps between runs of the
command reader thread, my previous advice (schedule the pingJob on a "fixed
delay" instead of a "fixed rate") would not have saved the build in this
case (though "fixed delay" is more correct and would save the build in
other possible cases). Our code base has some unit tests that rely on
"normal case" timing, but they now have the @Ignore attribute because they
would rarely, but too often, fail. For most teams and projects, unit tests
have to be more reliable than that.

So I think we need a solution that does not rely on preemptive scheduling
from the OS or on thread scheduling, but which ensures that commnads have
been read before running code depending on commands. There are multiple
ways to do that; maybe I can find time to write a solution later.


On Fri, Mar 22, 2019 at 3:46 PM Jason Young 
wrote:

> Thanks Tibor. I'll use `encodeAndWriteToOutput` later. For now, I hacked
> together some temporary changes to ForkedBooter so that it "logs" (just
> `System.out.println`) the current time every time:
>
>- the command reader thread reads a NOOP via stdin,
>- it starts the ping job,
>- it checks for NOOP (`canUseNewPingMechanism(..)` returns `false`),
>and
>- it would have exited because there is no NOOP (`!pingDone`).
>
> I also commented out the code to exit in that case so it keeps running and
> logging.
>
> The resulting output starts out showing that it's receiving ping every 10s
> and checking for NOOP every 30s, as expected. But then one large test
> begins, and:
>
> DEBUG 2019/03/22 19:14:40.223 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:02.723 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.139 GMT ForkedBooter: Running ping job (i.e.
> checking if we read a ping earlier).
> DEBUG 2019/03/22 19:18:03.172 GMT ForkedBooter: Checking for NOOP from
> Maven process.
> DEBUG 2019/03/22 19:18:03.186 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.187 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.187 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.187 GMT ForkedBooter: Running ping job (i.e.
> checking if we read a ping earlier).
> DEBUG 2019/03/22 19:18:03.188 GMT ForkedBooter: Checking for NOOP from
> Maven process.
> DEBUG 2019/03/22 19:18:03.187 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.243 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.307 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.308 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.202 GMT ForkedBooter: Running ping job (i.e.
> checking if we read a ping earlier).
> DEBUG 2019/03/22 19:18:03.323 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.337 GMT ForkedBooter: Checking for NOOP from
> Maven process.
> DEBUG 2019/03/22 19:18:03.351 GMT ForkedBooter: Received ping from Maven
> process.
> DEBUG 2019/03/22 19:18:03.351 GMT ForkedBooter: Running ping job (i.e.
> checking if we read a ping earlier).
> DEBUG 2019/03/22 19:18:03.351 GMT ForkedBooter: Received pi

Re: Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

DEBUG 2019/03/22 19:22:28.698 GMT ForkedBooter: Running ping job (i.e.
checking if we read a ping earlier).
DEBUG 2019/03/22 19:22:28.698 GMT ForkedBooter: Checking for NOOP from
Maven process.
WARN 2019/03/22 19:22:28.698 GMT ForkedBooter: Did not receive NOOP from
Maven process.
DEBUG 2019/03/22 19:22:28.698 GMT ForkedBooter: Running ping job (i.e.
checking if we read a ping earlier).
DEBUG 2019/03/22 19:22:28.698 GMT ForkedBooter: Checking for NOOP from
Maven process.
WARN 2019/03/22 19:22:28.698 GMT ForkedBooter: Did not receive NOOP from
Maven process.
DEBUG 2019/03/22 19:22:28.699 GMT ForkedBooter: Running ping job (i.e.
checking if we read a ping earlier).
DEBUG 2019/03/22 19:22:28.699 GMT ForkedBooter: Checking for NOOP from
Maven process.
WARN 2019/03/22 19:22:28.699 GMT ForkedBooter: Did not receive NOOP from
Maven process.
DEBUG 2019/03/22 19:22:28.699 GMT ForkedBooter: Running ping job (i.e.
checking if we read a ping earlier).
DEBUG 2019/03/22 19:22:28.699 GMT ForkedBooter: Checking for NOOP from
Maven process.
WARN 2019/03/22 19:22:28.699 GMT ForkedBooter: Did not receive NOOP from
Maven process.
DEBUG 2019/03/22 19:22:28.699 GMT ForkedBooter: Running ping job (i.e.
checking if we read a ping earlier).
DEBUG 2019/03/22 19:22:28.699 GMT ForkedBooter: Checking for NOOP from
Maven process.
WARN 2019/03/22 19:22:28.699 GMT ForkedBooter: Did not receive NOOP from
Maven process.
DEBUG 2019/03/22 19:22:28.699 GMT ForkedBooter: Running ping job (i.e.
checking if we read a ping earlier).
DEBUG 2019/03/22 19:22:28.699 GMT ForkedBooter: Checking for NOOP from
Maven process.
WARN 2019/03/22 19:22:28.699 GMT ForkedBooter: Did not receive NOOP from
Maven process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.703 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.704 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.704 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.704 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.704 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.704 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.704 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.704 GMT ForkedBooter: Received ping from Maven
process.
DEBUG 2019/03/22 19:22:28.704 GMT ForkedBooter: Received ping from Maven
process.
...

With these changes, a project which normally takes 15 to 20 minutes to run
actually took 80m. Would printing directly to stdout from within a test do
that somehow? If not, we have a problem in our code.

My comments on the above report to follow...

On Fri, Mar 22, 2019 at 11:58 AM Tibor Digana 
wrote:

> >> is there an -Xmx value you would recommend to ensure the parent process
> can send the NOOP to the surefire process?
>
> no such value is generic value to recommend. Each application or test must
> find our the value to prevent from OOM.
>
>
> >> I'm making a custom build of surefire-booter to workaround the initial
> problem by commenting out the code to exit.
>
> Very good step forward. Then post your findings and we can continue on
> making a pull request on GitHub.
>
>
> >> Is there a logging API which I can use instead?
>
> Yes, it is very low level of logging (sending events to parent process) but
> this will be turned to high level abstraction in several days (branch
> surefire-1222_2).
> Yet, pls use this function:
>
> encodeAndWriteToOutput()
>
>
> >>
>
>
>
>
>
> On Fri, Mar 22, 2019 at 4:44 PM Jason Young 
> wrote:
>
> > To clarify, this is the image ours is based on:
> > https://hub.docker.com/_/alpine; we are not using a vanilla Maven image
> as
> > we have to add a few other items to this image as well. I don't know who
> > maintains that or adds /etc/mavenrc; I'll bring that up in another topic
> in
> > another forum. I was just answering a previous question, and also
> throwing
> > out info for anyone struggling with e.g. OO

Re: Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

To clarify, this is the image ours is based on:
https://hub.docker.com/_/alpine; we are not using a vanilla Maven image as
we have to add a few other items to this image as well. I don't know who
maintains that or adds /etc/mavenrc; I'll bring that up in another topic in
another forum. I was just answering a previous question, and also throwing
out info for anyone struggling with e.g. OOME in their Maven process (a lot
of people have that problem when running the Sonar scanner, for example). I
don't think we need to worry about why that file is added for this topic.

But while we're on the subject, is there an -Xmx value you would recommend
to ensure the parent process can send the NOOP to the surefire process?

I'm making a custom build of surefire-booter to workaround the initial
problem by commenting out the code to exit. I am also adding logging via
stdout so I can see if ForkedBooter is indeed running the ping job in small
increments of time. Is there a logging API which I can use instead?

On Fri, Mar 22, 2019 at 5:44 AM Tibor Digana  wrote:

> The base images are developed in https://github.com/carlossg/docker-maven,
> right?
> Who creates "/etc/mavenrc"?
>
> On Thu, Mar 21, 2019 at 12:05 AM Jason Young 
> wrote:
>
> > Mikael, sorry I do not appear to have permission to view the link.
> >
> > I did some digging in the last couple of days. I see that the parent
> > process reads from stdin. I could not find anywhere that we are using
> > stdin. FWIW the failures nearly always happen at least 15m into a ~20m
> test
> > run, so perf is a likely culprit.
> >
> > I see also that ForkedBooter reads commands from stdin in one thread, and
> > uses an executor service to check for a past ping in
> > ForkedBooter.listenToShutdownCommands(..). When it checks, it also sets
> > pingDone to false. The executor is configured to run up to 2 threads
> > concurrently to handle the workload, and is set to run at a fixed rate
> (not
> > a fixed delay). If the test suite is busy with testing and GC and has
> lots
> > of threads running, it's entirely possible that a thread won't have a
> > chance to run for a long time (e.g. 5s). Maybe instead of a 30s delay,
> the
> > VM gets around to checking for a ping every 35s over a long span of time.
> > Because we're running at a "fixed rate" and not a "fixed delay", then
> after
> > a couple of minutes we might be a full 30s behind schedule. It's possible
> > the executor will create another thread to run the scheduled task because
> > it's running behind schedule. This new thread checks for a ping, finds
> it,
> > and sets pingDone to false. But then the original thread also runs, say,
> 2
> > seconds afterwards, checks pingDone, and finds it is false.
> >
> > So to mitigate the problem, can we a) make the executor run only 1 thread
> > and b) schedule the task at a fixed rate? For that matter, is there
> another
> > scheduled executor we can reuse? I understand why checking for ping
> > requires a separate executor. Should I ask in github?
> >
> > Regarding a previous question, I found out that Alpine's Maven package
> > comes with an /etc/mavenrc that sets `MAVEN_OPTS="$MAVEN_OPTS -Xmx512m"`
> > which cannot be undone by setting `MAVEN_OPTS` at the command line; you
> end
> > up with e.g. `-Xmx1g -Xmx512m`. (Note this applies to the Maven (parent)
> > process, not the surefire/failsafe (child) process.)
> >
> > On Wed, Mar 20, 2019 at 3:46 AM Bernd Eckenfels 
> > wrote:
> >
> > > I guess a timeout caused by FullGC can happen with TCP as well.
> > Increasing
> > > the timeout might not be nice but does look like it would help in both
> > > cases. (Problems with stdout are more related to unexpected JVM
> messages
> > I
> > > guess)
> > >
> > > Gruss
> > > Bernd
> > > --
> > > http://bernd.eckenfels.net
> > >
> > > 
> > > Von: Mikael Åsberg 
> > > Gesendet: Mittwoch, März 20, 2019 9:40 AM
> > > An: Maven Users List
> > > Betreff: Re: Failsafe: Killing self fork JVM. PING timeout elapsed.
> > >
> > > These issues regarding communication with forked JVMs, won't they be
> > > resolved once surefire moves to interprocess communication using
> > > tcp/ip sockets? This happens to be the target feature to be included
> > > in the next surefire 3.0.0 milestone:
> > > https://issues.apache.org/jira/projects/SUREFIRE/versions/12344668
> > >
> > > There are s many issues relating to surefire reading stdout of
> > 

Re: Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

Mikael, sorry I do not appear to have permission to view the link.

I did some digging in the last couple of days. I see that the parent
process reads from stdin. I could not find anywhere that we are using
stdin. FWIW the failures nearly always happen at least 15m into a ~20m test
run, so perf is a likely culprit.

I see also that ForkedBooter reads commands from stdin in one thread, and
uses an executor service to check for a past ping in
ForkedBooter.listenToShutdownCommands(..). When it checks, it also sets
pingDone to false. The executor is configured to run up to 2 threads
concurrently to handle the workload, and is set to run at a fixed rate (not
a fixed delay). If the test suite is busy with testing and GC and has lots
of threads running, it's entirely possible that a thread won't have a
chance to run for a long time (e.g. 5s). Maybe instead of a 30s delay, the
VM gets around to checking for a ping every 35s over a long span of time.
Because we're running at a "fixed rate" and not a "fixed delay", then after
a couple of minutes we might be a full 30s behind schedule. It's possible
the executor will create another thread to run the scheduled task because
it's running behind schedule. This new thread checks for a ping, finds it,
and sets pingDone to false. But then the original thread also runs, say, 2
seconds afterwards, checks pingDone, and finds it is false.

So to mitigate the problem, can we a) make the executor run only 1 thread
and b) schedule the task at a fixed rate? For that matter, is there another
scheduled executor we can reuse? I understand why checking for ping
requires a separate executor. Should I ask in github?

Regarding a previous question, I found out that Alpine's Maven package
comes with an /etc/mavenrc that sets `MAVEN_OPTS="$MAVEN_OPTS -Xmx512m"`
which cannot be undone by setting `MAVEN_OPTS` at the command line; you end
up with e.g. `-Xmx1g -Xmx512m`. (Note this applies to the Maven (parent)
process, not the surefire/failsafe (child) process.)

On Wed, Mar 20, 2019 at 3:46 AM Bernd Eckenfels 
wrote:

> I guess a timeout caused by FullGC can happen with TCP as well. Increasing
> the timeout might not be nice but does look like it would help in both
> cases. (Problems with stdout are more related to unexpected JVM messages I
> guess)
>
> Gruss
> Bernd
> --
> http://bernd.eckenfels.net
>
> 
> Von: Mikael Åsberg 
> Gesendet: Mittwoch, März 20, 2019 9:40 AM
> An: Maven Users List
> Betreff: Re: Failsafe: Killing self fork JVM. PING timeout elapsed.
>
> These issues regarding communication with forked JVMs, won't they be
> resolved once surefire moves to interprocess communication using
> tcp/ip sockets? This happens to be the target feature to be included
> in the next surefire 3.0.0 milestone:
> https://issues.apache.org/jira/projects/SUREFIRE/versions/12344668
>
> There are s many issues relating to surefire reading stdout of
> forked processes (which is my understanding that it is currently
> doing). Many of us are really looking forward to the next milestone.
>
> On Tue, Mar 19, 2019 at 8:59 PM Jason Young 
> wrote:
> >
> > Getting back to my original questions, I know that "ping" means to see
> if a
> > process is there, and "NOOP" implies it's not a command to do anything.
> But
> > what do the terms "ping" and "NOOP" mean in this context, i.e. how do the
> > processes communicate? I assume they don't sonar. Do other processes also
> > ping NOOPs? Can I PING Chrome with a NOOP from bash? Is it with TCP?
> >
> > I'm confused about what I should do regarding GC pauses. Previously I had
> > code that would write the amount of remaining heap space (or something
> like
> > that) to stdout after every test to troubleshoot OOMEs. Can writing to
> > stdout cause the communication failure somehow?
> >
> > On Wed, Mar 13, 2019 at 5:57 PM Jason Young 
> > wrote:
> >
> > > I upgraded failsafe and surefire to 3.0.0-M3 as advised; we encountered
> > > the same exception. (Still using -Xmx5g, will switch to OpenJ9 soon in
> case
> > > that helps.)
> > >
> > > BTW I also asked on StackOverflow previously, for anyone interested:
> > >
> https://stackoverflow.com/questions/54755846/killing-self-fork-jvm-ping-timeout-elapsed
> > >
> > > On Tue, Feb 26, 2019 at 6:40 PM Jason Young <
> jason.yo...@procentive.com>
> > > wrote:
> > >
> > >> Thanks again for the information.
> > >>
> > >> We had increased the RAM to 3g some time ago to prevent OOMEs. More
> > >> recently, I increased the RAM again to 5g for extra headroom since we
> had
> > >> mor

Re: Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

Getting back to my original questions, I know that "ping" means to see if a
process is there, and "NOOP" implies it's not a command to do anything. But
what do the terms "ping" and "NOOP" mean in this context, i.e. how do the
processes communicate? I assume they don't sonar. Do other processes also
ping NOOPs? Can I PING Chrome with a NOOP from bash? Is it with TCP?

I'm confused about what I should do regarding GC pauses. Previously I had
code that would write the amount of remaining heap space (or something like
that) to stdout after every test to troubleshoot OOMEs. Can writing to
stdout cause the communication failure somehow?

On Wed, Mar 13, 2019 at 5:57 PM Jason Young 
wrote:

> I upgraded failsafe and surefire to 3.0.0-M3 as advised; we encountered
> the same exception. (Still using -Xmx5g, will switch to OpenJ9 soon in case
> that helps.)
>
> BTW I also asked on StackOverflow previously, for anyone interested:
> https://stackoverflow.com/questions/54755846/killing-self-fork-jvm-ping-timeout-elapsed
>
> On Tue, Feb 26, 2019 at 6:40 PM Jason Young 
> wrote:
>
>> Thanks again for the information.
>>
>> We had increased the RAM to 3g some time ago to prevent OOMEs. More
>> recently, I increased the RAM again to 5g for extra headroom since we had
>> more headroom available; the problem hasn't happened since, but it hasn't
>> been very long.
>>
>> We use a more customized image based on Alpine 3.8.2. The JDK and Maven
>> are obtained via apk.
>>
>> I will try upgrading failsafe (and surefire while I'm at it) sooner, and
>> probably do some experimentation with JVMs another time (not pressing for
>> me ATM).
>>
>> On Tue, Feb 26, 2019 at 12:20 PM Tibor Digana 
>> wrote:
>>
>>> >> I'll try to enable some logging about GC pauses to see what's up
>>>
>>> Pls do not keep such setting after tuning the GC because this may
>>> sometime
>>> break the interprocess communication between Maven process and surefire
>>> process.
>>> It's worth to list GC information in a file and not in the console logs.
>>> This can be configured, I guess.
>>>
>>> >> Do you think the value is simply too low?
>>>
>>> GCing many objects may take some time and I remember we had a user who
>>> had
>>> this problem a year or two ago.
>>> We check every third NOOP (which is 3 x 10 sec) as a fix instead of every
>>> NOP. So 30 seconds looked satisfactory.
>>> I think you use old version 2.20 or something like that. The fixes for
>>> docker have been done so far, so please use the latest version 3.0.0-M3.
>>> See this page
>>> https://maven.apache.org/surefire/maven-surefire-plugin/docker.html, we
>>> used maven:3.5.3-jdk-8-alpine in this test. Which base image did you use?
>>>
>>> Cheers
>>> Tibor
>>>
>>> On Tue, Feb 26, 2019 at 5:24 PM Jason Young 
>>> wrote:
>>>
>>> > Thanks for the information. It's good to see someone understands a
>>> little
>>> > about this.
>>> >
>>> > Incidentally, we have been looking at other GCs and VMs for the
>>> application
>>> > in production environments, so I'll look into how these affect tests as
>>> > well. I'll try to enable some logging about GC pauses to see what's up.
>>> >
>>> > How would `-Xmx3g` cause long GC cycles? Do you think the value is
>>> simply
>>> > too low?
>>> >
>>> > FWIW we're running the Maven build in an Alpine-based Docker container.
>>> >
>>> > On Sat, Feb 23, 2019 at 6:36 AM Tibor Digana 
>>> > wrote:
>>> >
>>> > > Hi Jason,
>>> > >
>>> > > We spoke about this issue on our chat in ASF Slack:
>>> > > "I think his tests have been paused for a long GC periods and timed
>>> out
>>> > 3x
>>> > > PING period = 30 seconds. After this period forked JVM supposed the
>>> Maven
>>> > > process was killed by JenkinsCI and therefore all surefire processes
>>> are
>>> > > killed as well and all the file handlers and memory consumptions are
>>> > > freed."
>>> > >
>>> > > "But I have to say that `-Xmx3g` may cause long GC cycles, see
>>> > >
>>> > >
>>> >
>>> https://maven.apache.org/surefire/maven-surefire-plugin/examples/shutdown.html
>>> > > "
>>> > >
>>> >

Re: Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

I upgraded failsafe and surefire to 3.0.0-M3 as advised; we encountered the
same exception. (Still using -Xmx5g, will switch to OpenJ9 soon in case
that helps.)

BTW I also asked on StackOverflow previously, for anyone interested:
https://stackoverflow.com/questions/54755846/killing-self-fork-jvm-ping-timeout-elapsed

On Tue, Feb 26, 2019 at 6:40 PM Jason Young 
wrote:

> Thanks again for the information.
>
> We had increased the RAM to 3g some time ago to prevent OOMEs. More
> recently, I increased the RAM again to 5g for extra headroom since we had
> more headroom available; the problem hasn't happened since, but it hasn't
> been very long.
>
> We use a more customized image based on Alpine 3.8.2. The JDK and Maven
> are obtained via apk.
>
> I will try upgrading failsafe (and surefire while I'm at it) sooner, and
> probably do some experimentation with JVMs another time (not pressing for
> me ATM).
>
> On Tue, Feb 26, 2019 at 12:20 PM Tibor Digana 
> wrote:
>
>> >> I'll try to enable some logging about GC pauses to see what's up
>>
>> Pls do not keep such setting after tuning the GC because this may sometime
>> break the interprocess communication between Maven process and surefire
>> process.
>> It's worth to list GC information in a file and not in the console logs.
>> This can be configured, I guess.
>>
>> >> Do you think the value is simply too low?
>>
>> GCing many objects may take some time and I remember we had a user who had
>> this problem a year or two ago.
>> We check every third NOOP (which is 3 x 10 sec) as a fix instead of every
>> NOP. So 30 seconds looked satisfactory.
>> I think you use old version 2.20 or something like that. The fixes for
>> docker have been done so far, so please use the latest version 3.0.0-M3.
>> See this page
>> https://maven.apache.org/surefire/maven-surefire-plugin/docker.html, we
>> used maven:3.5.3-jdk-8-alpine in this test. Which base image did you use?
>>
>> Cheers
>> Tibor
>>
>> On Tue, Feb 26, 2019 at 5:24 PM Jason Young 
>> wrote:
>>
>> > Thanks for the information. It's good to see someone understands a
>> little
>> > about this.
>> >
>> > Incidentally, we have been looking at other GCs and VMs for the
>> application
>> > in production environments, so I'll look into how these affect tests as
>> > well. I'll try to enable some logging about GC pauses to see what's up.
>> >
>> > How would `-Xmx3g` cause long GC cycles? Do you think the value is
>> simply
>> > too low?
>> >
>> > FWIW we're running the Maven build in an Alpine-based Docker container.
>> >
>> > On Sat, Feb 23, 2019 at 6:36 AM Tibor Digana 
>> > wrote:
>> >
>> > > Hi Jason,
>> > >
>> > > We spoke about this issue on our chat in ASF Slack:
>> > > "I think his tests have been paused for a long GC periods and timed
>> out
>> > 3x
>> > > PING period = 30 seconds. After this period forked JVM supposed the
>> Maven
>> > > process was killed by JenkinsCI and therefore all surefire processes
>> are
>> > > killed as well and all the file handlers and memory consumptions are
>> > > freed."
>> > >
>> > > "But I have to say that `-Xmx3g` may cause long GC cycles, see
>> > >
>> > >
>> >
>> https://maven.apache.org/surefire/maven-surefire-plugin/examples/shutdown.html
>> > > "
>> > >
>> > > You are using java-1.8-openjdk. I guess you should use Shenandoah GC
>> > which
>> > > is an experimental algorithm in  JVM 1.8. This would significantly
>> short
>> > > the GC cycles.
>> > >
>> > > We should of cource provide a new configuration parameter to give you
>> a
>> > > chance to prolong the PING.
>> > >
>> > > Cheers
>> > > Tibor
>> > >
>> >
>> >
>> > --
>> >
>> > Jason Young
>> >
>>
>
>

Re: Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

Thanks again for the information.

We had increased the RAM to 3g some time ago to prevent OOMEs. More
recently, I increased the RAM again to 5g for extra headroom since we had
more headroom available; the problem hasn't happened since, but it hasn't
been very long.

We use a more customized image based on Alpine 3.8.2. The JDK and Maven are
obtained via apk.

I will try upgrading failsafe (and surefire while I'm at it) sooner, and
probably do some experimentation with JVMs another time (not pressing for
me ATM).

On Tue, Feb 26, 2019 at 12:20 PM Tibor Digana 
wrote:

> >> I'll try to enable some logging about GC pauses to see what's up
>
> Pls do not keep such setting after tuning the GC because this may sometime
> break the interprocess communication between Maven process and surefire
> process.
> It's worth to list GC information in a file and not in the console logs.
> This can be configured, I guess.
>
> >> Do you think the value is simply too low?
>
> GCing many objects may take some time and I remember we had a user who had
> this problem a year or two ago.
> We check every third NOOP (which is 3 x 10 sec) as a fix instead of every
> NOP. So 30 seconds looked satisfactory.
> I think you use old version 2.20 or something like that. The fixes for
> docker have been done so far, so please use the latest version 3.0.0-M3.
> See this page
> https://maven.apache.org/surefire/maven-surefire-plugin/docker.html, we
> used maven:3.5.3-jdk-8-alpine in this test. Which base image did you use?
>
> Cheers
> Tibor
>
> On Tue, Feb 26, 2019 at 5:24 PM Jason Young 
> wrote:
>
> > Thanks for the information. It's good to see someone understands a little
> > about this.
> >
> > Incidentally, we have been looking at other GCs and VMs for the
> application
> > in production environments, so I'll look into how these affect tests as
> > well. I'll try to enable some logging about GC pauses to see what's up.
> >
> > How would `-Xmx3g` cause long GC cycles? Do you think the value is simply
> > too low?
> >
> > FWIW we're running the Maven build in an Alpine-based Docker container.
> >
> > On Sat, Feb 23, 2019 at 6:36 AM Tibor Digana 
> > wrote:
> >
> > > Hi Jason,
> > >
> > > We spoke about this issue on our chat in ASF Slack:
> > > "I think his tests have been paused for a long GC periods and timed out
> > 3x
> > > PING period = 30 seconds. After this period forked JVM supposed the
> Maven
> > > process was killed by JenkinsCI and therefore all surefire processes
> are
> > > killed as well and all the file handlers and memory consumptions are
> > > freed."
> > >
> > > "But I have to say that `-Xmx3g` may cause long GC cycles, see
> > >
> > >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/shutdown.html
> > > "
> > >
> > > You are using java-1.8-openjdk. I guess you should use Shenandoah GC
> > which
> > > is an experimental algorithm in  JVM 1.8. This would significantly
> short
> > > the GC cycles.
> > >
> > > We should of cource provide a new configuration parameter to give you a
> > > chance to prolong the PING.
> > >
> > > Cheers
> > > Tibor
> > >
> >
> >
> > --
> >
> > Jason Young
> >
>


-- 
Jason Young
Software Engineer | PROCENTIVE
[image: Phone] 715 245 8000 x7609
[image: Mobile] 706 870 3540
[image: Web] procentive.com
Confidentiality Notice: This message is intended for the sole use of the
individual and entity to which it is addressed, and may contain information
that is privileged, confidential and exempt from disclosure under
applicable law. Any unauthorized review, use, disclosure or distribution of
this email message, including any attachment, is prohibited. If you are not
the intended recipient, please advise the sender by reply email and destroy
all copies of the original message.

Re: Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

Thanks for the information. It's good to see someone understands a little
about this.

Incidentally, we have been looking at other GCs and VMs for the application
in production environments, so I'll look into how these affect tests as
well. I'll try to enable some logging about GC pauses to see what's up.

How would `-Xmx3g` cause long GC cycles? Do you think the value is simply
too low?

FWIW we're running the Maven build in an Alpine-based Docker container.

On Sat, Feb 23, 2019 at 6:36 AM Tibor Digana  wrote:

> Hi Jason,
>
> We spoke about this issue on our chat in ASF Slack:
> "I think his tests have been paused for a long GC periods and timed out 3x
> PING period = 30 seconds. After this period forked JVM supposed the Maven
> process was killed by JenkinsCI and therefore all surefire processes are
> killed as well and all the file handlers and memory consumptions are
> freed."
>
> "But I have to say that `-Xmx3g` may cause long GC cycles, see
>
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/shutdown.html
> "
>
> You are using java-1.8-openjdk. I guess you should use Shenandoah GC which
> is an experimental algorithm in  JVM 1.8. This would significantly short
> the GC cycles.
>
> We should of cource provide a new configuration parameter to give you a
> chance to prolong the PING.
>
> Cheers
> Tibor
>


-- 

Jason Young

Failsafe: Killing self fork JVM. PING timeout elapsed.

[Jason Young]

We have a CI job running Maven including tests. Sometimes the failsfae
process will exit, e.g.:

[INFO] Results:
[INFO]
[WARNING] Tests run: 8152, Failures: 0, Errors: 0, Skipped: 31
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-failsafe-plugin:2.22.1:verify
(integration-test) on project app_server_core: There are test failures.
[ERROR]
[ERROR] Please refer to
/builds/App/Development/App/app_server_core/target/surefire-reports for the
individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump,
[date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The
forked VM terminated without properly saying goodbye. VM crash or
System.exit called?
[ERROR] Command was /bin/sh -c cd
/builds/App/Development/App/app_server_core &&
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx3g -jar
/builds/App/Development/App/app_server_core/target/surefire/surefirebooter7662621916357034130.jar
/builds/App/Development/App/app_server_core/target/surefire
2019-01-09T21-23-07_397-jvmRun1 surefire1770987927673067492tmp
surefire_37459604808221437221tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
...

Which test was running during the crash is different every time, though we
run in alphabetical order for this job. Failsafe makes a .dump file with
e.g.:

# Created at 2019-02-12T14:31:16.410
System.exit() or native command error interrupted process checker.
java.lang.IllegalStateException: Cannot use PPID 158 process information.
Going to use NOOP events.
at
org.apache.maven.surefire.booter.PpidChecker.checkProcessInfo(PpidChecker.java:155)
at
org.apache.maven.surefire.booter.PpidChecker.isProcessAlive(PpidChecker.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter$2.run(ForkedBooter.java:214)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

# Created at 2019-02-12T14:47:59.174
Killing self fork JVM. PING timeout elapsed.

I think that is the Failsafe process telling us why it called System.exit()
prematurely, but what does that mean? What are some reasons this would
happen?

We capture the stack trace whenever System.exit() is called; I can share
that stack in a reply (don't want to get black hole'd for too long an
email). We can compare the lines of code in that stack with Surefire code
at the right version to see what Surefire is doing.

Should I ask this in the developers mailing list?

-- 

Jason Young