Re: Questions about the build cache plugin

[Olivier Lamy]

On Mon, 12 Jun 2023 at 21:38, Benjamin Marwell  wrote:
>
> Am Mo., 12. Juni 2023 um 12:08 Uhr schrieb Olivier Lamy :
>
> > On Mon, 12 Jun 2023 at 18:53, Benjamin Marwell  wrote:
> > >
> > > Hello everyone!
> > >
> > > First of all thank you everyone working on the build cache plugin! It
> > > is amazing!
> > >
> > > At least some of  the following questions seem to be of interest to
> > > most users and might end up on the documentation, So here's a few
> > > things which came to my mind.
> > >
> > > 1.) Considering I always require PRs in my projects, would setting
> > > -Dmaven.build.cache.remote.save.enabled=true be a sensible thing only
> > > for the main branch?
> >
> > Personally, I would have the main branch always have a full build
> > without caching to be sure everything works fine but use cache only
> > for branches/PR :)
> > That's what we will do at Jetty project.
> > The build for 12 branch is around 50 minutes with the cache it goes
> > down to 8/9 minutes (only because something is always rebuilt and
> > retrigger a few small modules)
> > The idea is to reduce development/check of PR and use some incremental
> > build while the main is always fully build especially when the build
> > has few jdks as target.
> > But here it's up to you :)
>
> I can see your point, but who is updating the remote cache then?
>

every PR potentially updates the cache. depending on changes per PR
some module cache will be shared or not with other PRs.
We (Jetty project) have decided to not have any publicity open remote
cache setup per default as we do not have the infra for this (but CI
use a private one)
but in a company context or if you have the infra for this it might be
a better option.

>
>
> > > 2.) Can I maybe have  the branch name included in a path to a remote?
> > > This way I could have a different cache for maintenance branches. Just
> > > include the branch name into -Dmaven.build.cache.remote.url?
> >
> > why would need that?
> > if your branch has different sources (java, pom, etc..) the calculated
> > hash will be different so the hash key will be simply different no
> > need to configure a cache URL differently (well except if you want to
> > clean up caches per branch)
>
> It depends on what you might be trying to do. For projects with new
> major versions and some
> big refactoring done in the past, this did sound sensible to me.
> What is the cache worth if there's 0% hit on the maintenance branch?
>

hit ratio will depend on the changes you do.
parent pom change -> everything will be rebuilt
single java source in a module: only this module and dependant will be rebuilt.
Finally, there is no single rule on how to use/configure the cache,
the usage depends on how a team is working, what sort of change they
often have, etc...
first, have your build working (if it's complicated one you may have
some extra configuration to add)


>
>
> > > 3.) Can I somehow use option 1 & 2 to make caches available for
> > > colleagues without making them manually configure the remote URL for
> > > each branch they are working on?
> >
> > cache will be used by colleagues' build as long as a module have the
> > same calculated hash any local differences will have a different
> > calculated hash
>
> Same as above:
> What about devs working on a maintenance branch when main has seen big
> refactoring?
>
>
> > > 4.) The docs say there's also XX and XXMM algorithms, but it doesn't
> > > say WHEN to use them (only that they may leverage performance). Are
> > > there some example cases or does someone already have some experience
> > > we could benefit from?
> >
> > I have yet to see huge differences.
> > currently with a build already down from 50 minutes to 8/9 minutes not
> > sure this was my primary goal
> > So I can't really tell :)
>
> Well, it is listed under "performance tuning", so I concluded it might
> make a difference.
> Will probably go for XX (because it is mentioned as performance tuning
> without any drawbacks)
>  and switch to Metro once it is there.
>
>
> > > 5.) The section "filtering out artifacts" might want to give a size of
> > > artifacts which should be regarded as too large, depending on the
> > > environment probably.
> > >
> >
> > definitely depends on your environment.
>
> Yup.
> I might use different settings in a corporate environment, my thought.
>
> > >
> > > -
> > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> > > For additional commands, e-mail: users-h...@maven.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> > For additional commands, e-mail: users-h...@maven.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>

Re: Multi-module Maven projects and Scala-aware artifact IDs

[Francois Marot]

Hello Florent,

i'll suggest something while I know this is not exactly what you asked for.
But i'll try nevertheless, it may help :)

If you could change your requirements a bit to change the *version* instead
of the artifactId, then you could use the
ci-friendly concept in Maven (
https://maven.apache.org/maven-ci-friendly.html ). This in conjonction with
the flatten plugin
that ensures pom.xml are correctly alterned before being deployed could
help to have something quite similar to what you asked.

So my idea is to run a command similar to:

mvn deploy -Drevision=1.2.3-2.12
or
mvn deploy -Drevision=1.2.3-2.13

The consumer of your artifacts would have to update the version as well as
the artifactId, but they are usually colocated so it may be acceptable.
This is compatible with multi-module Maven projects where each child just
reference the parent pom with the ${revision} variable.
Hope this helps

François


Le lun. 12 juin 2023 à 15:21, Florent Biville  a
écrit :

> Hello,
>
> I'm working on a multi-module Maven project where part of the artifact ID
> includes the Scala version it is built against (2.12, 2.13, ...), as it is
> common with Scala projects, since Scala minor version bumps often break (or
> so I read).
>
> I'm currently trying to fix the project setup, since it relies on dynamic
> expressions in some of the module artifact IDs to achieve the above effect
> and that does not really work (and litters the build with warnings,
> rightfully so).
>
> My current approach is to script the whole thing, something like
> "./build-for-scala.sh 2.12" or "./build-for-scala-2.13". Sprinkle some sed
> and it kinda works.
>
> However, it is very fragile. Not only must the module artifact IDs change,
> but so must any reference to the parent POM and the artifact ID of each
> dependency sibling.
>
> Reading about this problem led me to
> https://stackoverflow.com/a/35023562/277128 but the solution is not very
> compelling.
>
> And of course, since that's an established project, I cannot afford to
> change the artifact IDs before the next major version.
>
> I would not even mind writing my own plugin or extension if that could
> solve this particular dynamic artifact ID resolution.
>
> Has anyone faced this before? How did you manage to solve this problem "the
> Maven way"?
>
> Best regards,
> Florent
>

Re: Multi-module Maven projects and Scala-aware artifact IDs

[Greg Chabala]

Sure, individual projects have done things as they saw fit. Guava has -jre
and -android flavors on their artifact ID instead of using classifiers.
Bouncycastle is using ant for their build process. I wouldn't want to
emulate either as best practice.

Scala, as an ecosystem, has decided on "_binaryVersion" appended to
artifact ID, which feels gross.

The Maven POM reference ( https://maven.apache.org/pom.html ) is pretty
clear: "The classifier distinguishes artifacts that were built from the
same POM but differ in content. It is some optional and arbitrary string
that - if present - is appended to the artifact name just after the version
number. As a motivation for this element, consider for example a project
that offers an artifact targeting Java 11 but at the same time also an
artifact that still supports Java 1.8."

Binary compatibility versions are noise that should be in the classifier,
if you're building from the same source tree, it only needs one artifact ID.

Re: Multi-module Maven projects and Scala-aware artifact IDs

[Delany]

Bouncycastle does similar, e.g. bcprov-jdk15to18-173.jar

https://www.bouncycastle.org/latest_releases.html

Delany

On Mon, 12 Jun 2023 at 18:51, Greg Chabala  wrote:

> >
> > Has anyone faced this before?
> >
>
> Yes. The fact that Scala's binary incompatibility leaked into artifact IDs
> rather than being a classifier irritates me still.
>
> I have no good solution for you. The mess that would have come from using
> Maven for cross version Scala builds led me to follow the status quo of
> using sbt for Scala projects. The question you linked is a good summary of
> the compromises.
>
> My belief is that if sbt didn't exist the Scala community would have been
> more motivated to improve this, with different artifact naming conventions
> and/or improvements to Maven.
>
> Greg
>

Re: Multi-module Maven projects and Scala-aware artifact IDs

[Greg Chabala]

>
> Has anyone faced this before?
>

Yes. The fact that Scala's binary incompatibility leaked into artifact IDs
rather than being a classifier irritates me still.

I have no good solution for you. The mess that would have come from using
Maven for cross version Scala builds led me to follow the status quo of
using sbt for Scala projects. The question you linked is a good summary of
the compromises.

My belief is that if sbt didn't exist the Scala community would have been
more motivated to improve this, with different artifact naming conventions
and/or improvements to Maven.

Greg

Multi-module Maven projects and Scala-aware artifact IDs

[Florent Biville]

Hello,

I'm working on a multi-module Maven project where part of the artifact ID
includes the Scala version it is built against (2.12, 2.13, ...), as it is
common with Scala projects, since Scala minor version bumps often break (or
so I read).

I'm currently trying to fix the project setup, since it relies on dynamic
expressions in some of the module artifact IDs to achieve the above effect
and that does not really work (and litters the build with warnings,
rightfully so).

My current approach is to script the whole thing, something like
"./build-for-scala.sh 2.12" or "./build-for-scala-2.13". Sprinkle some sed
and it kinda works.

However, it is very fragile. Not only must the module artifact IDs change,
but so must any reference to the parent POM and the artifact ID of each
dependency sibling.

Reading about this problem led me to
https://stackoverflow.com/a/35023562/277128 but the solution is not very
compelling.

And of course, since that's an established project, I cannot afford to
change the artifact IDs before the next major version.

I would not even mind writing my own plugin or extension if that could
solve this particular dynamic artifact ID resolution.

Has anyone faced this before? How did you manage to solve this problem "the
Maven way"?

Best regards,
Florent

Re: Questions about the build cache plugin

[Benjamin Marwell]

Am Mo., 12. Juni 2023 um 12:08 Uhr schrieb Olivier Lamy :

> On Mon, 12 Jun 2023 at 18:53, Benjamin Marwell  wrote:
> >
> > Hello everyone!
> >
> > First of all thank you everyone working on the build cache plugin! It
> > is amazing!
> >
> > At least some of  the following questions seem to be of interest to
> > most users and might end up on the documentation, So here's a few
> > things which came to my mind.
> >
> > 1.) Considering I always require PRs in my projects, would setting
> > -Dmaven.build.cache.remote.save.enabled=true be a sensible thing only
> > for the main branch?
>
> Personally, I would have the main branch always have a full build
> without caching to be sure everything works fine but use cache only
> for branches/PR :)
> That's what we will do at Jetty project.
> The build for 12 branch is around 50 minutes with the cache it goes
> down to 8/9 minutes (only because something is always rebuilt and
> retrigger a few small modules)
> The idea is to reduce development/check of PR and use some incremental
> build while the main is always fully build especially when the build
> has few jdks as target.
> But here it's up to you :)

I can see your point, but who is updating the remote cache then?



> > 2.) Can I maybe have  the branch name included in a path to a remote?
> > This way I could have a different cache for maintenance branches. Just
> > include the branch name into -Dmaven.build.cache.remote.url?
>
> why would need that?
> if your branch has different sources (java, pom, etc..) the calculated
> hash will be different so the hash key will be simply different no
> need to configure a cache URL differently (well except if you want to
> clean up caches per branch)

It depends on what you might be trying to do. For projects with new
major versions and some
big refactoring done in the past, this did sound sensible to me.
What is the cache worth if there's 0% hit on the maintenance branch?



> > 3.) Can I somehow use option 1 & 2 to make caches available for
> > colleagues without making them manually configure the remote URL for
> > each branch they are working on?
>
> cache will be used by colleagues' build as long as a module have the
> same calculated hash any local differences will have a different
> calculated hash

Same as above:
What about devs working on a maintenance branch when main has seen big
refactoring?


> > 4.) The docs say there's also XX and XXMM algorithms, but it doesn't
> > say WHEN to use them (only that they may leverage performance). Are
> > there some example cases or does someone already have some experience
> > we could benefit from?
>
> I have yet to see huge differences.
> currently with a build already down from 50 minutes to 8/9 minutes not
> sure this was my primary goal
> So I can't really tell :)

Well, it is listed under "performance tuning", so I concluded it might
make a difference.
Will probably go for XX (because it is mentioned as performance tuning
without any drawbacks)
 and switch to Metro once it is there.


> > 5.) The section "filtering out artifacts" might want to give a size of
> > artifacts which should be regarded as too large, depending on the
> > environment probably.
> >
>
> definitely depends on your environment.

Yup.
I might use different settings in a corporate environment, my thought.

> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> > For additional commands, e-mail: users-h...@maven.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Questions about the build cache plugin

[Xy Dav]

_Dmaven build cache.remote.save.anabled=türü
Bütün yetki sende

12 Haz 2023 Pzt 13:08 tarihinde Olivier Lamy  şunu yazdı:

> On Mon, 12 Jun 2023 at 18:53, Benjamin Marwell 
> wrote:
> >
> > Hello everyone!
> >
> > First of all thank you everyone working on the build cache plugin! It
> > is amazing!
> >
> > At least some of  the following questions seem to be of interest to
> > most users and might end up on the documentation, So here's a few
> > things which came to my mind.
> >
> > 1.) Considering I always require PRs in my projects, would setting
> > -Dmaven.build.cache.remote.save.enabled=true be a sensible thing only
> > for the main branch?
>
> Personally, I would have the main branch always have a full build
> without caching to be sure everything works fine but use cache only
> for branches/PR :)
> That's what we will do at Jetty project.
> The build for 12 branch is around 50 minutes with the cache it goes
> down to 8/9 minutes (only because something is always rebuilt and
> retrigger a few small modules)
> The idea is to reduce development/check of PR and use some incremental
> build while the main is always fully build especially when the build
> has few jdks as target.
> But here it's up to you :)
>
> >
> > 2.) Can I maybe have  the branch name included in a path to a remote?
> > This way I could have a different cache for maintenance branches. Just
> > include the branch name into -Dmaven.build.cache.remote.url?
>
> why would need that?
> if your branch has different sources (java, pom, etc..) the calculated
> hash will be different so the hash key will be simply different no
> need to configure a cache URL differently (well except if you want to
> clean up caches per branch)
>
> >
> > 3.) Can I somehow use option 1 & 2 to make caches available for
> > colleagues without making them manually configure the remote URL for
> > each branch they are working on?
>
> cache will be used by colleagues' build as long as a module have the
> same calculated hash any local differences will have a different
> calculated hash
>
> >
> > 4.) The docs say there's also XX and XXMM algorithms, but it doesn't
> > say WHEN to use them (only that they may leverage performance). Are
> > there some example cases or does someone already have some experience
> > we could benefit from?
>
>
> I have yet to see huge differences.
> currently with a build already down from 50 minutes to 8/9 minutes not
> sure this was my primary goal
> So I can't really tell :)
>
> >
> > 5.) The section "filtering out artifacts" might want to give a size of
> > artifacts which should be regarded as too large, depending on the
> > environment probably.
> >
>
> definitely depends on your environment.
>
> > Thanks!
> > - Ben
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> > For additional commands, e-mail: users-h...@maven.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>
>

Re: Questions about the build cache plugin

[Olivier Lamy]

On Mon, 12 Jun 2023 at 18:53, Benjamin Marwell  wrote:
>
> Hello everyone!
>
> First of all thank you everyone working on the build cache plugin! It
> is amazing!
>
> At least some of  the following questions seem to be of interest to
> most users and might end up on the documentation, So here's a few
> things which came to my mind.
>
> 1.) Considering I always require PRs in my projects, would setting
> -Dmaven.build.cache.remote.save.enabled=true be a sensible thing only
> for the main branch?

Personally, I would have the main branch always have a full build
without caching to be sure everything works fine but use cache only
for branches/PR :)
That's what we will do at Jetty project.
The build for 12 branch is around 50 minutes with the cache it goes
down to 8/9 minutes (only because something is always rebuilt and
retrigger a few small modules)
The idea is to reduce development/check of PR and use some incremental
build while the main is always fully build especially when the build
has few jdks as target.
But here it's up to you :)

>
> 2.) Can I maybe have  the branch name included in a path to a remote?
> This way I could have a different cache for maintenance branches. Just
> include the branch name into -Dmaven.build.cache.remote.url?

why would need that?
if your branch has different sources (java, pom, etc..) the calculated
hash will be different so the hash key will be simply different no
need to configure a cache URL differently (well except if you want to
clean up caches per branch)

>
> 3.) Can I somehow use option 1 & 2 to make caches available for
> colleagues without making them manually configure the remote URL for
> each branch they are working on?

cache will be used by colleagues' build as long as a module have the
same calculated hash any local differences will have a different
calculated hash

>
> 4.) The docs say there's also XX and XXMM algorithms, but it doesn't
> say WHEN to use them (only that they may leverage performance). Are
> there some example cases or does someone already have some experience
> we could benefit from?


I have yet to see huge differences.
currently with a build already down from 50 minutes to 8/9 minutes not
sure this was my primary goal
So I can't really tell :)

>
> 5.) The section "filtering out artifacts" might want to give a size of
> artifacts which should be regarded as too large, depending on the
> environment probably.
>

definitely depends on your environment.

> Thanks!
> - Ben
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Questions about the build cache plugin

[Benjamin Marwell]

Hello everyone!

First of all thank you everyone working on the build cache plugin! It
is amazing!

At least some of  the following questions seem to be of interest to
most users and might end up on the documentation, So here's a few
things which came to my mind.

1.) Considering I always require PRs in my projects, would setting
-Dmaven.build.cache.remote.save.enabled=true be a sensible thing only
for the main branch?

2.) Can I maybe have  the branch name included in a path to a remote?
This way I could have a different cache for maintenance branches. Just
include the branch name into -Dmaven.build.cache.remote.url?

3.) Can I somehow use option 1 & 2 to make caches available for
colleagues without making them manually configure the remote URL for
each branch they are working on?

4.) The docs say there's also XX and XXMM algorithms, but it doesn't
say WHEN to use them (only that they may leverage performance). Are
there some example cases or does someone already have some experience
we could benefit from?

5.) The section "filtering out artifacts" might want to give a size of
artifacts which should be regarded as too large, depending on the
environment probably.

Thanks!
- Ben

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org