Re: Gentoo packaging / reproducibility of Maven-based Java projects

[Thomas Broyer]

Disclaimer: I don't know Gentoo.

On Mon, Mar 20, 2017 at 4:19 PM Curtis Rueden  wrote:

>
> The main question I have is about Maven plugins: since Maven builds do some
> bootstrapping, but Gentoo builds are not supposed to download things from
> the network, it means we might need a pre-baked Maven local repository
> containing all commonly used plugins at various release versions? If so,
> that would be unfortunate. I was hoping for some insight from others here.
>

Well, either the POM declares explicit versions, or those from the
super-pom are used (if declared there; and those versions then depend on
the version of Maven itself), or a warning is issued and IIRC Maven will
use the LATEST version.
So AFAICT you could know exactly what Maven plugins are needed (modify the
POMs to specify a version if fall into the 2 former cases above), and
package them similarly to the other dependencies.

There would still be problems with:
 * plugins that will fetch dependencies as needed. Surefire is one of them,
dynamically fetching the appropriate "provider" depending on which testing
framework is used in the build (JUnit 3, JUnit 4, JUnit 4.7+, TestNG); also
the maven-compiler-plugin when using , and
probably many others, not necessarily "exotic" plugins. Those would have to
be "discovered" by building the project first and then adapting the Gentoo
build to add the appropriate dependencies (what I mean here is that those
dependencies can't be extracted automatically from the POMs, at least
without knowing the specifics of each plugin)
 * plugins that access the network during the build
 * plugins that don't lead to reproducible builds

Re: Gentoo packaging / reproducibility of Maven-based Java projects

[Curtis Rueden]

Hi Mark,

Thanks for the response.

> This is probably why you have so much trouble:  two package managers
> fighting for control.

I don't really think of Maven as a "package manager" but rather as a
structural tool, which should help make Portage's job _easier_, not harder.
Maven documents how the package dependencies are organized, and also makes
the actual build trivial (unless the Maven project in question requires
something non-standard, of course).

> The middle path might be:  package the dependencies individually, work
> out a way for Maven to see them as being present in the local
> repository, build the dependent happily using Maven.

Right, that is what we were discussing already on the linked issue thread.

> I don't know the guts of Maven well enough to say how hard it would be
> to plug in a new local-repository provider that uses the Gentoo
> /usr/share/*/lib layout.

We could do that, sure. Or we could just cheat with symlinks, to
manufacture a transient, lightweight local repository in a shape that Maven
knows already. :-)

The main question I have is about Maven plugins: since Maven builds do some
bootstrapping, but Gentoo builds are not supposed to download things from
the network, it means we might need a pre-baked Maven local repository
containing all commonly used plugins at various release versions? If so,
that would be unfortunate. I was hoping for some insight from others here.

> Binary dependencies are going to be a royal pain no matter how you do
> them.  Ugh!

Well, they need to be built from source. If they are cleanly Maven-based,
then we do things recursively the "easy" way. Otherwise, we need a custom
build script for each dependency with custom build requirements.
Dependencies which are too arduous to build from source will kill
downstream potential for inclusion with Gentoo, but that's always/already
true for a 100% source-based distro.

Regards,
Curtis

--
Curtis Rueden
LOCI software architect - https://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
Did you know ImageJ has a forum? http://forum.imagej.net/


On Mon, Mar 20, 2017 at 8:45 AM, Mark H. Wood  wrote:

> On Fri, Mar 17, 2017 at 02:11:20PM -0500, Curtis Rueden wrote:
> > I am currently discussing on GitHub with one of the Gentoo packagers the
> > possibility of packaging my group's Maven-based projects for Gentoo.
> >
> > The relevant issue is here:
> >  https://github.com/imagej/imagej/issues/162
> >
> > However, the issues are hardly unique to my projects. This has become a
> > very general discussion of rebuilding and packaging Maven projects.
> >
> > My questions are:
> > * Does anyone here use Gentoo?
>
> I do.
>
> > * Has anyone packaged their Maven-based stuff for it?
>
> No, I haven't contributed any packages to Gentoo yet.
>
> > * Does anyone know of a _general_ process for doing so?
>
> Typically a Gentoo package specifies all of its dependencies as Gentoo
> packages, and Portage does the dependency management.  This is
> probably why you have so much trouble:  two package managers fighting
> for control.  The "Gentoo Way" would seem to be:  first package all of
> the dependencies individually, get them into Portage, then add the
> dependent, adjust the POM to assume all dependencies will be supplied
> at runtime, and let Portage calculate the classpath.  Ick.
>
> There are some Gentoo packages based on projects which for some reason
> cannot or will not use stable releases of their dependencies, but
> instead package all of the source for those dependencies with the
> dependent code.  It's grating, but it works.  This is probably the
> easier route when using Maven in a Portage environment:  ship
> everything as subprojects and build it all together.
>
> The middle path might be:  package the dependencies individually, work
> out a way for Maven to see them as being present in the local
> repository, build the dependent happily using Maven.  I don't know the
> guts of Maven well enough to say how hard it would be to plug in a new
> local-repository provider that uses the Gentoo /usr/share/*/lib
> layout.
>
> Binary dependencies are going to be a royal pain no matter how you do
> them.  Ugh!  One thing this means is that your package can never exist
> for any architecture that the binary is not built for.  That's sadly
> not without precedent.
>
> --
> Mark H. Wood
> Lead Technology Analyst
>
> University Library
> Indiana University - Purdue University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> www.ulib.iupui.edu
>

Re: Gentoo packaging / reproducibility of Maven-based Java projects

[Mark H. Wood]

On Fri, Mar 17, 2017 at 02:11:20PM -0500, Curtis Rueden wrote:
> I am currently discussing on GitHub with one of the Gentoo packagers the
> possibility of packaging my group's Maven-based projects for Gentoo.
> 
> The relevant issue is here:
>  https://github.com/imagej/imagej/issues/162
> 
> However, the issues are hardly unique to my projects. This has become a
> very general discussion of rebuilding and packaging Maven projects.
> 
> My questions are:
> * Does anyone here use Gentoo?

I do.

> * Has anyone packaged their Maven-based stuff for it?

No, I haven't contributed any packages to Gentoo yet.

> * Does anyone know of a _general_ process for doing so?

Typically a Gentoo package specifies all of its dependencies as Gentoo
packages, and Portage does the dependency management.  This is
probably why you have so much trouble:  two package managers fighting
for control.  The "Gentoo Way" would seem to be:  first package all of
the dependencies individually, get them into Portage, then add the
dependent, adjust the POM to assume all dependencies will be supplied
at runtime, and let Portage calculate the classpath.  Ick.

There are some Gentoo packages based on projects which for some reason
cannot or will not use stable releases of their dependencies, but
instead package all of the source for those dependencies with the
dependent code.  It's grating, but it works.  This is probably the
easier route when using Maven in a Portage environment:  ship
everything as subprojects and build it all together.

The middle path might be:  package the dependencies individually, work
out a way for Maven to see them as being present in the local
repository, build the dependent happily using Maven.  I don't know the
guts of Maven well enough to say how hard it would be to plug in a new
local-repository provider that uses the Gentoo /usr/share/*/lib
layout.

Binary dependencies are going to be a royal pain no matter how you do
them.  Ugh!  One thing this means is that your package can never exist
for any architecture that the binary is not built for.  That's sadly
not without precedent.

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu


signature.asc
Description: PGP signature