Maven Build System Dependency Issue

[Phil Beadling]

Hi Guixers,

First let me say I'm a Maven novice, so it's possible I'm doing something
dumb on the Maven side of things.

I'm unable to make a bare-bones Maven project build in Guix.

This looks to be a problem with mismatched dependencies in Guix around
java-commons-codec.

I suggest what probably needs to be fixed upstream below (assuming my hunch
is correct!).

However I'm looking for advice on how to workaround this problem today to
get a basic Maven package building.

*My method:*

Following advice here:
https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html

Using an environment like so:
guix environment --ad-hoc maven

I create a new git repo and put this in the root (so pom.xml, etc, sit in
in the root of the repo)

mvn archetype:generate -DgroupId=com.quantile.app
-DartifactId=java-test-repo
-DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4
-DinteractiveMode=false



I'm able to package and run everything OK (obviously the dependencies are
being pulled from the web in this case):
mvn package
java -cp target/java-test-repo-1.0-SNAPSHOT.jar com.quantile.app.App

Next I try to create a bare-bones Guix package to build this (this time
there will be no web access during the build):

(define-public java-test-repo-integration
  (let ((commit-integration "b9e3894aa3629fc2d60ceadea2f655d4cb6a826b"))
(package
  (name "java-test-repo")
  (version integration-version)
  (source
   (git-checkout
(url "ssh://git@vcs:1234/ea/java-test-repo.git")
(commit commit-integration)))
  (build-system maven-build-system)
  (home-page "https://vcs/projects/EA/repos/java-test-repo/browse;)
  (synopsis "Java Test Repo Production")
  (description "Java/Maven bare-bones framework for testing")
  (license license-quantile

I get the following build failure:

[INFO]

[INFO] BUILD FAILURE
[INFO]

[INFO] Total time:  1.136 s
[INFO] Finished at: 2022-02-08T14:39:36Z
[INFO]

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test)
on project java-test-repo: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test failed: Plugin
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4 or one of its
dependencies could not be resolved: Failed to collect dependencies at
org.apache.maven.plugins:maven-surefire-plugin:jar:3.0.0-M4 ->
org.apache.maven.surefire:maven-surefire-common:jar:3.0.0-M4 ->
org.apache.maven.surefire:surefire-api:jar:3.0.0-M4 ->
*commons-codec:commons-codec:jar:1.15:
Failed to read artifact descriptor for
commons-codec:commons-codec:jar:1.15: *Cannot access central (
https://repo.maven.apache.org/maven2) in offline mode and the artifact
org.apache.commons:commons-parent:pom:52 has not been downloaded from it
before. -> [Help 1]


After doing some digging I think I know why:

$ guix show java-commons-codec

name: java-commons-codec

version: 1.15

outputs: out

systems: x86_64-linux i686-linux

dependencies: *apache-commons-parent-pom@50* java-commons-lang3@3.9
java-junit@4.12


So it's packaged with a propagated apache-commons-parent-pom at version 50.


See also here for package definition:

https://github.com/guix-mirror/guix/blob/9fe48723c8266ffe8e6a639be2ec2e362cf20cb5/gnu/packages/java.scm#L7671here


But if I look at the POM in the Gnu Store - it is showing version 52 for
the commons-parent which I am assuming (perhaps incorrectly?) should match
the version 50 above?


/gnu/store/7qgi821xidn9nbb07qw35g7wmgv84jln-java-commons-codec-1.15/lib/m2/./commons-codec/commons-codec/1.15/commons-codec-1.15.pom



  

org.apache.commons

commons-parent

*52*

  


Likewise if I look at the source code on Apache's website - it's 52 too:

https://dlcdn.apache.org//commons/codec/source/commons-codec-1.15-src.tar.gz




  

org.apache.commons

commons-parent

*52*

  


What I think has happened is that java-commons-codec has been promoted from
version 1.14 to version 1.15 without changing the propoagated input?

On version 1.14 everything would match:

https://archive.apache.org/dist/commons/codec/source/commons-codec-1.14-src.tar.gz

  

org.apache.commons

commons-parent

50

  

*So assuming this isn't going to cause an avalanche of other issues the
solution proper seems to be either downgrade java-common-codec to version
1.14, or upgrade the dependency on apache-commons-parent-pom to 52?*

What I'm looking for today is any workaround which allows me to use Maven
without having to wait to for upstream fix.

My first attempt was to try - adding an inherited package and then swapping
this in place of 1.15:

Re: Parallel guix builds can trample?

[Phil Beadling]

Hi all,

The issue also exists when using --with-commit - see below for a refined
test that makes it trivial to demonstrate with any package where the source
is retrieved from git.

On Tue, 18 Jan 2022 at 10:10, Phil  wrote:

>
> > Philip Beadling  writes:
>


> >No, and we probably should do, even in spite of this issue.  We're going
> >to repeat the experiment now using --with-commit, to confirm behaviour
> there.
>
>
Let's try and build 2 package variations - for this experiement we don't
even have to run them at the same time:

mkdir checkout-test
cd checkout-test
mkdir cache1 cache2
XDG_CACHE_HOME=`realpath ./cache1` guix build
--with-commit=simm=*3981f60d81d3c9592d78b3e88545259713a93194
*simm
XDG_CACHE_HOME=`realpath ./cache2` guix build
--with-commit=simm=*ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade
*simm

Now look at the caches:

$ ll cache1/guix/checkouts

total 4.0K
-rw-rw-r-- 1 phil.beadling phil.beadling  10 Jan 18 12:37
last-expiry-cleanup
drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:37
lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq
drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:37
mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:37
qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq
drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:37
smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:37
*xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq*

$ ll cache2/guix/checkouts

total 4.0K
-rw-rw-r-- 1 phil.beadling phil.beadling  10 Jan 18 12:38
last-expiry-cleanup
drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:38
lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq
drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:38
mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:39
qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq
drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:38
smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:38
*xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq*

$

Despite asking for different variants of the simm package, they are both
stored under the same SHA.  If they both used the same cache directory (the
default) - they'd have overwritten each other.  Thus it is not safe to run
--with-commit or any other variation in parallel for the same user.

To prove I've matched the right SHA - I can back-out the original commit
ids provided to --with-commit:

$ cd
cache1/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq
&& git rev-parse HEAD && cd -

*3981f60d81d3c9592d78b3e88545259713a93194*
~/checkout-test

$ cd
cache2/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq
&& git rev-parse HEAD && cd -

*ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade*
~/checkout-test

$

Whilst my original setup was a bit of a edge-case, I think this refined
example demonstrates that the problem can occur with typical/reasonable use
of guix switches?  I can even think of how this could occur without the
user running in parallel - for example if a package required via nth-order
dependencies to build 2 versions of the same repo - it's possible that a
single "guix build" command could cause the issue?

Thoughts?

Re: Parallel guix builds can trample?

[Phil Beadling]

Hi Ricardo, all,



I think we’ve worked out what the issue is, and have a proposed workaround,
and perhaps a case for solving the problem in Guix itself (depending on
what you people think!).



The issue is that despite each build being performed in its own isolated
container, these containers are fed by the same per-user cached source
directory.  In the case where *different* versions of the *same* repo are
built at once, this results in a race condition.



In our case we have one Linux account that does a lot of automated Guix
builds for us.



One example is this account watches our source control and automatically
rebuilds all outstanding Pull Requests (PRs) on a repo, after a separate
successful merge to our integration branch.  PRs are uniquely identified as
monotonically increasing PR numbers eg, PR-1, PR-2, PR-3 and so on.  Each
is a different branch on the same Repo with slightly different candidate
changes in it.  They are automatically kept up to date with the integration
branch.



To do this our watcher fires off (near) instantaneously dozens of guix
builds, each with their own local channel customized for the PR it is
building.  Doing them in parallel is important to make the system usably
responsive.



Each fired process does this:

   - Clone the channel containing the package into a local directory
   - Modify the commit id of the package to the new merged head of the PR
   - Modify the package version to some dummy version containing the PR
   number
   - Build the modified package using the local channel
   - Report the result (the build is effectively discarded; it is never
   used for anything)



What we think is happening is the following:



   - For each build that is kicked off in quick succession the local cache
   of the repo required updated by *update-cached-checkou*t
  -
  
https://github.com/guix-mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.scm#L476
  -
  
https://github.com/guix-mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.scm#L279
   - The problem with this is because each version is using the same cached
   repo --- before one has a chance to take a copy of the updated checkout,
   that checkout can be changed by a separate build process



Thus there is a race condition in this scenario.  We can provide a longer
test script to demo this if required – it’s quite straightforward to
reproduce just with a bash script, now we know what is causing it.



Our workaround has been to change XDG_CACHE_HOME for each PR build we do.
But this is a bit unsatisfactory as it effects processes beyond Guix – it
casts too wide of a net, but it does resolve the problem for the time being.



Do people think this is enough of an issue to make a switch available in
Guix to prevent sharing of cached clones?  This would be easy enough to
implement – a crude solution would be that each cache directory name would
simply be generated using a SHA of a string which includes the PID or
similar to ensure a unique name, and because it is never going to be reused
it could be deleted immediately after the build.



Whilst this is unlikely to happen at the console, as people script guix
build use-cases to fit their own problems (in particular building lots of
variations of a single piece of software) – I can see this causing a
headache?  I think at least the manual should make it clear that you cannot
build 2 packages referencing the same repo at the same time with the same
user (unless I’ve missed this bit I don’t think it’s made explicitly
clear?).  An even simpler change would be introduce a lock file that
refused the 2nd build and at least preventing the race condition happening,
and ensuring referential transparency, or simpler still just placed a
warning on stderr?



If people are amenable to adding a switch or other config option, we’d be
happy to look writing the patch?


Any thoughts/comments/advice?


Cheers!
Phil.


On Wed, 12 Jan 2022 at 09:37, Phil  wrote:

> Hi - more details below.
>
> Ricardo Wurmus writes:
>
> >
> > How are you using Guix with this?  Do you generate Guix package
> > expressions?  Do you use “guix build --with-commit”?
> >
>
> The situation is like this - if we had a directory of clones of my
> channel:
>
> - pr-1
> - pr-2
> - pr-3
> - pr-4
> ... and so on
>
> Initially all the clones are taken from the master branch of my
> channel and are all identical - but we change the version and commit to
> match the head of each PR branch as per below.
>
> Each clone looks like this:
> - pr-1
>   - my-package.scm
> - pr-2
>   - my-package.scm
> and so on
>
> Each my-package.scm has a package like below - the inital packages are all
> identical, but my system effectively seds the version and commit values
> like the below.  These values are never committed back to master they
> are used only as local channels to build each PR to test each build
> still passes.
>
> (define-public my-package
>   (package
>   

Issues with v910 of mypy

[Phil Beadling]

Hi all,

Was wondering if anyone had come across the issue with mypy using modular
stubs in v910.  As far as I can see installing for example
python-types-requests and python-mypy in Guix leads to mypy ignoring the
supplemented request stubs provided by the types-requests package.  I've
raised this on mypy, but it not really a mypy bug as there solution does
work with pip (although other package managers like PDM have reported
similar issues):

https://github.com/python/mypy/issues/11395

Was wondering if anyone has
a) experienced the same issue?
b) had a workaround?

I could consider downgrading mypy to the last non-modular stub version if
this issue is widely expreienced and there is no immediate workaround from
guix or mypy message boards?

As it standards the version in guix is pretty unusable to anyone developing
outside of python core library sets and using mypy - we'd be better on a
slightly older version that works for now, I think?

Thanks,
Phil.

Re: --with-source version not honored?

[Phil Beadling]

I was able to write a short manifest that avoided the overwriting of the
original "foobarpy" package in my case, and instead cleanly replace it with
a newer version specified using "with-source".

By setting, for example, GUIX_FOOBAR_VERSION=1.23 and
GUIX_TEST_PACKAGE=my-package - I can now create an environment which builds
the version of foobarpy I want from an https link generated from
GUIX_FOOBAR_VERSION, and then create a modified version of "my-package"
with foobarpy replaced, rather than overwritten.

This is much better than my original hack, but it's still not recursive so
whilst it does rebuild my-package with the new version of foobarpy as a
propagated input, it isn't rebuilding the whole dependency tree - other
inputs to my-package could also depend on foobarpy, and should be rebuilt
too - so what I've done is more of a graft, buthappens to be good enough
for my specific use case (the foobarpy is not used anywhere else in my
case).

Anyway thought I'd share this here as a template for simple non-recursive
cases.  I suspect it's not a massive effort to make it recursive, when I
have a moment I'll have a think about that too.  Any thoughts on how to
improve are welcome!

 (use-modules (guix transformations) ;; options->transformation
  (guix packages) ;; package/inherit
  (srfi srfi-1) ;; fold
  (srfi srfi-98)) ;; get-environment-variable


 (define (generate-foobar-link version)
   (string-append "https://packages.foobar.com/;
  (substring version 0 3) "/foobar"
  version "_linux64.tar.gz"))

 (define transform
   ;; The package transformation procedure.
   (let* ((foobar-version (get-environment-variable "GUIX_FOOBAR_VERSION"))
  (foobar-link (string-append
"foobarpy@" foobar-version
"=" (generate-foobar-link foobar-version
 (format #t "~%Setting Foobar Version: ~a~%" foobar-version)
 (options->transformation
  `((with-source . ,foobar-link)

 (define my-foobar-package (transform (specification->package "foobarpy")))

 (define-public my-package-with-my-foobar
   (let* ((test-package-string (get-environment-variable
"GUIX_TEST_PACKAGE"))
  (test-package (specification->package test-package-string)))
 (package/inherit test-package
   (propagated-inputs
`(("foobarpy" ,my-foobar-package)
  ,@(fold alist-delete (package-propagated-inputs test-package)
'("foobarpy")))

 (packages->manifest (list my-package-with-my-foobar))



On Sat, 23 Oct 2021 at 11:04, Phil  wrote:

> Hi,
>
> Ludovic Courtès writes:
>
> > For historical reasons, ‘--with-source’ only applies to leaf packages,
> > unlike most (all?) other transformation inputs.  Concretely, this works:
>
> Yes I think this is what I'm hitting.  I want to reference a package
> using --with-source, but then immediately build an environment which
> feeds this source as an input into a 2nd package - forcing it to
> rebuild.
>
> My hack works but for propagated dependencies in python - but it's not a
> proper solution, and is brittle.  I was going to try to improve
> it by writing a Guile script to avoid the overwriting - I can post any
> solution I find if I get something working.
>
> Currently I can only do 2 things at the command line it seems:
>
> 1. I can try package X without writing a package but only in isolation
> or as a leaf.
>
> 2. I can replace package X as a dependency of package Y, if I write a
> package
> for it.
>
> Being able to combine 1 and 2 at the command line - would be very
> useful.
>
> Cheers,
> Phil.
>

--with-source version not honored?

[Phil Beadling]

Hi all,

I'm using the following incantation:

guix build
--with-source=foobar@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64
<--with-source=gurobipy@9.5.0=/opt/thirdparty/gurobi/gurobi950_beta/linux64>
foobar


However the package build is failing with:

(copy-file "lib/libfoobar.so.9.0.1" "/gnu/store/gkawzac…")

In procedure copy-file: No such file or directory


That is the new version number 9.5.0 is not written to every place when
transforming the original package (version 9.0.1).  I think only the
package-version is updated, but the other package components are not then
regenerated, meaning that if they use the package-version as an input we
get a disjoint package.

In the example above I use version like so:






*(add-after 'install 'install-foobar-library
  (lambda* (#:key outputs #:allow-other-keys)
  (let* ((dir (string-append (assoc-ref outputs "out")

"/lib/python3.8/site-packages/foobar/"))
 (lib-to-install (string-append
"libfoobar.so." ,version))*

But ",version" is not updated, nor is it updated if I change this to call
(package-version foobar).


If I drop into Guile I can see this a bit more clearly by writing a
manifest - the code below gives exactly the same error however when the
package-version is displayed it correctly responds with 9.5.0.

Not sure if this should be considered a bug, or if there is a lazy way of
evaluating version so avoid the problem - I think it's unexpected from a
practicioners point of view as packages end up inconsistent.

I presume I can manually replace the arguments section of the package in
the manifest to workaround this - is there a standard way of doing this?

Any ideas or clarifications welcome!

Cheers,
Phil.
















*(use-modules (guix transformations) (guix packages))(define
transform  ;; The package transformation procedure.
(options->transformation   '((with-source .
"gurobipy@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64"(define
my-package (transform (specification->package "foobar")))(display
(package-version my-package)) ;; this will display version
9.5.0(newline)(packages->manifest (list my-package)) ;; building this will
fail because copy-file still looks for 9.0.1*