Re: broken references in jar manifests

[Ricardo Wurmus]

Chris Marusich  writes:

>>> >> we have a problem with jar manifests.  When we use the Class-Path
>>> >> property to ensure that an executable can find its dependencies on the
>>> >> class path at run-time, we end up with a manifest like this:
>>> >>
>>> >> --8<---cut here---start->8---
>>> >> Manifest-Version: 1.0
>>> >> Class-Path: /gnu/store/i28vi94r8z9f0x02zgkrv87w16ibmqkw-java-htsjdk-2.
>>> >>  10.1/share/java/htsjdk.jar
>>> >> Created-By: 1.8.0_151 (Oracle Corporation)
>>> >> Main-Class: picard.cmdline.PicardCommandLine
>>> >> --8<---cut here---end--->8---
>>> >>
>>> >> Note that the Class-Path property is broken into two lines.  This means
>>> >> that the reference scanner will miss it and grafting will fail.
>
> That is quite unfortunate.  :-(
>
>>> An alternative to recording full references in the manifest file is to
>>> install a “lib” directory that contains symlinks to dependencies.  The
>>> manifest can then contain relative paths to these symlinks.
>>>
>>>
>> I guess I would prefer to do this instead.
>
> Sounds good to me, too.

Actually, I think this wouldn’t work after all.  When the references in
the Class-Path property are split up they would be overlooked when
grafting.  The references in the separate custom file would be grafted.
The result is that the applications would keep trying to load the old,
ungrafted jars while Guix would think that the package only retained
references to the grafted packages.

Equally bad: the garbage collector would trash these ungrafted
references as it cannot detect them, thus breaking the application.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: broken references in jar manifests

[Chris Marusich]

Gábor Boskovits  writes:

> 2018-03-01 19:54 GMT+01:00 Ricardo Wurmus :
>
>>
>> Gábor Boskovits  writes:
>>
>> > 2018-03-01 18:11 GMT+01:00 Ricardo Wurmus 
>> :
>> >
>> >> Hi Guix,
>> >>
>> >> we have a problem with jar manifests.  When we use the Class-Path
>> >> property to ensure that an executable can find its dependencies on the
>> >> class path at run-time, we end up with a manifest like this:
>> >>
>> >> --8<---cut here---start->8---
>> >> Manifest-Version: 1.0
>> >> Class-Path: /gnu/store/i28vi94r8z9f0x02zgkrv87w16ibmqkw-java-htsjdk-2.
>> >>  10.1/share/java/htsjdk.jar
>> >> Created-By: 1.8.0_151 (Oracle Corporation)
>> >> Main-Class: picard.cmdline.PicardCommandLine
>> >> --8<---cut here---end--->8---
>> >>
>> >> Note that the Class-Path property is broken into two lines.  This means
>> >> that the reference scanner will miss it and grafting will fail.

That is quite unfortunate.  :-(

>> An alternative to recording full references in the manifest file is to
>> install a “lib” directory that contains symlinks to dependencies.  The
>> manifest can then contain relative paths to these symlinks.
>>
>>
> I guess I would prefer to do this instead.

Sounds good to me, too.

Ricardo Wurmus  writes:

> A possible work-around might be to patch the class loader of our JDKs to
> look at a custom Guix-specific file, which we will include in each jar.
> That file would be allowed to have longer lines.
>
> There are two disadvantages:
>
> 1) we need to patch the JDK
> 2) the jars would not do the right thing when executed with a different
> JDK (e.g. on a foreign distro).

We shouldn't be afraid to experiment with a custom class loader.  It may
turn out to be an effective way to "teach" Java how to follow store
references, and the class loader machinery is well pretty documented.

The tricks we (and Nix) do with the linker (e.g., ld-wrapper) to always
use rpaths for C programs and libraries is similar in spirit: if the
machine does not know our rules by default, we must teach the machine
how to play by our rules.

-- 
Chris


signature.asc
Description: PGP signature

Re: python-build-system: Delete .egg-info file created in phase check.

[Hartmut Goebel]

Hi Mark,
> For now, I changed the 'eqv?' to 'string=?' in commit
> 171a117c61224be10f2b97a6a880ad0f4c38ef6d on core-updates, but I'm

Thanks for spotting and fixing this!

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: guile-gdbm doesn't work with gdbm-1.14

[Mark H Weaver]

iyzs...@member.fsf.org (宋文武) writes:

> Hello, since version 1.14, gdbm doesn't export "gdbm_errno" anymore [1],
> so the guile-gdbm ffi binding code [2] need updates now (I'm not
> confident to do it myself...).
>
> [1] 
> http://git.gnu.org.ua/cgit/gdbm.git/commit/?id=c175231e2781abd17eabf412cfb597654a076c7b
> [2] https://github.com/ijp/guile-gdbm/blob/master/gdbm.scm#L156

Here's a preliminary fix.

  Mark


>From 353abb66adb756c0ad51b4784004034a8a06de8c Mon Sep 17 00:00:00 2001
From: Mark H Weaver 
Date: Sun, 18 Mar 2018 08:43:06 -0400
Subject: [PATCH] gnu: guile-gdbm-ffi: Add support for gdbm-1.14.

* gnu/packages/patches/guile-gdbm-ffi-support-gdbm-1.14.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/guile.scm (guile-gdbm-ffi)[native-inputs]: New field.
[inputs]: Move above arguments.  Add the patch, and the 'patch' program.
[propagated-inputs]: Move above arguments.
[arguments]: In the builder, add code to apply the patch.
---
 gnu/local.mk   |  1 +
 gnu/packages/guile.scm | 21 +++--
 .../patches/guile-gdbm-ffi-support-gdbm-1.14.patch | 53 ++
 3 files changed, 71 insertions(+), 4 deletions(-)
 create mode 100644 gnu/packages/patches/guile-gdbm-ffi-support-gdbm-1.14.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 010f1417f..a03b482df 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -742,6 +742,7 @@ dist_patch_DATA =		\
   %D%/packages/patches/guile-1.8-cpp-4.5.patch			\
   %D%/packages/patches/guile-2.2-default-utf8.patch		\
   %D%/packages/patches/guile-default-utf8.patch			\
+  %D%/packages/patches/guile-gdbm-ffi-support-gdbm-1.14.patch	\
   %D%/packages/patches/guile-linux-syscalls.patch		\
   %D%/packages/patches/guile-present-coding.patch		\
   %D%/packages/patches/guile-relocatable.patch			\
diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index 1c8eaa9ec..f6f8ef9b9 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -1134,6 +1134,15 @@ inspired by the SCSH regular expression system.")
(base32
 "1j8wrsw7v9w6qkl47xz0rdikg50v16nn6kbs3lgzcymjzpa7babj"
 (build-system trivial-build-system)
+(inputs
+ `(("guile" ,guile-2.2)
+   ;; patch-and-repack doesn't work for git checkouts,
+   ;; so we must apply the patch manually.
+   ("patch" ,patch)
+   ("patch-file" ,(search-patch
+   "guile-gdbm-ffi-support-gdbm-1.14.patch"
+(propagated-inputs
+ `(("gdbm" ,gdbm)))
 (arguments
  `(#:modules
((guix build utils))
@@ -1186,12 +1195,16 @@ inspired by the SCSH regular expression system.")
   (format #f "(dynamic-link \"~a/lib/libgdbm.so\")"
   (assoc-ref %build-inputs "gdbm"
 
+   ;; Apply the patch to add support for gdbm-1.14.
+   (let ((patch-command (string-append (assoc-ref %build-inputs "patch")
+   "/bin/patch"))
+ (patch-file (assoc-ref %build-inputs "patch-file")))
+ (with-directory-excursion (dirname gdbm.scm-dest)
+   (format #t "applying '~a'...~%" patch-file)
+   (invoke patch-command "--force" "--input" patch-file)))
+
;; compile to the destination
(compile-file gdbm.scm-dest gdbm.go-dest)
-(inputs
- `(("guile" ,guile-2.2)))
-(propagated-inputs
- `(("gdbm" ,gdbm)))
 (home-page "https://github.com/ijp/guile-gdbm;)
 (synopsis "Guile bindings to the GDBM library via Guile's FFI")
 (description
diff --git a/gnu/packages/patches/guile-gdbm-ffi-support-gdbm-1.14.patch b/gnu/packages/patches/guile-gdbm-ffi-support-gdbm-1.14.patch
new file mode 100644
index 0..e6b578bdb
--- /dev/null
+++ b/gnu/packages/patches/guile-gdbm-ffi-support-gdbm-1.14.patch
@@ -0,0 +1,53 @@
+From 1da99396dc65993ba34ac0370ca5d6acda6a3322 Mon Sep 17 00:00:00 2001
+From: Mark H Weaver 
+Date: Sun, 18 Mar 2018 07:02:37 -0400
+Subject: [PATCH] Add support for gdbm-1.14.
+
+As of gdbm-1.14, 'gdbm_errno' no longer exists as a binary interface.
+It has been replaced by 'gdbm_errno_location', a function that returns
+int*.  We now use this new interface if it's available.
+---
+ gdbm.scm | 18 --
+ 1 file changed, 16 insertions(+), 2 deletions(-)
+
+diff --git a/gdbm.scm b/gdbm.scm
+index b92992f..4d38cc3 100644
+--- a/gdbm.scm
 b/gdbm.scm
+@@ -17,6 +17,9 @@
+ ;; You should have received a copy of the GNU General Public License
+ ;; along with this program. If not, see .
+ 
++;; Modified by Mark H Weaver  in March 2018 to support
++;; gdbm-1.14 with its new 'gdbm_errno_location' interface.
++
+ (define-module (gdbm)
+   #:use-module (system foreign)
+   #:use-module (rnrs bytevectors)
+@@ -151,10 +154,21 @@
+ 
+ ;;; errors
+ 
+-(define %errno (dynamic-pointer 

Re: PYTHONPATH issue explanation

[宋文武]

iyzs...@member.fsf.org (宋文武) writes:

> [...]
>
> I'd like do more tests with the GUIX_PYTHON_X_Y_SITE_PACKAGES option
> (patch sent), hope it works :-)

Hello, I have write a shell script to do some tests, it looks good to me!


Updated 'GUIX_PYTHON_X_Y_SITE_PACKAGES' patch, target 'core-updates' at
commit 171a117c (you also have to comment out the "manual-database"
profile hook in the "guix/profiles.scm", as it's broken in that commit):

>From d807306d02aab0a84de4fa3ff457a5b97ac15520 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= 
Date: Sat, 17 Mar 2018 18:46:55 +0800
Subject: [PATCH] gnu: python-2.7, python-3.6: Honor
 'GUIX_PYTHON_X_Y_SITE_PACKAGES'.

This replace the use of 'PYTHONPATH' as search path specification, as
suggested by Hartmut Goebel .  See
 for
details.

* gnu/packages/python.scm (python-guix-search-path-specification)
(python-guix-sitecustomize.py): New procedures.
(python-2.7, python-3.6):
[native-search-paths]: Use 'python-guix-search-path-specification'.
[arguments]: Add 'install-sitecustomize.py' phase.
---
 gnu/packages/python.scm | 65 +++--
 1 file changed, 57 insertions(+), 8 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index f3a75c30e..45de8c527 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -136,6 +136,41 @@
   #:use-module (guix build-system trivial)
   #:use-module (srfi srfi-1))
 
+(define (python-guix-search-path-specification version)
+  "Return the search path specification for python VERSION."
+  (let* ((major.minor (version-major+minor version))
+ (variable(string-append
+   "GUIX_PYTHON_"
+   (string-replace-substring major.minor "." "_")
+   "_SITE_PACKAGES"))
+ (files   (list (string-append
+ "lib/python" major.minor "/site-packages"
+(search-path-specification
+ (variable variable)
+ (files files
+
+(define (python-guix-sitecustomize.py version)
+  "Return the content of @file{sitecustomize.py} for python VERSION."
+  (let* ((major.minor (version-major+minor version))
+ (variable(string-append
+   "GUIX_PYTHON_"
+   (string-replace-substring major.minor "." "_")
+   "_SITE_PACKAGES")))
+(format #f "# Append module search paths for guix packages to sys.path.
+import os
+import site
+
+SITE_PACKAGES = os.environ.get('~a')
+
+if SITE_PACKAGES is None:
+SITE_PACKAGES = []
+else:
+SITE_PACKAGES = SITE_PACKAGES.split(os.pathsep)
+
+for i in SITE_PACKAGES:
+site.addsitedir(i)
+" variable)))
+
 (define-public python-2.7
   (package
 (name "python2")
@@ -304,6 +339,16 @@
  "/site-packages")))
(install-file tkinter.so target)
(delete-file tkinter.so)
+#t)))
+  (add-after 'install 'install-sitecustomize.py
+(lambda* (#:key outputs #:allow-other-keys)
+  (let* ((out (assoc-ref outputs "out"))
+ (sitedir (car (find-files out "^site-packages$"
+   #:directories? #t
+(with-output-to-file
+(string-append sitedir "/sitecustomize.py")
+  (lambda ()
+(display ,(python-guix-sitecustomize.py version
 #t))
 (inputs
  `(("bzip2" ,bzip2)
@@ -318,9 +363,7 @@
 (native-inputs
  `(("pkg-config" ,pkg-config)))
 (native-search-paths
- (list (search-path-specification
-(variable "PYTHONPATH")
-(files '("lib/python2.7/site-packages")
+ (list (python-guix-search-path-specification version)))
 (home-page "https://www.python.org;)
 (synopsis "High-level, dynamically-typed programming language")
 (description
@@ -428,13 +471,19 @@ data types.")
  ,file)))
   (find-files out "\\.py$")))
   (list '() '("-O") '("-OO")))
+ #t)))
+   (replace 'install-sitecustomize.py
+ (lambda* (#:key outputs #:allow-other-keys)
+   (let* ((out (assoc-ref outputs "out"))
+  (sitedir (car (find-files out "^site-packages$"
+#:directories? #t
+ (with-output-to-file
+ (string-append sitedir "/sitecustomize.py")
+   (lambda ()
+ (display ,(python-guix-sitecustomize.py version
  #t)))
 (native-search-paths
- (list (search-path-specification
-(variable "PYTHONPATH")
-(files (list 

guile-gdbm doesn't work with gdbm-1.14

[宋文武]

Hello, since version 1.14, gdbm doesn't export "gdbm_errno" anymore [1],
so the guile-gdbm ffi binding code [2] need updates now (I'm not
confident to do it myself...).

[1] 
http://git.gnu.org.ua/cgit/gdbm.git/commit/?id=c175231e2781abd17eabf412cfb597654a076c7b
[2] https://github.com/ijp/guile-gdbm/blob/master/gdbm.scm#L156