Thanks for the responses Ludo and Eric.
On 02/10/16 23:33, Ludovic Courtès wrote:
Currently, we build a single set of x86_64 packages assuming SSE but
not SSE2 instructions, but sometimes it would be nice to use more
recent instructions like AVX.
Isn’t SSE2 part of the x86_64 base spec? I always forget.
I don't think so. I always refer to Mark's comment when I forget:
https://lists.gnu.org/archive/html/guix-devel/2016-07/msg01534.html
So I'm wondering if there is some way to specify a system more
specific than 'X86_64'? I tried simply adding '--with-arch=haswell' as
a configure argument in gcc-4.9 so that flag became the default for
gcc usage and saw some performance improvements, though I did have to
disable tests in gnutls.
Do you have performance figures for some CPU-intensive applications?
What software are you most interested in?
We tend to run software whose runtime is dependent on the input data, so
it is hard to say. But up to days or weeks of walltime in some cases. It
takes a lot of power researching climate change..
As an anecdote, adding "-march=haswell" shaved 13% off the runtime of
diamond, ~20% if the CPUs were contended.
Hardcoding that configure flag is definitely less than ideal, I'm
wondering if there is some better way that would enable us to share
package updates and even substitutes for these systems? My initial
thought is extending the triplet 'x86_64-unknown-linux-gnu' somehow,
but I suspect others have better ideas?
I’m not sure how to do this. Having, say, an x86_64avx-linux system
type (not triplet) would be impractical because it would be entirely
separate from x86_64-linux (different derivations).
Yes my original plan was to rebuild all packages locally, but that was
probably too optimistic as I reckon there will be too much work involved
in maintaining it.
Ideally, software for which using these CPU extensions makes a
significant difference would do what glibc does, which is to provide
several implementations of the relevant functions (one for SSE2, one for
AVX, etc.) and have the right one be selected at load time via an IFUNC
or similar mechanism.
That sounds useful in some cases, but it is probably too much of a
stretch for most bioinformatics packages.
In the end I think I'll just compile the specific packages we are
specifically interested in. I attached some example code in case anyone
is interested. But this brought up a few questions:
1) I also tried using --expression e.g. guix build --expression '(@@ (my
packages cpu-specific) diamond-cpu-specific)' but that fails to compile
as if the GUIX_PACKAGE_PATH is ignored, is that unexpected?
2) Is something amiss with gcc-toolchain-6? Compiling with it, diamond
complains of a missing stdlib.h.
Thanks,
ben
;;; Copyright © 2016 Ben Woodcroft <donttrust...@gmail.com>
;;;
;;; This code is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
(define-module (ben packages cpu-specific)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix packages)
#:use-module (guix utils)
#:use-module (guix download)
#:use-module (guix git-download)
#:use-module (guix hg-download)
#:use-module (guix build-system ant)
#:use-module (guix build-system gnu)
#:use-module (guix build-system cmake)
#:use-module (guix build-system perl)
#:use-module (guix build-system python)
#:use-module (guix build-system r)
#:use-module (guix build-system ruby)
#:use-module (guix build-system trivial)
#:use-module (gnu packages)
#:use-module (gnu packages autotools)
#:use-module (gnu packages algebra)
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages bison)
#:use-module (gnu packages boost)
#:use-module (gnu packages commencement)
#:use-module (gnu packages compression)
#:use-module (gnu packages cpio)
#:use-module (gnu packages curl)
#:use-module (gnu packages documentation)
#:use-module (gnu packages datastructures)
#:use-module (gnu packages file)
#:use-module (gnu packages gawk)
#:use-module (gnu packages gcc)
#:use-module (gnu packages gd)
#:use-module (gnu packages gtk)
#:use-module (gnu packages glib)
#:use-module (gnu packages groff)
#:use-module (gnu packages image)
#:use-module (gnu packages imagemagick)
#:use-module (gnu packages java)
#:use-module (gnu packages linux)
#:use-module (gnu packages logging)
#:use-module (gnu packages machine-learning)
#:use-module (gnu packages man)
#:use-module (gnu packages maths)
#:use-module (gnu packages mpi)
#:use-module (gnu packages ncurses)
#:use-module (gnu packages pcre)
#:use-module (gnu packages parallel)
#:use-module (gnu packages pdf)
#:use-module (gnu packages perl)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages popt)
#:use-module (gnu packages protobuf)
#:use-module (gnu packages python)
#:use-module (gnu packages readline)
#:use-module (gnu packages ruby)
#:use-module (gnu packages serialization)
#:use-module (gnu packages statistics)
#:use-module (gnu packages tbb)
#:use-module (gnu packages tex)
#:use-module (gnu packages texinfo)
#:use-module (gnu packages textutils)
#:use-module (gnu packages time)
#:use-module (gnu packages tls)
#:use-module (gnu packages vim)
#:use-module (gnu packages web)
#:use-module (gnu packages xml)
#:use-module (gnu packages xorg)
#:use-module (gnu packages zip)
#:use-module (srfi srfi-1)
#:use-module (gnu packages bioinformatics))
;; "sandybridge" for Ben's laptop
(define cpu "sandybridge")
(define-public gcc-cpu-specific
(let ((base gcc-5)) ; gcc-6 does not seem to work.
(package
(inherit base)
(name "gcc-cpu-specific")
(arguments
(substitute-keyword-arguments (package-arguments base)
((#:configure-flags configure-flags)
`(append ,configure-flags
(list (string-append
"--with-arch=" ,cpu)))))))))
(define-public (cpu-specific-package base-package)
(package
(inherit base-package)
(name (package-name base-package))
;; We must set a higher package version so this package is used instead of
;; the package in Guix proper.
(version (string-append (package-version base-package) "-cpu-specific"))
(inputs
`(,@(package-inputs base-package)
("gcc" ,((@@ (gnu packages commencement)
gcc-toolchain) gcc-cpu-specific))))))
(define-public diamond-cpu-specific (cpu-specific-package diamond))
(define-public fasttree-cpu-specific (cpu-specific-package fasttree))
(define-public blast+-cpu-specific (cpu-specific-package blast+))
(define-public bwa-cpu-specific (cpu-specific-package bwa))
(define-public metabat-cpu-specific (cpu-specific-package metabat))
