Re: Installing a wrapper guile script in /bin
> Anyway, thanks for the pointers! They motivated me to keep pushing until > something worked. Glad I could help you! signature.asc Description: This is a digitally signed message part
Re: Installing a wrapper guile script in /bin
Thank you for the thoughtful input. Maxime Devos wrote: > * forego representing `wrapper-script` as a gexp (using #~), instead represent > `wrapper-script` as something quasiquoted. Then write this expression to > a file (with 'write', and include an appropriate shebang line) > * The procedure ‘program-file’ turns a gexp into a representation of a store > item. After spending most of the day figuring out how to manually create and build derivations. I hit a fundamental problem including that code in a package def which made me realize that manually embedding scripts is a flawed approach. The issue is that a script itself has dependencies, e.g. guile, bash, whatever. Those dependencies also need to show up in the transitive closure of whatever package embeds this script. > * Define a wrapper package. See wrap-python3 in packages/python.scm > for an example. (This option probably has to be combined with one of the > first two options.) Your suggestion here eventually set me on the right track. Say we want the final executable to end up at /bin/X. The basic idea is this: 1) Creat original package with a name like X-lib, and leave /bin/X empty, 2) Write a wrapper script as a gexp argument to program-file. In this gexp we reference the X-lib package, and finally 3) Create an trivial-build-system package X that includes the program-file wrapper as an input. The build step will copy this input script to /bin/X! This way dependencies are correctly tracked, and there's no need for manual derivation building hackery. Anyway, thanks for the pointers! They motivated me to keep pushing until something worked.
Re: Installing a wrapper guile script in /bin
> The script contents are not what I'm confused about. I don't know how to turn > my gexp script into a file under /bin/. This is conceptually what I want: > > (package > (name "foo") > ... > (arguments >`(... > #:phases > (modify-phases %standard-phases > ... > (replace 'install >(lambda* (#:key outputs #:allow-other-keys) > (let ((out (assoc-ref outputs "out")) >(wrapper-script #~(...))) > ... ; Do normal install stuff > (copy-file wrapper-script (string-append out "/bin/foo")) > ... ; Finish install stuff > ))) > > Of course `copy-file` doesn't work here because `wrapper-script` is a gexp not > a file. What code do I replace this with? Three options I have in mind: * forego representing `wrapper-script` as a gexp (using #~), instead represent `wrapper-script` as something quasiquoted. Then write this expression to a file (with 'write', and include an appropriate shebang line) Something like this: (let ((wrapper-script-contents `(exec* ,(string-append out "/bin/.foo-real") "--extra-library-stuff" other-arguments))) rename-upstream-binary write-wrapper-binary ) * The procedure ‘program-file’ turns a gexp into a representation of a store item. Write your script to accept arguments like this: YOUR-SCRIPT REAL-BINARY ARGUMENTS And write a small wrapper shell script to /bin/foo. Maybe with something like this (format PORT "#!~a~%exec ~a ~a #$@" bin-sh-something #$THE-SCRIPT-GEXP /bin/.foo-real) (Warning: the above shell script might be incorrect. I'm not very familiar with using the shell as a programming language.) * Define a wrapper package. See wrap-python3 in packages/python.scm for an example. (This option probably has to be combined with one of the first two options.) Greetings, Maxime signature.asc Description: This is a digitally signed message part
Re: Installing a wrapper guile script in /bin
Maxime Devos wrote: > Let's presume the binary is called $X. > > What I would do: add a build phase after the "install" phase that renames > /bin/$X to /bin/.$X-real using the rename-file procedure. Create > your wrapper script at /bin/.$X-real with call-with-output-file, some > I/O procedures and chmod (to make the wrapper script executable). > > I hope that helps, Maxime. Thanks for the pointers. The script contents are not what I'm confused about. I don't know how to turn my gexp script into a file under /bin/. This is conceptually what I want: (package (name "foo") ... (arguments `(... #:phases (modify-phases %standard-phases ... (replace 'install (lambda* (#:key outputs #:allow-other-keys) (let ((out (assoc-ref outputs "out")) (wrapper-script #~(...))) ... ; Do normal install stuff (copy-file wrapper-script (string-append out "/bin/foo")) ... ; Finish install stuff ))) Of course `copy-file` doesn't work here because `wrapper-script` is a gexp not a file. What code do I replace this with? I am vaguely aware things like `build-expression->derivation` to reify a gexp into a derivation; however, I'm not sure what to do with the derivation object in this case or if this is even on the right track. Cheers!
Re: Installing a wrapper guile script in /bin
> Say I have a script that reads /proc/cpuinfo and runs my executable with the > correct flags to load the library with the best CPU features possible. How can > I embed such a script in the package definition (as a gexp?) and install it > under /bin/? Let's presume the binary is called $X. What I would do: add a build phase after the "install" phase that renames /bin/$X to /bin/.$X-real using the rename-file procedure. Create your wrapper script at /bin/.$X-real with call-with-output-file, some I/O procedures and chmod (to make the wrapper script executable). I hope that helps, Maxime. signature.asc Description: This is a digitally signed message part
Re: Installing a wrapper guile script in /bin
Ludovic Courtès wrote: > I wrote about this topic in the past: > > https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/ Very nice overview! Thanks for sharing (and writing!) that article. Definitely feeling the urge to jump down this rabbit hole and patch upstream now. Hehe. > I you’re the upstream author, I recommend using one of the techniques > given above to provide so-called “fat binaries” that contain several > implementations of the performance-sensitive functions; the loader > will pick the right implementation when the program starts. > > If you’re downstream… it depends on the specifics. The loader is also > able to pick a .so from the right lib/ sub-directory depending on the > micro-architecture. You can try: I'm downstream, unfortunately. However, the final executable actually provides a flag to explicitly specify a path to the lib, so that's not really a hurdle in this case. Given the small size of the build products, I feel like it would be nice to fake a fat binary at the filesystem level. Mind if we just entertain this idea for a second? Say I have a script that reads /proc/cpuinfo and runs my executable with the correct flags to load the library with the best CPU features possible. How can I embed such a script in the package definition (as a gexp?) and install it under /bin/?
Re: Installing a wrapper guile script in /bin
Hi! elaexuo...@wilsonb.com skribis: > More specifically, the package I have builds separate libraries for CPUs with > AVX, AVX2, and no AVX support. Since build-type isn't sufficiently specific to > distinguish such CPU features, I have, so far, opted to just build all three > libs and stuff them under /lib/. > > My idea is to have the linker script check CPU features at runtime (by parsing > /proc/cpuinfo or something) and executing the binary with the parameters to > load the correct binary. > > Perhaps there is a better overall approach? I wrote about this topic in the past: https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/ I you’re the upstream author, I recommend using one of the techniques given above to provide so-called “fat binaries” that contain several implementations of the performance-sensitive functions; the loader will pick the right implementation when the program starts. If you’re downstream… it depends on the specifics. The loader is also able to pick a .so from the right lib/ sub-directory depending on the micro-architecture. You can try: LD_DEBUG=files your program to see where the loader looks for shared libraries. HTH! Ludo’.
Installing a wrapper guile script in /bin
Hi elaexuotee, > More specifically, the package I have builds separate libraries for > CPUs with > AVX, AVX2, and no AVX support. Since build-type isn't sufficiently > specific to > distinguish such CPU features, I have, so far, opted to just build > all three > libs and stuff them under /lib/. That's certainly one approach to solving the issue of not knowing which CPU your code runs on. Another approach is described in [1] under the section "Dependency graph rewriting". TL;DR, just offer the base package with no AVX features, an AVX package and an AVX2 package. Then build any dependants against the base package. Users can afterwards decide to build against optimized versions on their own. Regards, Leo [1] https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/
Installing a wrapper guile script in /bin
Hello Guix, Writing package definition, I have need of a non-trivial wrapper script that decides how to execute the installed binary. How do I accomplish this? With my vague understanding, I am envisioning writing a gexp directly in the install phase and would like to somehow reify this into a guile script and install that file under /bin. Is this correct, at the high level? More specifically, the package I have builds separate libraries for CPUs with AVX, AVX2, and no AVX support. Since build-type isn't sufficiently specific to distinguish such CPU features, I have, so far, opted to just build all three libs and stuff them under /lib/. My idea is to have the linker script check CPU features at runtime (by parsing /proc/cpuinfo or something) and executing the binary with the parameters to load the correct binary. Perhaps there is a better overall approach? Appreciate your thoughts!
