Re: Guix infrastructure

[Catonano]

Wow, I had totally missed this thread !

2017-07-09 10:49 GMT+02:00 Ricardo Wurmus :

>
> myglc2  writes:
>
> > As you have experienced here, the learning/deployment costs and hassle
> > associated with each new type of server often dwarfs other costs. The
> > best way to minimize this is to minimize the number of types of servers
> > you own.
>
> The servers I’m preparing to add to the build farm once I’m back to the
> office are all of the same type.  There are dozens of them.
>

Ah, this is good news !

I can't wait :-)

Re: cuirass evaluate

[Mathieu Othacehe]

Hi Ludo,

> It’s a requirement because the evaluation process has side effects on
> the Guile that runs it; for instance, it loads tons of modules in it.

Ok thanks for the explanation.

I just sent a patch to add minimal error checking on "evaluate" errors.

Mathieu

Re: guile-dbi: Use libltdl and prefer GUILE_DBD_PATH.

[Linas Vepstas]

Sorry for the very late reply.  I applied these patches to

https://github.com/opencog/guile-dbi

which will have to serve as the official repo for guile-dbi, since gna.org
is now dead.

Maurizio Boriani, I was surprised to see you answer -- do you have some
other repo somwhere that contains the guile-dbi code?

--linas



On Thu, Sep 15, 2016 at 10:36 PM, 宋文武  wrote:

> Hi!
>
> To reall, curentlly, guile-dbi use dlopen to load its backends.
> So, for Guix we have to set LD_LIBRARY_PATH, which is not desired.
>
> > [...]
> > (me:)
> >> I think it will be great to introduce a new search path (eg:
> GUILE_DBD_PATH)
> >> in addition to LD_LIBRARY_PATH.  Find the absolute so filepath in it
> >> ourself, then pass to dlopen.
> >
> > (ludo:)
> > Why not, but then it would be redundant with LD_LIBRARY_PATH and
> > LTDL_LIBRARY_PATH, which would further complicate the search rules.
> >
>
> Yeah, but I just learn that libltdl has a convenient function to do
> this..
>
>
> So, here are patches for using libltdl and a new search path:
>
>
>
> Hope they are obvious :-)
>
>

Re: 01/03: gnu: Add ncurses-with-gpm.

[Leo Famulari]

On Tue, Jul 11, 2017 at 11:42:23AM +0200, Danny Milosavljevic wrote:
> With all this "ncurses/fixed" stuff in master, should I make it
> inherit from ncurses/fixed ?

Recently, Mark added the macro (package/inherit) which does the Right
Thing when inheriting from a grafted package:

https://git.savannah.gnu.org/cgit/guix.git/commit/?id=bedba064267490f2d9f7ee44e5ec103aacc82611

There are some recent examples in the commit log.


signature.asc
Description: PGP signature

Re: RPC pipelining

[Ludovic Courtès]

Hi Roel,

Roel Janssen  skribis:

> substitute: guix substitute: warning: ACL for archive imports seems to be 
> uninitialized, substitutes may be unavailable
> substitute: ;;; Failed to autoload make-session in (gnutls):
> substitute: ;;; ERROR: missing interface for module (gnutls)
> substitute: Backtrace:
> substitute:1 (primitive-load 
> "/gnu/repositories/guix/scripts/guix")
> substitute: In guix/ui.scm:
> substitute:   1352:12  0 (run-guix-command _ . _)
> substitute: 
> substitute: guix/ui.scm:1352:12: In procedure run-guix-command:
> substitute: guix/ui.scm:1352:12: In procedure module-lookup: Unbound 
> variable: make-session
> guix environment: error: build failed: writing to file: Broken pipe

This is because ‘guix substitute’, called by ‘guix-daemon’, doesn’t find
(gnutls) in its GUILE_LOAD_PATH.

Use either “sudo -E ./pre-inst-env guix-daemon …” so that guix-daemon
inherits GUILE_LOAD_PATH and GUILE_LOAD_COMPILED_PATH, or
--no-substitutes.

Thanks for testing!

Ludo’.

Re: RPC pipelining

[Roel Janssen]

Hello Ludo’!

Thanks for working so hard on this.
I run into trouble with my test setup..

[roel@hpcguix ~]$ time ./guixr environment --ad-hoc coreutils -- true

;;; (flush-pending-rpcs 170)

;;; (flush-pending-rpcs 4)
substitute: guix substitute: warning: ACL for archive imports seems to be 
uninitialized, substitutes may be unavailable
substitute: ;;; Failed to autoload make-session in (gnutls):
substitute: ;;; ERROR: missing interface for module (gnutls)
substitute: Backtrace:
substitute:1 (primitive-load "/gnu/repositories/guix/scripts/guix")
substitute: In guix/ui.scm:
substitute:   1352:12  0 (run-guix-command _ . _)
substitute: 
substitute: guix/ui.scm:1352:12: In procedure run-guix-command:
substitute: guix/ui.scm:1352:12: In procedure module-lookup: Unbound variable: 
make-session
guix environment: error: build failed: writing to file: Broken pipe

real0m8.679s
user0m1.199s
sys 0m0.202s


But FWIW, I think the time between no output and the "substitute: ..."
output is dramatically shorter.

I'll report back when I have a better testing environment ready.

Kind regards,
Roel Janssen


Ludovic Courtès writes:

> Hello Guix!
>
> One of the main sources of slowness when talking to a remote daemon, as
> with GUIX_DAEMON_SOCKET=guix://…, is the many RPCs that translate in
> lots of network round trips:
>
> --8<---cut here---start->8---
> $ GUIX_PROFILING=rpc ./pre-inst-env guix build inkscape -d --no-grafts
> /gnu/store/iymxyy5sn0qrkivppl6fn0javnmr3nss-inkscape-0.92.1.drv
> Remote procedure call summary: 1006 RPCs
>   built-in-builders  ... 1
>   add-to-store   ...   136
>   add-text-to-store  ...   869
> --8<---cut here---end--->8---
>
> In this example we’re making ~1,000 round trips; not good!
>
> Before changing the protocol, an idea that came to mind is to do “RPC
> pipelining”: send as many RPC requests at once, then read all the
> corresponding responses.
>
> It turns out to necessitate a small change in the daemon, though, but
> the attached patch demonstrates it: the client buffers all
> ‘add-text-to-store’ RPCs, and writes them all at once when another RPC
> is made (because other RPCs, which are not buffered, might depend on the
> effect of those ‘add-text-to-store’ RPCs) or when the connection is
> closed.  In practice, on the example above, it manages to buffer all 869
> RPCs and send them all at once.
>
> To estimate the effectiveness of this approach, I introduced delay on
> the loopback device with tc-netem(8) and measured execution time (the
> first run uses pipelining, the second doesn’t):
>
> --8<---cut here---start->8---
> $ sudo tc qdisc add dev lo root netem delay 150ms
> $ time GUIX_DAEMON_SOCKET=guix://localhost ./pre-inst-env guix build inkscape 
> -d --no-grafts
> accepted connection from 127.0.0.1
> /gnu/store/iymxyy5sn0qrkivppl6fn0javnmr3nss-inkscape-0.92.1.drv
>
> ;;; (flush-pending-rpcs 869)
>
> real  0m47.796s
> user  0m1.307s
> sys   0m0.056s
> $ time GUIX_DAEMON_SOCKET=guix://localhost guix build inkscape -d --no-grafts
> accepted connection from 127.0.0.1
> /gnu/store/iymxyy5sn0qrkivppl6fn0javnmr3nss-inkscape-0.92.1.drv
>
> real  5m7.226s
> user  0m1.392s
> sys   0m0.056s
> $ sudo tc qdisc del dev lo root
> --8<---cut here---end--->8---
>
> So the wall-clock time is divided by 6 thanks to ‘add-text-to-store’
> pipelining, but it’s still pretty high due to the 136 ‘add-to-store’
> RPCs which are still *not* pipelined.
>
> It’s less clear what to do with these.  Buffering them would require
> clients to compute the store file name of the files that are passed to
> ‘add-to-store’, which involves computing the hash of the files itself,
> which can be quite costly and redundant with what the daemon will do
> eventually anyway.  The CPU cost might be compensated for when latency
> is high, but not when latency is low.
>
> Anyway, food for thought!
>
> For now, if those using Guix on clusters are willing to test the patch
> below (notice that you need to run the patched guix-daemon as well), I’d
> be interested in seeing how representative the above test is!
>
> Ludo’.
>
> diff --git a/guix/store.scm b/guix/store.scm
> index b15da5485..1ba22cf2d 100644
> --- a/guix/store.scm
> +++ b/guix/store.scm
> @@ -40,6 +40,7 @@
>#:use-module (ice-9 regex)
>#:use-module (ice-9 vlist)
>#:use-module (ice-9 popen)
> +  #:use-module (ice-9 format)
>#:use-module (web uri)
>#:export (%daemon-socket-uri
>  %gc-roots-directory
> @@ -322,7 +323,7 @@
>  
>  (define-record-type 
>(%make-nix-server socket major minor
> -buffer flush
> +buffer flush pending-rpcs
>  ats-cache atts-cache)
>nix-server?
>(socket nix-server-socket)
> @@ -332,6 +333,10 @@
>(buffer nix-server-output-port)  

Re: guile-dbi: Use libltdl and prefer GUILE_DBD_PATH.

[宋文武]

Linas Vepstas  writes:

> Sorry for the very late reply. I applied these patches to 
>
> https://github.com/opencog/guile-dbi
>
> which will have to serve as the official repo for guile-dbi, since gna.org is 
> now dead.

Oh, thanks!

Is there a plan for a new release with tarballs?

Re: Installer, ISO9660, etc.

[Danny Milosavljevic]

Hi Ludo,

On Tue, 11 Jul 2017 11:23:46 +0200
l...@gnu.org (Ludovic Courtès) wrote:

> Is it really a problem to always have a partition table?

The ISO9660 filesystem code doesn't find its primary volume descriptor when 
it's looking for it on the (fake) partition.  (I'm not sure what the fake 
partition is for in the first place).

But /proc/partitions would have contained "sda" itself, Guix just filters it 
out.

Of course we could special-case ISO9660 - but even easier (and less surprising) 
would be to just use /proc/partitions as-is and not fiddle with it (ok, maybe 
filter out the ram disks - not sure what that's about).

WDYT?

Re: 01/03: gnu: Add ncurses-with-gpm.

[Danny Milosavljevic]

Hi Ludo,

On Fri, 07 Jul 2017 13:14:39 +0200
l...@gnu.org (Ludovic Courtès) wrote:

> I’m really happy to see you continue John’s heroic work on the
> installer!  I’ll be happy to give it a spin when you deem appropriate.

Okay.  Still some loose ends to tie up first :)

> > +(define-public ncurses/gpm
> > +  (package
> > +(inherit ncurses)
> > +(name "ncurses-with-gpm")
> > +(arguments
> > +(substitute-keyword-arguments (package-arguments ncurses)
> > + ((#:configure-flags cf)
> > +  `(cons (string-append "--with-gpm=" (assoc-ref %build-inputs 
> > "gpm") "/lib/libgpm.so.2") ,cf
> > +(inputs
> > + `(("gpm" ,gpm)  
> 
> Does it need the full file name of the .so?  Would be nicer if this
> could be avoided, to avoid breakage in the future.

I tried it - apparently it does need it.

With all this "ncurses/fixed" stuff in master, should I make it inherit from 
ncurses/fixed ?

Re: my recent messages

[Ludovic Courtès]

Hello ng0,

Thanks for your message.  Like Pjotr wrote, we all go through this at
times, that happens.

The technical issues you reported remain valid, and I hope we can all
work together on addressing them in the best of our capacity.

I wish you the best for your ongoing endeavors!

Ludo’.

Re: Installer, ISO9660, etc.

[Ludovic Courtès]

Heya,

Danny Milosavljevic  skribis:

> But this "partition?" thing would mean that having a partition table is now 
> mandatory, right?  Is that really supposed to be the case?  The partition 
> table has a magic mark (0x55 0xAA) in order for it *not* to be mandatory 
> (i.e. if you want to use the whole disk without partitioning anything, go 
> right ahead) and I can still remember using (floppy) disks which had no 
> partition table.

‘partition?’ is meant to return true when given a partition, so yes, it
assumes there’s a partition table somewhere.  :-)

In general, I think that’s a reasonable assumption.  Is it really a
problem to always have a partition table?

Ludo’.

Re: my recent messages

[Pjotr Prins]

No worries ng0. We all go through phases like that. The good side is
that it brings out that others state clearly where they are. Thumbs up
to the Guix team - this is a great effort and I am continuously
amazed with the quality of people here.

Pj.