Re: Avoiding the garbage collector for the Guix Data Service
Hi, Christopher Baines skribis: > I think so, it at least seems to greatly reduce the chance the > derivations get removed. I made the Guix Data Service change here [1], > and I haven't seen the issue since. > > 1: > https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=f59354ed23a9c50e43b7aaaeaddb5365feda4e29 Good! It looks like the right thing to do, to me. Ludo’.
Re: Avoiding the garbage collector for the Guix Data Service
Ludovic Courtès writes: > Hi Christopher, > > Christopher Baines skribis: > >> The Guix Data Service has Guix compute lots of derivations, and then it >> loads the data in to a database. I'm having some issues with the garbage >> collector though, as sometimes it removes data from the disk before I've >> had a chance to read it (at least I think this is what's happening, I >> think this job failed for this reason [1]). >> >> 1: http://data.guix.gnu.org/job/10638 >> >> I'm not quite sure what the best strategy is to combat this? I had a >> look, and I couldn't see how to acquire a lock to prevent the garbage >> collector from running. I thought about registering thousands of gc >> roots, but that seems to just move the race a bit, as there's still a >> period between generating the derivation and registering the root where >> it can be deleted. >> >> Any ideas? > > At the Guix Days we discussed using ‘add-temp-root’ and similar to > ensure that .drv files are at least GC-protected for the duration of the > session (connection to the daemon). > > Does that address your problem? I think so, it at least seems to greatly reduce the chance the derivations get removed. I made the Guix Data Service change here [1], and I haven't seen the issue since. 1: https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=f59354ed23a9c50e43b7aaaeaddb5365feda4e29 Thanks, Chris signature.asc Description: PGP signature
Re: Avoiding the garbage collector for the Guix Data Service
Hi Christopher, Christopher Baines skribis: > The Guix Data Service has Guix compute lots of derivations, and then it > loads the data in to a database. I'm having some issues with the garbage > collector though, as sometimes it removes data from the disk before I've > had a chance to read it (at least I think this is what's happening, I > think this job failed for this reason [1]). > > 1: http://data.guix.gnu.org/job/10638 > > I'm not quite sure what the best strategy is to combat this? I had a > look, and I couldn't see how to acquire a lock to prevent the garbage > collector from running. I thought about registering thousands of gc > roots, but that seems to just move the race a bit, as there's still a > period between generating the derivation and registering the root where > it can be deleted. > > Any ideas? At the Guix Days we discussed using ‘add-temp-root’ and similar to ensure that .drv files are at least GC-protected for the duration of the session (connection to the daemon). Does that address your problem? Thanks, Ludo’.
Avoiding the garbage collector for the Guix Data Service
Hey, The Guix Data Service has Guix compute lots of derivations, and then it loads the data in to a database. I'm having some issues with the garbage collector though, as sometimes it removes data from the disk before I've had a chance to read it (at least I think this is what's happening, I think this job failed for this reason [1]). 1: http://data.guix.gnu.org/job/10638 I'm not quite sure what the best strategy is to combat this? I had a look, and I couldn't see how to acquire a lock to prevent the garbage collector from running. I thought about registering thousands of gc roots, but that seems to just move the race a bit, as there's still a period between generating the derivation and registering the root where it can be deleted. Any ideas? Thanks, Chris signature.asc Description: PGP signature
