Re: Substitute timeouts
Hey Ludo, Thanks for taking the time to read my wall of text :D. > Yeah, it’s a double-edged sword. If this is a problem on the main ‘guix > publish’ server, we can lower the bypass threshold, which is currently > 50 MiB: > > > https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/services.scm#n450 > > WDYT? That would maybe help, but on the other hand, I would prefer to find a more definitive solution :). > First, in terms of UI, you’d have a command sitting there and doing > nothing, which can be off-putting. Second, clients have no idea how > long they’re going to wait; it could be that the nar is going to be > baked within seconds, or it could take 20mn if the baking queue is > already crowded or if the user is asking for a big store item like > libreoffice. Third, in many cases, building locally is likely to be > faster than waiting for substitutes to be available (the majority of > packages build very quickly, though the few most popular leaf packages > take a long time to build). It would be interesting to monitor the status of the baking workers. Could it really take 20 minutes to bake a substitute from your experience? Personally, I have always found this baking 404 and bypass cache a bit misleading. When substituting libreoffice, I would much rather wait a few minutes than trying to build it while there's an almost ready substitute. I get that this is a personal choice and maybe it should be an optional behaviour. >> It will also allow the Cuirass build farm to use directly the main guix >> publish server, simplifying the current CI setup. > > The only reason why Cuirass runs its own publish server is to avoid > overloading the main one? No, the main reason is that with the use of a publish cache, the Cuirass workers would probably hit 404 errors while the substitutes are being baked. Using a publish server without cache was a way to work around it. The motivation of the 202 waiting patch was to solve both problems at once. Maybe I should explore the narinfo dedicated thread solution as a short term solution, while starting to think about a more long term solution based on Fiber/Nginx. A Cuirass dedicated solution could also be to declare a build successful only when a nar is available and stop using a non-caching publish server. Thanks, Mathieu
Re: Substitute timeouts
Hi, Mathieu Othacehe skribis: > I have been investigating a problem that is visible both on the main > guix publish server at https://ci.guix.gnu.org[1] and on the Cuirass > build farm[2]. > > This error comes from the fact that the publish server does not accept > the "guix substitute" connection requests within the %fetch-timeout > duration of 5 seconds. Thanks for getting to the bottom of this! > The main guix publish server is using a cache. If a requested narinfo is > not in the cache, it will be baked and the client receives a 404 > error. Since ecaa102a58ad3ab0b42e04a3d10d7c761c05ec98 and the > introduction of the bypass mechanism, small store items are directly > returned. > > This means that the "narinfo-string" procedure can be called directly in > the main publish thread. Running perf on the main publish server reveals > that this procedure can be really expensive under IO pressure (GC > running for example) because it opens a lot of files. I have observed > that the "read-derivation-from-file" call can take up to 600 ms. > > If multiple clients were to ask narinfo of several items not yet cached, > under IO pressure, I think that the publish server could become > unresponsive and cause the timeout errors. Yeah, it’s a double-edged sword. If this is a problem on the main ‘guix publish’ server, we can lower the bypass threshold, which is currently 50 MiB: https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/services.scm#n450 WDYT? > The fact that Cuirass triggers the baking of successfully built > derivations probably doesn't help here. Could be. This threshold seemed to work fine earlier (and still does, mostly?). > Now regarding the timeout errors that are much more frequent on the > Cuirass build farm, the cause varies a bit. The Cuirass publish server > running on Berlin does not use a cache. This means that the > "narinfo-string" procedure is called for each request, in the main > thread. > > To fix those issues, a solution could be to run the "narinfo-string" in > a separate thread, but it will make the publish server code even harder > to understand. True! Though maybe it wouldn’t be that much worse. :-) The problem is that this thing is very much single-threaded, with exceptions in a couple of places. We could add one more exception like you write, or fiberize it, or run it behind nginx, possibly with a tiny bit of caching. > My proposition would be to get rid of the bypass mechanism and instead > implement a retry when some substitutes are reported as being baked, > as proposed by Miguel[3]. > > I think this is the most reasonable solution. This way, users won't > receive 404 errors and start building substitutes that are being > baked[4]. (If I followed correctly, the bypass mechanism is not at fault regarding timeouts on the Cuirass publish server since it’s not using a cache, right?) I don’t think it’s reasonable for ‘guix substitute’ to just wait upon 202 (or 404, that doesn’t matter). First, in terms of UI, you’d have a command sitting there and doing nothing, which can be off-putting. Second, clients have no idea how long they’re going to wait; it could be that the nar is going to be baked within seconds, or it could take 20mn if the baking queue is already crowded or if the user is asking for a big store item like libreoffice. Third, in many cases, building locally is likely to be faster than waiting for substitutes to be available (the majority of packages build very quickly, though the few most popular leaf packages take a long time to build). > It will also allow the Cuirass build farm to use directly the main guix > publish server, simplifying the current CI setup. The only reason why Cuirass runs its own publish server is to avoid overloading the main one? Thanks, Ludo’.
Substitute timeouts
Hello, I have been investigating a problem that is visible both on the main guix publish server at https://ci.guix.gnu.org[1] and on the Cuirass build farm[2]. This error comes from the fact that the publish server does not accept the "guix substitute" connection requests within the %fetch-timeout duration of 5 seconds. The main guix publish server is using a cache. If a requested narinfo is not in the cache, it will be baked and the client receives a 404 error. Since ecaa102a58ad3ab0b42e04a3d10d7c761c05ec98 and the introduction of the bypass mechanism, small store items are directly returned. This means that the "narinfo-string" procedure can be called directly in the main publish thread. Running perf on the main publish server reveals that this procedure can be really expensive under IO pressure (GC running for example) because it opens a lot of files. I have observed that the "read-derivation-from-file" call can take up to 600 ms. If multiple clients were to ask narinfo of several items not yet cached, under IO pressure, I think that the publish server could become unresponsive and cause the timeout errors. The fact that Cuirass triggers the baking of successfully built derivations probably doesn't help here. Now regarding the timeout errors that are much more frequent on the Cuirass build farm, the cause varies a bit. The Cuirass publish server running on Berlin does not use a cache. This means that the "narinfo-string" procedure is called for each request, in the main thread. To fix those issues, a solution could be to run the "narinfo-string" in a separate thread, but it will make the publish server code even harder to understand. My proposition would be to get rid of the bypass mechanism and instead implement a retry when some substitutes are reported as being baked, as proposed by Miguel[3]. I think this is the most reasonable solution. This way, users won't receive 404 errors and start building substitutes that are being baked[4]. It will also allow the Cuirass build farm to use directly the main guix publish server, simplifying the current CI setup. There's a proposed patch attached, WDYT? Thanks, Mathieu [1]: https://issues.guix.gnu.org/49089 [2]: https://issues.guix.gnu.org/48468 [3]: https://issues.guix.gnu.org/44193#2 [4]: http://issues.guix.gnu.org/33370 patch Description: Binary data
