On Sat, Nov 5, 2016 at 8:37 AM, Huji Lee <[email protected]> wrote: > Yeah I think it has to do with NFS too. The slow part is clearly the I/O > operations. > > I heard there is a new container we are working on in Labs (whose name I > forgot) which is faster that NFS. Is that virtualenv?
Webservices running with the kubernetes backend [0] have demonstrated much faster NFS access for things like Python virtualenvs. This is because the kubernetes containers (called "pods" in the upstream documentation) are configured to use an NFS cache layer that is not enabled on the OGE grid nodes or the bastions. The reasons that the grid and bastions do not have this NFS cache enabled are a bit murky, but at some point in the past it was the cache was disabled as a potential fix for a problem with stale data being read by some tools/bots. The techops team is cautious about changing the settings to re-enable the cache due lack of understanding about what could break as a result. Since the kubernetes environment is opt-in and new, they were confident that enabling the cache system there would not cause unknown problems. The larger issue of NFS slowness is to some extent a side effect of the growth of Tool Labs usage. More users, tools, and bots reading and writing files on the NFS shares cause more contention for the finite resources the underlying NFS servers have. A project has been underway for several months to migrate the NFS services to a new set of servers. This migration is currently scheduled to complete on November 14th [1]. There are no guarantees that NFS access will be significantly faster once the new servers are in use, but the setup will be better documented and more familiar to the Labs techops team. That should make it easier for them to troubleshoot issues and work towards providing a better experience for all of us. [0]: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes [1]: https://lists.wikimedia.org/pipermail/labs-announce/2016-November/000177.html Bryan -- Bryan Davis Wikimedia Foundation <[email protected]> [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855 _______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
