bd808 added a comment.
In https://phabricator.wikimedia.org/T134232#2258879, @AbdealiJK wrote: > I finally got mediawiki-core to clone successfully. I times it using the `time` command and here is the data: > > real 173m37.029s > user 9m37.520s > sys 29m47.641s > In https://phabricator.wikimedia.org/T134232#2259650, @Luke081515 wrote: > I make that experience too: at a normal medium labs instance, git clone is about two or three times faster than at the toollabs-bastion. Git operations are fairly file-system intensive. Many files are created, deleted, read, and/or stat'ed when working with a large repository. When these operations happen on Tool Labs host they are happening over NFS so that the resulting files are available on all grid hosts. Most other Labs projects do not use NFS and instead rely on host local storage which is much faster. File stat operations are especially slow over NFS which can be especially impactful for `git status` and `git add`. Generally Labs admins are working to improve the speed and reliability of the NFS servers constantly, but there is really only so much that can be done. There are some tips on StackOverflow <https://stackoverflow.com/questions/4994772/ways-to-improve-git-status-performance> for improving some git operations over NFS by tuning your git configuration. A trick that you can use to speed up cloning is to make your initial clones on a host-local filesystem (like `/tmp`) and the move the whole git repo to its final home on an NFS partition. $ time git clone --recursive https://gerrit.wikimedia.org/r/p/mediawiki/core.git mw-core-bd808-test Cloning into 'mw-core-bd808-test'... remote: Counting objects: 94277, done remote: Finding sources: 100% (21800/21800) remote: Getting sizes: 100% (2537/2537) remote: Compressing objects: 99% (79440/79441) remote: Total 631706 (delta 13485), reused 629664 (delta 12586) Receiving objects: 100% (631706/631706), 251.83 MiB | 9.90 MiB/s, done. Resolving deltas: 100% (522984/522984), done. Checking connectivity... done. Checking out files: 100% (6321/6321), done. real 3m14.461s user 5m6.477s sys 1m5.384s tools-bastion-02.tools:/tmp bd808$ time mv mw-core-bd808-test ~bd808 real 4m17.739s user 0m0.668s sys 0m28.939s About 7.5 minutes instead of the horrible 2.5 hours that was seen by @AbdealiJK when operating directly on NFS. TASK DETAIL https://phabricator.wikimedia.org/T134232 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bd808 Cc: bd808, zhuyifei1999, Luke081515, jayvdb, pywikibot-bugs-list, Aklapper, AbdealiJK, Zppix, Mdupont, yuvipanda, Gryllida, scfc, coren, valhallasw, chasemp _______________________________________________ pywikibot-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
