Hello all, The switch to the new NFS server hardware is now complete, with only 12 hours of delay. :-)
Seriously though, the filesystem copy took so long because - despite having prepared with a copy a day early so that only a quick rsync would suffice - there ended up being a bit over 4T worth of files that were touched in that 24h period. Log files, by their nature, are constantly updated. Normally, that shouldn't have been a major issue since the rsync would just copy them again... except that many of those log files were *hundreds of gigs* in size. This caused the filesystem resync to take a bit over 11 hours. While Cyberbot was the overall winner, with very nearly 2G of logs to his name, our winner for most impressive single log is yifeibot with 990G in a single log file! I would really much rather not have to turn quota on on our file system: it is very useful to be occasionally able to handle huge datasets. However, if users abuse the freedom I will have no choice but to do so in order to protect reliability and QoS. So, the delay having been that long, some things may have broken that I was no longer able to notice (I've been at this for 14h straight, now, and need sleep). I'll be on hand tomorrow to help work out any kinks that may have slipped in. On the positive side, however, we are now on new hardware for the NFS server and it seems to be working quite fine. Yeay! -- Marc _______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
