On 17/11/2020 11:51, Andi Christiansen wrote:
Hi all,
thanks for all the information, there was some interesting things
amount it..
I kept on going with rsync and ended up making a file with all top
level user directories and splitting them into chunks of 347 per
rsync session(total 42000 ish folders). yesterday we had only 14
sessions with 3000 folders in each and that was too much work for one
rsync session..
Unless you use something similar to my DB suggestion it is almost
inevitable that some of those rsync sessions are going to have issues
and you will have no way to track it or even know it has happened unless
you do a single final giant catchup/check rsync.
I should add that a copy of the sqlite DB is cover your backside
protection when a user pops up claiming that you failed to transfer one
of their vitally important files six months down the line and the old
system is turned off and scrapped.
JAB.
--
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
