Hi,I think you should probably should be looking not at TB/day but number of files (records) per unit time. Robinhood should report stats periodically in the log and this can be useful to see how your scan is going in more detail.
Those millions of tiny files will certainly have a cost in scan time. It should take just as long to read the file attributes for a 1K file as for a 100G file I think.
The bottleneck I think comes down to the database performance and MDS performance.
The robinhood developers probably have more insight. Scott On 4/20/2014 2:52 AM, Andrew Elwell wrote:
Hi folks, I suspect this is a "how long is a piece of string" question, but roughly what order of scan speed do other sites see on large systems? We have a 3PB /scratch hosted on sonnexion appliances (Cray) so I'm running 2 instances of robinhood (one on each of two esDM nodes) -- one as a lustre changelog, and the other performing a --scan -O --no-gc -d to help with the initial DB population (it's a fresh install of MariaDB10 on a 3rd host - dedicated LUN for /var/lib/mysql but without and SSD devices) I'm seeing an average of 15-20TB/day for the scan - is this normal? Also, some of our users have huge directory structures with millions of directories and tiny (o240k) files within them *cough* openfoam -- do other sites see this and how do they deal with the filetype mix? so far in (~7d) I have: type , count, volume, avg_size symlink , 269149, 19.68 MB, 77 dir , 41570192, 160.88 GB, 4.06 KB file , 194195639, 134.64 TB, 744.42 KB fifo , 3, 0, 0 Total: 236034983 entries, 148206163990949 bytes (134.79 TB) Many thanks Andrew ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech _______________________________________________ robinhood-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/robinhood-support
smime.p7s
Description: S/MIME Cryptographic Signature
------------------------------------------------------------------------------ Start Your Social Network Today - Download eXo Platform Build your Enterprise Intranet with eXo Platform Software Java Based Open Source Intranet - Social, Extensible, Cloud Ready Get Started Now And Turn Your Intranet Into A Collaboration Platform http://p.sf.net/sfu/ExoPlatform
_______________________________________________ robinhood-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/robinhood-support
