Hi Colin, It sounds like you are testing a lustre filesystem based off a local ramdisk or /dev/shm ?
To echo what Thomas said, you can check linux ulimit parameters for number of file locks and open files. You can also look into lustre client ldlm config for lru aging rate. When setting max_rpcs_in_flight keep in mind that robinhood uses the lustre mdc client far more than the osc client. What I found when running initial rbh scans is the host memory buffer cache would quickly fill up and then the scan rate was limited by the linux vm and lustre ldlm aging configuration. regards, chris hunter [email protected] > On Tue, Dec 15, 2015 at 12:48 PM, Thomas LEIBOVICI (mail perso) < > [email protected]> wrote: > >> Hi Colin, >> >> Not sure what is the limiting factor in your case: do you state it is the >> DB access or the filesystem access? >> This doc can help you determining what is limiting: >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cea-2Dhpc_robinhood_wiki_pipeline-5Ftuning&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=d_G2h_sZYG4xtHMeKo8QgjDmOcMVdQvYgM-5Dri1AOY&m=LNL-lcwqD09XG4P2eTAAYrfE93koH10fD0BOt6HdQAo&s=GwSuSW5fCs3EdmLI3ntEBl8fju3ZL5uoA0rk2rk754I&e= >> >> If it is the DB: I suggest disabling "accounting" feature which is a DB >> performance killer (and should be improved in next versions). >> Once disabled, you don't have to choose between bulk insert and high >> threads, robinhood can do both, which commonly increase DB ingest >> performance by x3-x4 >> (think about removing your previous tunings of batch size and pipeline >> stage threads, so you can just have a minimistic tuning of nb_threads of >> pipeline). >> >> If the FS operation latency is the limiting point, a track is to increase >> the number of operations that can be processed in parallel by the Lustre >> client >> by increasing "max_rpc_in_flight" and ko2iblnd peer_credits on robinhood >> host and on MDS accordingly. >> >> Thanks forward for any update, >> Thomas >> >> >> Le 09/12/2015 23:31, Colin Faber a ?crit : >> >> So I'm playing around with RBH on some reasonably powerful hardware, I've >> tried various database configurations of robinhood 3, against mariadb with >> both myisam and innodb tables but keep running into the same road blocks >> performance wise. >> >> My test is simple, on a quiet file system, generate 1 million files with 1 >> million records in changelogs, then fire off robinhood -r --once and time >> it (along with it's internal timing). >> >> The database an E5-2609 based system with 32 gb of memory. I'm allowing >> mariadb to eat up 26GB of it and have sliced a few gigs off for a memory >> backed file system which I run the tables off of (to eliminate possible >> array slowness). >> >> I've tried both strategies of high thread count, vs bulk inserts to the DB >> but seem to be generally limited performance wise. >> >> I'm wondering what tunings I should be focusing on here to achieve the >> results posted in the documentation. >> >> Thanks! >> >> -cf ------------------------------------------------------------------------------ _______________________________________________ robinhood-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/robinhood-support
