On 11/26/14 22:01, Craig Tierney - NOAA Affiliate wrote:
2014/11/26 20:59:19 [14613/2] STATS | ChangeLog reader #0:
2014/11/26 20:59:19 [14613/2] STATS | fs_name = lfs2
2014/11/26 20:59:19 [14613/2] STATS | mdt_name = MDT0000
2014/11/26 20:59:19 [14613/2] STATS | reader_id = cl1
2014/11/26 20:59:19 [14613/2] STATS | records read = 117135
2014/11/26 20:59:19 [14613/2] STATS | interesting records = 117135
2014/11/26 20:59:19 [14613/2] STATS | suppressed records = 0
2014/11/26 20:59:19 [14613/2] STATS | records pending = 0
2014/11/26 20:59:19 [14613/2] STATS | last received =
2014/11/26 20:59:17
2014/11/26 20:59:19 [14613/2] STATS | last read record time =
2014/11/26 20:55:06.685525
2014/11/26 20:59:19 [14613/2] STATS | last read record id =
265806017
2014/11/26 20:59:19 [14613/2] STATS | last pushed record id =
265806017
2014/11/26 20:59:19 [14613/2] STATS | last committed record id =
265796017
2014/11/26 20:59:19 [14613/2] STATS | last cleared record id =
265796017
2014/11/26 20:59:19 [14613/2] STATS | read speed =
672.94 record/sec (247.03 incl. idle time)
2014/11/26 20:59:19 [14613/2] STATS | processing speed ratio = 20.92
2014/11/26 20:59:19 [14613/2] STATS | status = terminating
2014/11/26 20:59:19 [14613/2] STATS | ChangeLog stats:
2014/11/26 20:59:19 [14613/2] STATS | MARK: 0, CREAT: 44, MKDIR: 0,
HLINK: 0, SLINK: 0, MKNOD: 0, UNLNK: 110499, RMDIR: 6592
2014/11/26 20:59:19 [14613/2] STATS | RENME: 0, RNMTO: 0, OPEN: 0,
CLOSE: 0, LYOUT: 0, TRUNC: 0, SATTR: 0, XATTR: 0, HSM: 0
2014/11/26 20:59:19 [14613/2] STATS | MTIME: 0, CTIME: 0, ATIME: 0
But right now it seems to be stuck.
"gstack 14613" can indicate where it is stuck (if it is pure userland).
If gstack is stuck too, rbh is likely stuck in a filesystem call. In
this case, you can use the "crash" utility to determine the kernel
function where is it stuck.
Regards.
Craig
I don't know why this would slow things down, I thought it was
just an update optimization. I ran some tests with a different
changelog user and it seemed dumping the changelogs and updating
the position should never be a limitation as I was able to grab
over 100,000 entries and reset the count in a few seconds.
OK.
So I updated batch_ack_count to 10,000. Now the change log
processing rate seemed to go up to 1666 logs/second (over 30
seconds). This is better. If the rate is limited by the
database performance, then there probably isn't much more I can
do (comparing to scan rates).
"grep STAT" into robinhood log would help to indentify the
limitation you hit.
If you want to sample stats for a shorter period that the default
(which is 15 or 20minutes), you can change the "stats_interval" in
the config.
What do people use for a value of batch_ack_count on large, PB
sized, filesystems?
I think a good value is a few seconds of changelog processing. So
10k is a good value in you case.
Regards
Thanks,
Craig
On Tue, Nov 18, 2014 at 3:00 AM, LEIBOVICI Thomas
<[email protected] <mailto:[email protected]>> wrote:
Hi Craig,
No, it is njot expected to get such a slow processing speed.
According to the Lustre versions you run, this slow
processing may be due to the following Lustre bug:
https://jira.hpdd.intel.com/browse/LU-5405
It is a MDS fix. For now the fix is only landed in Lustre
2.5.4. I don't know if it can be backported to Lustre2.4...
Regards,
Thomas
On 11/17/14 21:11, Craig Tierney - NOAA Affiliate wrote:
Hi,
I have just installed Robinhood 2.5.3 to monitor a Lustre
2.4.3 system. The client on the server is running the 2.5.3
version. When I did an initlal scan of another test system
I saw scan rates of about 1000-2000 entries per second.
While I had configured robinhood to monitor this new system,
the Robinhood server was not running when we started to copy
data to the new filesystem. From the changelog statistics,
I am about 144m events behind. Processing the change logs
seems only be going at 375 entries per second.
Is this typical? I would have expected the processing of
changelog events to be much faster than this or at least as
fast as a normal file scan.
Thanks,
Craig
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration &
more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
robinhood-support mailing list
[email protected]
<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/robinhood-support
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
robinhood-support mailing list
[email protected]
<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/robinhood-support
------------------------------------------------------------------------
<http://www.avast.com/>
Ce courrier électronique ne contient aucun virus ou logiciel
malveillant parce que la protection Antivirus avast!
<http://www.avast.com/> est active.
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
robinhood-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/robinhood-support
