Hello Thomas,
thanks for your email, following the details about the STATS during the
FSSCAN:
## BEST RESULT with user_acct group_acct disabled / multithreads DB :
2014/06/13 08:08:59 robinhood@robinhood[6946/4] STATS | DB_APPLY : last:
61E893EF/144115675168894784, status=processing
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | ====================
Dumping stats at 2014/06/13 08:09:59 =====================
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | ======== General
statistics =========
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | Daemon start time:
2014/06/13 08:06:58
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | Started modules: scan
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | ======== FS scan
statistics =========
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | current scan interval =
6.0h
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | scan is running:
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | started at :
2014/06/13 08:06:59 (3.0min ago)
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | last action:
2014/06/13 08:09:59 (00s ago)
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | progress :
2020110 entries scanned (0 errors)
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | avg. speed : 2.85
ms/entry/thread -> 11231.02 entries/sec
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | inst. speed: 4.20
ms/entry/thread -> 7618.37 entries/sec
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | ==== EntryProcessor
Pipeline Stats ===
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | Idle threads: 9
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | Id constraints count:
11 (hash min=0/max=1/avg=0.0)
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | Stage |
Wait | Curr | Done | Total | ms/op |
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | 0: GET_FID |
0 | 0 | 0 | 0 | 0.00 |
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | 1: GET_INFO_DB |
0 | 1 | 0 | 2021254 | 0.20 |
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | 2: GET_INFO_FS |
0 | 0 | 0 | 2021254 | 0.01 |
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | 3: REPORTING |
2 | 0 | 0 | 2021252 | 0.00 |
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | 4: PRE_APPLY |
0 | 1 | 0 | 2021251 | 0.00 |
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | 5: DB_APPLY |
0 | 7 | 0 | 2021244 | 0.95 |
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | 6: RM_OLD_ENTRIES |
0 | 0 | 0 | 0 | 0.00 |
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | DB ops:
get=89/ins=2021155/upd=89/rm=0
2014/06/13 08:09:59 robinhood@robinhood[6946/4] STATS | --- Pipeline stage
details --
Filesystem scan activity:
Current scan interval: 6.0h
Last filesystem scan:
status: done
start: 2014/06/13 08:06:59
end: 2014/06/13 08:10:40
duration: 03min 41s
Statistics:
entries scanned: 2523759
errors: 0
timeouts: 0
# threads: 32
average speed: 11428.57 entries/sec
##########################################
## RESULT with user_acct group_acct enabled / multithreads DB :
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | ====================
Dumping stats at 2014/06/13 08:40:27 =====================
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | ======== General
statistics =========
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | Daemon start time:
2014/06/13 08:30:27
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | Started modules: scan
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | ======== FS scan
statistics =========
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | current scan interval
= 6.0h
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | scan is running:
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | started at :
2014/06/13 08:30:27 (10.0min ago)
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | last action:
2014/06/13 08:40:27 (00s ago)
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | progress :
1214687 entries scanned (0 errors)
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | avg. speed :
15.25 ms/entry/thread -> 2098.43 entries/sec
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | inst. speed:
22.94 ms/entry/thread -> 1394.92 entries/sec
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | ==== EntryProcessor
Pipeline Stats ===
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | Idle threads: 16
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | Id constraints count:
100000 (hash min=8/max=19/avg=12.6)
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | Stage |
Wait | Curr | Done | Total | ms/op |
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | 0: GET_FID |
0 | 0 | 0 | 0 | 0.00 |
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | 1: GET_INFO_DB |
0 | 0 | 0 | 1248125 | 0.21 |
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | 2: GET_INFO_FS |
0 | 0 | 0 | 1248125 | 0.01 |
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | 3: REPORTING |
0 | 0 | 0 | 1248125 | 0.00 |
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | 4: PRE_APPLY |
0 | 0 | 0 | 1248125 | 0.00 |
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | 5: DB_APPLY
|99983 | 16 | 0 | 1148126 | 8.17 |
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | 6: RM_OLD_ENTRIES |
0 | 0 | 0 | 0 | 0.00 |
2014/06/13 08:40:27 robinhood@robinhood[23160/4] STATS | DB ops:
get=89/ins=1148037/upd=89/rm=0
Filesystem scan activity:
Current scan interval: 6.0h
Last filesystem scan:
status: done
start: 2014/06/13 08:30:27
end: 2014/06/13 08:52:28
duration: 22min 01s
Statistics:
entries scanned: 2523760
errors: 0
timeouts: 0
# threads: 32
average speed: 1923.08 entries/sec
###################################
## RESULT with user_acct group_acct enabled / batch DB :
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | ======== FS scan
statistics =========
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | current scan interval
= 6.0h
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | scan is running:
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | started at :
2014/06/13 09:00:54 (5.0min ago)
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | last action:
2014/06/13 09:05:54 (00s ago)
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | progress :
811449 entries scanned (0 errors)
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | avg. speed :
11.73 ms/entry/thread -> 2728.74 entries/sec
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | inst. speed:
12.80 ms/entry/thread -> 2499.13 entries/sec
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | ==== EntryProcessor
Pipeline Stats ===
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | Idle threads: 8
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | Id constraints count:
99999 (hash min=4/max=24/avg=12.6)
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | Stage |
Wait | Curr | Done | Total | ms/op |
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | 0: GET_FID |
0 | 0 | 0 | 0 | 0.00 |
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | 1: GET_INFO_DB |
0 | 0 | 0 | 817744 | 0.17 |
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | 2: GET_INFO_FS |
0 | 0 | 0 | 817744 | 0.01 |
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | 3: REPORTING |
0 | 0 | 0 | 817744 | 0.00 |
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | 4: PRE_APPLY |
0 | 0 | 0 | 817744 | 0.00 |
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | 5: DB_APPLY
|99995 | 4 | 0 | 717745 | 0.36 | 89.96% batched (avg batch size: 9.7)
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | 6: RM_OLD_ENTRIES |
0 | 0 | 0 | 0 | 0.00 |
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | DB ops:
get=89/ins=717656/upd=89/rm=0
2014/06/13 09:05:54 robinhood@santisrbh01[7739/4] STATS | --- Pipeline stage
details ---
Filesystem scan activity:
Current scan interval: 6.0h
Last filesystem scan:
status: done
start: 2014/06/13 09:00:54
end: 2014/06/13 09:15:41
duration: 14min 47s
Statistics:
entries scanned: 2523760
errors: 0
timeouts: 0
# threads: 32
average speed: 2862.25 entries/sec
#########################
## RESULT with user_acct group_acct disabled / batch DB :
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | ======== FS scan
statistics =========
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | current scan interval
= 6.0h
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | scan is running:
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | started at :
2014/06/13 09:18:40 (1.0min ago)
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | last action:
2014/06/13 09:19:40 (00s ago)
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | progress :
358138 entries scanned (0 errors)
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | avg. speed : 5.25
ms/entry/thread -> 6098.09 entries/sec
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | inst. speed:
26.34 ms/entry/thread -> 1214.79 entries/sec
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | ==== EntryProcessor
Pipeline Stats ===
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | Idle threads: 1
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | Id constraints count:
89022 (hash min=6/max=20/avg=11.2)
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | Stage |
Wait | Curr | Done | Total | ms/op |
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | 0: GET_FID |
0 | 0 | 0 | 0 | 0.00 |
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | 1: GET_INFO_DB
|11928 | 2 | 23902 | 347795 | 0.15 |
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | 2: GET_INFO_FS |
0 | 0 | 0 | 323893 | 0.01 |
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | 3: REPORTING |
0 | 0 | 0 | 323893 | 0.00 |
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | 4: PRE_APPLY |
0 | 0 | 0 | 323893 | 0.00 |
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | 5: DB_APPLY
|53175 | 11 | 0 | 270707 | 0.18 | 89.33% batched (avg batch size: 9.3)
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | 6: RM_OLD_ENTRIES |
0 | 0 | 0 | 0 | 0.00 |
2014/06/13 09:19:40 robinhood@robinhood[24169/4] STATS | DB ops:
get=24/ins=270683/upd=24/rm=0
Filesystem scan activity:
Current scan interval: 6.0h
Last filesystem scan:
status: done
start: 2014/06/13 09:18:40
end: 2014/06/13 09:27:31
duration: 08min 51s
Statistics:
entries scanned: 2523760
errors: 0
timeouts: 0
# threads: 32
average speed: 4790.42 entries/sec
#########################################
As you can see the result with multithread DB is much higher than batch DB
11428.57 entries/sec vs 4790.42 entries/sec
Something that could be helpful in the documentation I think would be the fact
that formatting the file system were the mysql is running as XFS will offer
better results in terms of scan performance. Indeed with ext4 I reached at
maximum 3800 entries/sec with user_acct group_acct disabled, instead with XFS I
reached 11000 entries/sec.
Is it correct to say that the user_acct group_acct are only needed for the
webgui? Because if I'm not wrong from command line, even if I clean up the DB
and I scan without user_acc group_acct, I got all the needed data, but I cannot
use the web gui.
Cheers
Fabio
--
- Fabio Verzelloni - CSCS - Swiss National Supercomputing Centre
via Trevano 131 - 6900 Lugano, Switzerland
Tel: +41 (0)91 610 82 04
________________________________________
From: LEIBOVICI Thomas [[email protected]]
Sent: Wednesday, June 11, 2014 4:38 PM
To: Verzelloni Fabio; [email protected]
Subject: Re: [robinhood-support] question about 'best configuration practice' &
'mysql errors'
Hi Fabio,
Congratulations. I'm impressed of your in-depth tuning of robinhood,
mysql and your system.
Robinhood should dump stats in its logs at regular interval.
It would help identifying bottlenecks if you could send an extract of
it, like this:
2014/05/21 21:08:34 [13362/4] STATS | ======== FS scan statistics =========
2014/05/21 21:08:34 [13362/4] STATS | scan is running:
2014/05/21 21:08:34 [13362/4] STATS | started at : 2014/05/21
09:08:26 (12.0h ago)
2014/05/21 21:08:34 [13362/4] STATS | last action: 2014/05/21
21:08:33 (01s ago)
2014/05/21 21:08:34 [13362/4] STATS | progress : 6349237 entries
scanned (0 errors)
2014/05/21 21:08:34 [13362/4] STATS | avg. speed : 103.84
ms/entry/thread -> 154.09 entries/sec
2014/05/21 21:08:34 [13362/4] STATS | inst. speed: 41.30
ms/entry/thread -> 387.39 entries/sec
2014/05/21 21:08:34 [13362/4] STATS | ==== EntryProcessor Pipeline Stats ===
2014/05/21 21:08:34 [13362/4] STATS | Idle threads: 15
2014/05/21 21:08:34 [13362/4] STATS | Id constraints count: 10000 (hash
min=0/max=7/avg=1.3)
2014/05/21 21:08:34 [13362/4] STATS | Stage | Wait | Curr | Done |
Total | ms/op |
2014/05/21 21:08:34 [13362/4] STATS | 0: GET_FID | 0 | 0 | 0
| 0 | 0.00 |
2014/05/21 21:08:34 [13362/4] STATS | 1: GET_INFO_DB | 0 | 0 |
0 | 6603345 | 0.25 |
2014/05/21 21:08:34 [13362/4] STATS | 2: GET_INFO_FS | 0 | 0 |
0 | 6603345 | 0.03 |
2014/05/21 21:08:34 [13362/4] STATS | 3: REPORTING | 0 | 0 | 0
| 6603345 | 0.00 |
2014/05/21 21:08:34 [13362/4] STATS | 4: PRE_APPLY | 0 | 0 | 0
| 6603345 | 0.00 |
2014/05/21 21:08:34 [13362/4] STATS | 5: DB_APPLY | 9999 | 1 | 0
| 6593345 | 6.54 | 98.35% batched (avg batch size: 85.1)
2014/05/21 21:08:34 [13362/4] STATS | 6: RM_OLD_ENTRIES | 0 | 0
| 0 | 0 | 0.00 |
I noticed that you disabled DB batching to use multithreaded DB
operations. Did you get better results this way ?
Your hardware looks appropriate. The stats dump will give use more
information.
Regards
On 06/06/14 09:29, Verzelloni Fabio wrote:
> Hello folks,
> I'm doing some test on robinhood in our environment, some details
> regarding the hardware in use:
>
> ## Robinhood server ##
>
> IBM M3550 X4
> 128Gb RAM
> 256G HD SSD for mysql
> 2* Intel Xeon Processor E5-2650v2 8C 2.6GHz 20MB Cache 1866MHz
> Lustre 2.5.1
> Mysql 5.5
>
> ## Luster Version in production ##
> lustre 2.1
>
> ## Robinhood.conf ##
>
> FS_Scan {
> nb_threads_scan = 32;
> nb_prealloc_tasks=10000;
> }
>
> EntryProcessor {
> nb_threads = 32;
> STAGE_GET_FID_threads_max = 16;
> STAGE_GET_INFO_DB_threads_max = 4;
> STAGE_GET_INFO_FS_threads_max = 4;
> STAGE_REPORTING_threads_max = 1;
> STAGE_DB_APPLY_threads_max = 16;
> STAGE_CHGLOG_CLR_threads_max = 1;
> STAGE_RM_OLD_ENTRIES_threads_max = 1;
> max_pending_operations = 1000;
> max_batch_size=1;
> }
>
> ## My.cnf ##
>
> [mysqld]
> large-pages
> datadir=/var/lib/mysql
> socket=/var/lib/mysql/mysql.sock
> user=mysql
> # Disabling symbolic-links is recommended to prevent assorted security risks
> symbolic-links=0
> innodb_flush_log_at_trx_commit = 0
> # possibly the most important setting
> max_connections= 512
> innodb_buffer_pool_size= 60G
> # ~50% of memory
> innodb_max_dirty_pages_pct= 15
> innodb_thread_concurrency= 32
> innodb_log_file_size= 100M
> innodb_log_buffer_size= 50M
> innodb_data_file_path= ibdata1:1G:autoextend
> # kernel must be configured for support
> table-open-cache= 2000
> sort-buffer-size= 32M
> read-buffer-size= 16M
> read-rnd-buffer-size= 4M
> thread-cache-size= 128
> query-cache-size= 40M
> query-cache-limit= 1M
> tmp-table-size= 16M
>
> [mysqld_safe]
> log-error=/var/log/mysqld.log
> pid-file=/var/run/mysqld/mysqld.pid
>
> ## vm.nr_hugepages ##
>
> vm.nr_hugepages = 50000
> vm.nr_hugepages_mempolicy = 50000
> vm.hugetlb_shm_group = 27
> vm.hugepages_treat_as_movable = 0
> vm.nr_overcommit_hugepages = 0
>
> ## sysctl ##
>
> kernel.shmmax = 118111600640
> kernel.shmall = 118111600640
>
> ## limits.conf ##
>
> mysql hard memlock unlimited
> mysql soft memlock unlimited
> ---
>
> I'm trying to find the best configuration to reach the best "entries/sec",
> and with this configuration the best number I can get is 2600 ~ entries/sec.
> Do you think that based on the HW in use is it possible to improve the speed
> of the scan?
> What's the best practice to better configure the server to perform the best
> speed of scan?
>
> While I'm running the initial scan I see a lot of the following messages:
>
> ...
> 2014/06/06 08:34:20 [10535/15] ListMgr | Retryable DB error in ListMgr_Insert
> l.218. Restarting transaction in 1 sec...
> 2014/06/06 08:34:21 [10535/15] ListMgr | DB deadlock detected
> ...
>
> I was hoping to reach 4000 / 5000 entries/sec do you think with the HW I have
> available I can manage to reach that result? Suggestions or questions are
> welcome.
>
> Regards
> Fabio
>
>
> --
> - Fabio Verzelloni - CSCS - Swiss National Supercomputing Centre
> via Trevano 131 - 6900 Lugano, Switzerland
> Tel: +41 (0)91 610 82 04
>
>
> ------------------------------------------------------------------------------
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and their
> applications. Written by three acclaimed leaders in the field,
> this first edition is now available. Download your free book today!
> http://p.sf.net/sfu/NeoTech
> _______________________________________________
> robinhood-support mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/robinhood-support
------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
_______________________________________________
robinhood-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/robinhood-support
