Hi Doug -
Pardon the frustration :) Anyway, the CMDB jobs are running on dedicated
servers in the server group.
We've done all of the performance tuning with threads, etc. For example, all
of the CMDB pluginsvr_config.xml files have this set:
<numCoreThreads>30</numCoreThreads>
<numSelectorThreads>2</numSelectorThreads>
The first value was set under support's help. Everything in the recon console
is set to use RPC 390698 with a min/max of 8/16 defined for those. All 16 of
the threads under that RPC are in use, and the idle times are low (although
it's hard to tell, because the times include time where the recon engine was
not doing anything). But at peak usage when running jobs we are definitly
using them all.
I just check Base Relationship and it has 9,664,346 records - so in short it's
LOTS of records. This is running on a 3-node Oracle RAC system and the
database is rarely using a lot of resources. It's just doing table scans quite
often due to the way the lookups are written.
I did also engage our DBA's to see if there's anything they can do.
From: Action Request System discussion list(ARSList)
[mailto:[email protected]] On Behalf Of Mueller, Doug
Sent: Monday, June 23, 2014 1:29 PM
To: [email protected]
Subject: Re: Reconciliation speed
**
William,
I don't have an point answer, but some thoughts....
First, remember there is a major difference between a FIRST run of
reconciliation to get all the data
established and the ongoing nightly (or hourly or continuous or whatever you
configure) work once the
CMDB is established. Reconciliation will run only on CHANGED things and the
number of changes in the
CMDB on a daily basis is tiny compared to the size of your CMDB - even with
millions of CIs, changes are
in the 10s of thousands or less range. So, even at the rate you are seeing
now, you would easily keep up
in a steady state environment.
Now, 7000 an hour are about 2 per second. We commonly see rates at the 20 to
30 per second range with
reconciliation. So, there is an issue.
Do you have reconciliation configured single threaded or multi-threaded? If
single threaded, you may be
getting what you should expect. If multi-threaded, you should be getting much
better throughput than
you are seeing.
Are you running ONE reconciliation job to do everything or multiple jobs? One
idea is to run one job per
class so you can parallelize the work - mainly something to consider for an
initial load rather than steady
state where things can keep up without issue. Make sure you do the CI classes
first and finish them before
starting work on the Relationships so that all your CIs are correctly merged
and in place when you try and
establish relationships.
If you have a class with a million items, you could split that further by
having reconciliation jobs that segment
a single class and run in parallel there too. So, you could have something on
something like Computer System
where anything with a host name starting with a-f is in one job, g-o in another
and p-z in a third (or however
many you want to split into.
I have run situations where we changed reconcile speed from the 1 to 2 per
second to the 25 per second range
by using multi-threading and running parallel reconciliation jobs. (and this
is just one example).
NOTE: Don't overdo it. I would not have more than say 10 to 12 parallel jobs
running or you start getting
diminishing returns with contention in the DB.
You didn't mention version of the CMDB (but the following are all generic
performance tunes to the CMDB)
Make sure things like Entry ID block size are turned ON for the CMDB tables
(especially base element and
base relationship)
Make sure that Status History is turned OFF for every CMDB data form
(especially base element and
base relationship) This should be done in current releases but if you have
gone through a set of upgrades,
make sure this is done as it does save time and CMDB does not use status
history at all on the data tables.
Don't worry about it on the CMDB metadata tables.
If you have ITSM installed and you are pre-8.0, the Asset fields are attributes
in the CMDB. 7 or 8 of them are
currency types and they default to 5 or 6 functional currencies. Removing all
the functional currencies you
don't need from these currency fields (all are in base element) will help
create/modify performance. NOTE
the version here is of ITSM not the CMDB that is important. 8.0 and later ITSM
does not have this issue as
there are no Asset lifecycle fields (and so no currency fields) put into the
CMDB.
Also, depending on the version and patch level, there have been a couple of key
design changes to the
reconciliation engine itself that have a major affect on throughput of the
engine (older versions did some
SQL COUNT operations that were terribly slow and the new versions do not do
this COUNT and it yields a
significant performance gain - the count operation could take minutes to
execute). So, something to check
if you are using an older version of the CMDB.
These are all things to look at today.
Going forward, there are a couple of additional significant tunes that are
being done to both the
reconciliation engine itself and to the CMDB structure to further improve
throughput and performance. News
about those will come as the capabilities are added to the system.
I hope this information is useful,
Doug Mueller
From: Action Request System discussion list(ARSList)
[mailto:[email protected]] On Behalf Of William Rentfrow
Sent: Monday, June 23, 2014 9:43 AM
To: [email protected]
Subject: Reconciliation speed
**
Hi -
We're trying to get our recon jobs faster. We have hundreds of thousands of
CI's in certain classes and we will have over a million in some other classes.
Right now we're getting about 7000 per hour - which means the tool are
basically unusable because it takes waaaaaaaaaaaaaaaaaaay too long to go
through a recon job.
This is with dedicated servers and having gone through many (all?) of BMC's
performance tuning papers we're still not able to get much faster than that.
William Rentfrow
[email protected]<mailto:[email protected]>
Office: 715-204-3061
Cell: 715-398-5056
_ARSlist: "Where the Answers Are" and have been for 20 years_
________________________________
No virus found in this message.
Checked by AVG - www.avg.com<http://www.avg.com>
Version: 2014.0.4592 / Virus Database: 3972/7728 - Release Date: 06/23/14
_ARSlist: "Where the Answers Are" and have been for 20 years_
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
"Where the Answers Are, and have been for 20 years"
