William, I don't have an point answer, but some thoughts....
First, remember there is a major difference between a FIRST run of reconciliation to get all the data established and the ongoing nightly (or hourly or continuous or whatever you configure) work once the CMDB is established. Reconciliation will run only on CHANGED things and the number of changes in the CMDB on a daily basis is tiny compared to the size of your CMDB - even with millions of CIs, changes are in the 10s of thousands or less range. So, even at the rate you are seeing now, you would easily keep up in a steady state environment. Now, 7000 an hour are about 2 per second. We commonly see rates at the 20 to 30 per second range with reconciliation. So, there is an issue. Do you have reconciliation configured single threaded or multi-threaded? If single threaded, you may be getting what you should expect. If multi-threaded, you should be getting much better throughput than you are seeing. Are you running ONE reconciliation job to do everything or multiple jobs? One idea is to run one job per class so you can parallelize the work - mainly something to consider for an initial load rather than steady state where things can keep up without issue. Make sure you do the CI classes first and finish them before starting work on the Relationships so that all your CIs are correctly merged and in place when you try and establish relationships. If you have a class with a million items, you could split that further by having reconciliation jobs that segment a single class and run in parallel there too. So, you could have something on something like Computer System where anything with a host name starting with a-f is in one job, g-o in another and p-z in a third (or however many you want to split into. I have run situations where we changed reconcile speed from the 1 to 2 per second to the 25 per second range by using multi-threading and running parallel reconciliation jobs. (and this is just one example). NOTE: Don't overdo it. I would not have more than say 10 to 12 parallel jobs running or you start getting diminishing returns with contention in the DB. You didn't mention version of the CMDB (but the following are all generic performance tunes to the CMDB) Make sure things like Entry ID block size are turned ON for the CMDB tables (especially base element and base relationship) Make sure that Status History is turned OFF for every CMDB data form (especially base element and base relationship) This should be done in current releases but if you have gone through a set of upgrades, make sure this is done as it does save time and CMDB does not use status history at all on the data tables. Don't worry about it on the CMDB metadata tables. If you have ITSM installed and you are pre-8.0, the Asset fields are attributes in the CMDB. 7 or 8 of them are currency types and they default to 5 or 6 functional currencies. Removing all the functional currencies you don't need from these currency fields (all are in base element) will help create/modify performance. NOTE the version here is of ITSM not the CMDB that is important. 8.0 and later ITSM does not have this issue as there are no Asset lifecycle fields (and so no currency fields) put into the CMDB. Also, depending on the version and patch level, there have been a couple of key design changes to the reconciliation engine itself that have a major affect on throughput of the engine (older versions did some SQL COUNT operations that were terribly slow and the new versions do not do this COUNT and it yields a significant performance gain - the count operation could take minutes to execute). So, something to check if you are using an older version of the CMDB. These are all things to look at today. Going forward, there are a couple of additional significant tunes that are being done to both the reconciliation engine itself and to the CMDB structure to further improve throughput and performance. News about those will come as the capabilities are added to the system. I hope this information is useful, Doug Mueller From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of William Rentfrow Sent: Monday, June 23, 2014 9:43 AM To: [email protected] Subject: Reconciliation speed ** Hi - We're trying to get our recon jobs faster. We have hundreds of thousands of CI's in certain classes and we will have over a million in some other classes. Right now we're getting about 7000 per hour - which means the tool are basically unusable because it takes waaaaaaaaaaaaaaaaaaay too long to go through a recon job. This is with dedicated servers and having gone through many (all?) of BMC's performance tuning papers we're still not able to get much faster than that. William Rentfrow [email protected]<mailto:[email protected]> Office: 715-204-3061 Cell: 715-398-5056 _ARSlist: "Where the Answers Are" and have been for 20 years_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the Answers Are, and have been for 20 years"
