All, Problem resolved – the problem was a corrupt edb file on two of management servers. Found this when I decided to clear the caches on my Mgt servers and the edb files were permanently locked after all System Center processed stopped.
Peter Hakesley | Monitoring & Automation Technical Lead Engineer, Data Centre Services t: +44(0)845 155 6556 ext: 4006 e: [email protected] | w: www.scc.com<http://www.scc.com/> a: SCC, CV1, Cole Valley, 20 Westwood Avenue, Tyseley, Birmingham B11 3RZ From: [email protected] [mailto:[email protected]] On Behalf Of Pete Hakesley Sent: 07/September/2016 10:32 To: [email protected] Subject: RE: [msmom] RE: Aleret.AlertStage Yes for UR 9 I applied all SQL scripts. If I truncate the table within 20 mins I get over 110Million rows back. If I keep truncating that table will it eventually sort itself out? Peter Hakesley | Monitoring & Automation Technical Lead Engineer, Data Centre Services t: +44(0)845 155 6556 ext: 4006 e: [email protected]<mailto:[email protected]> | w: www.scc.com<http://www.scc.com/> a: SCC, CV1, Cole Valley, 20 Westwood Avenue, Tyseley, Birmingham B11 3RZ From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Kevin Holman Sent: 07/September/2016 08:52 To: [email protected]<mailto:[email protected]> Subject: RE: [msmom] RE: Aleret.AlertStage No, alerts are synched from the opsdb to these. There is an alert synch role hosted by the all management servers resource pool that manages this. So all alerts in the opsdb will try and synch. Are you sure when you applied ur9 you did the scripts in SQL? There were some fixes for managed entity synch that could be related. ________________________________ From: Pete Hakesley<mailto:[email protected]> Sent: 9/7/2016 2:49 AM To: [email protected]<mailto:[email protected]> Subject: [msmom] RE: Aleret.AlertStage Thnaks Kevin, That was the procedure I was working to but it is still a problem. Can you tell me where the data for the alertstage comes from it it directly from the management servers? Peter Hakesley | Monitoring & Automation Technical Lead Engineer, Data Centre Services t: +44(0)845 155 6556 ext: 4006 e: [email protected]<mailto:[email protected]> | w: www.scc.com<http://www.scc.com/> a: SCC, CV1, Cole Valley, 20 Westwood Avenue, Tyseley, Birmingham B11 3RZ From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Kevin Holman Sent: 06/September/2016 15:26 To: [email protected]<mailto:[email protected]> Subject: [msmom] RE: Aleret.AlertStage Generally when I see alertstage backups, there is a problem with an alert(s) being inserted, and you can find the events on the management server event logs discussing the problem. There is an issue with a managed entity, or a specific alert that cannot be inserted due to an issue. You need the error events from the management servers – or run the maintenance manually to output the error. One possibility is if you flood the environment with WAY more alerts than it is designed to handle, such as a storm from a badly written MP, you could run into a RESEED issue, such as: https://blogs.technet.microsoft.com/sudheesn/2009/07/27/getting-31552-gtexception-sqlexception-timeout-expired-very-frequently-in-scom-server/ I cannot say that’s your problem, we need the error/warning events from the management servers to see what the root issue is. From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Pete Hakesley Sent: Tuesday, September 6, 2016 2:51 AM To: [email protected]<mailto:[email protected]> Subject: [msmom] RE: Aleret.AlertStage Hi Kevin, Thanks for the response I had checked the dataset and it is there SchemaName Alert DataSetID A9FD….. Also yes in my stored procedures I have a dbo.AlertProcessStaging My Alert.AlertStaging row count is currently 111,577,637 !! Peter Hakesley | Monitoring & Automation Technical Lead Engineer, Data Centre Services t: +44(0)845 155 6556 ext: 4006 e: [email protected]<mailto:[email protected]> | w: www.scc.com<http://www.scc.com/> a: SCC, CV1, Cole Valley, 20 Westwood Avenue, Tyseley, Birmingham B11 3RZ From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Kevin Holman Sent: 05/September/2016 17:13 To: [email protected]<mailto:[email protected]> Subject: [msmom] RE: Aleret.AlertStage If you cannot find this, that might be your issue. That is not normal. What does this return? select * from StandardDataset where SchemaName = 'Alert' If you look at your stored procs in the DW – do you have dbo.AlertProcessStaging ? From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Pete Hakesley Sent: Monday, September 5, 2016 10:04 AM To: [email protected]<mailto:[email protected]> Subject: [msmom] Aleret.AlertStage All, I have a very large Alert.AlertStage table >500GB I have decided that truncation is the simplist method to resolve this however. |Following various articles some as to disable the DW maintenance rule for alert dataset. However, I cannot find this dataset in the DataWarehouse rule Question is this dataset now been removed for override? Also every time I truncate the table it just fills with 100K’s or rows of data and fills the dataset. Can anyone help. Ps. SCOM 2012 R2 UR9 and I do not get any errors in the Operations Manager Logs on any of my MS servers? e.g. a. Disable Alert Dataset maintenance rule from Operations Console. Go to Authoring pane-> Expand Management Pack Objects–> Select Rules Scope the Rules to “Standard Data Set” (use View All Targets option). Now we will see a rule called “Standard Data Warehouse Data Set Maintenance Rule”. Override for a Specific object –> select “Alert Data Set” and save it in any Unsealed MP. Peter Hakesley | Monitoring & Automation Technical Lead Engineer, Data Centre Services t: +44(0)845 155 6556 ext: 4006 e: [email protected]<mailto:[email protected]> | w: www.scc.com<http://www.scc.com/> a: SCC, CV1, Cole Valley, 20 Westwood Avenue, Tyseley, Birmingham B11 3RZ
