Kevin (and all): I forgot to follow up on this. It turned out to be a service who's associated AD account password was reset. This caused Event IDs 7000, 7031, and 7038 to be generated at ~60events per second on that monitored server. We updated the service account password, which calmed down the alerts. We did an internal review and decided there is no need to collect "Service or Driver Failed to Start" events. We disabled that Event Collection rule. We also disabled the collection rule for unexpected service terminations.
Once that was taken care of, we used the dwdatarp utility to adjust some of the retention/grooming on the Data Warehouse database. We are now back to "normal" event loads from our environment. I would also note in the support reps closure email there were multiple reference links-all but one of them were links to your own blog. So thank you...again! From: [email protected] [mailto:[email protected]] On Behalf Of Kevin Holman Sent: Monday, November 27, 2017 6:55 AM To: [email protected] Subject: [msmom] RE: Large DW Growth ATTENTION: This email came from an external source. DO NOT open attachments or click on links from unknown senders or unexpected emails. I would simply find out what event it likely is, and go from there: --Most Common Events by event number: (This helps us know which event ID's are the most common in the database) SELECT top 50 EventDisplayNumber, COUNT(*) AS 'TotalEvents' FROM Event.vEvent GROUP BY EventDisplayNumber ORDER BY TotalEvents DESC --Most Common Events by event number going back a set number of days: (This helps us know which event ID's are the most common in the database recently) SELECT top 50 EventDisplayNumber, COUNT(*) AS 'TotalEvents' FROM Event.vEvent WHERE DateTime > dateadd(dd,-10,getutcdate()) GROUP BY EventDisplayNumber ORDER BY TotalEvents DESC From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Orlebeck, Geoffrey Sent: Monday, November 20, 2017 4:34 PM To: [email protected]<mailto:[email protected]> Subject: [msmom] Large DW Growth All: I'm trying to trace back to something that may have changed on a specific date. I'm using a query from Kevin Holman to see the 'NumEventsPerDay' values, and on a specific date (11/06/17), our NumEventsPerDay has spiked drastically from a previous average of right around 5-6k to today which has (so far) 4,382,291. This is my first time having to dive into the DataWarehouse DB, so I've been reading up some of Kevin's stuff and TechNet, but I'm having trouble figuring out a meaningful query to determine what may be the root cause. There are usually only two of us that modify Management Packs, but neither of us recall any specific changes that correlate with the date range-though I'm pretty convinced it was a change we introduced. Any help is greatly appreciated. Kevin Holman query and the results: SELECT CASE WHEN(GROUPING(CONVERT(VARCHAR(20), DateTime, 101)) = 1) THEN 'All Days' ELSE CONVERT(VARCHAR(20), DateTime, 101) END AS DayAdded, COUNT(*) AS NumEventsPerDay FROM Event.vEvent GROUP BY CONVERT(VARCHAR(20), DateTime, 101) WITH ROLLUP ORDER BY DayAdded DESC DayAdded NumEventsPerDay 11/20/2017 4382291 11/19/2017 5279380 11/18/2017 5308107 11/17/2017 5324428 11/16/2017 5031736 11/15/2017 5065012 11/14/2017 4908262 11/13/2017 4481308 11/12/2017 4452455 11/11/2017 3516115 11/10/2017 1027510 11/09/2017 37229 11/08/2017 25581 11/07/2017 21306 11/06/2017 10804 11/05/2017 5578 11/04/2017 5737 11/03/2017 5818 11/02/2017 5988 11/01/2017 5993 10/31/2017 5857 10/30/2017 5800 10/29/2017 5740 Confidentiality Notice: This is a transmission from Montage Health. This message and any attached documents may be confidential and contain information protected by state and federal medical privacy statutes. They are intended only for the use of the addressee. If you are not the intended recipient, any disclosure, copying, or distribution of this information is strictly prohibited. If you received this transmission in error, please accept our apologies and notify the sender. Thank you. Confidentiality Notice: This is a transmission from Montage Health. This message and any attached documents may be confidential and contain information protected by state and federal medical privacy statutes. They are intended only for the use of the addressee. If you are not the intended recipient, any disclosure, copying, or distribution of this information is strictly prohibited. If you received this transmission in error, please accept our apologies and notify the sender. Thank you.
