You have two major issues.
1. Old statechanges. You have 95 million total statechanges in the DB, but of
these, 91 million are older than your statechange retention. This is caused by
one or both:
a. flooding of statechanges for monitors that are still enabled but
unhealthy when grooming runs.
b. old monitors that flooded data in the past but are now disabled or
otherwise unloaded.
You need to run the cleanup script I have posted here:
http://blogs.technet.com/b/kevinholman/archive/2009/12/21/tuning-tip-do-you-have-monitors-constantly-flip-flopping.aspx
That is a VERY resource intensive script, it cleans this stuff up in batches to
keep the transaction logs from filling, but man, I have never had to clean up
60GB before..... so this could take a LONG time and absolutely hammer your
database server.... so I'd do this on a weekend and monitor the progress.
2. Flooding statechanges. You have 400,000 statechanges coming in a day.
That's a lot, even for a large environment. You should spend some time tuning
this environment, starting with the most offensive monitors: From:
http://blogs.technet.com/b/kevinholman/archive/2007/10/18/useful-operations-manager-2007-sql-queries.aspx
Noisiest monitors changing state in the database in the last 7 days:
select distinct top 50 count(sce.StateId) as NumStateChanges,
m.DisplayName as MonitorDisplayName,
m.Name as MonitorIdName,
mt.typename AS TargetClass
from StateChangeEvent sce with (nolock)
join state s with (nolock) on sce.StateId = s.StateId
join monitorview m with (nolock) on s.MonitorId = m.Id
join managedtype mt with (nolock) on m.TargetMonitoringClassId =
mt.ManagedTypeId
where m.IsUnitMonitor = 1
-- Scoped to within last 7 days
AND sce.TimeGenerated > dateadd(dd,-7,getutcdate())
group by m.DisplayName, m.Name,mt.typename
order by NumStateChanges desc
Resolving #1 above will give you back a LOT of free space.
Resolving #2 above will keep it that way....otherwise you will be back into
this boat in NO TIME.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Narendra Bathula
Sent: Wednesday, January 21, 2015 4:30 PM
To: [email protected]
Subject: RE: [msmom] SV: scom 2007 R2 db size is huge
Oh...My God...
I have run the queries and the out put is below and attached the state changes
for day report.
state changes you are inserting per day:
DayGenerated State changes per day
All Days 94736294
2015.01.21 409944
2015.01.20 400424
2015.01.19 430796
2015.01.18 928245
2015.01.17 397198
2015.01.16 390819
2015.01.15 398827
2015.01.14 390893
2015.01.13 301351
2015.01.12 307750
2015.01.11 446894
2015.01.10 298380
2015.01.09 302589
2015.01.08 302818
2015.01.07 301092
2015.01.06 298855
2015.01.05 304668
2015.01.04 436911
2015.01.03 307480
2015.01.02 303202
2015.01.01 297335
2014.12.31 295587
2014.12.30 296586
2014.12.29 298869
2014.12.28 440319
2014.12.27 299234
2014.12.26 298075
2014.12.25 297859
2014.12.24 297568
2014.12.23 296091
2014.12.22 309781
2014.12.21 776963
2014.12.20 325212
2014.12.19 292141
2014.12.18 296494
2014.12.17 297879
2014.12.16 297396
2014.12.15 301138
How old your statechange data is:
Total state changes with in grooming retention >Grooming retentiion
>30 days >90 days >365days
94736338 3356643 91379695
83959536 63637681 12783381
Thanks & Regards,
Narendra
________________________________________
From: [email protected] [[email protected]] on behalf
of Kevin Holman [[email protected]]
Sent: Wednesday, January 21, 2015 11:47 PM
To: [email protected]
Subject: RE: [msmom] SV: scom 2007 R2 db size is huge
You have 60 GIG of state changes. That should be pretty obvious.... no?
Quite frankly, I am not sure how this environment is still standing.
:-)
I'm serious.
Ok, the next step is to figure out - are these still ongoing problems flooding
your SCOM database, or are these old statechanges that are failing to groom or
not trying to groom for other reasons.
Next (logical) steps:
Find out how many state changes you are inserting per day:
--------------------------------------------
SELECT CASE WHEN(GROUPING(CONVERT(VARCHAR(20), TimeGenerated, 102)) = 1) THEN
'All Days' ELSE CONVERT(VARCHAR(20), TimeGenerated, 102) END AS DayGenerated,
COUNT(*) AS StateChangesPerDay FROM StateChangeEvent WITH (NOLOCK) GROUP BY
CONVERT(VARCHAR(20), TimeGenerated, 102) WITH ROLLUP ORDER BY DayGenerated DESC
---------------------------------------------
Then find out how old your statechange data is:
-------------------------------------------------
declare @statedaystokeep INT
SELECT @statedaystokeep = DaysToKeep from PartitionAndGroomingSettings WHERE
ObjectName = 'StateChangeEvent'
SELECT COUNT(*) as 'Total StateChanges', count(CASE WHEN sce.TimeGenerated >
dateadd(dd,-@statedaystokeep,getutcdate()) THEN sce.TimeGenerated ELSE NULL
END) as 'within grooming retention', count(CASE WHEN sce.TimeGenerated <
dateadd(dd,-@statedaystokeep,getutcdate()) THEN sce.TimeGenerated ELSE NULL
END) as '> grooming retention', count(CASE WHEN sce.TimeGenerated <
dateadd(dd,-30,getutcdate()) THEN sce.TimeGenerated ELSE NULL END) as '> 30
days', count(CASE WHEN sce.TimeGenerated < dateadd(dd,-90,getutcdate()) THEN
sce.TimeGenerated ELSE NULL END) as '> 90 days', count(CASE WHEN
sce.TimeGenerated < dateadd(dd,-365,getutcdate()) THEN sce.TimeGenerated ELSE
NULL END) as '> 365 days'
from StateChangeEvent sce
---------------------------------------------------
Report back.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Narendra Bathula
Sent: Wednesday, January 21, 2015 11:54 AM
To: [email protected]
Subject: RE: [msmom] SV: scom 2007 R2 db size is huge
Below is the large table query output:
Also attached the output file. Thank you
table name reserved row_count data
index_size unused l1 schemaname
StateChangeEvent 60706432 94881409 53284224 7378376
43832 1 dbo
PerformanceSignatureData1296976 12312525 788856 507704 416 0
dbo
PerformanceData_39 1165424 11321884 806304 357776 1344 1
dbo
PerformanceData_38 961168 11581675 697840 263208 120 0
dbo
PerformanceData_34 955408 11512841 693664 261608 136 1
dbo
PerformanceData_35 952848 11482890 691840 260904 104 0
dbo
PerformanceData_33 934160 11259200 678264 255760 136 1
dbo
PerformanceData_36 912144 10994943 662320 249736 88 0
dbo
PerformanceData_37 864656 10424159 627832 236680 144 1
dbo
Event_39 504416 511693 489848 14488 80
0 dbo
Event_38 500912 505466 487936 12360 616
1 dbo
PerformanceData_41 472320 3625515 345912 125952 456 0 dbo
ManagementPack 331264 252 328544 16 2704 1 dbo
Event_34 326376 376501 317192 8984 200
0 dbo
Event_33 295272 353742 286856 8352 64
1 dbo
Event_35 287920 337236 279752 7976 192
0 dbo
PerformanceData_40 241808 1839619 177464 64248 96 1 dbo
Event_37 218608 231363 212992 5432 184
0 dbo
RecursiveMembership 212816 1867630 130944 81808 64 1 dbo
Event_36 193512 197495 188768 4600 144
0 dbo
Event_41 159528 180647 151824 6448 1256
1 dbo
State 141968 980531 79392 62488 88 0
dbo
Relationship 123728 278030 26544 95032 2152
1 dbo
KnowledgeArticle 86648 14763 83048 1568 2032 0 dbo
BaseManagedEntity 77464 101981 67832 9512 120 1 dbo
PerformanceSource 75152 271711 44584 30520 48 0 dbo
Event_40 70296 70659 67240 2944 112 1 dbo
Alert 68344 21156 55032 11808 1504 0 dbo
AlertHistory 67280 25815 55096 11088 1096 1 dbo
LocalizedText 61856 107340 42024 19792 40 0 dbo
MaintenanceModeHistory 47056 192083 36528 10440 88 1 dbo
TypedManagedEntity 37592 179897 14536 22912 144 0 dbo
MT_Certificate 34456 55230 34200 240 16 1 dbo
DiscoverySourceToRelationship 32336 305485 18800 13408 128 0
dbo
Module 28112 23432 26752 1136 224 1 dbo
ModuleType 21808 1371 21560 112 136 0 dbo
DiscoverySourceToTypedManagedEntity 21328 200816 12360 8832 136
1 dbo
DiscoverySource 16720 111726 11320 5336 64 0 dbo
MaintenanceMode 14632 82722 11528 2872 232 1 dbo
MonitorType 14320 2389 14008 176 136 0 dbo
Report 12136 86 12080 16 40 1 dbo
Monitor 11120 5077 10360 648 112 0 dbo
Thanks & Regards,
Narendra
________________________________________
From: [email protected] [[email protected]] on behalf
of Kevin Holman [[email protected]]
Sent: Wednesday, January 21, 2015 7:49 PM
To: [email protected]
Subject: RE: [msmom] SV: scom 2007 R2 db size is huge
A file size of 140GB is not a "problem" although it might feel like a problem
to you, because your disk is also 140GB.
What matters most is "is there enough free space in the DB for a supported
operation". In your case - with 75GB used and 140GB DB size, then yes, this is
good.
First, let me say - we NEVER recommend shrinking the OpsMgr database DB file.
This should be left alone. Shrinking this file can cause some performance
issues that can only be resolved by creating a new file, and moving data over
to the new file, and this process can be a bit painful. In SQL, it is quite
easy to shrink a file, but it causes some level of fragmentation that a simple
reindex will not fully resolve. So if it can be avoided, we recommend it.
Ok, next, moving on to the root question.... with 850 agents, why do I have a
75GB used space in my database? Is that large? The answer is - yes, for most
deployments, that is a lot of data. There could be a lot of reasons for this.
Perhaps your grooming is broken.
Perhaps you are keeping some datatype for a really long time.
Perhaps you are massive overcollecting some datatype, like perf, or event data.
Perhaps you don't manage your environment at all, and you are flooded with
state changes and alert data from untuned management packs.
Perhaps you have written terrible rules and monitors and they are flooding SCOM
with data.
Perhaps you have something very unique, like a LARGE number of URL's, or APM,
or network monitoring going on.
The next logical step in troubleshooting.... is to tell us "what is in your
database"? Which you haven't done.
Run the large table query here:
http://blogs.technet.com/b/kevinholman/archive/2007/10/18/useful-operations-manager-2007-sql-queries.aspx
Tell us what you see - which tables are the biggest, and how big are they.
Copy the output (in a readable format please) to this thread.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Narendra Bathula
Sent: Wednesday, January 21, 2015 8:08 AM
To: [email protected]
Subject: RE: [msmom] SV: scom 2007 R2 db size is huge
Thank you for replying. I have checked the both the sites mentioned and
applied on my DB. But the opsmgrdb is still on high usage. i have also tried
to removed orphned entries from the DB. That helped me little. most of the
microsoft management pack alerts are disabled. we have approximately 100 custom
monitors configured.
mdf file size =140 GB and disk size where the mdf file located is also 140 GB.
That means the drive is fully occupied.
Current Status below:
File_size_MB : 140 GB
space_used_MB : 75 GB
Free_Space_MB : 65 GB
my gromming settings are configued to 4 days for all.
I am not able to understand why operations databse is this much bigger...Please
suggest to resolve the issue. Thank you
Thanks & Regards,
Narendra
________________________________________
From: [email protected] [[email protected]] on behalf
of David Biot [[email protected]]
Sent: Wednesday, January 21, 2015 5:18 PM
To: [email protected]
Subject: Re: [msmom] SV: scom 2007 R2 db size is huge
Narendra,
do you mean that you only have 4 GB of free space in the OperationsManager
database? The database needs at least 40% of free space, preferable 50% for the
grooming jobs. More information can be found in the following blog post:
http://blogs.technet.com/b/kevinholman/archive/2013/10/03/opsmgr-2012-grooming-deep-dive-in-the-operationsmanager-database.aspx
This could explain why your database is so huge. According to the official
sizing tool (http://www.microsoft.com/en-us/download/details.aspx?id=23016),
your databse should have a size of about 20,44 GB and your datawarehouse 605,56.
Besides the localizedtext-issue, there is also another issue that is described
in the following blog post:
http://blogs.technet.com/b/kevinholman/archive/2009/12/21/tuning-tip-do-you-have-monitors-constantly-flip-flopping.aspx.
Best regards,
David Biot
On Wed, Jan 21, 2015 at 12:23 PM, Henrik Andersen
<[email protected]<mailto:[email protected]>> wrote:
Which version of SQL Server are you on?
For SQL Server 2008 please visit
https://msdn.microsoft.com/en-us/library/ms189035.aspx
/Henrik
-----Oprindelig meddelelse-----
Fra: [email protected]<mailto:[email protected]>
[mailto:[email protected]<mailto:[email protected]>]
På vegne af Narendra Bathula
Sendt: 21. januar 2015 12:07
Til: [email protected]<mailto:[email protected]>
Emne: [msmom] scom 2007 R2 db size is huge
Hi All,
I am running with scom 2007 R2 and my operationsmanager database size is 140GB
occupied. i am having only 850 server under monitoring. How to reduce this
size..?
I believe is huge size that scom DB is using. I have tried Kevin holman
localised text script and got 4gb size freed which is showing in internal
database free space. The free space is not showing in total drive space. it
is showing in only internal database free space.
How to make some free space in logical drive where database is hosted. Please
suggest me how to get some free space on drive. Thank you.
Thanks & Regards,
Narendra
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended
for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information
could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
may contain viruses in transmission. The e mail and its contents (with or
without referred errors) shall therefore not attach any liability on the
originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the
author and may not necessarily reflect the views or opinions of HCL or its
affiliates. Any form of reproduction, dissemination, copying, disclosure,
modification, distribution and / or publication of this message without the
prior written consent of authorized representative of HCL is strictly
prohibited. If you have received this email in error please delete it and
notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and
other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
