Re: TSM7.1.7 DB growing with Dedup

2017-04-20 Thread Bill Boyer 
We'll be doing this now as it turns out. The dbvolume ran out of space and the 
instance is down at the moment. As this was upgraded to the 7.1.7 version from 
a V6.

-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Stefan 
Folkerts
Sent: Thursday, April 20, 2017 9:38 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] TSM7.1.7 DB growing with Dedup

You're welcome Bill.

And yes, you need to bring down TSM for an offline reorg, they are very fast 
however, especially if you run them on SSD's as a target location for the reorg 
and you do it one table at a time so you can stop after 2 if you are running 
out of time for instance and do the others in another window.

I've seen massive performance enhancements after an offline reorg.

I would recommend using the analyze_db2_formulas.pl script, the output is very 
usefull and looks like this:


tsminst1@hostname:~/20170420-0818> cat summary.out BEGIN SUMMARY
"db2 alter tablespace BFABFIDXSPACE reduce max" will return = 28.2G to the 
operating system file system
"db2 alter tablespace BFBFEXTIDXSPACE reduce max" will return = 31.5G to the 
operating system file system
BF_BITFILE_EXTENTS needs to be reorganized. estimated savings Table5
GB, Index2 GB
BF_QUEUED_CHUNKS needs to be reorganized. estimated savings Table1 GB,
Index  163 GB
Total estimated savings 171 GB
END SUMMARY
tsminst1@hostname:~/20170420-0818>

You can see what tablespaces need a reorg and what need a reduce max. The 
reduce max estimates can be a bit to optimistic but still, never good 
indication or where your attention should be.

Good luck

  Stefan


On Thu, Apr 20, 2017 at 2:28 PM, Bill Boyer  wrote:

> Offline re-org? Does that mean TSM needs to be down? I'm going to be 
> verifying the DB2 database version/stats this morning.
>
> Thanks!
>
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf 
> Of Stefan Folkerts
> Sent: Friday, April 14, 2017 2:02 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] TSM7.1.7 DB growing with Dedup
>
> Hi Bill,
>
> I don't know what is going on the the protect stgpool command vs 
> replicate node in regards to DB usage but the conversion growth...i've 
> been there a few times before, it happens with every conversion 
> because the directory containerpool stores it's metadata differently than the 
> filepool.
> It's more efficient in the end but during the conversion you need 
> extra space on the database filesystems.
>
> You can however reclaim that space, I use offline reorgs and reduce 
> max commands on the tablespaces that hold the old filepool metadata.
>
> http://www-01.ibm.com/support/docview.wss?uid=swg21683633
>
> Every conversion I have done has ended up with at least 30% less 
> database space used but not on it's own, they all need some help.
> You can run those reorgs and reduce max commands half way during the 
> conversion without problems if you are running out of space. Well, at 
> least that has been my experience.
>
> If you are having issues with this I suggest you contact IBM support, 
> they can determine what tables to reorg and reduce max to get the 
> maximum amount of space (and performance) back.
>
> I would also check out this page :
> http://www-01.ibm.com/support/docview.wss?uid=swg1IT15619
> I would advise you to run those selects on the database (might take a 
> day to complete), we have a few cases of orphaned chunks that 
> according to us MIGHT to be related (IBM is investigating) to the use 
> of the protect stgpool command, if you have them I would also suggest 
> asking IBM for support.
>
>
>
> On Wed, Apr 12, 2017 at 9:03 PM, Bill Boyer  wrote:
>
> > I converted a server over to directory container polls and replication.
> > Their previous FILE stgpool was around 140TB of data. After defining 
> > everything I used the TSMOC dialogs to enable replication and then 
> > started running the CONVERT STGPOOL command. At the start of this I 
> > had 430GB of available DBSPACE. I was only running the PROTECT 
> > schedules. The REPLICATE schedule was marked inactive until the 
> > convert completed. I stopped the convert when I got down to only 
> > 70GB of DBspace left and around 40TB to convert. Since then the DB 
> > has been shrinking about 2-4GB per day. We're ordering more SSD 
> > drives so we can increase the DBSPACE but I may run out of space before 
> > then.
> >
> >
> >
> > Is the fact that all nodes are set for replication holding on to 
> > some DB and DIR pool space? And my target container pool seems to be 
> > a lot different used% even though my PROTECT processes are running 
> > to completion.
> >
> >
> >
> > I'm thinking if I remove replication from all the nodes until I'm 
> > done with the conversion it should help? Then change each node back 
> > for replication and start the schedule.
> >
> >
> >
> > TIA,
> >
> > Bill Boyer
> >
> > "There are

Re: TSM7.1.7 DB growing with Dedup

2017-04-20 Thread Stefan Folkerts 
You're welcome Bill.

And yes, you need to bring down TSM for an offline reorg, they are very
fast however, especially if you run them on SSD's as a target location for
the reorg and you do it one table at a time so you can stop after 2 if you
are running out of time for instance and do the others in another window.

I've seen massive performance enhancements after an offline reorg.

I would recommend using the analyze_db2_formulas.pl script, the output is
very usefull and looks like this:


tsminst1@hostname:~/20170420-0818> cat summary.out
BEGIN SUMMARY
"db2 alter tablespace BFABFIDXSPACE reduce max" will return = 28.2G to the
operating system file system
"db2 alter tablespace BFBFEXTIDXSPACE reduce max" will return = 31.5G to
the operating system file system
BF_BITFILE_EXTENTS needs to be reorganized. estimated savings Table5
GB, Index2 GB
BF_QUEUED_CHUNKS needs to be reorganized. estimated savings Table1 GB,
Index  163 GB
Total estimated savings 171 GB
END SUMMARY
tsminst1@hostname:~/20170420-0818>

You can see what tablespaces need a reorg and what need a reduce max. The
reduce max estimates can be a bit to optimistic but still, never good
indication or where your attention should be.

Good luck

  Stefan


On Thu, Apr 20, 2017 at 2:28 PM, Bill Boyer  wrote:

> Offline re-org? Does that mean TSM needs to be down? I'm going to be
> verifying the DB2 database version/stats this morning.
>
> Thanks!
>
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of
> Stefan Folkerts
> Sent: Friday, April 14, 2017 2:02 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] TSM7.1.7 DB growing with Dedup
>
> Hi Bill,
>
> I don't know what is going on the the protect stgpool command vs replicate
> node in regards to DB usage but the conversion growth...i've been there a
> few times before, it happens with every conversion because the directory
> containerpool stores it's metadata differently than the filepool.
> It's more efficient in the end but during the conversion you need extra
> space on the database filesystems.
>
> You can however reclaim that space, I use offline reorgs and reduce max
> commands on the tablespaces that hold the old filepool metadata.
>
> http://www-01.ibm.com/support/docview.wss?uid=swg21683633
>
> Every conversion I have done has ended up with at least 30% less database
> space used but not on it's own, they all need some help.
> You can run those reorgs and reduce max commands half way during the
> conversion without problems if you are running out of space. Well, at least
> that has been my experience.
>
> If you are having issues with this I suggest you contact IBM support, they
> can determine what tables to reorg and reduce max to get the maximum amount
> of space (and performance) back.
>
> I would also check out this page :
> http://www-01.ibm.com/support/docview.wss?uid=swg1IT15619
> I would advise you to run those selects on the database (might take a day
> to complete), we have a few cases of orphaned chunks that according to us
> MIGHT to be related (IBM is investigating) to the use of the protect
> stgpool command, if you have them I would also suggest asking IBM for
> support.
>
>
>
> On Wed, Apr 12, 2017 at 9:03 PM, Bill Boyer  wrote:
>
> > I converted a server over to directory container polls and replication.
> > Their previous FILE stgpool was around 140TB of data. After defining
> > everything I used the TSMOC dialogs to enable replication and then
> > started running the CONVERT STGPOOL command. At the start of this I
> > had 430GB of available DBSPACE. I was only running the PROTECT
> > schedules. The REPLICATE schedule was marked inactive until the
> > convert completed. I stopped the convert when I got down to only 70GB
> > of DBspace left and around 40TB to convert. Since then the DB has been
> > shrinking about 2-4GB per day. We're ordering more SSD drives so we
> > can increase the DBSPACE but I may run out of space before then.
> >
> >
> >
> > Is the fact that all nodes are set for replication holding on to some
> > DB and DIR pool space? And my target container pool seems to be a lot
> > different used% even though my PROTECT processes are running to
> > completion.
> >
> >
> >
> > I'm thinking if I remove replication from all the nodes until I'm done
> > with the conversion it should help? Then change each node back for
> > replication and start the schedule.
> >
> >
> >
> > TIA,
> >
> > Bill Boyer
> >
> > "There are seldom good technological solutions to behavioral problems."
> -??
> >
>

Re: TSM7.1.7 DB growing with Dedup

2017-04-20 Thread Bill Boyer 
Offline re-org? Does that mean TSM needs to be down? I'm going to be verifying 
the DB2 database version/stats this morning.

Thanks!

-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Stefan 
Folkerts
Sent: Friday, April 14, 2017 2:02 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] TSM7.1.7 DB growing with Dedup

Hi Bill,

I don't know what is going on the the protect stgpool command vs replicate node 
in regards to DB usage but the conversion growth...i've been there a few times 
before, it happens with every conversion because the directory containerpool 
stores it's metadata differently than the filepool.
It's more efficient in the end but during the conversion you need extra space 
on the database filesystems.

You can however reclaim that space, I use offline reorgs and reduce max 
commands on the tablespaces that hold the old filepool metadata.

http://www-01.ibm.com/support/docview.wss?uid=swg21683633

Every conversion I have done has ended up with at least 30% less database space 
used but not on it's own, they all need some help.
You can run those reorgs and reduce max commands half way during the conversion 
without problems if you are running out of space. Well, at least that has been 
my experience.

If you are having issues with this I suggest you contact IBM support, they can 
determine what tables to reorg and reduce max to get the maximum amount of 
space (and performance) back.

I would also check out this page :
http://www-01.ibm.com/support/docview.wss?uid=swg1IT15619
I would advise you to run those selects on the database (might take a day to 
complete), we have a few cases of orphaned chunks that according to us MIGHT to 
be related (IBM is investigating) to the use of the protect stgpool command, if 
you have them I would also suggest asking IBM for support.



On Wed, Apr 12, 2017 at 9:03 PM, Bill Boyer  wrote:

> I converted a server over to directory container polls and replication.
> Their previous FILE stgpool was around 140TB of data. After defining 
> everything I used the TSMOC dialogs to enable replication and then 
> started running the CONVERT STGPOOL command. At the start of this I 
> had 430GB of available DBSPACE. I was only running the PROTECT 
> schedules. The REPLICATE schedule was marked inactive until the 
> convert completed. I stopped the convert when I got down to only 70GB 
> of DBspace left and around 40TB to convert. Since then the DB has been 
> shrinking about 2-4GB per day. We're ordering more SSD drives so we 
> can increase the DBSPACE but I may run out of space before then.
>
>
>
> Is the fact that all nodes are set for replication holding on to some 
> DB and DIR pool space? And my target container pool seems to be a lot 
> different used% even though my PROTECT processes are running to 
> completion.
>
>
>
> I'm thinking if I remove replication from all the nodes until I'm done 
> with the conversion it should help? Then change each node back for 
> replication and start the schedule.
>
>
>
> TIA,
>
> Bill Boyer
>
> "There are seldom good technological solutions to behavioral problems." -??
>

Re: TSM7.1.7 DB growing with Dedup

2017-04-14 Thread Stefan Folkerts 
Bill, I forgot to link this document from IBM, some good info here :
https://www-01.ibm.com/support/docview.wss?uid=swg21992410



On Fri, Apr 14, 2017 at 8:02 AM, Stefan Folkerts 
wrote:

> Hi Bill,
>
> I don't know what is going on the the protect stgpool command vs replicate
> node in regards to DB usage but the conversion growth...i've been there a
> few times before, it happens with every conversion because the directory
> containerpool stores it's metadata differently than the filepool.
> It's more efficient in the end but during the conversion you need extra
> space on the database filesystems.
>
> You can however reclaim that space, I use offline reorgs and reduce max
> commands on the tablespaces that hold the old filepool metadata.
>
> http://www-01.ibm.com/support/docview.wss?uid=swg21683633
>
> Every conversion I have done has ended up with at least 30% less database
> space used but not on it's own, they all need some help.
> You can run those reorgs and reduce max commands half way during the
> conversion without problems if you are running out of space. Well, at least
> that has been my experience.
>
> If you are having issues with this I suggest you contact IBM support, they
> can determine what tables to reorg and reduce max to get the maximum amount
> of space (and performance) back.
>
> I would also check out this page : http://www-01.ibm.com/
> support/docview.wss?uid=swg1IT15619
> I would advise you to run those selects on the database (might take a day
> to complete), we have a few cases of orphaned chunks that according to us
> MIGHT to be related (IBM is investigating) to the use of the protect
> stgpool command, if you have them I would also suggest asking IBM for
> support.
>
>
>
> On Wed, Apr 12, 2017 at 9:03 PM, Bill Boyer  wrote:
>
>> I converted a server over to directory container polls and replication.
>> Their previous FILE stgpool was around 140TB of data. After defining
>> everything I used the TSMOC dialogs to enable replication and then started
>> running the CONVERT STGPOOL command. At the start of this I had 430GB of
>> available DBSPACE. I was only running the PROTECT schedules. The REPLICATE
>> schedule was marked inactive until the convert completed. I stopped the
>> convert when I got down to only 70GB of DBspace left and around 40TB to
>> convert. Since then the DB has been shrinking about 2-4GB per day. We're
>> ordering more SSD drives so we can increase the DBSPACE but I may run out
>> of
>> space before then.
>>
>>
>>
>> Is the fact that all nodes are set for replication holding on to some DB
>> and
>> DIR pool space? And my target container pool seems to be a lot different
>> used% even though my PROTECT processes are running to completion.
>>
>>
>>
>> I'm thinking if I remove replication from all the nodes until I'm done
>> with
>> the conversion it should help? Then change each node back for replication
>> and start the schedule.
>>
>>
>>
>> TIA,
>>
>> Bill Boyer
>>
>> "There are seldom good technological solutions to behavioral problems."
>> -??
>>
>
>

Re: TSM7.1.7 DB growing with Dedup

2017-04-14 Thread Stefan Folkerts 
Hi Bill,

I don't know what is going on the the protect stgpool command vs replicate
node in regards to DB usage but the conversion growth...i've been there a
few times before, it happens with every conversion because the directory
containerpool stores it's metadata differently than the filepool.
It's more efficient in the end but during the conversion you need extra
space on the database filesystems.

You can however reclaim that space, I use offline reorgs and reduce max
commands on the tablespaces that hold the old filepool metadata.

http://www-01.ibm.com/support/docview.wss?uid=swg21683633

Every conversion I have done has ended up with at least 30% less database
space used but not on it's own, they all need some help.
You can run those reorgs and reduce max commands half way during the
conversion without problems if you are running out of space. Well, at least
that has been my experience.

If you are having issues with this I suggest you contact IBM support, they
can determine what tables to reorg and reduce max to get the maximum amount
of space (and performance) back.

I would also check out this page :
http://www-01.ibm.com/support/docview.wss?uid=swg1IT15619
I would advise you to run those selects on the database (might take a day
to complete), we have a few cases of orphaned chunks that according to us
MIGHT to be related (IBM is investigating) to the use of the protect
stgpool command, if you have them I would also suggest asking IBM for
support.



On Wed, Apr 12, 2017 at 9:03 PM, Bill Boyer  wrote:

> I converted a server over to directory container polls and replication.
> Their previous FILE stgpool was around 140TB of data. After defining
> everything I used the TSMOC dialogs to enable replication and then started
> running the CONVERT STGPOOL command. At the start of this I had 430GB of
> available DBSPACE. I was only running the PROTECT schedules. The REPLICATE
> schedule was marked inactive until the convert completed. I stopped the
> convert when I got down to only 70GB of DBspace left and around 40TB to
> convert. Since then the DB has been shrinking about 2-4GB per day. We're
> ordering more SSD drives so we can increase the DBSPACE but I may run out
> of
> space before then.
>
>
>
> Is the fact that all nodes are set for replication holding on to some DB
> and
> DIR pool space? And my target container pool seems to be a lot different
> used% even though my PROTECT processes are running to completion.
>
>
>
> I'm thinking if I remove replication from all the nodes until I'm done with
> the conversion it should help? Then change each node back for replication
> and start the schedule.
>
>
>
> TIA,
>
> Bill Boyer
>
> "There are seldom good technological solutions to behavioral problems." -??
>