Re: EXPIRE INVENTORY in TSM 6.3.6

[Zoltan Forray]

My experience (which confounded IBM) is the expires would never end (let it
run for over 9-days) even when picking a single node to expire.  I was
restarting the TSM server on a daily basis, it not multiple times a day -
especially when I was going the expires 1-by-1 to determine which nodes
were causing the lockups (the expiration would not cancel). Before were
decided to push ahead with the upgrades to 7.1.6.300, I had written a
server script with single expire statements for each node, exluding
8-troublemakers - i.e. the ones that would never complete.

On Fri, Nov 11, 2016 at 3:40 AM, Henrik Ahlgren  wrote:

> Thanks. I think the reason why EXPIRE INVENTORY without NODES only seems
> to expire a small amount of nodes is that it tries to continue where it
> left the last time it stopped – if you happen to normally run expire
> with DURATION= limitation. If TSM cancels the expire process
> after duration is exceeded, it does not emit any errors or warnings
> (unlike ANR4927W for reclamations) to the activity log, so this can
> easily go unnoticed for few days unless there is a specific monitoring
> for slow expires in place.
>
> On Thu, 2016-11-10 at 14:59 -0500, Zoltan Forray wrote:
> > Here is the 6.3.6.000 expiration problem APAR description:
> >
> > IT17642: SLOW EXPIRATION AFTER UPGRADE TO 6.3.6.000 SERVER
> >
> > APAR status - OPEN -
> >
> > *Error description*
> > After upgrading the server to 6.3.6.000 (or higher) and 7.1.3(or higher)
> > expiration might "hang" on a node while expiring backup data only.
> > Expiration is not actually hung it is still processing but very slowly
> due
> > to a non-optimized SQL/Select. A change to this SQL/Select occurred
> > between 6.3.5.0 and 6.3.6.0. This code change was implemented in servers
> > 6.3.6.0+ and 7.1.3.0 to 7.1.7. There have been no reported problems at
> > 7.1.3 or above.
>



-- 
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator (in training)
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zfor...@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://infosecurity.vcu.edu/phishing.html

Re: EXPIRE INVENTORY in TSM 6.3.6

[Henrik Ahlgren]

Thanks. I think the reason why EXPIRE INVENTORY without NODES only seems
to expire a small amount of nodes is that it tries to continue where it
left the last time it stopped – if you happen to normally run expire
with DURATION= limitation. If TSM cancels the expire process
after duration is exceeded, it does not emit any errors or warnings
(unlike ANR4927W for reclamations) to the activity log, so this can
easily go unnoticed for few days unless there is a specific monitoring
for slow expires in place.

On Thu, 2016-11-10 at 14:59 -0500, Zoltan Forray wrote:
> Here is the 6.3.6.000 expiration problem APAR description:
> 
> IT17642: SLOW EXPIRATION AFTER UPGRADE TO 6.3.6.000 SERVER
> 
> APAR status - OPEN -
> 
> *Error description*
> After upgrading the server to 6.3.6.000 (or higher) and 7.1.3(or higher)
> expiration might "hang" on a node while expiring backup data only.
> Expiration is not actually hung it is still processing but very slowly due
> to a non-optimized SQL/Select. A change to this SQL/Select occurred
> between 6.3.5.0 and 6.3.6.0. This code change was implemented in servers
> 6.3.6.0+ and 7.1.3.0 to 7.1.7. There have been no reported problems at
> 7.1.3 or above.

Re: EXPIRE INVENTORY in TSM 6.3.6

[Zoltan Forray]

Here is the 6.3.6.000 expiration problem APAR description:

IT17642: SLOW EXPIRATION AFTER UPGRADE TO 6.3.6.000 SERVER

APAR status - OPEN -

*Error description*
After upgrading the server to 6.3.6.000 (or higher) and 7.1.3(or higher)
expiration might "hang" on a node while expiring backup data only.
Expiration is not actually hung it is still processing but very slowly due
to a non-optimized SQL/Select. A change to this SQL/Select occurred
between 6.3.5.0 and 6.3.6.0. This code change was implemented in servers
6.3.6.0+ and 7.1.3.0 to 7.1.7. There have been no reported problems at
7.1.3 or above.

*Customer/L2 Diagnostics* (if applicable)
>From the server activity log - review expiration entries. Expiration
processes by node/filespace. If it is seen that 1 node does not complete
expiration in a timely manner your server may be experiencing this issue.
By default Expiration restarts where it left off. If it starts on the same
node repeatedly this may also be another indicator you are hitting this
issue.

Servermon.pl output can be gathered for 1 hour and needs to collect the db2
information. From the *db2.txt output search for the "ELAPSED_TIME_SEC"
section. If you are experiencing this issue you will find the following
select and a high Elapsed execution time - in this example 25310 seconds
(which is a little over 7 hrs):

ELAPSED_TIME_SEC STATE STMT_TEXT
 -
25310 EXEC SELECT IMBK.OBJID, IMBK.BFSIZE, CASE WHEN IMBK.GROUPTYPE IS NOT
NULL AND BITAND(IMBK.GROUPTYPE,458752)>0 THEN 1 END AS ISIMGLLEAD, CASE
WHEN IMBK.GROUPTYPE IS NOT NULL AND BITAND(IMBK.GROUPTYPE,7)>0 THEN 1 END
AS ISIMGLMEMB FROM TSMDB1.BACKUP_OBJECTS IMBK WHERE IMBK.NODEID=? AND
IMBK.FSID=? AND IMBK.OBJTYPE=? AND IMBK.STATE=2 AND IMBK.MCID=? AND
(BITAND(IMBK.GROUPTYPE,CAST(? AS INT))=0 OR IMBK.GROUPTYPE=131072 OR
IMBK.GROUPTYPE IS NULL ) AND ( IMBK.DEACDATE<='1956-10-11 00:00:00' OR (
IMBK.DEACDATE<=? AND

  1 record(s) selected.

*NOTE*
The ELAPSED_TIME_SEC select has been truncated in the DB2 collection and
this is how it will appear in the output.

*Platforms affected:*
Windows, Unix/Linux 6.3.6, 7.1.3 and higher

*Initial Impact:* Medium

*Additional Keywords:* IT07612

|MDVREGR 6.3.6.0| |MDVREGR 7.1.3.0|

*Local fix*
1) Contact Tivoli Storage Manager Support for a "Diagnostic" server
(6.3.6.000 only) which has been made available to correct this problem
until the fix is available
2) If you can not apply the DIAG server the work-around would be to run
expiration:   excluding the "slow" node(s) and restarting expiration at the
first node. This can be done by adding the 2 following
undocumented parameters to the Expire Inventory command:
  restart=no
  excludenodes=node1,node2,etc (where node1, node2 are the
names of the slow nodes)

On Thu, Nov 10, 2016 at 12:14 PM, Zoltan Forray  wrote:

> Not sure.  We have always used "expire inventory resource=8" with no
> issues until 6.3.6.000
>
> Since the APAR info is behind IBM's support wall (don't get me started on
> the inability to get firmware updates for my IBM TS3500 library from IBM),
> I don't know if I can post it here.  Hopefully someone from IBM can
> chime-in.
>
>
>
> On Thu, Nov 10, 2016 at 11:34 AM, Henrik Ahlgren 
> wrote:
>
>> Thanks Zoltan, – I somehow missed your earlier posting about this topic.
>>  I'll try to get my hands on the IT17642 article (why on earth does IBM
>> show it only to "authorized" users? I hate this vendor BS.).
>>
>> But in addition to slowness, it seems that EXPIRE INVENTORY NODE=* at
>> least attempts to expire all nodes, unlike  EXPIRE INVENTORY without the
>> NODE or DOMAIN options. Have you seen this?
>>
>> On Thu, Nov 10, 2016, at 18:07, Zoltan Forray wrote:
>> > Henrik,
>> >
>> > If you check recent posts here, you will see there is a major issue with
>> > 6.3.6.000 server and expirations.  I have been fighting it since I
>> > upgrade
>> > one server.  IBM has a hotfix to address it or you can simply upgrade to
>> > 7.1.6, which is what I did. You must contact IBM support for it.  They
>> > say
>> > the impact has been minimal and hit-or-miss (not everyone who upgrades
>> to
>> > 6.3.6.000 has the problem) so an official patch has not been released so
>> > far.
>> >
>> > Search Google for IT17642.
>> >
>> > On Thu, Nov 10, 2016 at 10:50 AM, Henrik Ahlgren 
>> > wrote:
>> >
>> > > The documentation for EXPIRE INVENTORY command says "If you do not
>> > > specify either NODE or DOMAIN with value, data for all nodes is
>> > > processes". That is in fact how it works up to 6.3.6.100, but it seems
>> > > that there is a bug in 6.3.6 that causes it to only expire a handful
>> of
>> > > randomly selected nodes, unless you specify "NODE=*".
>> > >
>> > > Have any of you run into this? If you're using 6.3.6, I would urge you
>> > > to check the expire history, i.e.
>> > >
>> > > select date(start_time) as date,count(*) as nodes,sum(affected) as
>> > > expired
>> > > from summary

Re: EXPIRE INVENTORY in TSM 6.3.6

[Zoltan Forray]

Not sure.  We have always used "expire inventory resource=8" with no issues
until 6.3.6.000

Since the APAR info is behind IBM's support wall (don't get me started on
the inability to get firmware updates for my IBM TS3500 library from IBM),
I don't know if I can post it here.  Hopefully someone from IBM can
chime-in.



On Thu, Nov 10, 2016 at 11:34 AM, Henrik Ahlgren 
wrote:

> Thanks Zoltan, – I somehow missed your earlier posting about this topic.
>  I'll try to get my hands on the IT17642 article (why on earth does IBM
> show it only to "authorized" users? I hate this vendor BS.).
>
> But in addition to slowness, it seems that EXPIRE INVENTORY NODE=* at
> least attempts to expire all nodes, unlike  EXPIRE INVENTORY without the
> NODE or DOMAIN options. Have you seen this?
>
> On Thu, Nov 10, 2016, at 18:07, Zoltan Forray wrote:
> > Henrik,
> >
> > If you check recent posts here, you will see there is a major issue with
> > 6.3.6.000 server and expirations.  I have been fighting it since I
> > upgrade
> > one server.  IBM has a hotfix to address it or you can simply upgrade to
> > 7.1.6, which is what I did. You must contact IBM support for it.  They
> > say
> > the impact has been minimal and hit-or-miss (not everyone who upgrades to
> > 6.3.6.000 has the problem) so an official patch has not been released so
> > far.
> >
> > Search Google for IT17642.
> >
> > On Thu, Nov 10, 2016 at 10:50 AM, Henrik Ahlgren 
> > wrote:
> >
> > > The documentation for EXPIRE INVENTORY command says "If you do not
> > > specify either NODE or DOMAIN with value, data for all nodes is
> > > processes". That is in fact how it works up to 6.3.6.100, but it seems
> > > that there is a bug in 6.3.6 that causes it to only expire a handful of
> > > randomly selected nodes, unless you specify "NODE=*".
> > >
> > > Have any of you run into this? If you're using 6.3.6, I would urge you
> > > to check the expire history, i.e.
> > >
> > > select date(start_time) as date,count(*) as nodes,sum(affected) as
> > > expired
> > > from summary
> > > where activity='EXPIRED'
> > > group by date(start_time)
> > >
> >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator (in training)
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zfor...@vcu.edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit http://infosecurity.vcu.edu/phishing.html
>



-- 
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator (in training)
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zfor...@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://infosecurity.vcu.edu/phishing.html

Re: EXPIRE INVENTORY in TSM 6.3.6

[Henrik Ahlgren]

Thanks Zoltan, – I somehow missed your earlier posting about this topic.
 I'll try to get my hands on the IT17642 article (why on earth does IBM
show it only to "authorized" users? I hate this vendor BS.). 

But in addition to slowness, it seems that EXPIRE INVENTORY NODE=* at
least attempts to expire all nodes, unlike  EXPIRE INVENTORY without the
NODE or DOMAIN options. Have you seen this?

On Thu, Nov 10, 2016, at 18:07, Zoltan Forray wrote:
> Henrik,
> 
> If you check recent posts here, you will see there is a major issue with
> 6.3.6.000 server and expirations.  I have been fighting it since I
> upgrade
> one server.  IBM has a hotfix to address it or you can simply upgrade to
> 7.1.6, which is what I did. You must contact IBM support for it.  They
> say
> the impact has been minimal and hit-or-miss (not everyone who upgrades to
> 6.3.6.000 has the problem) so an official patch has not been released so
> far.
> 
> Search Google for IT17642.
> 
> On Thu, Nov 10, 2016 at 10:50 AM, Henrik Ahlgren 
> wrote:
> 
> > The documentation for EXPIRE INVENTORY command says "If you do not
> > specify either NODE or DOMAIN with value, data for all nodes is
> > processes". That is in fact how it works up to 6.3.6.100, but it seems
> > that there is a bug in 6.3.6 that causes it to only expire a handful of
> > randomly selected nodes, unless you specify "NODE=*".
> >
> > Have any of you run into this? If you're using 6.3.6, I would urge you
> > to check the expire history, i.e.
> >
> > select date(start_time) as date,count(*) as nodes,sum(affected) as
> > expired
> > from summary
> > where activity='EXPIRED'
> > group by date(start_time)
> >
> 
> 
> 
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator (in training)
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zfor...@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://infosecurity.vcu.edu/phishing.html

Re: EXPIRE INVENTORY in TSM 6.3.6

[Zoltan Forray]

Henrik,

If you check recent posts here, you will see there is a major issue with
6.3.6.000 server and expirations.  I have been fighting it since I upgrade
one server.  IBM has a hotfix to address it or you can simply upgrade to
7.1.6, which is what I did. You must contact IBM support for it.  They say
the impact has been minimal and hit-or-miss (not everyone who upgrades to
6.3.6.000 has the problem) so an official patch has not been released so
far.

Search Google for IT17642.

On Thu, Nov 10, 2016 at 10:50 AM, Henrik Ahlgren 
wrote:

> The documentation for EXPIRE INVENTORY command says "If you do not
> specify either NODE or DOMAIN with value, data for all nodes is
> processes". That is in fact how it works up to 6.3.6.100, but it seems
> that there is a bug in 6.3.6 that causes it to only expire a handful of
> randomly selected nodes, unless you specify "NODE=*".
>
> Have any of you run into this? If you're using 6.3.6, I would urge you
> to check the expire history, i.e.
>
> select date(start_time) as date,count(*) as nodes,sum(affected) as
> expired
> from summary
> where activity='EXPIRED'
> group by date(start_time)
>



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator (in training)
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zfor...@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://infosecurity.vcu.edu/phishing.html