Re: DATA Corruption using Deduplication in TSM 6.3.1.1 WARNING

[Ray Carlson]

Can anyone tell me what is in PMR 62476,033,000?  Apparently IBM believes that 
someone following the steps in that PMR caused the lose.  
Thanks,
Ray
On May 30, 2012, at 7:39 AM, Ray Carlson wrote:

> It was a brand new 2008 Server with a 6.3.0 install, which has been upgraded 
> to 6.3.1.1 because of other problems. Then the database was converted from a 
> 5.5 version.  
> It started with completely empty and new disk storage pools.  Then data was 
> moved from tape back to the disks.  Finally, Deduplication and Replication 
> were implemented.  It had been running for several months before the problem 
> arose.  I don't know if it is old data or fresh data since I haven't been 
> able to identify exactly which file is involved.  Unfortunately I do not have 
> the call reference number since that part is being handled by an outside 
> company.  I have also not had the opportunity attempt a restore using the DR 
> tapes since that would involve recalling multiple tapes from an offsite 
> company.  Basically, this is stopping some of my Generate backupsets and it 
> required that I do a disk to disk copy of the data I was trying to restore.  
> I still had the original data on the server and was simply using restore to 
> move it to a replacement server.  Thankfully, I haven't lost my original data 
> on the servers, just the ability to restore it from tsm if it was lost.
> Ray
> On May 30, 2012, at 4:09 AM, DeGroat, Steve wrote:
> 
>> Same here, we are about to go "live" on a new 6.3.1 server expecting client 
>> and server side deduplication.  Any additional information about the 
>> environment would be helpful. 
>> 
>> Steve DeGroat
>> Storage Manager
>> ITS Production Services
>> Yale University
>> 203.436.4540
>> 
>> "If you build it, they will come."
>> 
>> 
>> -Original Message-
>> From: ADSM: Dist Stor Manager [mailto:ADSM-L@vm.marist.edu] On Behalf Of 
>> Steven Langdale
>> Sent: Wednesday, May 30, 2012 3:24 AM
>> To: ADSM-L@vm.marist.edu
>> Subject: Re: [ADSM-L] DATA Corruption using Deduplication in TSM 6.3.1.1 
>> WARNING
>> 
>> Ray,  on top of what Remco posted, woudl you also post your call ref?
>> 
>> I'm about to implement a new 6.3 environ using dedupe so would like to press 
>> our IBM technical rep on the issue.  You never know, with a few people 
>> asking about it, it mya get fixed quicker.
>> 
>> Best of luck with your issue though!
>> 
>> Steven
>> 
>> On 30 May 2012 07:08, Remco Post  wrote:
>> 
>>> Hi Ray,
>>> 
>>> Thanks for the warning.
>>> 
>>> I was wondering if you could tell us a bit more about this TSM server. 
>>> Is it a converted 5.5 server? At which 6.x level did you start using 
>>> TSM version 6 for this server, 6.1, 6.2 or 6.3? Do the errors occur 
>>> with any particular kind of data? Is it 'old' data, or fresh data 
>>> recently written to TSM? Are you able to restore the files from 
>>> copypool if this error occurs?
>>> 
>>> On 29 mei 2012, at 20:46, Ray Carlson wrote:
>>> 
 Or as IBM called it, "Orphaned deduplicate references".
 
 We are running TSM 6.3.1.1 on a Windows 2008 Server, and using the
>>> Identify command to do deduplication on the Server, not the client.
 
 Interestingly, everything seemed to be mostly working.  We had a few
>>> volumes that would not be reclaimed or moved because it said the 
>>> deduplicated data had not been backed up to the copy pool, but that 
>>> was jut an annoyance.
 
 Then we discovered that we could not do restores of various servers.
>>> The error we got was:
 "05/21/2012 20:52:45 ANRD_2547000324 bfRtrv(bfrtrv.c:1161)
>>> Thread<129>: Error  obtaining deduplication information for object
>>> 254560532 in super bitfile 664355697 in pool 7 (SESSION: 8235, PROCESS:
>>> 375)".
 
 A Severity 1 trouble ticket was opened with IBM back on 5/21 and 
 various
>>> information was gathered and provided to IBM.  So far IBM has not been 
>>> able to identify the root cause or provide a fix.  They have 
>>> transferred the ticket to the Development team.
 
 So here I sit, not knowing which servers, if any, I could restore if
>>> needed.  Unfortunately, most operations appear to be fine and report 
>>> Success.  Only when I try to do a Generate Backupset, or do a Restore, 
>>> do I discover that there is a problem and the job fails.  Also, it 
>>> doesn't just skip the file/files that it can't restore and restore 
>>> everything else, it simply stops the restore and says it failed.
 
 I'm wondering how many other people are in the same situation, but 
 do
>>> not realize it.
 
 BEWARE Deduplication
 
 Ray Carlson
>>> 
>>> --
>>> Met vriendelijke groeten/Kind Regards,
>>> 
>>> Remco Post
>>> r.p...@plcs.nl
>>> +31 6 248 21 622
>>> 
> 

Re: There are not enough scratch volumes available

[Richard Sims]

As an administrator, you need to perform the following regularly:
 'Query Volume ACCess=UNAVailable,DESTroyed'
 'Query Volume ACCess=READOnly STATus=FIlling'
 'Query Volume ACCess=READOnly STATus=EMPty'
 'Query Volume STatus=PENding Format=Detailed'
to find tapes which TSM has given up on, as per messages like ANR1411W.
Also check for tapes which may have been dedicated via Define Volume and which 
might be put back into the scratch pool.
This is to say that you may be artificially out of immediately usable scratch 
tapes, but that potentially usable tapes may be identified and rendered 
candidates once again.

You should not check primary storage tapes out of a library except in dire 
circumstances, and then judiciously.  Capacity planning monitoring should be in 
place to anticipate the need for more library resources in advance of 
exhaustion.

Richard Sims

Re: There are not enough scratch volumes available

[Stackwick, Stephen]

You are right, check for "unavailable" tapes. I've seen bad tape drives take 
out all the tapes in a library, one by one, while you weren't looking. Check 
that, for sure!

STEPHEN STACKWICK | Senior Consultant | 301.518.6352 (m) | 
stephen.stackw...@icfi.com | icfi.com
ICF INTERNATIONAL | 410 E. Pratt Street Suite 2214, Baltimore, MD 21202 | 
410.539.1135 (o)



From: ADSM: Dist Stor Manager [ADSM-L@VM.MARIST.EDU] on behalf of Chavdar 
Cholev [chavdar.cho...@gmail.com]
Sent: Friday, June 01, 2012 15:31
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] There are not enough scratch volumes available

Also check vlume status RO, RW, unavailabel, etc

On 6/1/12, Stackwick, Stephen  wrote:
> Doncha hate it when that happens on a Friday?
>
> "Q DRM" should show which tapes are copypool tapes and can be removed. You
> may also have some DB backup volumes in there you could possibly even delete
> to free up scratch tapes. Or, you may have some PENDing tapes; in an
> emergency, you could delete the oldest ones. Doing that means you couldn't
> restore the TSM DB back to to that time, but in a pinch, it's a possibility.
>
> Why out of scratch, though? I'd check to see if reclamation has been running
> successfully.
>
> Steve
>
> STEPHEN STACKWICK | Senior Consultant | 301.518.6352 (m) |
> stephen.stackw...@icfi.com | icfi.com
> ICF INTERNATIONAL | 410 E. Pratt Street Suite 2214, Baltimore, MD 21202 |
> 410.539.1135 (o)
>
>
> 
> From: ADSM: Dist Stor Manager [ADSM-L@VM.MARIST.EDU] on behalf of danesh1
> [tsm-fo...@backupcentral.com]
> Sent: Monday, May 28, 2012 20:29
> To: ADSM-L@VM.MARIST.EDU
> Subject: [ADSM-L] There are not enough scratch volumes available
>
> I'm getting messages saying "There are not enough scratch volumes available"
>
> At the moment the tape library is fully populated with 211 tape volumes.
>
> Can some please tell me which tape volumes I need to eject from the tape
> library slots to be sent offiste so that I can load and check in more
> scratch volumes?
>
> +--
> |This was sent by d.qure...@mdx.ac.uk via Backup Central.
> |Forward SPAM to ab...@backupcentral.com.
> +--
>
>

Re: Address in ANR0406I message

[Vandeventer, Harold [BS]]

I fight firewall issues with a couple of tests.


From the TSM server:
PING 
TELNET  

Typically, the node is listening on 1501 I believe.  It should be specified in 
the node dsm.opt file.

Try stopping/starting the nodes TSM Scheduler service, then try the TELNET 
again.

If PING works, but the TELNET doesn't, you've likely got a firewall block.




Harold Vandeventer
Systems Programmer
State of Kansas - Department of Administration - Office of Information 
Technology Services
harold.vandeven...@ks.gov
(785) 296-0631

-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Thomas 
Denier
Sent: Friday, June 01, 2012 1:54 PM
To: ADSM-L@VM.MARIST.EDU
Subject: [ADSM-L] Address in ANR0406I message

We are seeing strange behavior with one of our client systems since it was 
moved behind a firewall. The client is a Windows XP system with TSM
6.2.2.0 client code. It sends backups to a TSM 6.2.2.0 server running under 
zSeries Linux. Before the firewall was introduced, the client was able to run 
backups with prompted mode scheduling. When the firewall was introduced, the 
client was given a new IP address and the old address was transferred to the 
firewall. The backups stopped running when the firewall was introduced. The 
messages reporting failed attempts to contact the client system indicated that 
the TSM server was trying to contact the client using the old IP address. 
Backups started running again after the client system administrator added a 
'tcpclientaddress' option specifying the new IP address to dsm.opt.

According to the firewall administrator, the only traffic the firewall is 
currently blocking is for automatic distribution of software patches (the 
system is subject to regulatory requirements for explicit control of 
configuration changes).

The 'TCP/IP Address' line in the output from 'query node' with 'f=d' shows the 
new address, and did so before the introduction of the 'tcpclientaddress'
option. All ANR0406I session start messages for the client show the old address 
(now the firewall address), and did so before the introduction of the 
'tcpclientaddress' option. According to the firewall administrator, the 
firewall is not configured for NAT (Network Address Translation).

I suspect that the address in an ANR0406I message is the source address of one 
of the packets involved in the TCP session initiation process, but I cannot 
find any documentation that says this explicitly. Does anyone know of an 
authoritative statement on this point? Does anyone have a theory that would 
account for the combination of behaviors described above?

Thomas Denier
Thomas Jefferson University Hospital

[Confidentiality notice:]
***
This e-mail message, including attachments, if any, is intended for the
person or entity to which it is addressed and may contain confidential
or privileged information.  Any unauthorized review, use, or disclosure
is prohibited.  If you are not the intended recipient, please contact
the sender and destroy the original message, including all copies,
Thank you.
***

Re: There are not enough scratch volumes available

[Chavdar Cholev]

Also check vlume status RO, RW, unavailabel, etc

On 6/1/12, Stackwick, Stephen  wrote:
> Doncha hate it when that happens on a Friday?
>
> "Q DRM" should show which tapes are copypool tapes and can be removed. You
> may also have some DB backup volumes in there you could possibly even delete
> to free up scratch tapes. Or, you may have some PENDing tapes; in an
> emergency, you could delete the oldest ones. Doing that means you couldn't
> restore the TSM DB back to to that time, but in a pinch, it's a possibility.
>
> Why out of scratch, though? I'd check to see if reclamation has been running
> successfully.
>
> Steve
>
> STEPHEN STACKWICK | Senior Consultant | 301.518.6352 (m) |
> stephen.stackw...@icfi.com | icfi.com
> ICF INTERNATIONAL | 410 E. Pratt Street Suite 2214, Baltimore, MD 21202 |
> 410.539.1135 (o)
>
>
> 
> From: ADSM: Dist Stor Manager [ADSM-L@VM.MARIST.EDU] on behalf of danesh1
> [tsm-fo...@backupcentral.com]
> Sent: Monday, May 28, 2012 20:29
> To: ADSM-L@VM.MARIST.EDU
> Subject: [ADSM-L] There are not enough scratch volumes available
>
> I'm getting messages saying "There are not enough scratch volumes available"
>
> At the moment the tape library is fully populated with 211 tape volumes.
>
> Can some please tell me which tape volumes I need to eject from the tape
> library slots to be sent offiste so that I can load and check in more
> scratch volumes?
>
> +--
> |This was sent by d.qure...@mdx.ac.uk via Backup Central.
> |Forward SPAM to ab...@backupcentral.com.
> +--
>
>

Address in ANR0406I message

[Thomas Denier]

We are seeing strange behavior with one of our client systems since it
was moved behind a firewall. The client is a Windows XP system with TSM
6.2.2.0 client code. It sends backups to a TSM 6.2.2.0 server running 
under zSeries Linux. Before the firewall was introduced, the client was
able to run backups with prompted mode scheduling. When the firewall was
introduced, the client was given a new IP address and the old address was
transferred to the firewall. The backups stopped running when the firewall
was introduced. The messages reporting failed attempts to contact the
client system indicated that the TSM server was trying to contact the client
using the old IP address. Backups started running again after the client
system administrator added a 'tcpclientaddress' option specifying the new
IP address to dsm.opt.

According to the firewall administrator, the only traffic the firewall is
currently blocking is for automatic distribution of software patches (the
system is subject to regulatory requirements for explicit control of
configuration changes).

The 'TCP/IP Address' line in the output from 'query node' with 'f=d' shows
the new address, and did so before the introduction of the 'tcpclientaddress'
option. All ANR0406I session start messages for the client show the old
address (now the firewall address), and did so before the introduction of
the 'tcpclientaddress' option. According to the firewall administrator,
the firewall is not configured for NAT (Network Address Translation).

I suspect that the address in an ANR0406I message is the source address of
one of the packets involved in the TCP session initiation process, but I
cannot find any documentation that says this explicitly. Does anyone know
of an authoritative statement on this point? Does anyone have a theory that
would account for the combination of behaviors described above?

Thomas Denier
Thomas Jefferson University Hospital

Re: There are not enough scratch volumes available

[Stackwick, Stephen]

Doncha hate it when that happens on a Friday?

"Q DRM" should show which tapes are copypool tapes and can be removed. You may 
also have some DB backup volumes in there you could possibly even delete to 
free up scratch tapes. Or, you may have some PENDing tapes; in an emergency, 
you could delete the oldest ones. Doing that means you couldn't restore the TSM 
DB back to to that time, but in a pinch, it's a possibility.

Why out of scratch, though? I'd check to see if reclamation has been running 
successfully.

Steve

STEPHEN STACKWICK | Senior Consultant | 301.518.6352 (m) | 
stephen.stackw...@icfi.com | icfi.com
ICF INTERNATIONAL | 410 E. Pratt Street Suite 2214, Baltimore, MD 21202 | 
410.539.1135 (o)



From: ADSM: Dist Stor Manager [ADSM-L@VM.MARIST.EDU] on behalf of danesh1 
[tsm-fo...@backupcentral.com]
Sent: Monday, May 28, 2012 20:29
To: ADSM-L@VM.MARIST.EDU
Subject: [ADSM-L] There are not enough scratch volumes available

I'm getting messages saying "There are not enough scratch volumes available"

At the moment the tape library is fully populated with 211 tape volumes.

Can some please tell me which tape volumes I need to eject from the tape 
library slots to be sent offiste so that I can load and check in more scratch 
volumes?

+--
|This was sent by d.qure...@mdx.ac.uk via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--

There are not enough scratch volumes available

[danesh1]

I'm getting messages saying "There are not enough scratch volumes available"

At the moment the tape library is fully populated with 211 tape volumes.

Can some please tell me which tape volumes I need to eject from the tape 
library slots to be sent offiste so that I can load and check in more scratch 
volumes?

+--
|This was sent by d.qure...@mdx.ac.uk via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--

Re: TSM4VE NBD data path

[Stackwick, Stephen]

Stefan,

The proxy/backup server/datamover/whateverIBMiscallingitnow is not a VM, so 
hotadd is not an option. Agree that NBD isn't ideal, but the customer has NAS 
VM storage, so it's either NBD or hotadd.

Steve

STEPHEN STACKWICK | Senior Consultant | 301.518.6352 (m) | 
stephen.stackw...@icfi.com | icfi.com
ICF INTERNATIONAL | 410 E. Pratt Street Suite 2214, Baltimore, MD 21202 | 
410.539.1135 (o)



From: ADSM: Dist Stor Manager [ADSM-L@VM.MARIST.EDU] on behalf of Stefan 
Folkerts [stefan.folke...@gmail.com]
Sent: Friday, June 01, 2012 12:11
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] TSM4VE NBD data path

Hi Steven,

I haven't but why not multihome the datamover vm and try to let it connect
to the storage & vCenter server that way?
I might be missing something but why would you want to use NBD over hotadd?

Regards,
  Stefan

On Fri, Jun 1, 2012 at 5:20 PM, Stackwick, Stephen <
stephen.stackw...@icfi.com> wrote:

> I have a customer with a VMware environment using NFS storage. He has a
> dedicated private network connecting his ESX hosts to his network storage.
> He wondered if it was possible to attach the backup/proxy server to this
> private LAN, in order to isolation the backup traffic. Has anyone tried
> this? The problem I see is that the proxy wants to talk to vCenter, and
> vCenter isn't reachable on the storage interface. It seems if you ditched
> the vCenter plugin approach, and manually configured the datamover dsm.opt
> files to point to ESX server IP addresses, it might work, right?
>
> Steve
>
> STEPHEN STACKWICK | Senior Consultant | 301.518.6352 (m) |
> stephen.stackw...@icfi.com | icfi.com<
> http://www.icfi.com/>
> ICF INTERNATIONAL | 410 E. Pratt Street Suite 2214, Baltimore, MD 21202 |
> 410.539.1135 (o)
>

Re: DB Space usage 6.2

[Andrew Carlson]

I saw behaviour like this when I had 2 DB volumes, and added 2 more on
the fly.  After talking with IBM, I tried a restart, and that didn't
help.  What I had to do was restore the database to all 4 volumes.  If
you had all 4 volumes there from the start, then I would open a PMR
with IBM.

On Fri, Jun 1, 2012 at 10:54 AM, Huebner,Andy,FORT WORTH,IT
 wrote:
> I am a recent convert to v6.2.
> I have 2 TSM servers that are not equally using the DB volumes.  Is this 
> normal?  A 3rd server is equally using its volumes.
> These 2 servers are new installs, nothing was retained from the previous 
> server.
> AIX
> TSM 6.2.3.0
>
> Location                       Total Space(MB)  Used Space(MB)  Free Space(MB)
> -- --- --- ---
> /tsm_db04                            20,544.00       20,543.85            0.15
> /tsm_db01                            20,544.00       16,788.45        3,755.55
> /tsm_db02                            20,544.00       16,788.30        3,755.70
> /tsm_db03                            20,544.00       16,788.52        3,755.48
>
> Location                       Total Space(MB)  Used Space(MB)  Free Space(MB)
> -- --- --- ---
> /tsm_db01                            20,608.00       16,006.84        4,601.16
> /tsm_db02                            20,608.00        4,150.68       16,457.32
> /tsm_db03                            20,608.00        4,150.53       16,457.47
> /tsm_db04                            20,608.00        4,150.53       16,457.47
>
> Andy Huebner
>
>
> This e-mail (including any attachments) is confidential and may be legally 
> privileged. If you are not an intended recipient or an authorized 
> representative of an intended recipient, you are prohibited from using, 
> copying or distributing the information in this e-mail or its attachments. If 
> you have received this e-mail in error, please notify the sender immediately 
> by return e-mail and delete all copies of this message and any attachments.
>
> Thank you.



-- 
Andy Carlson
---
Gamecube:$150,PSO:$50,Broadband Adapter: $35, Hunters License: $8.95/month,
The feeling of seeing the red box with the item you want in it:Priceless.

Re: TSM4VE NBD data path

[Stefan Folkerts]

Hi Steven,

I haven't but why not multihome the datamover vm and try to let it connect
to the storage & vCenter server that way?
I might be missing something but why would you want to use NBD over hotadd?

Regards,
  Stefan

On Fri, Jun 1, 2012 at 5:20 PM, Stackwick, Stephen <
stephen.stackw...@icfi.com> wrote:

> I have a customer with a VMware environment using NFS storage. He has a
> dedicated private network connecting his ESX hosts to his network storage.
> He wondered if it was possible to attach the backup/proxy server to this
> private LAN, in order to isolation the backup traffic. Has anyone tried
> this? The problem I see is that the proxy wants to talk to vCenter, and
> vCenter isn't reachable on the storage interface. It seems if you ditched
> the vCenter plugin approach, and manually configured the datamover dsm.opt
> files to point to ESX server IP addresses, it might work, right?
>
> Steve
>
> STEPHEN STACKWICK | Senior Consultant | 301.518.6352 (m) |
> stephen.stackw...@icfi.com | icfi.com<
> http://www.icfi.com/>
> ICF INTERNATIONAL | 410 E. Pratt Street Suite 2214, Baltimore, MD 21202 |
> 410.539.1135 (o)
>

DB Space usage 6.2

[Huebner,Andy,FORT WORTH,IT]

I am a recent convert to v6.2.
I have 2 TSM servers that are not equally using the DB volumes.  Is this 
normal?  A 3rd server is equally using its volumes.
These 2 servers are new installs, nothing was retained from the previous server.
AIX
TSM 6.2.3.0

Location   Total Space(MB)  Used Space(MB)  Free Space(MB)
-- --- --- ---
/tsm_db0420,544.00   20,543.850.15
/tsm_db0120,544.00   16,788.453,755.55
/tsm_db0220,544.00   16,788.303,755.70
/tsm_db0320,544.00   16,788.523,755.48

Location   Total Space(MB)  Used Space(MB)  Free Space(MB)
-- --- --- ---
/tsm_db0120,608.00   16,006.844,601.16
/tsm_db0220,608.004,150.68   16,457.32
/tsm_db0320,608.004,150.53   16,457.47
/tsm_db0420,608.004,150.53   16,457.47

Andy Huebner


This e-mail (including any attachments) is confidential and may be legally 
privileged. If you are not an intended recipient or an authorized 
representative of an intended recipient, you are prohibited from using, copying 
or distributing the information in this e-mail or its attachments. If you have 
received this e-mail in error, please notify the sender immediately by return 
e-mail and delete all copies of this message and any attachments.

Thank you.

TSM4VE NBD data path

[Stackwick, Stephen]

I have a customer with a VMware environment using NFS storage. He has a 
dedicated private network connecting his ESX hosts to his network storage. He 
wondered if it was possible to attach the backup/proxy server to this private 
LAN, in order to isolation the backup traffic. Has anyone tried this? The 
problem I see is that the proxy wants to talk to vCenter, and vCenter isn't 
reachable on the storage interface. It seems if you ditched the vCenter plugin 
approach, and manually configured the datamover dsm.opt files to point to ESX 
server IP addresses, it might work, right?

Steve

STEPHEN STACKWICK | Senior Consultant | 301.518.6352 (m) | 
stephen.stackw...@icfi.com | 
icfi.com
ICF INTERNATIONAL | 410 E. Pratt Street Suite 2214, Baltimore, MD 21202 | 
410.539.1135 (o)