ANR1162W Space reclamation skipping damaged file on volume
Seeing these errors on replicated TSM server during reclaim process, both servers have been upgraded recently to 6.3.3 ANR1162W Space reclamation skipping damaged file on volume D:\WIN2K_PRIM2\0001A370.BFS: Node NEREUS, Type Backup, File space \\nereus\d$file:///\\nereus\d$, File name \STELLENT\IDCM1\WEBLAYOUT\GROUPS\RECORDSGROUP\DOCUMEN- TS\RM_INTERNALINFO\AKK.CXPC2PZPK5ANS08KDXD 821.HTM. Then if I audit the volume this is the result, I know I can delete the volume and re-replicate it It is a de-deduplicated storage pool. Prior to 6.3.3 we saw this but never the ANRD messages audit vol D:\WIN2K_PRIM2\0001A370.BFS fix=yes ANR2312I Audit Volume (Repair) process started for volume D:\WIN2K_PRIM2\0001A370.BFS (process ID 175). ANR8340I FILE volume D:\WIN2K_PRIM2\0001A370.BFS mounted. ANR0512I Process 175 opened input volume D:\WIN2K_PRIM2\0001A370.BFS. ANRD_2164256325 BfGetBitfileExtents(bfaggrut.c:4572) Thread38: BFBE Entry not found for object 702767727 ANRD Thread38 issued message from: ANRD Thread38 07FEE92F9839 OutDiagToCons()+159 ANRD Thread38 07FEE92F33BC outDiagfExt()+fc ANRD Thread38 07FEE8CAD44B BfGetBitfileExtents()+e0b ANRD Thread38 07FEE8CBFBA8 bfRtrvExt()+1328 ANRD Thread38 07FEE8D00ED4 AfAuditBitfile()+d4 ANRD Thread38 07FEE8CC3716 bfAuditBitfile()+cc6 ANRD Thread38 07FEE8BD4508 AdmVolDelThread()+1708 ANRD Thread38 07FEE8BD6875 AdmAuditVolThread()+7b5 ANRD Thread38 07FEE8AE6384 startThread()+124 ANRD Thread38 74431D9F endthreadex()+43 ANRD Thread38 74431E3B endthreadex()+df ANRD Thread38 7704652D BaseThreadInitThunk()+d ANRD Thread38 7727C521 RtlUserThreadStart()+21 ANRD_2547000324 bfRtrvExt(bfrtrv.c:1444) Thread38: Error 9979 obtaining deduplication information for object 702767726 in super bitfile 702767540 in pool 16 ANRD Thread38 issued message from: ANRD Thread38 07FEE92F9839 OutDiagToCons()+159 ANRD Thread38 07FEE92F33BC outDiagfExt()+fc ANRD Thread38 07FEE8CBFBFB bfRtrvExt()+137b ANRD Thread38 07FEE8D00ED4 AfAuditBitfile()+d4 ANRD Thread38 07FEE8CC3716 bfAuditBitfile()+cc6 ANRD Thread38 07FEE8BD4508 AdmVolDelThread()+1708 ANRD Thread38 07FEE8BD6875 AdmAuditVolThread()+7b5 ANRD Thread38 07FEE8AE6384 startThread()+124 ANRD Thread38 74431D9F endthreadex()+43 ANRD Thread38 74431E3B endthreadex()+df ANRD Thread38 7704652D BaseThreadInitThunk()+d ANRD Thread38 7727C521 RtlUserThreadStart()+21 ANR2327W Audit volume process terminated for volume D:\WIN2K_PRIM2\0001A370.BFS - internal server error detected. ANR0515I Process 175 closed volume D:\WIN2K_PRIM2\0001A370.BFS. ANR0985I Process 175 for AUDIT VOLUME (REPAIR) running in the BACKGROUND completed with completion state FAILURE at 08:01:10. ANR1893E Process 175 for AUDIT VOLUME (REPAIR) completed with a completion Thanks, Tim Brown Supervisor Computer Operations Central Hudson Gas Electric 284 South Ave Poughkeepsie, NY 12601
Re: ANR1162W Space reclamation skipping damaged file on volume
Check a recent APAR https://www-304.ibm.com/support/entdocview.wss?uid=swg1IC95938myns=aparmynp=DOCTYPEcomponentmync=E for a similar problem. I was running node replication and had some damaged files. The files had been de-duped by the node. Volume audits with fix=yes resolved the trouble. The above APAR was created shortly after my PMR was closed. -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Tim Brown Sent: Tuesday, September 24, 2013 7:08 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] ANR1162W Space reclamation skipping damaged file on volume Seeing these errors on replicated TSM server during reclaim process, both servers have been upgraded recently to 6.3.3 ANR1162W Space reclamation skipping damaged file on volume D:\WIN2K_PRIM2\0001A370.BFS: Node NEREUS, Type Backup, File space \\nereus\d$file:///\\nereus\d$, File name \STELLENT\IDCM1\WEBLAYOUT\GROUPS\RECORDSGROUP\DOCUMEN- TS\RM_INTERNALINFO\AKK.CXPC2PZPK5ANS08KDXD 821.HTM. Then if I audit the volume this is the result, I know I can delete the volume and re-replicate it It is a de-deduplicated storage pool. Prior to 6.3.3 we saw this but never the ANRD messages audit vol D:\WIN2K_PRIM2\0001A370.BFS fix=yes ANR2312I Audit Volume (Repair) process started for volume D:\WIN2K_PRIM2\0001A370.BFS (process ID 175). ANR8340I FILE volume D:\WIN2K_PRIM2\0001A370.BFS mounted. ANR0512I Process 175 opened input volume D:\WIN2K_PRIM2\0001A370.BFS. ANRD_2164256325 BfGetBitfileExtents(bfaggrut.c:4572) Thread38: BFBE Entry not found for object 702767727 ANRD Thread38 issued message from: ANRD Thread38 07FEE92F9839 OutDiagToCons()+159 ANRD Thread38 07FEE92F33BC outDiagfExt()+fc ANRD Thread38 07FEE8CAD44B BfGetBitfileExtents()+e0b ANRD Thread38 07FEE8CBFBA8 bfRtrvExt()+1328 ANRD Thread38 07FEE8D00ED4 AfAuditBitfile()+d4 ANRD Thread38 07FEE8CC3716 bfAuditBitfile()+cc6 ANRD Thread38 07FEE8BD4508 AdmVolDelThread()+1708 ANRD Thread38 07FEE8BD6875 AdmAuditVolThread()+7b5 ANRD Thread38 07FEE8AE6384 startThread()+124 ANRD Thread38 74431D9F endthreadex()+43 ANRD Thread38 74431E3B endthreadex()+df ANRD Thread38 7704652D BaseThreadInitThunk()+d ANRD Thread38 7727C521 RtlUserThreadStart()+21 ANRD_2547000324 bfRtrvExt(bfrtrv.c:1444) Thread38: Error 9979 obtaining deduplication information for object 702767726 in super bitfile 702767540 in pool 16 ANRD Thread38 issued message from: ANRD Thread38 07FEE92F9839 OutDiagToCons()+159 ANRD Thread38 07FEE92F33BC outDiagfExt()+fc ANRD Thread38 07FEE8CBFBFB bfRtrvExt()+137b ANRD Thread38 07FEE8D00ED4 AfAuditBitfile()+d4 ANRD Thread38 07FEE8CC3716 bfAuditBitfile()+cc6 ANRD Thread38 07FEE8BD4508 AdmVolDelThread()+1708 ANRD Thread38 07FEE8BD6875 AdmAuditVolThread()+7b5 ANRD Thread38 07FEE8AE6384 startThread()+124 ANRD Thread38 74431D9F endthreadex()+43 ANRD Thread38 74431E3B endthreadex()+df ANRD Thread38 7704652D BaseThreadInitThunk()+d ANRD Thread38 7727C521 RtlUserThreadStart()+21 ANR2327W Audit volume process terminated for volume D:\WIN2K_PRIM2\0001A370.BFS - internal server error detected. ANR0515I Process 175 closed volume D:\WIN2K_PRIM2\0001A370.BFS. ANR0985I Process 175 for AUDIT VOLUME (REPAIR) running in the BACKGROUND completed with completion state FAILURE at 08:01:10. ANR1893E Process 175 for AUDIT VOLUME (REPAIR) completed with a completion Thanks, Tim Brown Supervisor Computer Operations Central Hudson Gas Electric 284 South Ave Poughkeepsie, NY 12601 [Confidentiality notice:] *** This e-mail message, including attachments, if any, is intended for the person or entity to which it is addressed and may contain confidential or privileged information. Any unauthorized review, use, or disclosure is prohibited. If you are not the intended recipient, please contact the sender and destroy the original message, including all copies, Thank you. ***
Scheduled backups not working on Hyper-V Hosts to backup VM'
TSM client 6.4 is installed on Hyper-V Hosts so we can back up guest VM's on each host. Problem we are facing has to do with scheduled jobs. Scheduled jobs are not starting. Having to login to each host, fire up baclient, and backup VM's manually. Anyone experience this issue? +-- |This was sent by dusl...@yahoo.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +--
San question on Lanfree backup
Hi to all Hi have two exchange node I backup with lanfree version 6.3.4 , the configuration are correct success to backup them with lanfree TSM server version 6.3.4 TDP for Exchange version 6.4.1 For Node ExchsrvAn_SA: (Incr backup) 09/24/2013 19:26:00 ANE4971I (Session: 5707, Node: EXCHSRVAN_DB) LanFree databytes:19.41 GB (SESSION: 5707) For Node ExchsrvBn_SA: (Incr backup) 09/24/2013 20:16:00 ANE4971I (Session: 5803, Node: EXCHSRVBN_DB) LanFree data bytes: 23.51 GB (SESSION: 5803) I have sandiscovery on both dsmsta.opt I run on each server Q SAN , on ExchsrvAn_SA I see the correct output . 10 virtual drives from DataDomain and 2 physical LTO5 drives + Changer. But on ExchsrvBn_SA I see only the 2 physical LTO5 drives + Changer and do not see the virtual drives !! As I said when running a backup on ExchsrvB I see the correct drives mounted on the correct paths define ! Here the output’s tsm: ADSM2exchsrvan_sa:q san ANR1699I Resolved EXCHSRVAN_SA to 1 server(s) - issuing command Q SAN against server(s). ANR1687I Output for command 'Q SAN' issued against server EXCHSRVAN_SA follows: Device Vendor Product Serial Number Device Type -- DRIVEIBM ULTRIUM-TD3 G030260041 Tape0 DRIVEIBM ULTRIUM-TD3 G030260044 Tape1 DRIVEIBM ULTRIUM-TD3 G030260046 Tape2 DRIVEIBM ULTRIUM-TD3 G030260047 Tape3 DRIVEIBM ULTRIUM-TD3 G03026005B Tape4 DRIVEIBM ULTRIUM-TD3 G03026005C Tape5 DRIVEIBM ULTRIUM-TD3 G03026005D Tape6 DRIVEIBM ULTRIUM-TD3 G03026005E Tape7 DRIVEIBM ULTRIUM-TD3 G03026005F Tape8 DRIVEIBM ULTRIUM-TD3 G030260060 Tape9 DRIVEIBM ULT3580-HH5 1068090833 Tape10 LIBRARY IBM 3573-TL 00L4U78T4363_LL0 Changer0 DRIVEIBM ULT3580-HH5 1068090782 Tape11 ANR1688I Output for command 'Q SAN' issued against server EXCHSRVAN_SA completed. ANR1694I Server EXCHSRVAN_SA processed command 'Q SAN' and completed successfully. ANR1697I Command 'Q SAN' processed by 1 server(s): 1 successful, 0 with warnings, and 0 with errors. tsm: ADSM2exchsrvbn_sa:q san ANR1699I Resolved EXCHSRVBN_SA to 1 server(s) - issuing command Q SAN against server(s). ANR1687I Output for command 'Q SAN' issued against server EXCHSRVBN_SA follows: Device Vendor Product Serial Number Device Type -- DRIVEIBM ULT3580-HH5 1068090833 Tape10 LIBRARY IBM 3573-TL 00L4U78T4363_LL0 Changer0 DRIVEIBM ULT3580-HH5 1068090782 Tape11 ANR1688I Output for command 'Q SAN' issued against server EXCHSRVBN_SA completed. ANR1694I Server EXCHSRVBN_SA processed command 'Q SAN' and completed successfully. ANR1697I Command 'Q SAN' processed by 1 server(s): 1 successful, 0 with warnings, and 0 with errors. I miss something ? any ideas ? Best Regards Robert
Re: San question on Lanfree backup
Your storage agent should not see the changer. The TSM server owns the library and will mount and unmount all tapes. Verify that the zoning is correct for the missing drives. The STA must the zoned to the ports you present the drives on. Verify that the access group(s) are correct for the missing drives. The DD must allow access (masking for the SAN people) to the drives from the port that the STA is zoned to. This access is controlled by the WWN, so if you have more than 1 access group be sure the correct WWN is in all of the groups. Andy Huebner -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Robert Ouzen Sent: Tuesday, September 24, 2013 12:16 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] San question on Lanfree backup Hi to all Hi have two exchange node I backup with lanfree version 6.3.4 , the configuration are correct success to backup them with lanfree TSM server version 6.3.4 TDP for Exchange version 6.4.1 For Node ExchsrvAn_SA: (Incr backup) 09/24/2013 19:26:00 ANE4971I (Session: 5707, Node: EXCHSRVAN_DB) LanFree databytes:19.41 GB (SESSION: 5707) For Node ExchsrvBn_SA: (Incr backup) 09/24/2013 20:16:00 ANE4971I (Session: 5803, Node: EXCHSRVBN_DB) LanFree data bytes: 23.51 GB (SESSION: 5803) I have sandiscovery on both dsmsta.opt I run on each server Q SAN , on ExchsrvAn_SA I see the correct output . 10 virtual drives from DataDomain and 2 physical LTO5 drives + Changer. But on ExchsrvBn_SA I see only the 2 physical LTO5 drives + Changer and do not see the virtual drives !! As I said when running a backup on ExchsrvB I see the correct drives mounted on the correct paths define ! Here the output's tsm: ADSM2exchsrvan_sa:q san ANR1699I Resolved EXCHSRVAN_SA to 1 server(s) - issuing command Q SAN against server(s). ANR1687I Output for command 'Q SAN' issued against server EXCHSRVAN_SA follows: Device Vendor Product Serial Number Device Type -- DRIVEIBM ULTRIUM-TD3 G030260041 Tape0 DRIVEIBM ULTRIUM-TD3 G030260044 Tape1 DRIVEIBM ULTRIUM-TD3 G030260046 Tape2 DRIVEIBM ULTRIUM-TD3 G030260047 Tape3 DRIVEIBM ULTRIUM-TD3 G03026005B Tape4 DRIVEIBM ULTRIUM-TD3 G03026005C Tape5 DRIVEIBM ULTRIUM-TD3 G03026005D Tape6 DRIVEIBM ULTRIUM-TD3 G03026005E Tape7 DRIVEIBM ULTRIUM-TD3 G03026005F Tape8 DRIVEIBM ULTRIUM-TD3 G030260060 Tape9 DRIVEIBM ULT3580-HH5 1068090833 Tape10 LIBRARY IBM 3573-TL 00L4U78T4363_LL0 Changer0 DRIVEIBM ULT3580-HH5 1068090782 Tape11 ANR1688I Output for command 'Q SAN' issued against server EXCHSRVAN_SA completed. ANR1694I Server EXCHSRVAN_SA processed command 'Q SAN' and completed successfully. ANR1697I Command 'Q SAN' processed by 1 server(s): 1 successful, 0 with warnings, and 0 with errors. tsm: ADSM2exchsrvbn_sa:q san ANR1699I Resolved EXCHSRVBN_SA to 1 server(s) - issuing command Q SAN against server(s). ANR1687I Output for command 'Q SAN' issued against server EXCHSRVBN_SA follows: Device Vendor Product Serial Number Device Type -- DRIVEIBM ULT3580-HH5 1068090833 Tape10 LIBRARY IBM 3573-TL 00L4U78T4363_LL0 Changer0 DRIVEIBM ULT3580-HH5 1068090782 Tape11 ANR1688I Output for command 'Q SAN' issued against server EXCHSRVBN_SA completed. ANR1694I Server EXCHSRVBN_SA processed command 'Q SAN' and completed successfully. ANR1697I Command 'Q SAN' processed by 1 server(s): 1 successful, 0 with warnings, and 0 with errors. I miss something ? any ideas ? Best Regards Robert
Re: Scheduled backups not working on Hyper-V Hosts to backup VM'
Assuming you are using client side schedules: Please confirm your scheduled job configuration. Specifically, check the General tab under Security options to confirm the job is configured to Run whether user is logged on or not. Ray Storer NIBCO INC. 574.295.3457 -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of duslune Sent: Tuesday, September 24, 2013 10:14 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Scheduled backups not working on Hyper-V Hosts to backup VM' TSM client 6.4 is installed on Hyper-V Hosts so we can back up guest VM's on each host. Problem we are facing has to do with scheduled jobs. Scheduled jobs are not starting. Having to login to each host, fire up baclient, and backup VM's manually. Anyone experience this issue? +-- |This was sent by dusl...@yahoo.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
Re: San question on Lanfree backup
Hi Andy Thank you for the reply I double check in D.D configuration and the parameters are correct (WWPN , symbolic Port Name) Initiator Group Status WWNN WWPN Port - - -- --- --- exchsrvbvtl_exchangeb Online 20:00:00:1b:32:18:7a:24 21:00:00:1b:32:18:7a:24 6a Initiator Symbolic Port Name Address Method - --- -- exchsrvbQLE2460 FW:v5.07.02 DVR:v9.1.10.28 auto On the access groups vtl_echangeb all my LUN (10) are configure with the initiators WWPN 21:00:00:1b:32:18:7a:24 End what it's very strange the lanfree backup run fine , with the drives configure in D.D and path's configure in Tsm Run the command: validate lanfree exchsrvbn_db exchsrvbn_sa EXCHSRVBN_DB EXCHSRVBN_SA BACKUPMGEXCHANGE VTL_EXCHANGE Yes EXCHSRVBN_DB EXCHSRVBN_SA BACKUPMGLTO5 LTO5 Yes ANR1706I Ping for server 'EXCHSRVBN_SA' was able to establish a connection. ANR0388I Node EXCHSRVBN_DB using storage agent EXCHSRVBN_SA has 2 storage pools capable of LAN-free data movement and 0 storage pools not capable of LAN-free data movement. One is D.D Storage , the second is LTO5 storage I did a check from the exchange server running the tsmdlst.exe utility and got the correct output c:\Program Files (x86)\Tivoli\TSM\consoletsmdlst.exe Tivoli Storage Manager -- Device List Utility Computer Name: EXCHSRVB OS Version: 6.1 OS Build #: 7601 TSM Device Driver: TSMScsi - Not Running 3 HBAs were detected. ManufacturerModel Driver VersionFirmware Description Emulex Corporation LPe1150-E elxstor 7.2.30.016 2.70A5 EMC LPe1150-E 4Gb PCIe Fibre Channel Adapter Emulex Corporation LPe1150-E elxstor 7.2.30.016 2.80X4 EMC LPe1150-E 4Gb PCIe Fibre Channel Adapter QLogic Corporation QLE2460ql2300.sys 9.1.10.28 5.07.02 QLogic QLE2460 Fibre Channel Adapter TSM Name ID LUN Bus Port SSN WWN TSM Type Driver Device Identifier - mt0.10.0.5 010 05G030260067 - LTO IBM IBM ULTRIUM-TD3 8711 mt0.11.0.5 011 05G030260068 - LTO IBM IBM ULTRIUM-TD3 8711 mt0.12.0.5 012 05G030260069 - LTO IBM IBM ULTRIUM-TD3 8711 mt0.13.0.5 013 05G03026006A - LTO IBM IBM ULTRIUM-TD3 8711 mt0.14.0.5 014 05G03026006B - LTO IBM IBM ULTRIUM-TD3 8711 mt0.15.0.5 015 05G03026006C - LTO IBM IBM ULTRIUM-TD3 8711 mt0.16.0.5 016 05G03026006D - LTO IBM IBM ULTRIUM-TD3 8711 mt0.17.0.5 017 05G03026006E - LTO IBM IBM ULTRIUM-TD3 8711 mt0.8.0.50805G030260065 - LTO IBM IBM ULTRIUM-TD3 8711 mt0.9.0.50905G030260066 - LTO IBM IBM ULTRIUM-TD3 8711 mt1.0.0.510051068090833 2002000E1114EA72 LTO IBM IBM ULT3580-HH5 D2A1 lb1.1.0.5110500L4U78T4363_LL02002000E1114EA72 LIBRARY IBM IBM 3573-TL B.50 mt2.0.0.520051068090782 2005000E1114EA72 LTO IBM IBM ULT3580-HH5 D2A1 So if the backup run correctly meaning using lanfree with the correct drives and paths configured why the q san still give me an incorrect output Best Regards Robert -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Huebner, Andy Sent: Tuesday, September 24, 2013 10:19 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] San question on Lanfree backup Your storage agent should not see the changer. The TSM server owns the library and will mount and unmount all tapes. Verify that the zoning is correct for the missing drives. The STA must the zoned to the ports you present the drives on.
