We finally resolved this problem. I worked with IBM support trying to identify why TSM/TDPO objects would not go away when we would run TDPOSYNC. We traced a TDPOSYNC run and clearly saw that the objects in TSM _were_ in rman - TDPOSYNC was working correctly! Somewhere/somehow the objects were still in rman, but not easily identified or located. Once we had hard evidence as to where the problem was, the DBA dug into the RMAN catalog and found the objects. I do not know what he deleted or how he deleted the data out or RMAN, but he did it.
Summary: We preformed a RMAN restore which included a Oracle DBID change. This in effect became a NEW Oracle db in RMAN which was backed up to the same node in TDPO/TSM. The old backups became stranded in RMAN and TSM. To TDPOSYNC everything was in-sync since these old objects were still in RMAN. I personally have not used/run RMAN, so I do not know what this looked like, but apparently it was not obvious that the old DB objects were still in RMAN. After the DBA found and deleted these old objects, TDPOSYNC deleted the TDPO/TSM objects. Here are the IBM instructions for tracing TDPOSYNC, and further down is a explanation of just how TSPOSYNC works. #There is a way we can check the RMAN and the TDPOSYNC outputs. #The background is in the following technote: # http://www-01.ibm.com/support/docview.wss?rs=663&q1=1142277&uid=swg21142277&loc=en_US&cs=utf-8&lang=en # I want to gather the outputs from this tdposync processing. # There are some temporary files as well as a trace that will help #to see the problem. If we trace the tdposync processing, this will #give us the path that the SQLxxxx files will be placed in and the #trace will also cause the SQLxxx files to not be deleted at the end #of the TDPOSYNC SYNCDB processing. This will prevent the #temporary files from being deleted after the TDPOSYNC completes #and we can gather them at leisure. #To trace the TDP Oracle client: #In the TPDO_OPTFILE: /opt/tivoli/tsm/client/oracle/bin/tdpo.opt you will need to place the trace variables: # TDPO_TRACE_FILE /directory/and/Filename # TDPO_TRACE_FLAGS orclevel2 #Be sure to specify a directory and filename that has writeable permissions for the Oracle user. #Then run the "TDPOSYNC SYNCDB" And, here is a very good explanation from lvl2 support sent of how TDPOSYNC works. #During the TDPOSYNC processing, it will initially connect to #SQLPLUS in order to query the RMAN catalog and pull the entire #list of backuppieces stored there (placing the list within a temporary #file in the temp directory). There is no specification for the target #database. The query list will be for all the objects stored in the RMAN #catalog. The TDPOSYNC will then connect to the TSM Server and #verify the list of objects from there based on the NodeName and Filespace. #The comparison is only one direction, which checks if the object that is on #the TSM Server exists in the RMAN catalog. A list is output of those items #that are on TSM but not within the RMAN catalog (additional objects may #exist in the RMAN catalog). Rick Richard L. Rhodes/FirstEnerg y To "ADSM: Dist Stor Manager" 02/10/2009 03:17 <[email protected]> PM cc Subject TDPO problem - can't delete old tdpo backups(Document link: Richard L. Rhodes) We've been using rman/tdpo backups for over a year now, and are still trying to get our heads fully around it. I've been working with TSM support on a problem and not getting very far, so I thought I'd ask the list. We have several TDPO nodes in a TSM server that have old backups we cannot get rid of. In all other ways TDP/TSM is working great. When we run TDPOSYNC it says the RMAN catalog and TDPO is INSYNC - nothing to cleanup. But, a list of objects form the backups table in TSM for the node shows old backups. we keep backups for 50 days - these old files are well back into last year. What we think happened: Back in Oct/Nov/Dec time frame, we did RESTORES on several databases. According to the DBA's, part of these restore was Oracle creating a new DBID (Database ID). In essence, after the restore we had a completely NEW DATABASE. >From RMAN's perspective, these were NEW DATABASES. The old objects in TSM that we can't get rid of appear to be from PRIOR TO THE RESTORE. It's as if both RMAN and TDPO have no knowledge of these objects. The DBA who handles rman/tdpo (who is very good!) says that these old objects are NOT in RMAN. RMAN/TDPO/TSM is CORRECTLY backing up and deleting backups SINCE the restore. Also, we setup rman/tdpo so that every Oracle db has it's own TSM node. IBM suggested 1) delete them with the BA client. It doesn't work. It seems the files in TSM have no owner, and the ba client won't/can't see them. 2) rename the node and start over. Yes, I can do this, but it seems to be running around the problem. or 3) User error (the dba and myself) - we don't understand something about RMAN/TDPO/TSM. It seems to me that I have either a RMAN bug, or a TDPOSYNC bug. what I wish is that tdposync would list out the full set of files that it is comparing from rman and tsm. Yes, I'm aware of the "delete object" command . . . I really don't want to go there! ================= Here is an example of a the files in one of the problem nodes: ==> tsmsap2 TDPO-ONSITE SAPEQ1D1_DB ==> q node: SAPEQ1D1_DB AIX TDPO-ONSITE 1 1 No <these lines are the old backups that tdposync/rman doesn't see> i0jo7odu_1_1 ACTIVE_VERSION 2008-08-17 i8jo7tuc_1_1 ACTIVE_VERSION 2008-08-17 <deleted lines> u7jrej66_1_1 ACTIVE_VERSION 2008-09-25 u8jrejb3_1_1 ACTIVE_VERSION 2008-09-25 <deleted lines> ucjrejk0_1_1 ACTIVE_VERSION 2008-09-25 udjrejlo_1_1 ACTIVE_VERSION 2008-09-25 <GAP - no files - the lines below are GOOD backups that are created/deleted correctly> 4mk2gkvp_1_1 ACTIVE_VERSION 2008-12-17 4tk2gv5l_1_1 ACTIVE_VERSION 2008-12-17 <deleted lines> o0k3bpge_1_1 ACTIVE_VERSION 2008-12-28 o5k3bvvt_1_1 ACTIVE_VERSION 2008-12-28 <delted lines> 6pk410jh_1_1 ACTIVE_VERSION 2009-01-05 6uk4173b_1_1 ACTIVE_VERSION 2009-01-05 <deleted lines> btk4iavp_1_1 ACTIVE_VERSION 2009-01-11 buk4ib01_1_1 ACTIVE_VERSION 2009-01-11 Thanks Rick ----------------------------------------- The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
