Been there - read the book - saw the movie....etc. We went through the same problem plus other issues with the log stair-stepping (more transactions that it could handle - things slowing down until it could catch up - flooded with transactions again - rinse-lather-repeat) generating false log-full conditions all night long.
Spent numerous hours/days working with IBM who swore that there weren't any changes that would effect this process and that my server was simply overloaded. I argued that I hadn't added any new nodes or processes to the server having the problem. They said we simply hit the "saturation wall" and the only answer was to redistribute/reduce the workload. I too setup cron jobs to monitor the "transaction log % full values" and issues "show logpinned" commands and send me emails (wore out the batteries in my BB). Yes there were some Oracle backups causing some of the problems (eventhough they hadn't changed in years) and I did get the Oracle folks to split the backup transactions into chunks (this is in the book) but the problems still persisted until I forced folks off this server to another TSM server. We also worked hard at redistributing the scheduled backups to smooth out the peaks. Eventually the problems stopped. Yes, what IBM told be to do "fixed" the problem but considering the workload mix had not changed, I still argued that since the only change was the TSM server level, there must be a problem in the server update. I am somewhat heartened to see other folks experiencing the same issues I went through and confirming that I was not going crazy. Zoltan Forray TSM Software & Hardware Administrator Virginia Commonwealth University UCC/Office of Technology Services [email protected] - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://infosecurity.vcu.edu/phishing.html From: Nancy L Leugemors <[email protected]> To: [email protected] Date: 04/16/2011 11:42 PM Subject: [ADSM-L] TSM Recovery log is pinning since upgrade to 5.5.5.0 code Sent by: "ADSM: Dist Stor Manager" <[email protected]> Hello, We just upgraded our TSM Server this from 5.5.4.0 to 5.5.5.0 to fix APAR IC66116. Since the server upgrade we have experienced several recovery log pinning incidents from a few different database backup clients. I haven't found any hits on new bugs related to our issue and I just opened up a case with support but was wondering if anyone had a similar issue with this level of server and client code or know of any APARS this might fit? Our recovery log has always been set to 13GB so I can't expand it anymore. Running a database backup doesn't free any space while the client database backup is running either. We run 1 full TSM database backup and multiple incrementals throughout the day. The only difference is the upgrade to the latest code. I'm not seeing any other errors, just showing different database clients pinning the log after issuing show log pinned command. Sometimes the client backup finishes before hitting 80% and sometimes TSM server cancels the longest running backup that is pinning the log. example of error: 29857) 04/16/11 04:03:26 ANR0524W Transaction failed for session 24418 for node DEVNODE_API (SQL-BACKTRACK) - data transfer interrupted. (SESSION: 24418) 04/16/11 04:03:26 ANR2997W The server log is 81 percent full. The server will delay transactions by 3 milliseconds. (SESSION: 29846) TSM Background: TSM Server: 5.5.5.0 TSM OS: AIX 5300 -12 TSM Clients: 5.5.2.0 TSM Client OS: AIX 5.3 (UDB-DB2 & SQL-Backtrack Sybase seen so far) Thank You, Nancy Leugemors Enterprise Systems HealthNow, NY 716-887-7979 CONFIDENTIALITY NOTICE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain proprietary, confidential, trade secret or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited and may be a violation of law. If you are not the intended recipient or a person responsible for delivering this message to an intended recipient, please contact the sender by reply email and destroy all copies of the original message.
