sql tdp hang issue
I have a dba that has one sql server that has tsm hang issues according to him. They have a full backup job that backs up 7 sql db's. The size is about 80GB total. Sometime it backs them all up with no issues, sometime it says waiting on tsm server but if I look at q se there are no active sessions. Right now there is one running from last night and it has only backed up 12.5GB in 9 hours. below are some client details and log details. Anyone have any ideas. Other sql backups run just fine along with the other 15TB that backup each night, but of course DBA's say it's a TSM issue. Here is what they see when it hangs: Full: 3 Read: 658505728 Written: 65536 Rate: 480.95 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 480.59 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 480.23 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 479.87 Kb/Sec Waiting for TSM server.. .. clint info: Node Name: WHQWPSQLCL1RS01_SQL Platform: TDP MSSQL Win32 Client OS Level: 5.02 Client Version: Version 5, Release 3, Level 3.0 Policy Domain Name: SQL Last Access Date/Time: 12/14/07 08:00:05 Days Since Last Access: 1 thanks, Rod This email and any files transmitted with it are confidential and intended solely for the use of the addressee. If you are not the intended addressee, then you have received this email in error and any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. Please notify us immediately of your unintended receipt by reply and then delete this email and your reply. Tyson Foods, Inc. and its subsidiaries and affiliates will not be held liable to any person resulting from the unintended or unauthorized use of any information contained in this email or as a result of any additions or deletions of information originally contained in this email.
Re: sql tdp hang issue
You have received a secure message from Ben Bullock entitled, RE: [ADSM-L] sql tdp hang issue. You may view the message (before 06/21/2008) at the following web address: https://mail.bcidaho.com:443/messenger/msg?x=d-42733-qh6GFAGG8617 - Blue Cross of Idaho, 3000 E. Pine Ave., Meridian, ID 83642. 1-208-331-7421
Re: sql tdp hang issue
Did you examine the TSM Server activity log closely? Was there a problem where a tape did not get mounted in time? How many stripes are you using for the backup? If you are using multiple stripes and one of those stripes is waiting for the TSM server to mount a tape, it can cause this. I have seen this when the TSM Server does not have enough available tapes mounted to satisfy the DP/SQL request. The Waiting on TSM server indicates that DP/SQL is waiting for some event on the TSM Server to occur before it can continue sending any more data. Is it possible that some TSM Server event pre-empted the DP/SQL backup and took one of the tapes that was needed to satisfy a multiple stripe DP/SQL backup request? You should start on the TSM Server... in the activity log to find out what it thinks happened. Thanks, Del ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU wrote on 12/14/2007 09:52:59 AM: I have a dba that has one sql server that has tsm hang issues according to him. They have a full backup job that backs up 7 sql db's. The size is about 80GB total. Sometime it backs them all up with no issues, sometime it says waiting on tsm server but if I look at q se there are no active sessions. Right now there is one running from last night and it has only backed up 12.5GB in 9 hours. below are some client details and log details. Anyone have any ideas. Other sql backups run just fine along with the other 15TB that backup each night, but of course DBA's say it's a TSM issue. Here is what they see when it hangs: Full: 3 Read: 658505728 Written: 65536 Rate: 480.95 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 480.59 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 480.23 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 479.87 Kb/Sec Waiting for TSM server.. .. clint info: Node Name: WHQWPSQLCL1RS01_SQL Platform: TDP MSSQL Win32 Client OS Level: 5.02 Client Version: Version 5, Release 3, Level 3.0 Policy Domain Name: SQL Last Access Date/Time: 12/14/07 08:00:05 Days Since Last Access: 1 thanks, Rod
Re: sql tdp hang issue
I don't see anything in the activity log. The backups are going straight to diskpools and the backups are single threaded. I did add disk space to my diskpool this week in case that was the issue. Looking at my diskvols, data doesn't seem to be striped across all disk evenly, would that be that problem if some fill up and others don't, causing it to wait on tsm? tsm: TSMCORP1q vol stg=sql_diskpool Volume Name Storage Device Estimated Pct Volume Pool Name Class Name Capacity Util Status --- -- - - /dev/rsqldiskvol1SQL_DISKPO- DISK51,072.0 0.2 On-Line OL /dev/rsqldiskvol10 SQL_DISKPO- DISK51,072.0 31.4 On-Line OL /dev/rsqldiskvol11 SQL_DISKPO- DISK51,072.0 100.0 On-Line OL /dev/rsqldiskvol12 SQL_DISKPO- DISK51,072.0 36.6 On-Line OL /dev/rsqldiskvol13 SQL_DISKPO- DISK51,072.0 100.0 On-Line OL /dev/rsqldiskvol14 SQL_DISKPO- DISK51,072.0 92.8 On-Line OL /dev/rsqldiskvol15 SQL_DISKPO- DISK51,072.0 98.8 On-Line OL /dev/rsqldiskvol16 SQL_DISKPO- DISK51,072.0 81.9 On-Line OL /dev/rsqldiskvol2SQL_DISKPO- DISK51,072.0 3.2 On-Line OL /dev/rsqldiskvol3SQL_DISKPO- DISK51,072.0 91.3 On-Line OL /dev/rsqldiskvol4SQL_DISKPO- DISK51,072.0 21.0 On-Line OL /dev/rsqldiskvol5SQL_DISKPO- DISK51,072.0 11.5 On-Line OL /dev/rsqldiskvol6SQL_DISKPO- DISK51,072.0 69.4 On-Line OL /dev/rsqldiskvol7SQL_DISKPO- DISK51,072.0 15.4 On-Line OL /dev/rsqldiskvol8SQL_DISKPO- DISK51,072.0 99.9 On-Line OL /dev/rsqldiskvol9SQL_DISKPO- DISK51,072.0 44.3 On-Line -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Del Hoobler Sent: Friday, December 14, 2007 9:45 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] sql tdp hang issue Did you examine the TSM Server activity log closely? Was there a problem where a tape did not get mounted in time? How many stripes are you using for the backup? If you are using multiple stripes and one of those stripes is waiting for the TSM server to mount a tape, it can cause this. I have seen this when the TSM Server does not have enough available tapes mounted to satisfy the DP/SQL request. The Waiting on TSM server indicates that DP/SQL is waiting for some event on the TSM Server to occur before it can continue sending any more data. Is it possible that some TSM Server event pre-empted the DP/SQL backup and took one of the tapes that was needed to satisfy a multiple stripe DP/SQL backup request? You should start on the TSM Server... in the activity log to find out what it thinks happened. Thanks, Del ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU wrote on 12/14/2007 09:52:59 AM: I have a dba that has one sql server that has tsm hang issues according to him. They have a full backup job that backs up 7 sql db's. The size is about 80GB total. Sometime it backs them all up with no issues, sometime it says waiting on tsm server but if I look at q se there are no active sessions. Right now there is one running from last night and it has only backed up 12.5GB in 9 hours. below are some client details and log details. Anyone have any ideas. Other sql backups run just fine along with the other 15TB that backup each night, but of course DBA's say it's a TSM issue. Here is what they see when it hangs: Full: 3 Read: 658505728 Written: 65536 Rate: 480.95 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 480.59 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 480.23 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 479.87 Kb/Sec Waiting for TSM server.. .. clint info: Node Name: WHQWPSQLCL1RS01_SQL Platform: TDP MSSQL Win32 Client OS Level: 5.02 Client Version: Version 5, Release 3, Level 3.0
Re: sql tdp hang issue
This sounds like PMR#64846-122, which is still open after months of work. The last thing in the traces we ran is a TCPFLUSH. After that things hang forever. Fred Johanson TSM Administrator University of Chicago 773-702-8464 -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Park, Rod Sent: Friday, December 14, 2007 9:50 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] sql tdp hang issue I don't see anything in the activity log. The backups are going straight to diskpools and the backups are single threaded. I did add disk space to my diskpool this week in case that was the issue. Looking at my diskvols, data doesn't seem to be striped across all disk evenly, would that be that problem if some fill up and others don't, causing it to wait on tsm? tsm: TSMCORP1q vol stg=sql_diskpool Volume Name Storage Device Estimated Pct Volume Pool Name Class Name Capacity Util Status --- -- - - /dev/rsqldiskvol1SQL_DISKPO- DISK51,072.0 0.2 On-Line OL /dev/rsqldiskvol10 SQL_DISKPO- DISK51,072.0 31.4 On-Line OL /dev/rsqldiskvol11 SQL_DISKPO- DISK51,072.0 100.0 On-Line OL /dev/rsqldiskvol12 SQL_DISKPO- DISK51,072.0 36.6 On-Line OL /dev/rsqldiskvol13 SQL_DISKPO- DISK51,072.0 100.0 On-Line OL /dev/rsqldiskvol14 SQL_DISKPO- DISK51,072.0 92.8 On-Line OL /dev/rsqldiskvol15 SQL_DISKPO- DISK51,072.0 98.8 On-Line OL /dev/rsqldiskvol16 SQL_DISKPO- DISK51,072.0 81.9 On-Line OL /dev/rsqldiskvol2SQL_DISKPO- DISK51,072.0 3.2 On-Line OL /dev/rsqldiskvol3SQL_DISKPO- DISK51,072.0 91.3 On-Line OL /dev/rsqldiskvol4SQL_DISKPO- DISK51,072.0 21.0 On-Line OL /dev/rsqldiskvol5SQL_DISKPO- DISK51,072.0 11.5 On-Line OL /dev/rsqldiskvol6SQL_DISKPO- DISK51,072.0 69.4 On-Line OL /dev/rsqldiskvol7SQL_DISKPO- DISK51,072.0 15.4 On-Line OL /dev/rsqldiskvol8SQL_DISKPO- DISK51,072.0 99.9 On-Line OL /dev/rsqldiskvol9SQL_DISKPO- DISK51,072.0 44.3 On-Line -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Del Hoobler Sent: Friday, December 14, 2007 9:45 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] sql tdp hang issue Did you examine the TSM Server activity log closely? Was there a problem where a tape did not get mounted in time? How many stripes are you using for the backup? If you are using multiple stripes and one of those stripes is waiting for the TSM server to mount a tape, it can cause this. I have seen this when the TSM Server does not have enough available tapes mounted to satisfy the DP/SQL request. The Waiting on TSM server indicates that DP/SQL is waiting for some event on the TSM Server to occur before it can continue sending any more data. Is it possible that some TSM Server event pre-empted the DP/SQL backup and took one of the tapes that was needed to satisfy a multiple stripe DP/SQL backup request? You should start on the TSM Server... in the activity log to find out what it thinks happened. Thanks, Del ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU wrote on 12/14/2007 09:52:59 AM: I have a dba that has one sql server that has tsm hang issues according to him. They have a full backup job that backs up 7 sql db's. The size is about 80GB total. Sometime it backs them all up with no issues, sometime it says waiting on tsm server but if I look at q se there are no active sessions. Right now there is one running from last night and it has only backed up 12.5GB in 9 hours. below are some client details and log details. Anyone have any ideas. Other sql backups run just fine along with the other 15TB that backup each night, but of course DBA's say it's a TSM issue. Here is what they see when it hangs: Full: 3 Read: 658505728 Written: 65536 Rate: 480.95 Kb/Sec Full: 3 Read: 658505728 Written: 65536 Rate: 480.59 Kb/Sec Full: 3 Read
Re: sql tdp hang issue
I would look at your network connection first. Based on the throughput stats, I'm guessing you have a nic/switch mismatch between either the speed or duplex setting. Steve Schaub Systems Engineer, WNI BlueCross BlueShield of Tennessee 423-535-6574 (desk) 423-785-7347 (cell) ***public*** -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Park, Rod Sent: Friday, December 14, 2007 10:50 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] sql tdp hang issue I don't see anything in the activity log. The backups are going straight to diskpools and the backups are single threaded. I did add disk space to my diskpool this week in case that was the issue. Looking at my diskvols, data doesn't seem to be striped across all disk evenly, would that be that problem if some fill up and others don't, causing it to wait on tsm? tsm: TSMCORP1q vol stg=sql_diskpool Volume Name Storage Device Estimated Pct Volume Pool Name Class Name Capacity Util Status --- -- - - /dev/rsqldiskvol1SQL_DISKPO- DISK51,072.0 0.2 On-Line OL /dev/rsqldiskvol10 SQL_DISKPO- DISK51,072.0 31.4 On-Line OL /dev/rsqldiskvol11 SQL_DISKPO- DISK51,072.0 100.0 On-Line OL /dev/rsqldiskvol12 SQL_DISKPO- DISK51,072.0 36.6 On-Line OL /dev/rsqldiskvol13 SQL_DISKPO- DISK51,072.0 100.0 On-Line OL /dev/rsqldiskvol14 SQL_DISKPO- DISK51,072.0 92.8 On-Line OL /dev/rsqldiskvol15 SQL_DISKPO- DISK51,072.0 98.8 On-Line OL /dev/rsqldiskvol16 SQL_DISKPO- DISK51,072.0 81.9 On-Line OL /dev/rsqldiskvol2SQL_DISKPO- DISK51,072.0 3.2 On-Line OL /dev/rsqldiskvol3SQL_DISKPO- DISK51,072.0 91.3 On-Line OL /dev/rsqldiskvol4SQL_DISKPO- DISK51,072.0 21.0 On-Line OL /dev/rsqldiskvol5SQL_DISKPO- DISK51,072.0 11.5 On-Line OL /dev/rsqldiskvol6SQL_DISKPO- DISK51,072.0 69.4 On-Line OL /dev/rsqldiskvol7SQL_DISKPO- DISK51,072.0 15.4 On-Line OL /dev/rsqldiskvol8SQL_DISKPO- DISK51,072.0 99.9 On-Line OL /dev/rsqldiskvol9SQL_DISKPO- DISK51,072.0 44.3 On-Line -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Del Hoobler Sent: Friday, December 14, 2007 9:45 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] sql tdp hang issue Did you examine the TSM Server activity log closely? Was there a problem where a tape did not get mounted in time? How many stripes are you using for the backup? If you are using multiple stripes and one of those stripes is waiting for the TSM server to mount a tape, it can cause this. I have seen this when the TSM Server does not have enough available tapes mounted to satisfy the DP/SQL request. The Waiting on TSM server indicates that DP/SQL is waiting for some event on the TSM Server to occur before it can continue sending any more data. Is it possible that some TSM Server event pre-empted the DP/SQL backup and took one of the tapes that was needed to satisfy a multiple stripe DP/SQL backup request? You should start on the TSM Server... in the activity log to find out what it thinks happened. Thanks, Del ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU wrote on 12/14/2007 09:52:59 AM: I have a dba that has one sql server that has tsm hang issues according to him. They have a full backup job that backs up 7 sql db's. The size is about 80GB total. Sometime it backs them all up with no issues, sometime it says waiting on tsm server but if I look at q se there are no active sessions. Right now there is one running from last night and it has only backed up 12.5GB in 9 hours. below are some client details and log details. Anyone have any ideas. Other sql backups run just fine along with the other 15TB that backup each night, but of course DBA's say it's a TSM issue. Here is what they see when it hangs: Full: 3 Read: 658505728 Written: 65536 Rate: 480.95 Kb/Sec Full: 3 Read: 658505728
