Re: TSM db backup delayed or hung for more than few hours
Ah, OK, I was thinking in relation to tape devices, I wouldn't expect SAN discovery to be in play for disk, or at least not in a way that's visible to TSM. Definitely let the list know what the results of the PMR are, I suspect many of us will be curious... On Tue, May 30, 2023 at 11:52:14PM +0800, Saravanan Palanisamy wrote: > Hi Skylar > > We use disk for db backup and it will finish within 30 minutes once db > backup started at db2 level for 2 tb database. > > There is no errors or warnings reported during db backup and only concern > it takes more than 30 min to see db backup message in db2diag log though > tsm side backup db command issued. > > We just need to ensure db backup get complete two times to clear all the > logs. > > Sometime we noticed there was no db backup message in db2 diag log and db > backup process hang with 0 bytes even after 12 hours. > > There is more stress for us to monitor db backup process very closely as it > may get hung anytime. > > Our data fully resides on all flash disk ( db , active , archive and > directory container ) > > > > > > On Tue, 30 May 2023 at 11:29 PM, Skylar Thompson wrote: > > > What's the device type for the DB backup? We'll see this when SAN discovery > > hasn't caught up with new device paths after a replacement. The telltale > > will be ANR8975I messages on the library manager and ANR8974I messages on > > the clients. > > > > As others noted, though, it can also be an API package problem. It might be > > fruitful looking in the activity log around the time backups start, along > > with the db2diag.log and tsmdbmgr.log files. > > > > On Mon, May 29, 2023 at 05:16:50AM +0800, Saravanan Palanisamy wrote: > > > !---| > > > This Message Is From an Untrusted Sender > > > You have not previously corresponded with this sender. > > > See https://itconnect.uw.edu/email-tags for additional > > > information.Please contact the UW-IT Service Center, > > > h...@uw.edu 206.221.5000, for assistance. > > > |---! > > > > > > V8.1.18 > > > > > > We have noticed tsm backup started at tsm level but didn???t issue any > > message to db2 and tsm level db backup appears hung > > > > > > db2 list utilities show details - No process > > > db2diag.log - didn???t see any message like ( Starting an online db > > backup ) > > > > > > Tried to restart tsm server to clear db2 hung process but no luck and db > > backup again hung. > > > > > > Main issue here : > > > > > > When we start tsm db backup it???s not able start and archive log became > > full. Tried to increase archive log few times. > > > > > > We tried almost 3 times ( we waited more than 1 hour ) but no luck. > > > > > > we always run db backup to clear archive log so that server will up and > > running. > > > > > > Same issue observed on last week in different server and today one more > > server. > > > > > > We have raised PMR to check this issue. > > > > > > Did anyone face this issue in v8.1.18 ? > > > > > > We have never seen this issue earlier but noticed after upgrade to > > v8.1.18 > > > > > > > > > Usually when we issue db backup command it will immediately start and we > > will see tsmdbmgr sessions in tsm server. It looks like strange issue and > > causing outage for production jobs. > > > > > > > > > Regards > > > Sarav > > > +65 9857 8665 > > > > -- > > -- Skylar Thompson (skyl...@u.washington.edu) > > -- Genome Sciences Department (UW Medicine), System Administrator > > -- Foege Building S046, (206)-685-7354 > > -- Pronouns: He/Him/His > > > -- > Thanks & Regards, > Saravanan > Mobile: +65-8228 4384 -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His
Re: TSM db backup delayed or hung for more than few hours
Hi Skylar We use disk for db backup and it will finish within 30 minutes once db backup started at db2 level for 2 tb database. There is no errors or warnings reported during db backup and only concern it takes more than 30 min to see db backup message in db2diag log though tsm side backup db command issued. We just need to ensure db backup get complete two times to clear all the logs. Sometime we noticed there was no db backup message in db2 diag log and db backup process hang with 0 bytes even after 12 hours. There is more stress for us to monitor db backup process very closely as it may get hung anytime. Our data fully resides on all flash disk ( db , active , archive and directory container ) On Tue, 30 May 2023 at 11:29 PM, Skylar Thompson wrote: > What's the device type for the DB backup? We'll see this when SAN discovery > hasn't caught up with new device paths after a replacement. The telltale > will be ANR8975I messages on the library manager and ANR8974I messages on > the clients. > > As others noted, though, it can also be an API package problem. It might be > fruitful looking in the activity log around the time backups start, along > with the db2diag.log and tsmdbmgr.log files. > > On Mon, May 29, 2023 at 05:16:50AM +0800, Saravanan Palanisamy wrote: > > !---| > > This Message Is From an Untrusted Sender > > You have not previously corresponded with this sender. > > See https://itconnect.uw.edu/email-tags for additional > > information.Please contact the UW-IT Service Center, > > h...@uw.edu 206.221.5000, for assistance. > > |---! > > > > V8.1.18 > > > > We have noticed tsm backup started at tsm level but didn???t issue any > message to db2 and tsm level db backup appears hung > > > > db2 list utilities show details - No process > > db2diag.log - didn???t see any message like ( Starting an online db > backup ) > > > > Tried to restart tsm server to clear db2 hung process but no luck and db > backup again hung. > > > > Main issue here : > > > > When we start tsm db backup it???s not able start and archive log became > full. Tried to increase archive log few times. > > > > We tried almost 3 times ( we waited more than 1 hour ) but no luck. > > > > we always run db backup to clear archive log so that server will up and > running. > > > > Same issue observed on last week in different server and today one more > server. > > > > We have raised PMR to check this issue. > > > > Did anyone face this issue in v8.1.18 ? > > > > We have never seen this issue earlier but noticed after upgrade to > v8.1.18 > > > > > > Usually when we issue db backup command it will immediately start and we > will see tsmdbmgr sessions in tsm server. It looks like strange issue and > causing outage for production jobs. > > > > > > Regards > > Sarav > > +65 9857 8665 > > -- > -- Skylar Thompson (skyl...@u.washington.edu) > -- Genome Sciences Department (UW Medicine), System Administrator > -- Foege Building S046, (206)-685-7354 > -- Pronouns: He/Him/His > -- Thanks & Regards, Saravanan Mobile: +65-8228 4384
Re: TSM db backup delayed or hung for more than few hours
What's the device type for the DB backup? We'll see this when SAN discovery hasn't caught up with new device paths after a replacement. The telltale will be ANR8975I messages on the library manager and ANR8974I messages on the clients. As others noted, though, it can also be an API package problem. It might be fruitful looking in the activity log around the time backups start, along with the db2diag.log and tsmdbmgr.log files. On Mon, May 29, 2023 at 05:16:50AM +0800, Saravanan Palanisamy wrote: > !---| > This Message Is From an Untrusted Sender > You have not previously corresponded with this sender. > See https://itconnect.uw.edu/email-tags for additional > information.Please contact the UW-IT Service Center, > h...@uw.edu 206.221.5000, for assistance. > |---! > > V8.1.18 > > We have noticed tsm backup started at tsm level but didn???t issue any > message to db2 and tsm level db backup appears hung > > db2 list utilities show details - No process > db2diag.log - didn???t see any message like ( Starting an online db backup ) > > Tried to restart tsm server to clear db2 hung process but no luck and db > backup again hung. > > Main issue here : > > When we start tsm db backup it???s not able start and archive log became > full. Tried to increase archive log few times. > > We tried almost 3 times ( we waited more than 1 hour ) but no luck. > > we always run db backup to clear archive log so that server will up and > running. > > Same issue observed on last week in different server and today one more > server. > > We have raised PMR to check this issue. > > Did anyone face this issue in v8.1.18 ? > > We have never seen this issue earlier but noticed after upgrade to v8.1.18 > > > Usually when we issue db backup command it will immediately start and we will > see tsmdbmgr sessions in tsm server. It looks like strange issue and causing > outage for production jobs. > > > Regards > Sarav > +65 9857 8665 -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His
Re: TSM db backup delayed or hung for more than few hours
Hi Vikas We are using latest tsm client (8.1.17.2) and server version (8.1.18) Regards Sarav +65 9857 8665 > On 30 May 2023, at 2:27 AM, Vikas Arora wrote: > > Hi Sarav, > > May I ask which tsm client version (dsmadmc) you are using? Can you please > try to update it with the latest version and then re-try? I had a similar > experience a few days ago and that helped! > > thanks > > > > Grüße > Vikas Arora > > *e Mail* : vikar...@gmail.com > > >> On Mon, 29 May 2023 at 13:38, Saravanan Palanisamy < >> evergreen.sa...@gmail.com> wrote: >> >> Today it took more than 90 min to initiate db backup at db2 level where as >> admin schedule issues backup db command 90 min before and backup got >> completed. We will monitor closely. >> >> Regards >> Sarav >> +65 9857 8665 >> >> >>>> On 29 May 2023, at 3:29 PM, David L.A. De Leeuw wrote: >>> >>> Hi Sarav >>> >>> We had a similar case, just yesterday. I noticed the daily full database >> backup schedule was missed a number of days. >>> Then I found that five days ago the daily full backup just hanged. >>> Canceling the process hung as well on "pending". >>> >>> I contacted our system administrator and she reported that db2 got >> "stuck". >>> She rebooted the server, and the next backup ran as usual. >>> If this repeats, we will open a PMR as well. >>> >>> Our version: 8.1.17 >>> >>> Best wishes >>> >>> David de Leeuw >>> >>> -Original Message- >>> From: ADSM: Dist Stor Manager On Behalf Of >> Saravanan Palanisamy >>> Sent: Monday, May 29, 2023 12:17 AM >>> To: ADSM-L@VM.MARIST.EDU >>> Subject: [ADSM-L] TSM db backup delayed or hung for more than few hours >>> >>> V8.1.18 >>> >>> We have noticed tsm backup started at tsm level but didn’t issue any >> message to db2 and tsm level db backup appears hung >>> >>> db2 list utilities show details - No process db2diag.log - didn’t see >> any message like ( Starting an online db backup ) >>> >>> Tried to restart tsm server to clear db2 hung process but no luck and db >> backup again hung. >>> >>> Main issue here : >>> >>> When we start tsm db backup it’s not able start and archive log became >> full. Tried to increase archive log few times. >>> >>> We tried almost 3 times ( we waited more than 1 hour ) but no luck. >>> >>> we always run db backup to clear archive log so that server will up and >> running. >>> >>> Same issue observed on last week in different server and today one more >> server. >>> >>> We have raised PMR to check this issue. >>> >>> Did anyone face this issue in v8.1.18 ? >>> >>> We have never seen this issue earlier but noticed after upgrade to >> v8.1.18 >>> >>> >>> Usually when we issue db backup command it will immediately start and we >> will see tsmdbmgr sessions in tsm server. It looks like strange issue and >> causing outage for production jobs. >>> >>> >>> Regards >>> Sarav >>> +65 9857 8665 >>
Re: TSM db backup delayed or hung for more than few hours
Hi Sarav, May I ask which tsm client version (dsmadmc) you are using? Can you please try to update it with the latest version and then re-try? I had a similar experience a few days ago and that helped! thanks Grüße Vikas Arora *e Mail* : vikar...@gmail.com On Mon, 29 May 2023 at 13:38, Saravanan Palanisamy < evergreen.sa...@gmail.com> wrote: > Today it took more than 90 min to initiate db backup at db2 level where as > admin schedule issues backup db command 90 min before and backup got > completed. We will monitor closely. > > Regards > Sarav > +65 9857 8665 > > > > On 29 May 2023, at 3:29 PM, David L.A. De Leeuw wrote: > > > > Hi Sarav > > > > We had a similar case, just yesterday. I noticed the daily full database > backup schedule was missed a number of days. > > Then I found that five days ago the daily full backup just hanged. > > Canceling the process hung as well on "pending". > > > > I contacted our system administrator and she reported that db2 got > "stuck". > > She rebooted the server, and the next backup ran as usual. > > If this repeats, we will open a PMR as well. > > > > Our version: 8.1.17 > > > > Best wishes > > > > David de Leeuw > > > > -Original Message----- > > From: ADSM: Dist Stor Manager On Behalf Of > Saravanan Palanisamy > > Sent: Monday, May 29, 2023 12:17 AM > > To: ADSM-L@VM.MARIST.EDU > > Subject: [ADSM-L] TSM db backup delayed or hung for more than few hours > > > > V8.1.18 > > > > We have noticed tsm backup started at tsm level but didn’t issue any > message to db2 and tsm level db backup appears hung > > > > db2 list utilities show details - No process db2diag.log - didn’t see > any message like ( Starting an online db backup ) > > > > Tried to restart tsm server to clear db2 hung process but no luck and db > backup again hung. > > > > Main issue here : > > > > When we start tsm db backup it’s not able start and archive log became > full. Tried to increase archive log few times. > > > > We tried almost 3 times ( we waited more than 1 hour ) but no luck. > > > > we always run db backup to clear archive log so that server will up and > running. > > > > Same issue observed on last week in different server and today one more > server. > > > > We have raised PMR to check this issue. > > > > Did anyone face this issue in v8.1.18 ? > > > > We have never seen this issue earlier but noticed after upgrade to > v8.1.18 > > > > > > Usually when we issue db backup command it will immediately start and we > will see tsmdbmgr sessions in tsm server. It looks like strange issue and > causing outage for production jobs. > > > > > > Regards > > Sarav > > +65 9857 8665 >
Re: TSM db backup delayed or hung for more than few hours
Today it took more than 90 min to initiate db backup at db2 level where as admin schedule issues backup db command 90 min before and backup got completed. We will monitor closely. Regards Sarav +65 9857 8665 > On 29 May 2023, at 3:29 PM, David L.A. De Leeuw wrote: > > Hi Sarav > > We had a similar case, just yesterday. I noticed the daily full database > backup schedule was missed a number of days. > Then I found that five days ago the daily full backup just hanged. > Canceling the process hung as well on "pending". > > I contacted our system administrator and she reported that db2 got "stuck". > She rebooted the server, and the next backup ran as usual. > If this repeats, we will open a PMR as well. > > Our version: 8.1.17 > > Best wishes > > David de Leeuw > > -Original Message- > From: ADSM: Dist Stor Manager On Behalf Of Saravanan > Palanisamy > Sent: Monday, May 29, 2023 12:17 AM > To: ADSM-L@VM.MARIST.EDU > Subject: [ADSM-L] TSM db backup delayed or hung for more than few hours > > V8.1.18 > > We have noticed tsm backup started at tsm level but didn’t issue any message > to db2 and tsm level db backup appears hung > > db2 list utilities show details - No process db2diag.log - didn’t see any > message like ( Starting an online db backup ) > > Tried to restart tsm server to clear db2 hung process but no luck and db > backup again hung. > > Main issue here : > > When we start tsm db backup it’s not able start and archive log became full. > Tried to increase archive log few times. > > We tried almost 3 times ( we waited more than 1 hour ) but no luck. > > we always run db backup to clear archive log so that server will up and > running. > > Same issue observed on last week in different server and today one more > server. > > We have raised PMR to check this issue. > > Did anyone face this issue in v8.1.18 ? > > We have never seen this issue earlier but noticed after upgrade to v8.1.18 > > > Usually when we issue db backup command it will immediately start and we will > see tsmdbmgr sessions in tsm server. It looks like strange issue and causing > outage for production jobs. > > > Regards > Sarav > +65 9857 8665
Re: TSM db backup delayed or hung for more than few hours
Hi Sarav We had a similar case, just yesterday. I noticed the daily full database backup schedule was missed a number of days. Then I found that five days ago the daily full backup just hanged. Canceling the process hung as well on "pending". I contacted our system administrator and she reported that db2 got "stuck". She rebooted the server, and the next backup ran as usual. If this repeats, we will open a PMR as well. Our version: 8.1.17 Best wishes David de Leeuw -Original Message- From: ADSM: Dist Stor Manager On Behalf Of Saravanan Palanisamy Sent: Monday, May 29, 2023 12:17 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] TSM db backup delayed or hung for more than few hours V8.1.18 We have noticed tsm backup started at tsm level but didn’t issue any message to db2 and tsm level db backup appears hung db2 list utilities show details - No process db2diag.log - didn’t see any message like ( Starting an online db backup ) Tried to restart tsm server to clear db2 hung process but no luck and db backup again hung. Main issue here : When we start tsm db backup it’s not able start and archive log became full. Tried to increase archive log few times. We tried almost 3 times ( we waited more than 1 hour ) but no luck. we always run db backup to clear archive log so that server will up and running. Same issue observed on last week in different server and today one more server. We have raised PMR to check this issue. Did anyone face this issue in v8.1.18 ? We have never seen this issue earlier but noticed after upgrade to v8.1.18 Usually when we issue db backup command it will immediately start and we will see tsmdbmgr sessions in tsm server. It looks like strange issue and causing outage for production jobs. Regards Sarav +65 9857 8665
TSM db backup delayed or hung for more than few hours
V8.1.18 We have noticed tsm backup started at tsm level but didn’t issue any message to db2 and tsm level db backup appears hung db2 list utilities show details - No process db2diag.log - didn’t see any message like ( Starting an online db backup ) Tried to restart tsm server to clear db2 hung process but no luck and db backup again hung. Main issue here : When we start tsm db backup it’s not able start and archive log became full. Tried to increase archive log few times. We tried almost 3 times ( we waited more than 1 hour ) but no luck. we always run db backup to clear archive log so that server will up and running. Same issue observed on last week in different server and today one more server. We have raised PMR to check this issue. Did anyone face this issue in v8.1.18 ? We have never seen this issue earlier but noticed after upgrade to v8.1.18 Usually when we issue db backup command it will immediately start and we will see tsmdbmgr sessions in tsm server. It looks like strange issue and causing outage for production jobs. Regards Sarav +65 9857 8665
