Oops - ignore first sentence in my reply - was sending that to the OS tech responsible for this box.....
On Fri, Oct 6, 2017 at 11:48 AM, Zoltan Forray <[email protected]> wrote: > Can you get me this file: > ANS2250W A IBM Spectrum Protect core file or crash report was found: > C:\Program Files\Tivoli\TSM\baclient\dsmcrash.dmp > > Hi Andy, > > Thanks for the reply. I have plenty of logs but obviously I can't upload > them here. This includes the instrumentation data. The upgrade was done > last Friday. > > I will pull down and remove the dsmcrash.dmp file. > > I pulled this from one of the dsminstr.log files. As you can see, the > processing time has jumped a lot. > > Elapsed processing time: 00:00:35 > Elapsed processing time: 00:00:18 > Elapsed processing time: 00:00:52 > Elapsed processing time: 00:00:28 > Elapsed processing time: 00:00:34 > Elapsed processing time: 00:00:28 > Elapsed processing time: 00:01:05 > Elapsed processing time: 00:00:27 > Elapsed processing time: 00:02:52 > Elapsed processing time: 00:02:54 > Elapsed processing time: 00:02:36 > Elapsed processing time: 00:02:25 > > This is for a small node with <1,400 objects. > > Total number of objects inspected: 1,321 > Total number of objects inspected: 1,324 > Total number of objects inspected: 1,327 > Total number of objects inspected: 1,331 > Total number of objects inspected: 1,332 > Total number of objects inspected: 1,339 > Total number of objects inspected: 1,339 > Total number of objects inspected: 1,339 > Total number of objects inspected: 1,340 > Total number of objects inspected: 1,342 > Total number of objects inspected: 1,344 > Total number of objects inspected: 1,348 > > > ---------- Forwarded message ---------- > From: Andrew Raibeck <[email protected]> > Date: Fri, Oct 6, 2017 at 11:26 AM > Subject: Re: [ADSM-L] Problems after upgrading client from 7.1.6.3 to > 8.1.0.2 > To: [email protected] > > > Hi Zoltan, > > It sounds like you are seeing multiple issues, best to deal with them one > at a time. > > 1) For the crash dump messages, it is possible that there is only one dump, > and the message just keeps repeating because the client detects the same > file over and over again. Check the corresponding dsmcrash.log file (start > at the bottom of that file) to see if crashes are constantly occurring, and > when they occurred. If the crashes are persistent, you should open a PMR > and upload the dump, so it can be reviewed. If it is a one-time crash, you > can do the same thing, or you can delete the dump file (dsmcrash.dmp), > which should eliminate the messages. If the problem reoccurs, open a PMR, > etc. > > 2) I am not sure what the hung sessions are. Sessions in IdleW state mean > that the session is not in transaction with the server. Most likely these > are consumer sessions waiting for work, possibly related to item 3, below. > > 3) You mentioned the backups run longer, and you observe "Process Dirs" as > the category with the most time in it. "Process Dirs" is the time the > client spends traversing the file system, searching for changed files. A > lengthy "Process Dirs" time is not necessarily a "problem" for file systems > with a large number, e.g., hundreds of thousands, or millions, of files; > and that number can be the largest if the number of changed files is very > small in comparison. This could also explain consumer sessions in IdleW > state (item 2, above). > > Does the dsminstr.log file include data from when the client level was > 7.1.6.3? If yes, then you can compare the latest 7.1.6.3 metrics with the > earliest 8.1.0.2 metrics to see if there was truly some big difference > introduced at the time you upgraded the client. Off the top of my head, I > cannot think of anything in the client code itself that would suddenly > cause such a performance issue. > > Without actual data to see what is going on (complete dsmsched.log file, > dsmerror.log file, dsminstr.log file, dsm.opt, etc.) it is hard to provide > more specific suggestions. > > Regards, > > Andy > > ____________________________________________________________ > ________________ > > Andrew Raibeck | IBM Spectrum Protect Level 3 | [email protected] > > IBM Tivoli Storage Manager links: > Product support: > https://www.ibm.com/support/entry/portal/product/tivoli/tivo > li_storage_manager > > Online documentation: > http://www.ibm.com/support/knowledgecenter/SSGSG7/landing/ > welcome_ssgsg7.html > > Product Wiki: > https://www.ibm.com/developerworks/community/wikis/home/ > wiki/Tivoli%20Storage%20Manager > > "ADSM: Dist Stor Manager" <[email protected]> wrote on 2017-10-06 > 10:36:15: > > > From: Zoltan Forray <[email protected]> > > To: [email protected] > > Date: 2017-10-06 10:37 > > Subject: Problems after upgrading client from 7.1.6.3 to 8.1.0.2 > > Sent by: "ADSM: Dist Stor Manager" <[email protected]> > > > > We have this server that is used to backup DFS mounts/files. It has 30+ > > TSM nodes configured to it. > > > > In the past, everything has been running like it should, albeit a little > > slow sometimes due to the size of some of the DFS mounts but the backups > > would usually finish. > > > > After upgrading the client to 8.1.0.2, we are experiencing constant > > problems. Many of the backups are not completing, some node sessions are > > running 10+ hours to examine 1000 files and backup 7MB in changes, etc. > > > > Looking at the logs, we are seeing numerous "A IBM Spectrum Protect core > > file or crash report was found:" crash-dumps for most of the nodes, often > > numerous times within the same backup session. > > > > Instrumentation statistics on a few nodes I examined say the largest time > > consumer is "Process Dirs". > > > > From the SP server perspective, these nodes are in a constant "Idle > Wait". > > Session info shows lots of metadata send to the client, but very little > > coming from the client. > > > > The only change has been the SP client upgrade. Yes we have rebooted the > > server (Windows 2012) multiple times to clear the hung sessions > (sometimes > > running multiple days and never ending). > > > > Any way to determine is going on? > > > > > > > > > > > > -- > > *Zoltan Forray* > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > Xymon Monitor Administrator > > VMware Administrator > > Virginia Commonwealth University > > UCC/Office of Technology Services > > www.ucc.vcu.edu > > [email protected] - 804-828-4807 > > Don't be a phishing victim - VCU and other reputable organizations will > > never use email to request that you reply with your password, social > > security number or confidential personal information. For more details > > visit https://urldefense.proofpoint.com/v2/url? > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=bnJkpUfY38- > > > nYutnVeKi_lreUncb4usTMD65HpEa4ko&s=hvKEF7hPnJ83ZRrIKoqxy35Cf > ELFHGsrMocoLUGDXZU&e= > > > > > > > -- > *Zoltan Forray* > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > Xymon Monitor Administrator > VMware Administrator > Virginia Commonwealth University > UCC/Office of Technology Services > www.ucc.vcu.edu > [email protected] - 804-828-4807 <(804)%20828-4807> > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit http://phishing.vcu.edu/ > -- *Zoltan Forray* Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu [email protected] - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/
