Hi Zoltan, I do not think the problem is really with MSVCR110.dll, and the stack traces in the dsmcrash.log file are not going to be particularly insightful (other than knowing when a crash occurs, how often, and exception type). I suppose if you upgraded the client but did not reboot if necessary, that *might* be a contributing factor. But we would have to see the corresponding dsmcrash.dmp file to get a better sense of why it occurs.
Regards, Andy ____________________________________________________________________________ Andrew Raibeck | IBM Spectrum Protect Level 3 | [email protected] IBM Tivoli Storage Manager links: Product support: https://www.ibm.com/support/entry/portal/product/tivoli/tivoli_storage_manager Online documentation: http://www.ibm.com/support/knowledgecenter/SSGSG7/landing/welcome_ssgsg7.html Product Wiki: https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%20Storage%20Manager "ADSM: Dist Stor Manager" <[email protected]> wrote on 2017-10-06 12:28:07: > From: Zoltan Forray <[email protected]> > To: [email protected] > Date: 2017-10-06 12:29 > Subject: Re: Problems after upgrading client from 7.1.6.3 to 8.1.0.2 > Sent by: "ADSM: Dist Stor Manager" <[email protected]> > > Andy, > > From the dsmcrash.log file: > > 2017/10/04 10:25:54 > IBM Spectrum Protect > Version: 8.1.0.2 > Build date: Mon May 01 16:15:45 2017 > > dsmcsvc.exe caused exception C0000005 (EXCEPTION_ACCESS_VIOLATION) at > 00000000722C7C65 > > with pointers to MSVCR110.dll. > > Can you tell me what level/version this dll is supposed to be at? I > suspect something didn't get updated when the client was upgraded. > > > On Fri, Oct 6, 2017 at 11:26 AM, Andrew Raibeck <[email protected]> wrote: > > > Hi Zoltan, > > > > It sounds like you are seeing multiple issues, best to deal with them one > > at a time. > > > > 1) For the crash dump messages, it is possible that there is only one dump, > > and the message just keeps repeating because the client detects the same > > file over and over again. Check the corresponding dsmcrash.log file (start > > at the bottom of that file) to see if crashes are constantly occurring, and > > when they occurred. If the crashes are persistent, you should open a PMR > > and upload the dump, so it can be reviewed. If it is a one-time crash, you > > can do the same thing, or you can delete the dump file (dsmcrash.dmp), > > which should eliminate the messages. If the problem reoccurs, open a PMR, > > etc. > > > > 2) I am not sure what the hung sessions are. Sessions in IdleW state mean > > that the session is not in transaction with the server. Most likely these > > are consumer sessions waiting for work, possibly related to item 3, below. > > > > 3) You mentioned the backups run longer, and you observe "Process Dirs" as > > the category with the most time in it. "Process Dirs" is the time the > > client spends traversing the file system, searching for changed files. A > > lengthy "Process Dirs" time is not necessarily a "problem" for file systems > > with a large number, e.g., hundreds of thousands, or millions, of files; > > and that number can be the largest if the number of changed files is very > > small in comparison. This could also explain consumer sessions in IdleW > > state (item 2, above). > > > > Does the dsminstr.log file include data from when the client level was > > 7.1.6.3? If yes, then you can compare the latest 7.1.6.3 metrics with the > > earliest 8.1.0.2 metrics to see if there was truly some big difference > > introduced at the time you upgraded the client. Off the top of my head, I > > cannot think of anything in the client code itself that would suddenly > > cause such a performance issue. > > > > Without actual data to see what is going on (complete dsmsched.log file, > > dsmerror.log file, dsminstr.log file, dsm.opt, etc.) it is hard to provide > > more specific suggestions. > > > > Regards, > > > > Andy > > > > ____________________________________________________________ > > ________________ > > > > Andrew Raibeck | IBM Spectrum Protect Level 3 | [email protected] > > > > IBM Tivoli Storage Manager links: > > Product support: > > https://www.ibm.com/support/entry/portal/product/tivoli/ > > tivoli_storage_manager > > > > Online documentation: > > http://www.ibm.com/support/knowledgecenter/SSGSG7/ > > landing/welcome_ssgsg7.html > > > > Product Wiki: > > https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli% > > 20Storage%20Manager > > > > "ADSM: Dist Stor Manager" <[email protected]> wrote on 2017-10-06 > > 10:36:15: > > > > > From: Zoltan Forray <[email protected]> > > > To: [email protected] > > > Date: 2017-10-06 10:37 > > > Subject: Problems after upgrading client from 7.1.6.3 to 8.1.0.2 > > > Sent by: "ADSM: Dist Stor Manager" <[email protected]> > > > > > > We have this server that is used to backup DFS mounts/files. It has 30+ > > > TSM nodes configured to it. > > > > > > In the past, everything has been running like it should, albeit a little > > > slow sometimes due to the size of some of the DFS mounts but the backups > > > would usually finish. > > > > > > After upgrading the client to 8.1.0.2, we are experiencing constant > > > problems. Many of the backups are not completing, some node sessions are > > > running 10+ hours to examine 1000 files and backup 7MB in changes, etc. > > > > > > Looking at the logs, we are seeing numerous "A IBM Spectrum Protect core > > > file or crash report was found:" crash-dumps for most of the nodes, often > > > numerous times within the same backup session. > > > > > > Instrumentation statistics on a few nodes I examined say the largest time > > > consumer is "Process Dirs". > > > > > > From the SP server perspective, these nodes are in a constant "Idle > > Wait". > > > Session info shows lots of metadata send to the client, but very little > > > coming from the client. > > > > > > The only change has been the SP client upgrade. Yes we have rebooted the > > > server (Windows 2012) multiple times to clear the hung sessions > > (sometimes > > > running multiple days and never ending). > > > > > > Any way to determine is going on? > > > > > > > > > > > > > > > > > > -- > > > *Zoltan Forray* > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > Xymon Monitor Administrator > > > VMware Administrator > > > Virginia Commonwealth University > > > UCC/Office of Technology Services > > > www.ucc.vcu.edu > > > [email protected] - 804-828-4807 > > > Don't be a phishing victim - VCU and other reputable organizations will > > > never use email to request that you reply with your password, social > > > security number or confidential personal information. For more details > > > visit https://urldefense.proofpoint.com/v2/url? > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=bnJkpUfY38- > > > > > nYutnVeKi_lreUncb4usTMD65HpEa4ko&s=hvKEF7hPnJ83ZRrIKoqxy35CfELFHG > > srMocoLUGDXZU&e= > > > > > > > > > > > -- > *Zoltan Forray* > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > Xymon Monitor Administrator > VMware Administrator > Virginia Commonwealth University > UCC/Office of Technology Services > www.ucc.vcu.edu > [email protected] - 804-828-4807 > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit https://urldefense.proofpoint.com/v2/url? > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=SSNQ9Hyzp- > uXTAnZNWKcnvvYR9klhqPdbivWnLFH6Gk&s=pOYK0915YykOlToqNC0KdOnwwh2r93uz68JThlrbepU&e= >
