Re: TSM server performance continuing

[Kizzire, Chris]

We were having performance issues. I still think there are some. I could tell 
it was having issues when Q PR took way too long to show- if at all. But much 
improved now after RAM increase.

Specs for our environment:
SP 8.1.4.0
IBM 750- AIX 7.2 
3.1 TB DB on SSD
128GB RAM increased to 180GB

DR site:
SP 8.1.4.0
IBM 770- AIX 7.2
3.1TB DB on HDD
104GB RAM increased to 250GB

WOW at the difference.
We did increase Quedepth. The big difference was after RAM increase. DeDup & 
compression are memory hogs.
We still plan to open a PMR for AIX performance. We are now maxed out on RAM.

Chris Kizzire
Backup Administrator (Network Engineer II)

BROOKWOOD BAPTIST HEALTH
Information Systems
O:   205.820.5973

chris.kizz...@bhsala.com
BROOKWOODBAPTISTHEALTH.COM

-Original Message-
From: ADSM: Dist Stor Manager  On Behalf Of Loon, Eric 
van (ITOP NS) - KLM
Sent: Wednesday, August 21, 2019 10:39 AM
To: ADSM-L@VM.MARIST.EDU
Subject: [ADSM-L] TSM server performance continuing


CAUTION: ***EXTERNAL EMAIL*** Do NOT click links or open attachments unless you 
recognize the sender and know the content is safe. If you are unsure, please 
use PhishAlarm to report suspicious emails.

Hi guys,

A few weeks ago I already wrote about the severe performance issues we have 
with our TSM 7.1 servers. In the 'old days' we used to back up our clients to 
TSM 6.3 servers with Data Domains attached. Smaller clients backed up through 
the LAN, large ones through the SAN.
Our newer servers use LAN-only with directory containers and the performance of 
these servers really sucks. Setting up a session takes sometimes almost one 
minute and a q stg also takes 30 to 50 seconds. I noticed that performance is 
OK when there are no TDP for Oracle sessions running, but as soon as they are 
started the performance starts to drop drastically.
We are really lost on where to look for the cause. I sent numerous logs and 
traces to IBM, but I guess they are out of ideas too since I don't hear 
anything back from them lately. The only thing is that supports notices delays 
in DB2, but they don't know why...
What I noticed on my TSM server is that as soon as there is a load on the 
server, the blocked queue starts to rise. I would like to know if that's 
something to focus on or not.
Can some of you please run the "vmstat 1" command on their Linux server 
(preferably one with directory containers too) and let me know if you too see 
values other than 0 in the B column?
Thank you very much for your help in advance!

Kind regards,
Eric van Loon
Air France/KLM Storage & Backup

For information, services and offers, please visit our web site: 
http://www.klm.com. This e-mail and any attachment may contain confidential and 
privileged material intended for the addressee only. If you are not the 
addressee, you are notified that no part of the e-mail or any attachment may be 
disclosed, copied or distributed, and that any other action related to this 
e-mail or attachment is strictly prohibited, and may be unlawful. If you have 
received this e-mail by error, please notify the sender immediately by return 
e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its 
employees shall not be liable for the incorrect or incomplete transmission of 
this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch 
Airlines) is registered in Amstelveen, The Netherlands, with registered number 
33014286


--- Confidentiality Notice: The 
information contained in this email message is privileged and confidential 
information and intended only for the use of the individual or entity named in 
the address. If you are not the intended recipient, you are hereby notified 
that any dissemination, distribution, or copying of this information is 
strictly prohibited. If you received this information in error, please notify 
the sender and delete this information from your computer and retain no copies 
of any of this information.

Re: deletion performance of large deduplicated files

[Kizzire, Chris]

OK. Thanks. Michael. We'll give it a try.

Chris Kizzire
Backup Administrator (Network Engineer II)

BROOKWOOD BAPTIST HEALTH
Information Systems
O:   205.820.5973

chris.kizz...@bhsala.com
BROOKWOODBAPTISTHEALTH.COM


-Original Message-
From: ADSM: Dist Stor Manager  On Behalf Of Michael Prix
Sent: Friday, July 19, 2019 10:14 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] deletion performance of large deduplicated files


CAUTION: ***EXTERNAL EMAIL*** Do NOT click links or open attachments unless you 
recognize the sender and know the content is safe. If you are unsure, please 
use PhishAlarm to report suspicious emails.

Chris,

  yours is easy to answer: You have a performance problem, not a TSM problem.
For VM-backup I have a dedicated LPAR in a S842, 3vCPU, 128GB RAM, SR-IOV 10GB. 
Storage is a V7000, SSD/SAS. There has never been a backup or restore problem 
performance wise, everything is running with wire-speed.
 The only problem we hsd was client side dedup, II28096 hit us bad, 5 days of 
audit for each TSM-instance over and over again until the reason was identified.

For a start: keep the size of hdisk small for the db. Better 2 x 50GB that 1 
100GB. chfs -e x and reorgvg is your friend. mount option "rbrw,noatime" is of 
great use for db, log, arc and stg pools. Check you queue_depth and set it to 
max.

As for SQL: use as many streams as the client can handle. CPU is there to 
handle load, not to waste energy by idleing.

--
Michael Prix

On Fri, 2019-07-19 at 13:02 +0000, Kizzire, Chris wrote:
> Alas... we are not the only ones I knew it...
> We went from TSM 6.3 to SP 8.1.4.0. Performance is 5 times slower in 
> the new Environment overall. We use Container Pools, Dedup, & 
> Compression. We use SP for VE for most VM's. Baclient & SQL for physical 
> machines & vm's w/ SQL.
> It took about 17 hours to restore 4.5TB SQL DB the other day.
> Main Server:
>   IBM Power 750 running AIX 7.2 .
>   3.1TB DB is on SSD.
>   128GB RAM
> Server at DR site
>   IBM Power 770 running AIX 7.2
>   3.1TB DB NOT on SSD
>   100ish GB RAM
> Container Pool is on 1.5 year old IBM V5030 w/ Identical V5030 at DR 
> site.
>
> IBM says open a Performance PMR for AIX- which we have yet to do.
> Protect Stgpool runs for days & we have to cancel because we get too 
> far behind on Replication. If we are lucky we might can replicate 12TB 
> in a 24 hour period w/ 100 sessions (maxsessions=50)
>
>
> Chris Kizzire
> Backup Administrator (Network Engineer II)
>
> BROOKWOOD BAPTIST HEALTH
> Information Systems
> O:   205.820.5973
>
> chris.kizz...@bhsala.com
> BROOKWOODBAPTISTHEALTH.COM
>
>
> -Original Message-
> From: ADSM: Dist Stor Manager  On Behalf Of 
> Michael Prix
> Sent: Friday, July 19, 2019 5:11 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] deletion performance of large deduplicated files
>
>
> CAUTION: ***EXTERNAL EMAIL*** Do NOT click links or open attachments 
> unless you recognize the sender and know the content is safe. If you 
> are unsure, please use PhishAlarm to report suspicious emails.
>
> Hello Eric,
>
>   welcome to my nightmares. Take a seat, wanna have a drink?
>
> I had the pleasure of performance and data corruption PMRs during the 
> last two years with TDP Oracle. Yes, at first the customer got blamed 
> for not adhering completely to to blueprints, but after some weeks it 
> boild down to  ...
> silence.
> Data corruption was because of what ended in IT28096 - now fixed.
> Performance is interesting, but resembles to what you have written. We 
> work with MAXPIECESIZE settings on RMAN to keep the backup pieces 
> small and got some interesting values, pending further observation, 
> but we might be on a cheerful way. I'm talking about database sizes of 
> 50TB here, warehouse style.
>   In between we moved the big DBs to a dedicated server to prove that 
> the performance drop is because of the big DBs, and the remaining 
> "small" DBs  - size of 500MB up to 5TB - didn't put any measurable 
> stress on the DB in terms of expiration and protect stgpool. Even the 
> big DBs on their dedicated server performed better in terms of 
> expiration and protect stgpool, which might have been a coincidence of 
> these DBs holding nearly the same data and having the same retention period.
>
> What I can't observe is a slowness of the DB. Queries are answered in 
> the normal time - depending on the query. a count(*) from 
> backupobjects naturally takes some time, considerably longer when you 
> use dedup, but the daily queries are answered in the "normal" timeframe.
>
> What helped immediately was some tuning:
> - More LUNS and filesystems for the TSM-DB
> - smaller disk

Re: deletion performance of large deduplicated files

[Kizzire, Chris]

Alas... we are not the only ones I knew it...
We went from TSM 6.3 to SP 8.1.4.0. Performance is 5 times slower in the new 
Environment overall. We use Container Pools, Dedup, & Compression. We use SP 
for VE for most VM's. Baclient & SQL for physical machines & vm's w/ SQL.
It took about 17 hours to restore 4.5TB SQL DB the other day.
Main Server:
IBM Power 750 running AIX 7.2 .
3.1TB DB is on SSD.
128GB RAM
Server at DR site
IBM Power 770 running AIX 7.2
3.1TB DB NOT on SSD
100ish GB RAM
Container Pool is on 1.5 year old IBM V5030
w/ Identical V5030 at DR site.

IBM says open a Performance PMR for AIX- which we have yet to do.
Protect Stgpool runs for days & we have to cancel because we get too far behind 
on Replication. If we are lucky we might can replicate 12TB in a 24 hour period 
w/ 100 sessions (maxsessions=50)


Chris Kizzire
Backup Administrator (Network Engineer II)

BROOKWOOD BAPTIST HEALTH
Information Systems
O:   205.820.5973

chris.kizz...@bhsala.com
BROOKWOODBAPTISTHEALTH.COM


-Original Message-
From: ADSM: Dist Stor Manager  On Behalf Of Michael Prix
Sent: Friday, July 19, 2019 5:11 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] deletion performance of large deduplicated files


CAUTION: ***EXTERNAL EMAIL*** Do NOT click links or open attachments unless you 
recognize the sender and know the content is safe. If you are unsure, please 
use PhishAlarm to report suspicious emails.

Hello Eric,

  welcome to my nightmares. Take a seat, wanna have a drink?

I had the pleasure of performance and data corruption PMRs during the last two 
years with TDP Oracle. Yes, at first the customer got blamed for not adhering 
completely to to blueprints, but after some weeks it boild down to  ...
silence.
Data corruption was because of what ended in IT28096 - now fixed.
Performance is interesting, but resembles to what you have written. We work 
with MAXPIECESIZE settings on RMAN to keep the backup pieces small and got some 
interesting values, pending further observation, but we might be on a cheerful 
way. I'm talking about database sizes of 50TB here, warehouse style.
  In between we moved the big DBs to a dedicated server to prove that the 
performance drop is because of the big DBs, and the remaining "small" DBs  - 
size of 500MB up to 5TB - didn't put any measurable stress on the DB in terms 
of expiration and protect stgpool. Even the big DBs on their dedicated server 
performed better in terms of expiration and protect stgpool, which might have 
been a coincidence of these DBs holding nearly the same data and having the 
same retention period.

What I can't observe is a slowness of the DB. Queries are answered in the 
normal time - depending on the query. a count(*) from backupobjects naturally 
takes some time, considerably longer when you use dedup, but the daily queries 
are answered in the "normal" timeframe.

What helped immediately was some tuning:
- More LUNS and filesystems for the TSM-DB
- smaller disks, but more of them, for each filesystem.
  changing the disks from 100GB to 2 x 50GB for each DB-filesystem got me a 
performance boost of 200% in expiration and backup db. Unbelievable, but true.
Yes, I'm using SSD. And SVC. And multiple storage systems. Performance isn't 
the problem, we are measuring 2ms respone time for write AND read.
- stripeset for each fileset


--
Michael Prix

On Fri, 2019-07-19 at 07:29 +, Loon, Eric van (ITOP NS) - KLM wrote:
> Hi TSM/SP-ers,
>
> We are struggling with the performance of our TSM servers for months 
> now. We are running several servers with hardware (Data Domain) dedup 
> for years without any problems, but on our new servers with directory 
> container pools performance is really, really bad.
> The servers and storage are designed according to the Blueprints and 
> they are working fine as long as you do not add large database (Oracle 
> and SAP) client to them. As soon as you do, the overall server 
> performance becomes very bad: client and admin session initiation 
> takes 20 to 40 seconds, SQL queries run for minutes where they should 
> take a few seconds and q query stgpool sometimes takes more than a minute to 
> respond!
> I have two cases open for this. In one case we focused a lot on the OS 
> and disk performance, but during that process I noticed that the 
> performance is most likely caused by the way TSM processes large (few 
> hundred MB) files. I performed a large amount of tests and came to the 
> conclusion that it takes TSM a huge amount of time to delete large 
> deduplicated files, both in container pools as deduplicated file 
> pools. As test I use an TDP for Oracle client which uses a backup 
> piece size of 900 MB. The client contains about
> 5000 files. Deleting the files from a container pool takes more than 
> an hour. When you run a delete object for the files individually I see 
> that most files take more than a second(!) to delete. If I put that 

SP for VE error

[Kizzire, Chris]

Hello everyone.
I have been getting this error (GVM5113E) for about 6 months or so in the 
vSphere Webclient. No green check by "View schedules and manage data movers". 
When I click Configure schedules and data movers I get the GVM5113E.
IBM Error messages seem to stop at GVM5112E. I feel like there might be 
something wrong with tagging, but am not sure. Schedules are running. I also 
get GVM5002E every time I submit an on-demand backup. However the backup 
usually runs, but just seems like a delayed start. I am not sure if they are 
related.
Any help would be appreciated.

Wording is:
GVM5113E: An error occurred getting the backup management information. Contact 
the IBM Spectrum Protect server administrator for assistance.



We are running SP 8.1.4.0 on AIX 7.2
SP4VE 8.1.4.0






Chris Kizzire
Backup Administrator (Network Engineer II)

BROOKWOOD BAPTIST HEALTH
Information Systems
O:   205.820.5973

chris.kizz...@bhsala.com
BROOKWOODBAPTISTHEALTH.COM


--- Confidentiality Notice: The 
information contained in this email message is privileged and confidential 
information and intended only for the use of the individual or entity named in 
the address. If you are not the intended recipient, you are hereby notified 
that any dissemination, distribution, or copying of this information is 
strictly prohibited. If you received this information in error, please notify 
the sender and delete this information from your computer and retain no copies 
of any of this information.

SharePoint

[Kizzire, Chris]

Is anyone using SP 8.1.x to backup & restore SharePoint?
If so, are you having any issues. My understanding SP will back it up, but 
cannot do a file level restore. We would like get red of of our 3rd party 
backup solution for SharePoint & only use SP.

Chris Kizzire
Backup Administrator (Network Engineer II)

BROOKWOOD BAPTIST HEALTH
Information Systems
O:   205.820.5973

chris.kizz...@bhsala.com
BROOKWOODBAPTISTHEALTH.COM



--- Confidentiality Notice: The 
information contained in this email message is privileged and confidential 
information and intended only for the use of the individual or entity named in 
the address. If you are not the intended recipient, you are hereby notified 
that any dissemination, distribution, or copying of this information is 
strictly prohibited. If you received this information in error, please notify 
the sender and delete this information from your computer and retain no copies 
of any of this information.

Share Point

[Kizzire, Chris]

Hello,
Question:  Is anyone successfully usually Spectrum Protect 8.1 to backup Share 
Point (Doc Ave) for file level restore capabilities?


Chris Kizzire
Network Engineer II

BROOKWOOD BAPTIST HEALTH
Information Systems
O:   205.820.5973

chris.kizz...@bhsala.com
BROOKWOODBAPTISTHEALTH.COM


--- Confidentiality Notice: The 
information contained in this email message is privileged and confidential 
information and intended only for the use of the individual or entity named in 
the address. If you are not the intended recipient, you are hereby notified 
that any dissemination, distribution, or copying of this information is 
strictly prohibited. If you received this information in error, please notify 
the sender and delete this information from your computer and retain no copies 
of any of this information.