Re: [Veritas-bu] Shared memory issue with Solaris 10
Hi This is what we usually do to find the best performance in an environment. It is good and well to say 64 buffers with a size of 262144 but this is not necessarily the best in another environment. For you environment, do the following. 1. Take a 2 to 3 GB file and back it up without any buffer files. Take note of throughput. This will be base throughput (8 x 65536). 2. Do same test with 16 buffers and size of 65536, then 32 with 65536 and then 64 with 65536. Take note of best throughput with 65536 size. 3. Do same test with 16 buffers and size of 131072, then 32 with 131072 and then 64 with 131072. Take note of best throughput with 131072 size. 4. Do same test with 16 buffers and size of 262144, then 32 with 262144 and then 64 with 262144. Take note of best throughput with 262144 size. 5. Then set your buffers to the best throughput from tests from 1,2 and 3. 6. Remember to add same buffer files on your media servers. Usually the best settings are: In /usr/openv/netbackup/db/config - NUMBER_DATA_BUFFERS = 32 NUMBER_DATA_BUFFERS_RESTORE = 32 SIZE_DATA_BUFFERS = 131072 As everybody said, 3072 will def give you shared memory issues. JR -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steve Quan Sent: Thursday, July 24, 2008 7:40 PM To: VERITAS-BU@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Shared memory issue with Solaris 10 Sure, NUMBER_DATA_BUFFERS 64 SIZE_DATA_BUFFERS 262144 /Steve --- On Thu, 24 Jul 2008, NBU wrote: Steve, Can you share what is the Number / Size of data buffers set in your environment. +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Shared memory issue with Solaris 10
Steve, Can you share what is the Number / Size of data buffers set in your environment. +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Shared memory issue with Solaris 10
1. I really REALLY hope you do not have number of data buffers set to 3072; that's ludicrously high. A reasonable number for LTO3 drives would be 32 with size data buffers of 262144. Don't forget that the amount of shared memory required by NBU is size data buffers * num data buffers * mpx * number of drives. With your settings your looking at 400Mb of shared memory before you take into account mpx and 2. With solaris 10 you should not be changing the /etc/system file - a lot of settings that you'd put in there for Sol9 and earlier are now deprecated or obsolete (as per SUN's documentation in system tuning); IPC tuning is now done by using solaris 10 projects (/etc/projects file). -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of NBU Sent: 23 July 2008 13:40 To: VERITAS-BU@mailman.eng.auburn.edu Subject: [Veritas-bu] Shared memory issue with Solaris 10 Friends, Fine tunning has been done in /etc/system also but no luck. Following is systems setting: We have set Net_BUFFER SIZE and /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS = 131072 /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS= 3072 Following is the kernel parameter of Solaris 10 OS [EMAIL PROTECTED] # prtctl $$ -bash: prtctl: command not found [EMAIL PROTECTED] # prctl $$ process: 15640: -bash NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT process.max-port-events privileged 65.5K - deny - system 2.15G max deny - process.max-msg-messages privileged 8.19K - deny - system 4.29G max deny - process.max-msg-qbytes privileged 64.0KB - deny - system 16.0EB max deny - process.max-sem-ops privileged 512 - deny - system 2.15G max deny - process.max-sem-nsems privileged 512 - deny - system 32.8K max deny - process.max-address-space privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-descriptor basic 256 - deny 15640 privileged 65.5K - deny - system 2.15G max deny - process.max-core-size privileged 8.00EB max deny - system 8.00EB max deny - process.max-stack-size basic 8.00MB - deny 15640 privileged 8.00EB - deny - system 8.00EB max deny - process.max-data-size privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-size privileged 8.00EB max deny,signal=XFSZ - system 8.00EB max deny - process.max-cpu-time privileged 18.4Es inf signal=XCPU - system 18.4Es inf none - task.max-cpu-time system 18.4Es inf none - task.max-lwps system 2.15G max deny - project.max-contracts privileged 10.0K - deny - system 2.15G max deny - project.max-device-locked-memory privileged 3.92GB - deny - system 16.0EB max deny - project.max-locked-memory system 16.0EB max deny - project.max-port-ids privileged 8.19K - deny - system 65.5K max deny - project.max-shm-memory privileged 48.0GB - deny - system 16.0EB max deny - project.max-shm-ids privileged 512 - deny - system 16.8M max deny - project.max-msg-ids privileged 256 - deny - system 16.8M max deny - project.max-sem-ids privileged 512 - deny - system 16.8M max deny - project.max-crypto-memory privileged 15.7GB - deny - system 16.0EB max deny - project.max-tasks system 2.15G max deny - project.max-lwps system 2.15G max deny - project.cpu-shares privileged 1 - none - system 65.5K max none - zone.max-swap system 16.0EB max deny - zone.max-locked-memory system 16.0EB max deny - zone.max-shm-memory system 16.0EB max deny - zone.max-shm-ids system 16.8M max deny - zone.max-sem-ids system 16.8M max deny - zone.max-msg-ids system 16.8M max deny - zone.max-lwps system 2.15G max deny - zone.cpu-shares privileged 1 - none - system 65.5K max none - +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Shared memory issue with Solaris 10
Sure, NUMBER_DATA_BUFFERS 64 SIZE_DATA_BUFFERS 262144 /Steve --- On Thu, 24 Jul 2008, NBU wrote: Steve, Can you share what is the Number / Size of data buffers set in your environment. +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Shared memory issue with Solaris 10
Hi forum, I have a Master and 3 media server on solaris 9 having Netbackup 6.0 MP4. Recently added new media server with solaris 10. Problem which i am facing is when the load increases (Schedule starts) backup start failing with error code 89. Experts need your help on this. thanks +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Shared memory issue with Solaris 10
On Wed, 23 Jul 2008, NBU wrote: Hi forum, I have a Master and 3 media server on solaris 9 having Netbackup 6.0 MP4. Recently added new media server with solaris 10. Problem which i am facing is when the load increases (Schedule starts) backup start failing with error code 89. Experts need your help on this. thanks +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu Error 89 is shared memory problems, read the archive on how to fix that for Solaris 10, generally you need to increase your SHMMAX value in the kernel (Linux) in Solaris it has been awhile :) Justin. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Shared memory issue with Solaris 10
I'd be interested in this too, as i'd heard you didnt need to bother with /etc/system tuning in Solaris 10. Cheers Justin Piszcz wrote: On Wed, 23 Jul 2008, NBU wrote: Hi forum, I have a Master and 3 media server on solaris 9 having Netbackup 6.0 MP4. Recently added new media server with solaris 10. Problem which i am facing is when the load increases (Schedule starts) backup start failing with error code 89. Experts need your help on this. thanks +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu Error 89 is shared memory problems, read the archive on how to fix that for Solaris 10, generally you need to increase your SHMMAX value in the kernel (Linux) in Solaris it has been awhile :) Justin. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Shared memory issue with Solaris 10
Normally you shouldn't but some of the shared momory settings are still working and can only be tuned by /etc/system Dave Markham a écrit : I'd be interested in this too, as i'd heard you didnt need to bother with /etc/system tuning in Solaris 10. Cheers Justin Piszcz wrote: On Wed, 23 Jul 2008, NBU wrote: Hi forum, I have a Master and 3 media server on solaris 9 having Netbackup 6.0 MP4. Recently added new media server with solaris 10. Problem which i am facing is when the load increases (Schedule starts) backup start failing with error code 89. Experts need your help on this. thanks +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu Error 89 is shared memory problems, read the archive on how to fix that for Solaris 10, generally you need to increase your SHMMAX value in the kernel (Linux) in Solaris it has been awhile :) Justin. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Shared memory issue with Solaris 10
There is your problem right there. $ cat NUMBER_DATA_BUFFERS 32 $ cat SIZE_DATA_BUFFERS 262144 For LTO-2 and LTO-3 you should be using the SIZE_DATA_BUFFERS as shown above, for the number of DATA_BUFFERS, anything above 32 is usually overkill/makes no differnece in performance. Justin. On Wed, 23 Jul 2008, NBU wrote: Friends, Fine tunning has been done in /etc/system also but no luck. Following is systems setting: We have set Net_BUFFER SIZE and /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS = 131072 /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS= 3072 Following is the kernel parameter of Solaris 10 OS [EMAIL PROTECTED] # prtctl $$ -bash: prtctl: command not found [EMAIL PROTECTED] # prctl $$ process: 15640: -bash NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT process.max-port-events privileged 65.5K - deny - system 2.15G max deny - process.max-msg-messages privileged 8.19K - deny - system 4.29G max deny - process.max-msg-qbytes privileged 64.0KB - deny - system 16.0EB max deny - process.max-sem-ops privileged 512 - deny - system 2.15G max deny - process.max-sem-nsems privileged 512 - deny - system 32.8K max deny - process.max-address-space privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-descriptor basic 256 - deny 15640 privileged 65.5K - deny - system 2.15G max deny - process.max-core-size privileged 8.00EB max deny - system 8.00EB max deny - process.max-stack-size basic 8.00MB - deny 15640 privileged 8.00EB - deny - system 8.00EB max deny - process.max-data-size privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-size privileged 8.00EB max deny,signal=XFSZ - system 8.00EB max deny - process.max-cpu-time privileged 18.4Es inf signal=XCPU - system 18.4Es inf none - task.max-cpu-time system 18.4Es inf none - task.max-lwps system 2.15G max deny - project.max-contracts privileged 10.0K - deny - system 2.15G max deny - project.max-device-locked-memory privileged 3.92GB - deny - system 16.0EB max deny - project.max-locked-memory system 16.0EB max deny - project.max-port-ids privileged 8.19K - deny - system 65.5K max deny - project.max-shm-memory privileged 48.0GB - deny - system 16.0EB max deny - project.max-shm-ids privileged 512 - deny - system 16.8M max deny - project.max-msg-ids privileged 256 - deny - system 16.8M max deny - project.max-sem-ids privileged 512 - deny - system 16.8M max deny - project.max-crypto-memory privileged 15.7GB - deny - system 16.0EB max deny - project.max-tasks system 2.15G max deny - project.max-lwps system 2.15G max deny - project.cpu-shares privileged 1 - none - system 65.5K max none - zone.max-swap system 16.0EB max deny - zone.max-locked-memory system 16.0EB max deny - zone.max-shm-memory system 16.0EB max deny - zone.max-shm-ids system 16.8M max deny - zone.max-sem-ids system 16.8M max deny - zone.max-msg-ids system 16.8M max deny - zone.max-lwps system 2.15G max deny - zone.cpu-shares privileged 1 - none - system 65.5K max none - +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Shared memory issue with Solaris 10
I'll concur with Justin. 262144 SIZE is a good performer, and 32 NUMBER is a sweet spot for LTO-4. If you have the ram and lots of inbound connectivity, you can go to 64 buffers, and it seems to work well. With 131072 SIZE and 3072 NUMBER, if you have 1 drive and multiplexing set to 1, that tries to allocate a 400MB segment of memory. Add in number of drives and multiplexing, I'm not surprised it cannot allocate memory. The formula from the docs is (SIZE_DATA_BUFFERS * NUMBER_DATA_BUFFERS * Num_Drives * MPX_Factor) -Jon - Original Message From: Justin Piszcz [EMAIL PROTECTED] To: NBU [EMAIL PROTECTED] Cc: VERITAS-BU@mailman.eng.auburn.edu Sent: Wednesday, July 23, 2008 9:11:06 AM Subject: Re: [Veritas-bu] Shared memory issue with Solaris 10 There is your problem right there. $ cat NUMBER_DATA_BUFFERS 32 $ cat SIZE_DATA_BUFFERS 262144 For LTO-2 and LTO-3 you should be using the SIZE_DATA_BUFFERS as shown above, for the number of DATA_BUFFERS, anything above 32 is usually overkill/makes no differnece in performance. Justin. On Wed, 23 Jul 2008, NBU wrote: Friends, Fine tunning has been done in /etc/system also but no luck. Following is systems setting: We have set Net_BUFFER SIZE and /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS = 131072 /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS= 3072 Following is the kernel parameter of Solaris 10 OS [EMAIL PROTECTED] # prtctl $$ -bash: prtctl: command not found [EMAIL PROTECTED] # prctl $$ process: 15640: -bash NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT process.max-port-events privileged 65.5K - deny - system 2.15G max deny - process.max-msg-messages privileged 8.19K - deny - system 4.29G max deny - process.max-msg-qbytes privileged 64.0KB - deny - system 16.0EB max deny - process.max-sem-ops privileged 512 - deny - system 2.15G max deny - process.max-sem-nsems privileged 512 - deny - system 32.8K max deny - process.max-address-space privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-descriptor basic 256 - deny 15640 privileged 65.5K - deny - system 2.15G max deny - process.max-core-size privileged 8.00EB max deny - system 8.00EB max deny - process.max-stack-size basic 8.00MB - deny 15640 privileged 8.00EB - deny - system 8.00EB max deny - process.max-data-size privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-size privileged 8.00EB max deny,signal=XFSZ - system 8.00EB max deny - process.max-cpu-time privileged 18.4Es inf signal=XCPU - system 18.4Es inf none - task.max-cpu-time system 18.4Es inf none - task.max-lwps system 2.15G max deny - project.max-contracts privileged 10.0K - deny - system 2.15G max deny - project.max-device-locked-memory privileged 3.92GB - deny - system 16.0EB max deny - project.max-locked-memory system 16.0EB max deny - project.max-port-ids privileged 8.19K - deny - system 65.5K max deny - project.max-shm-memory privileged 48.0GB - deny - system 16.0EB max deny - project.max-shm-ids privileged 512 - deny - system 16.8M max deny - project.max-msg-ids privileged 256 - deny - system 16.8M max deny - project.max-sem-ids privileged 512 - deny - system 16.8M max deny - project.max-crypto-memory privileged 15.7GB - deny - system 16.0EB max deny - project.max-tasks system 2.15G max deny - project.max-lwps system 2.15G max deny - project.cpu-shares privileged 1 - none - system 65.5K max none - zone.max-swap system 16.0EB max deny - zone.max-locked-memory system 16.0EB max deny - zone.max-shm-memory system 16.0EB max deny - zone.max-shm-ids system 16.8M max deny - zone.max-sem-ids system 16.8M max deny - zone.max-msg-ids system 16.8M max deny - zone.max-lwps system 2.15G max deny - zone.cpu-shares privileged 1 - none - system 65.5K max none - +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Shared memory issue with Solaris 10
I will throw this one out for everyone. In test after test what we found was 65536 size and 128 num buffers worked best. We are a mixed Win / RedHat shop (with other stuff thrown in) . Win2k3 master and 2 media servers to an i2000 with 10 lto4 drives. We make sure the size is set on all the clients as well. When I say test I mean we used the settings noted above with the settings mentioned below on regular full backups. Our settings ran about 15-20% faster than the typical 256/32 or 131/32. David Spearman County of Henrico From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jon Bousselot Sent: Wednesday, July 23, 2008 11:04 AM To: Justin Piszcz; NBU Cc: VERITAS-BU@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Shared memory issue with Solaris 10 I'll concur with Justin. 262144 SIZE is a good performer, and 32 NUMBER is a sweet spot for LTO-4. If you have the ram and lots of inbound connectivity, you can go to 64 buffers, and it seems to work well. With 131072 SIZE and 3072 NUMBER, if you have 1 drive and multiplexing set to 1, that tries to allocate a 400MB segment of memory. Add in number of drives and multiplexing, I'm not surprised it cannot allocate memory. The formula from the docs is (SIZE_DATA_BUFFERS * NUMBER_DATA_BUFFERS * Num_Drives * MPX_Factor) -Jon - Original Message From: Justin Piszcz [EMAIL PROTECTED] To: NBU [EMAIL PROTECTED] Cc: VERITAS-BU@mailman.eng.auburn.edu Sent: Wednesday, July 23, 2008 9:11:06 AM Subject: Re: [Veritas-bu] Shared memory issue with Solaris 10 There is your problem right there. $ cat NUMBER_DATA_BUFFERS 32 $ cat SIZE_DATA_BUFFERS 262144 For LTO-2 and LTO-3 you should be using the SIZE_DATA_BUFFERS as shown above, for the number of DATA_BUFFERS, anything above 32 is usually overkill/makes no differnece in performance. Justin. On Wed, 23 Jul 2008, NBU wrote: Friends, Fine tunning has been done in /etc/system also but no luck. Following is systems setting: We have set Net_BUFFER SIZE and /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS = 131072 /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS= 3072 Following is the kernel parameter of Solaris 10 OS [EMAIL PROTECTED] # prtctl $$ -bash: prtctl: command not found [EMAIL PROTECTED] # prctl $$ process: 15640: -bash NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT process.max-port-events privileged 65.5K - deny - system 2.15G max deny - process.max-msg-messages privileged 8.19K - deny - system 4.29G max deny - process.max-msg-qbytes privileged 64.0KB - deny - system 16.0EB max deny - process.max-sem-ops privileged 512 - deny - system 2.15G max deny - process.max-sem-nsems privileged 512 - deny - system 32.8K max deny - process.max-address-space privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-descriptor basic 256 - deny 15640 privileged 65.5K - deny - system 2.15G max deny - process.max-core-size privileged 8.00EB max deny - system 8.00EB max deny - process.max-stack-size basic 8.00MB - deny 15640 privileged 8.00EB - deny - system 8.00EB max deny - process.max-data-size privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-size privileged 8.00EB max deny,signal=XFSZ - system 8.00EB max deny - process.max-cpu-time privileged 18.4Es inf signal=XCPU - system 18.4Es inf none - task.max-cpu-time system 18.4Es inf none - task.max-lwps system 2.15G max deny - project.max-contracts privileged 10.0K - deny - system 2.15G max deny - project.max-device-locked-memory privileged 3.92GB - deny - system 16.0EB max deny - project.max-locked-memory system 16.0EB max deny - project.max-port-ids privileged 8.19K - deny - system 65.5K max deny - project.max-shm-memory privileged 48.0GB - deny - system 16.0EB max deny - project.max-shm-ids privileged 512 - deny - system 16.8M max deny - project.max-msg-ids privileged 256 - deny - system 16.8M max deny - project.max-sem-ids privileged 512 - deny - system 16.8M max deny - project.max-crypto-memory privileged 15.7GB - deny - system 16.0EB max deny - project.max-tasks system 2.15G max deny - project.max-lwps system 2.15G max deny - project.cpu-shares privileged 1 - none - system 65.5K max none - zone.max-swap system 16.0EB max deny - zone.max-locked-memory system 16.0EB max deny - zone.max-shm-memory system 16.0EB max deny - zone.max-shm-ids system 16.8M max deny - zone.max-sem-ids system 16.8M max deny - zone.max-msg-ids system 16.8M max deny - zone.max-lwps system 2.15G max deny - zone.cpu-shares privileged 1 - none - system 65.5K max none - +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http
Re: [Veritas-bu] Shared memory issue with Solaris 10
Hm I will add that to my notes the next time I do some benchmarking, thanks. Justin. On Wed, 23 Jul 2008, Spearman, David wrote: I will throw this one out for everyone. In test after test what we found was 65536 size and 128 num buffers worked best. We are a mixed Win / RedHat shop (with other stuff thrown in) . Win2k3 master and 2 media servers to an i2000 with 10 lto4 drives. We make sure the size is set on all the clients as well. When I say test I mean we used the settings noted above with the settings mentioned below on regular full backups. Our settings ran about 15-20% faster than the typical 256/32 or 131/32. David Spearman County of Henrico From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jon Bousselot Sent: Wednesday, July 23, 2008 11:04 AM To: Justin Piszcz; NBU Cc: VERITAS-BU@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Shared memory issue with Solaris 10 I'll concur with Justin. 262144 SIZE is a good performer, and 32 NUMBER is a sweet spot for LTO-4. If you have the ram and lots of inbound connectivity, you can go to 64 buffers, and it seems to work well. With 131072 SIZE and 3072 NUMBER, if you have 1 drive and multiplexing set to 1, that tries to allocate a 400MB segment of memory. Add in number of drives and multiplexing, I'm not surprised it cannot allocate memory. The formula from the docs is (SIZE_DATA_BUFFERS * NUMBER_DATA_BUFFERS * Num_Drives * MPX_Factor) -Jon - Original Message From: Justin Piszcz [EMAIL PROTECTED] To: NBU [EMAIL PROTECTED] Cc: VERITAS-BU@mailman.eng.auburn.edu Sent: Wednesday, July 23, 2008 9:11:06 AM Subject: Re: [Veritas-bu] Shared memory issue with Solaris 10 There is your problem right there. $ cat NUMBER_DATA_BUFFERS 32 $ cat SIZE_DATA_BUFFERS 262144 For LTO-2 and LTO-3 you should be using the SIZE_DATA_BUFFERS as shown above, for the number of DATA_BUFFERS, anything above 32 is usually overkill/makes no differnece in performance. Justin. On Wed, 23 Jul 2008, NBU wrote: Friends, Fine tunning has been done in /etc/system also but no luck. Following is systems setting: We have set Net_BUFFER SIZE and /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS = 131072 /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS= 3072 Following is the kernel parameter of Solaris 10 OS [EMAIL PROTECTED] # prtctl $$ -bash: prtctl: command not found [EMAIL PROTECTED] # prctl $$ process: 15640: -bash NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT process.max-port-events privileged 65.5K - deny - system 2.15G max deny - process.max-msg-messages privileged 8.19K - deny - system 4.29G max deny - process.max-msg-qbytes privileged 64.0KB - deny - system 16.0EB max deny - process.max-sem-ops privileged 512 - deny - system 2.15G max deny - process.max-sem-nsems privileged 512 - deny - system 32.8K max deny - process.max-address-space privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-descriptor basic 256 - deny 15640 privileged 65.5K - deny - system 2.15G max deny - process.max-core-size privileged 8.00EB max deny - system 8.00EB max deny - process.max-stack-size basic 8.00MB - deny 15640 privileged 8.00EB - deny - system 8.00EB max deny - process.max-data-size privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-size privileged 8.00EB max deny,signal=XFSZ - system 8.00EB max deny - process.max-cpu-time privileged 18.4Es inf signal=XCPU - system 18.4Es inf none - task.max-cpu-time system 18.4Es inf none - task.max-lwps system 2.15G max deny - project.max-contracts privileged 10.0K - deny - system 2.15G max deny - project.max-device-locked-memory privileged 3.92GB - deny - system 16.0EB max deny - project.max-locked-memory system 16.0EB max deny - project.max-port-ids privileged 8.19K - deny - system 65.5K max deny - project.max-shm-memory privileged 48.0GB - deny - system 16.0EB max deny - project.max-shm-ids privileged 512 - deny - system 16.8M max deny - project.max-msg-ids privileged 256 - deny - system 16.8M max deny - project.max-sem-ids privileged 512 - deny - system 16.8M max deny - project.max-crypto-memory privileged 15.7GB - deny - system 16.0EB max deny - project.max-tasks system 2.15G max deny - project.max-lwps system 2.15G max deny - project.cpu-shares privileged 1 - none - system 65.5K max none - zone.max-swap system 16.0EB max deny - zone.max-locked-memory system 16.0EB max deny - zone.max-shm-memory system 16.0EB max deny - zone.max-shm-ids system 16.8M max deny - zone.max-sem-ids system 16.8M max deny - zone.max-msg-ids system 16.8M max deny - zone.max-lwps system 2.15G max deny - zone.cpu-shares privileged 1 - none - system 65.5K max none - +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED
Re: [Veritas-bu] Shared memory issue with Solaris 10
We have Solaris10 master media servers here without any changes to /etc/system but I'm also interested in what others are doing. /Steve --- On Wed, 23 Jul 2008, Dave Markham wrote: I'd be interested in this too, as i'd heard you didnt need to bother with /etc/system tuning in Solaris 10. Cheers Justin Piszcz wrote: On Wed, 23 Jul 2008, NBU wrote: Hi forum, I have a Master and 3 media server on solaris 9 having Netbackup 6.0 MP4. Recently added new media server with solaris 10. Problem which i am facing is when the load increases (Schedule starts) backup start failing with error code 89. Experts need your help on this. thanks +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu Error 89 is shared memory problems, read the archive on how to fix that for Solaris 10, generally you need to increase your SHMMAX value in the kernel (Linux) in Solaris it has been awhile :) Justin. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Shared memory issue with Solaris 10
Sorry, I didn't informed that i am having LTO3 drives. Other media server with Solaris 9 is having same settings which is giving a better output. +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Shared memory issue with Solaris 10
Steve, You are right. When you read the latest Symantec and SUN doco on this it tells you that all previous /etc/system recommendations are now obsolete in Solaris 10. I believe I have posted this before but to recap, Sol 10 boots with a 1/4 of physical memory assigned to shared memory. All previous settings that I've seen do not come close to this on new systems. We are now talking GBs instead of MBs of old. Therefore no one should need to touch this. But here is a caveat, I have noticed that some systems running Sol 10 can experience shared memory like symptoms under heavy load. But it is not the cause, it just looks like it and the following has solved the issue, but alas, it's an /etc/system entry that SUN have tried to eliminate but is still used in Sol 10. shmsys:shminfo_shmmni=512 Should be 512 or 1024, depends on how big the system is. I recommend reading the SUN doco if you need to play around with it. But it may fix the issue if you see it. Just like below: the load increases (Schedule starts) backup start failing with error code 89 regards, Dominik Steve Quan [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 24/07/2008 01:20 AM To Dave Markham [EMAIL PROTECTED] cc VERITAS-BU@mailman.eng.auburn.edu Subject Re: [Veritas-bu] Shared memory issue with Solaris 10 We have Solaris10 master media servers here without any changes to /etc/system but I'm also interested in what others are doing. /Steve --- On Wed, 23 Jul 2008, Dave Markham wrote: I'd be interested in this too, as i'd heard you didnt need to bother with /etc/system tuning in Solaris 10. Cheers Justin Piszcz wrote: On Wed, 23 Jul 2008, NBU wrote: Hi forum, I have a Master and 3 media server on solaris 9 having Netbackup 6.0 MP4. Recently added new media server with solaris 10. Problem which i am facing is when the load increases (Schedule starts) backup start failing with error code 89. Experts need your help on this. thanks +-- |This was sent by [EMAIL PROTECTED] via Backup Central. |Forward SPAM to [EMAIL PROTECTED] +-- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu Error 89 is shared memory problems, read the archive on how to fix that for Solaris 10, generally you need to increase your SHMMAX value in the kernel (Linux) in Solaris it has been awhile :) Justin. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu National Australia Bank Ltd - ABN 12 004 044 937 This email may contain confidential information. If you are not the intended recipient, please immediately notify us at [EMAIL PROTECTED] or by replying to the sender, and then destroy all copies of this email. Except where this email indicates otherwise, views expressed in this email are those of the sender and not of National Australia Bank Ltd. Advice in this email does not take account of your objectives, financial situation, or needs. It is important for you to consider these matters and, if the e-mail refers to a product(s), you should read the relevant Product Disclosure Statement(s)/other disclosure document(s) before making any decisions. If you do not want email marketing from us in future, forward this email with unsubscribe in the subject line to [EMAIL PROTECTED] in order to stop marketing emails from this sender. National Australia Bank Ltd does not represent that this email is free of errors, viruses or interference. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
