Hi All, Thanks for all the information and suggestions it's given me a lot to look at. Hopefully I've covered all your questions below...
> I've checked and there's no evidence I can find that anyone has made any changes against the arrays, either via the command line or vea - the latter which we don't use. However, looking at the tuneable values and what they're set to and what the defaults are there are two that are different to the defaults... fuj411:/root# vxdmpadm gettune all Tunable Current Value Default Value ------------------------------- ------------- ------------- dmp_failed_io_threshold 57600 57600 dmp_retry_count 5 5 dmp_pathswitch_blks_shift 11 11 dmp_queue_depth 32 32 dmp_cache_open off off dmp_daemon_count 10 10 dmp_scsi_timeout 30 30 dmp_delayq_interval 15 15 dmp_path_age 0 300 dmp_stat_interval 1 1 dmp_health_time 0 60 dmp_probe_idle_lun on on dmp_log_level 1 1 dmp_retry_timeout 0 0 fuj411:/root# I don't know who would've changed these or when, or at all. There's a lot of disk seen down each path - physical HBA. We have an A and a B side to the SAN with the disk being presented down both sides to separate HBA's on the server. The errors and other messages are always to the same array (IBM_SHARK1) which is the one that has the different value set, and is the one that see's the throttling against it. > The version of VxVM is 4.1 MP2 so I know we are downlevel and doesn't have the new default value of 512 that's come in with 5.1 MP3. > All paths are active and i/o is seen going down both paths to all disks. I can see this at the server and the SAN port level. The storage team have checked and the cannot find anything untoward. I've also had a look at the HBA's as well (fcinfo hba-port -l <HBA_WWN>) > The recovery option on both arrays is the same and set to the following... fuj411:/root# vxdmpadm getattr enclosure IBM_SHARK0 recoveryoption ENCLR-NAME RECOVERY-OPTION DEFAULT[VAL] CURRENT[VAL] ============================================================================== IBM_SHARK0 Throttle Timebound[10] Timebound [10] IBM_SHARK0 Error-Retry Fixed-Retry[5] Fixed-Retry [5] fuj411:/root# fuj411:/root# vxdmpadm getattr enclosure IBM_SHARK1 recoveryoption ENCLR-NAME RECOVERY-OPTION DEFAULT[VAL] CURRENT[VAL] ============================================================================== IBM_SHARK1 Throttle Timebound[10] Timebound [10] IBM_SHARK1 Error-Retry Fixed-Retry[5] Fixed-Retry [5] fuj411:/root# My suspicion is that we have some kind of fault it's just identifying where. I suspect this will involve getting the server up to the latest patch levels in the o/s and within the version installed for starters and getting the storage team to carry out a full end to end test of the hardware. In the meantime thanks again for all your help with this, very much appreciated. I think I will take Dmitry's advice and log a call with Symantec and see if they can explain the difference and whether it's just the way this array type operates. Thanks again. Cheers Phil. Dmitry Glushenok <gl...@jet.msk.su > To William Havey <bbha...@gmail.com> 26/07/2012 21:09 phil.cole...@ba.com cc veritas-vx@mailman.eng.auburn.edu Subject Re: [Veritas-vx] Question over DMP partitionsize Hello William, Phil, 26.07.2012, в 21:25, William Havey написал(а): Phil, I don't believe any DMP values are changed by vxconfigd automatically. To be certain the partitionsize has not been manually changed, Is VxVM command line logging enabled? run vxcmdlog -l to find out If yes, look in /var/adm/vx/cmdlog for any vxdmpadm setattr commands. Phil using VCS, so cmdlog gets rewritten very fast. If /etc/vx/dmppolicy.info contains nothing about partition size and other VCS nodes have same partition size differences - it looks like ASL for SHARK (or vxdmp itself) is making the changes. And only way to verify this is contact symantec :) Do you use vea? If so, its commands are logged in /var/adm/vx/veacmdlog. But, I don't believe these values can beset in vea. I believe the "throttle" and "unthrottle" in the dmpevents log file means holding back I/O from being addressed and sent out to storage but I certainly can't say the two (partitionsize of 512 and log entries for the SHARK1 with that size) are related or not. Path can be throttled if it holds SCSI command for more than specified amount of time or if queue depth for the path reaches specified limit. It can be checked with "vxdmpadm getattr enclosure <enc_name> recoveryoption" and verified with iostat: asvc_t and actv/wait columns (not sure which queue vxdmp counts, active or wait). Are all paths being used for I/O just about the same? vxdmpadm iostat start vxdmpadm iostat reset vxdmpadm iostat show enclosure=IBM_SHARK0 enclosure=IBM_SHARK1 interval=5 The first iteration of the utility shows cumulative statistics from the time of mounting the file systems. So, ignore the first output. If totals are about equal, then the value the partitionsize value may be irrelevant. Perhaps I/O is truly random so that no I/O address is within 512 sectors of the previous I/O. Even in a random environment sometimes numerous successive I/O's can be within 512 sectors of each other so that throttling those I/Os kicks-in. IHTH, Bill On Thu, Jul 26, 2012 at 11:56 AM, <phil.cole...@ba.com> wrote: Hi William, Thanks for the reply. Not sure how to get this track cach size? What's confusing me most here is that the values are so different between the two arrays. They're identical models set-up up the same and with the same number of disks allocated to the servers - sorry, forgot to mention they're in an HA pair using VCS. The only difference is that IBM_SHARK0 is local to the server where the workload is currently running, and IBM_SHARK1 is in another building about 1.5KM's away. It's this disparity that is confusing me and making me wonder whether an issue we are seeing is being caused by this, or if it's an indication of an issue, though I've been informed by our storage people that there's nothing wrong with either array. This is certainly not something we've changed so I don't know if VxVM/DMP is throttling things back because it's seeing an issue. I have seen messages in the /etc/vx/dmpevents.log file for disks in the IBM_SHARK1 array reporting 'Throttled Path' and then 'Un-throttled Path' and I'm trying to work out if the two are linked. Cheers Phil. William Havey <bbha...@gmail.co m> To phil.cole...@ba.com 26/07/2012 15:48 cc Subject Re: [Veritas-vx] Question over DMP partitionsize Partitionsize is in play when the iopolicy is Balanced, which is the default policy. You have Balanced. It is defined as "The partitionsize attribute: Each successive I/O starting within in this range (default is 2048 sectors) goes through the same path as the previous I/O" The man page has "Takes the track cache into consideration when balancing I/O across paths" What is the track cache size on each array? Seems that the partitionsize should be the same value as the track cache size. On Thu, Jul 26, 2012 at 7:09 AM, <phil.cole...@ba.com> wrote: Hi, I'm trying to understand why one array has a wildly different partitionsize value to the other... fuj411:/root# vxdmpadm getattr arrayname IBM_SHARK partitionsize ENCLR_NAME DEFAULT CURRENT ============================================ IBM_SHARK0 2048 2048 IBM_SHARK1 256 512 fuj411:/root# Both arrays are IBM ESS SHARK 2105's which are active/active and operating in a Balanced i/o policy... fuj411:/root# vxdmpadm getattr arrayname IBM_SHARK iopolicy ENCLR_NAME DEFAULT CURRENT ============================================ IBM_SHARK0 Balanced Balanced IBM_SHARK1 Balanced Balanced fuj411:/root# The server is a Fujitsu PW850 running Solaris 10 with VxVM v4.1 MP2. The volume are mirrored between the two arrays. I'm trying to get a better understanding of what this means and why it would be so different between the two arrays. Any help greatly appreciated. Cheers Phil. -- This message is private and confidential and may also be legally privileged. If you have received this message in error, please email it back to the sender and immediately permanently delete it from your computer system. Please do not read, print, re-transmit, store or act in reliance on it or any attachments. British Airways may monitor email traffic data and also the content of emails, where permitted by law, for the purposes of security and staff training and in order to prevent or detect unauthorised use of the British Airways email system. Virus checking of emails (including attachments) is the responsibility of the recipient. British Airways Plc is a public limited company registered in England and Wales. Registered number: 1777777. Registered office: Waterside, PO Box 365, Harmondsworth, West Drayton, Middlesex, England, UB7 0GB. Additional terms and conditions are available on our website: www.ba.com _______________________________________________ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx -- This message is private and confidential and may also be legally privileged. If you have received this message in error, please email it back to the sender and immediately permanently delete it from your computer system. Please do not read, print, re-transmit, store or act in reliance on it or any attachments. British Airways may monitor email traffic data and also the content of emails, where permitted by law, for the purposes of security and staff training and in order to prevent or detect unauthorised use of the British Airways email system. Virus checking of emails (including attachments) is the responsibility of the recipient. British Airways Plc is a public limited company registered in England and Wales. Registered number: 1777777. Registered office: Waterside, PO Box 365, Harmondsworth, West Drayton, Middlesex, England, UB7 0GB. Additional terms and conditions are available on our website: www.ba.com _______________________________________________ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx -- Dmitry Glushenok Jet Infosystems ------------------------------------------------------------------------------------------------------------ Get the best from British Airways at ba.com http://www.ba.com -- This message is private and confidential and may also be legally privileged. If you have received this message in error, please email it back to the sender and immediately permanently delete it from your computer system. Please do not read, print, re-transmit, store or act in reliance on it or any attachments. British Airways may monitor email traffic data and also the content of emails, where permitted by law, for the purposes of security and staff training and in order to prevent or detect unauthorised use of the British Airways email system. Virus checking of emails (including attachments) is the responsibility of the recipient. British Airways Plc is a public limited company registered in England and Wales. Registered number: 1777777. Registered office: Waterside, PO Box 365, Harmondsworth, West Drayton, Middlesex, England, UB7 0GB. Additional terms and conditions are available on our website: www.ba.com _______________________________________________ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx