Glav,
· Actually staging is 32bit and production 64bit. I just swapped production to 32bit to see if that will make a difference. So far the issue still exists though even with the switch, hopefully there is a slight performance improvement in general. · Ive looked at request execution time and its a bit bizarre. Most of the time it is 15ms on a request that takes longer than it should, but sometimes it suddenly jumps up to over 10 seconds for a rather simple page. Monitoring the firewall shows query execution time between the web server and the database server is done in less than 2 seconds (firewall doesnt measure less than 1 second and includes TCP socket open and close). · The firewall shows that the TCP socket is getting reset on the times when these issues occur, meaning that there is no TCP end frame. Yet those times I can clearly see that from the IIS logs that request execution time is less than a seconds to return the page. Why would IIS not return/close the socket? · Havent noticed any app restarts in the event logs, but Ill put this under a 48 hour performance log. · No havent noticed any memory leaks as yet. Although Ill turn the GC counter on in the performance log as well. Thanks ----- Michael From: [email protected] [mailto:[email protected]] On Behalf Of Paul Glavich Sent: Thursday, 26 May 2011 8:37 AM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts A few things you can look at are: · Is staging a 32 bit environment or is everything all 64bit? If memory requirements are not large you can look at running IIS in 32bit mode in the VMs which can yield good CPU utilisation benefits. Generally, VMs suck IMHO compared to physical. I have personally used 32bit mode IIS within a 64bit VM and seen very good CPU improvement. · Do you use stored procedures and if so, do many of the stored procedures have conditional logic in there that cause different SQL statements to be executed based in differing input parameters (not talking about constructing SQL strings and using sp_executesql tho)? This kind of thing can cause SQL to use incorrect query plans for a proc and cause really long execution times of that proc until it is recompiled/query plan removed from cache. · Look at the Asp.NET request execution time counter to see if ASP.NET itself is taking a long time to process any requests. If not, then you know its outside of the ASP.NET app space. If yes, it could be doing either a long bit of processing or waiting on something like a SQL query to execute · Also look at Asp.NET Application Restart and worker process restarts. App restarts may be causing app recompilation and thus making requests take longer. · I assume you have looked at the memory counter to ensure no memory leaks are present (ever increasing memory usage). If you suspect this, you may also want to look at the GC counts in the Asp.Net Memory counters to ensure an excessive amount of garbage collection is not holding up processing (although u would see this on staging as well). - Glav From: [email protected] [mailto:[email protected]] On Behalf Of Michael Lyons Sent: Tuesday, 24 May 2011 2:29 PM To: 'ozDotNet' Subject: Website request slow performance / timeouts Ive been working on an ASP.Net solution which has a slow performance issues and it has got me baffled. Problem: The production server randomly slows down when serving asp.net requests and even times out. System architecture: The solution is hosted on a dedicated box which is running VmWares ESXi with 4 VM servers sitting on it (1 per core). Each VM is on its own network. All network communication is done through a dedicated hardware firewall, even between VMs (unfortunately the auditor has to have it this way). Database is on 1 VM while another has the web server. ASP.Net is v4 running on IIS 7.5 while database is SQL Server 2008R2 all on top of Windows Server 2008 R2 Analysis to date: Ive run a profiler over the solution and so far come up with nothing that really needs to be optimised. Our staging environment is running the same way as our production system architecture minus the hardware firewall and has a lot lower hardware specs but performs better than the production environment. When Im talking slower, Im talking ¼ of the memory and a 7 year old CPU. Production IIS logs show some randomly high request execution times. Theories to date: ESXi is doing something weird and causing VMs to run slow. Firewall is blocking requests randomly or is having performance issues, although I dont see it. IIS is randomly running slow. Sql Server is randomly running slow My questions: What would Windows performance counters would you watch? Besides the typical CPU, Disk, memory and ASP.Net 4.0 counters? Does the IIS logs request execution times include the time to send the network data? Eg. From time of socket open to time of socket closed? Or is it just the pipeline without the TCP time included eg. Serving a straight html file would just really be time to read the file from disk. What else would you look at? ------ Michael Lyons
