Sorry for the late response. I missed this reply.
For Apache settings the worker and prefork configurations are the exact 
same between the two vms:

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild 100
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>


*Symptoms from our old production vm:*
Previously, only about 2-3 times a day at random times, we would get a 
build of Apache processes that would hit the server at the same time, which 
results in the load average on top going up to 100-200 in the worse case 
scenario.  During this time, any operations done in the website are 
extremely slow and often times users will report not receiving an email 
after a publish.  Since then, we've increased the sendmail Queue and Refuse 
limits from their default values of (12 and 15) to (20 and 150) 
respectively.
The stats on this vm were:
- 10 GB RAM
- 23 GB Swap
- 4 Cores
- RHEL 5.3
- Red Hat Enterprise Linux Server release 5.3 (Tikanga)
- Server version: Apache/2.2.3

*Symptoms on our new production vm:*
We moved the production to RHEL6.4 as recommended by our IT team and have 
since been noticing that we we get these stalled processes more often and 
users tend to notice the performance hits much more.  Another odd thing 
that we noticed from our performance monitoring tool Zenoss is that the IO 
spikes on writes every 5 minutes, which was on the case previously 
(screenshots attached).

The stats on this vm were:
- 16 GB RAM
- 4 GB Swap
- 4 Cores
- Red Hat Enterprise Linux Server release 6.4 (Santiago)
- Server version: Apache/2.2.15 (Unix)

We're trying a lot of different things on our end, but if you have any 
ideas or if anyone has seen this issue, it would help.

<https://lh4.googleusercontent.com/-L5GVIinnSbo/UxpXxkdzJDI/AAAAAAAACkg/Aq9DYWW6AXs/s1600/Screen+Shot+2014-03-07+at+3.35.13+PM.png>

<https://lh6.googleusercontent.com/-7FD3HifikjY/UxpXjy3QUwI/AAAAAAAACkY/BFO-tGDUwoE/s1600/Screen+Shot+2014-03-07+at+3.33.09+PM.png>
  
  


Ze

On Thursday, March 6, 2014 8:21:00 PM UTC-8, Christian Hammond wrote:
>
> Okay, well, I was hoping it'd be simple :)
>
> Can you give me some examples of operations that are very slow, and 
> operations that remain fast? Or does everything basically slow to a grind?
>
> How do the Apache settings (worker vs prefork, and their config) compare 
> between installs?
>
> Christian
>
>
> On Thursday, March 6, 2014, Ze Xiao <ilackno...@gmail.com <javascript:>> 
> wrote:
>
>> Thanks for the quick reply.  Yes, memcached is running.  Here is what I 
>> see from the Admin Server Cache page
>>
>> I've got it running on two different vms, which I've obfuscated as "VM1" 
>> and "VM2"
>>
>> SERVER CACHE
>>  Cache backend:
>>
>> django.core.cache.backends.memcached.CacheClass
>>  vm1
>> Memory usage:
>>
>> 1.8 GB
>> Keys in cache:
>>
>> 61079 of 257077
>> Cache hits:
>>
>> 5289571 of 5458860: 96%
>> Cache misses:
>>
>> 169289 of 5458860: 3%
>> Cache evictions:
>>
>> 139881
>> Cache traffic:
>>
>> 10.2 GB in, 27.9 GB out
>> Uptime:
>>
>> 3683047 seconds
>> vm2
>> Memory usage:
>>
>> 1.8 GB
>> Keys in cache:
>>
>> 54978 of 401980
>> Cache hits:
>>
>> 5999634 of 6277198: 95%
>> Cache misses:
>>
>> 277564 of 6277198: 4%
>> Cache evictions:
>>
>> 307751
>> Cache traffic:
>>
>> 16.8 GB in, 26.2 GB out
>> Uptime:
>>
>> 938019 seconds
>>
>>
>> On Thu, Mar 6, 2014 at 5:20 PM, Christian Hammond <chip...@chipx86.com>wrote:
>>
>> Hi Ze,
>>
>> Those warnings are probably unrelated.
>>
>> I want to get a better sense of the performance problems. First thing I 
>> want to check is that your server is properly accessing and using 
>> memcached. If you log into the admin UI, do you see any stats on memcached, 
>> and any keys stored in the cache?
>>
>> Christian
>>
>> -- 
>> Christian Hammond - chip...@chipx86.com
>> Review Board - http://www.reviewboard.org
>> Beanbag, Inc. - http://www.beanbaginc.com
>>
>>
>> On Thu, Mar 6, 2014 at 4:52 PM, Ze Lin Xiao <ilacknormal...@gmail.com>wrote:
>>
>> Hi Christian,
>>
>> We're facing some pretty bad performance issues on our production system 
>> after we moved our application to a different vm with RHEL6.4.
>>
>> We notice that our performance issues occur especially when the log shows 
>> this:
>> [Fri Mar 07 00:18:19 2014] [error] 
>> /opt/software/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg/Crypto/Util/number.py:57:
>>  
>> PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using 
>> libgmp >= 5 to avoid timing attack vulnerability.
>>
>> However, it is important to note that we've seen these warning issues for 
>> the last 1.5 years, so I doubt it has to do with it.  Nonetheless, do you 
>> know what specific operations one could do to trigger this warning?  I'm 
>> trying to see if I can reproduce the performance spikes.
>>
>> Thanks,
>> Ze
>>
>> On Wednesday, February 6, 2013 12:22:49 AM UTC-8, Christian Hammond wrote:
>>
>> Hi Chuck,
>>
>> Sorry for failing to respond to the previous e-mail. Missed it.
>>
>> I haven't seen that particular warning before. It'll probably have a log 
>> entry any time pycrypto is imported. What distro/version are you using? 
>> Sounds like maybe it's an older one? You may need to hand-upgrade libgmp, 
>> I'm not sure.
>>
>> From your previous e-mail:
>>
>> Doing a site backup never hurts, but generally isn't important.
>>
>> Review Board won't delete any files. At most, it'd add some new 
>> directories and tell you to change permissions, but I don't think we've 
>> done that since 1.5. We have provided instructions on other sorts of manual 
>> updates that need to be made, though.
>>
>> We don't have any documentation right now on p4python's SSL support. This 
>> is only needed if you're using SSL-backed Perforce repositories. It's 
>> unfortunately not something we can automate well right now, but 
>> essentially, you'd have to install OpenSSL 1.0.1 on your distro and install 
>> its development package (I don't know if newer versions work -- hopefully 
>> other 1.0.x releases do). You'd then need to manually compile/install 
>> p4python. Yes, it's a pain, but it's something Perforce will need to make 
>> easier for us.
>>
>> From the e-mail you just posted while I was replying to this, you'd need 
>> to check the reviewboard.log file and see what error it's reporting before 
>> I can say what happened.
>>
>> Christian
>>
>> -- 
>> Christian Hammond - chi...@chipx86.com
>>
>> Review Board - http://www.reviewboard.org
>> VMware, Inc. - http://www.vmware.com 
>>
>> On Feb 6, 2013, at 12:10 AM, chuck j <cjerr...@gmail.com> wrote:
>>
>> Hi Christian,
>>
>> I would like to thank you for your response about upgrade.
>>
>> I went through with your comments and i was able to bring my server to 
>> 1.7.4.
>>
>> Also also want to bring to your notice regarding below warning i got 
>> after while upgrading my site.
>>
>> /usr/local/lib/python2.7/site-
>>
>> -- 
>> Ze Lin Xiao
>>  
>> -- 
>> Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
>> ---
>> Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
>> ---
>> Happy user? Let us know at http://www.reviewboard.org/users/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "reviewboard" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to reviewboard+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> -- 
> -- 
> Christian Hammond - chi...@chipx86.com <javascript:>
> Review Board - http://www.reviewboard.org
> Beanbag, Inc. - http://www.beanbaginc.com
>
>

-- 
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to