Ok now just type the following from the wp docroot to find your culprit:
egrep -Rn 'sm_type|is_unique|connect_hash|reqType'
wp-content/{plugins,themes}On Tue, Aug 2, 2016 at 10:58 AM, Ayberk Kimsesiz <[email protected]> wrote: > Hi Andrei, > > Here are the results: > > T *****:56538 -> ******:8080 [AP] > POST /wp-admin/admin-ajax.php HTTP/1.1. > Host: *******.com. > Accept-Language: tr-tr. > User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 10_0 like Mac OS X) > AppleWebKit/6 02.1.40 (KHTML, like Gecko) Version/10.0 Mobile/14A5309d > Safari/602.1. > X-Requested-With: XMLHttpRequest. > Accept: application/json, text/javascript, */*; q=0.01. > Referer: http://******/. > Content-Type: application/x-www-form-urlencoded; charset=UTF-8. > Content-Length: 130. > Origin: http://****.com. > X-Actual-IP: 5.46.70.115. > Cookie: > pps_show_100=Tue%20Aug%2002%202016%2018%3A50%3A21%20GMT+0300%20%28EEST%2 > 9; > pps_times_showed_100=1. > X-Forwarded-For: 5.46.70.115, 5.46.70.115, 5.46.70.115. > Accept-Encoding: gzip. > X-Varnish: 39754073. > . > mod=statistics&action=add&id=100&type=show&sm_type=0&is_unique=1&connect_hash=c1 > > 532a201e2ee25540c61d3199e0e960&pl=pps&reqType=ajax > > > > T *****:56652 -> ******:8080 [AP] > POST /wp-admin/admin-ajax.php HTTP/1.1. > Host: *****.com. > Content-Length: 130. > Origin: http://*****.com. > X-Requested-With: XMLHttpRequest. > User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:46.0) > Gecko/20100101 Firefox/46.19. > Content-Type: application/x-www-form-urlencoded; charset=UTF-8. > Accept: application/json, text/javascript, */*; q=0.01. > Referer: https://www.yandex.com.tr/. > Accept-Language: en-us,en. > Accept-Charset: iso-8859-1,*,utf-8. > X-Actual-IP: 78.173.206.179. > Cookie: PHPSESSID=br4avjrtmgd0e2j04eoelev4r6; > pps_show_100=Tue%20Aug%2002%202016%2018%3A51%3A14%20GMT+0300%20%28Turkey%20Daylight%20 > Time%29; pps_times_showed_100=2. > X-Forwarded-For: 78.173.206.179, 78.173.206.179, 78.173.206.179. > Accept-Encoding: gzip. > X-Varnish: 37456291. > . > > mod=statistics&action=add&id=100&type=show&sm_type=0&is_unique=0&connect_hash=c1532a201e2ee25540c61d3199e0e960&pl=pps&reqType=ajax > > > T *****:56630 -> *****.10:8080 [AP] > POST /wp-admin/admin-ajax.php HTTP/1.1. > Host: *****.com. > Content-Length: 130. > Origin: http://*****.com. > X-Requested-With: XMLHttpRequest. > User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:46.0) > Gecko/20100101 Firefox/46.8. > Content-Type: application/x-www-form-urlencoded; charset=UTF-8. > Accept: application/json, text/javascript, */*; q=0.01. > Referer: https://google.com.tr/. > Accept-Language: en-us,en. > Accept-Charset: iso-8859-1,*,utf-8. > X-Actual-IP: 46.197.96.35. > Cookie: > pps_show_100=Tue%20Aug%2002%202016%2018%3A51%3A27%20GMT+0300%20%28T%FCrkiye%20Yaz%20Saati%29; > pps_times_showed_100=1. > X-Forwarded-For: 46.197.96.35, 46.197.96.35, 46.197.96.35. > Accept-Encoding: gzip. > X-Varnish: 35061669. > > Heartbeat plugin didn't fix it by the way. > > 2016-08-02 18:45 GMT+03:00 Andrei <[email protected]>: > >> Those admin-ajax.php POST requests won't get cached, and are likely >> related to WordPress heartbeats, or plugins. The quickest way to see what >> those requests actually are, which will help you identify the plugin/theme >> option is using ngrep: ngrep 'admin-ajax' -d any dst port 8080 -W byline -q >> >> On Tue, Aug 2, 2016 at 9:43 AM, MAGNIEN, Thierry <[email protected] >> > wrote: >> >>> Hi, >>> >>> >>> >>> There are known CPU issues due to WordPress HearBeat API, for example, >>> or misbehaving plugins. >>> >>> >>> >>> See >>> http://www.inmotionhosting.com/support/website/wordpress/heartbeat-ajax-php-usage >>> for example. >>> >>> >>> >>> Maybe one of your plugins has some difficulties going through varnish. >>> >>> >>> >>> If you enable debug on your browser, can you see specific calls to >>> /wp-admin/admin-ajax.php that take a very long time with varnish and not >>> when varnish is disabled ? >>> >>> >>> >>> Thierry >>> >>> >>> >>> *De :* [email protected] >>> [mailto:[email protected]] *De >>> la part de* Ayberk Kimsesiz >>> *Envoyé :* mardi 2 août 2016 15:18 >>> *À :* Stig Bakken >>> *Cc :* varnish-misc >>> *Objet :* Re: Varnish CPU Usage >>> >>> >>> >>> I used Default.VCL in two different ways. The first with default >>> settings and the second with settings customized for Wordpress (either >>> case, the CPU usage increases). That is as follows: >>> >>> >>> >>> Also i couldn't find a Centos 6 installation guide for Prometheus. >>> >>> >>> >>> */* SET THE HOST AND PORT OF WORDPRESS* >>> >>> * * *********************************************************/* >>> >>> *vcl 4.0;* >>> >>> *import std;* >>> >>> >>> >>> *backend default {* >>> >>> * .host = "SERVER IP";* >>> >>> * .port = "8080";* >>> >>> * .first_byte_timeout = 60s;* >>> >>> * .connect_timeout = 300s;* >>> >>> *}* >>> >>> >>> >>> *# SET THE ALLOWED IP OF PURGE REQUESTS* >>> >>> *# ##########################################################* >>> >>> *acl purge {* >>> >>> * "localhost";* >>> >>> * "127.0.0.1";* >>> >>> *}* >>> >>> >>> >>> *#THE RECV FUNCTION* >>> >>> *# ##########################################################* >>> >>> *sub vcl_recv {* >>> >>> >>> >>> *# set realIP by trimming CloudFlare IP which will be used for various >>> checks* >>> >>> *set req.http.X-Actual-IP = regsub(req.http.X-Forwarded-For, "[, ].*$", >>> ""); * >>> >>> >>> >>> * # FORWARD THE IP OF THE REQUEST* >>> >>> * if (req.restarts == 0) {* >>> >>> * if (req.http.x-forwarded-for) {* >>> >>> * set req.http.X-Forwarded-For =* >>> >>> * req.http.X-Forwarded-For + ", " + client.ip;* >>> >>> * } else {* >>> >>> * set req.http.X-Forwarded-For = client.ip;* >>> >>> * }* >>> >>> * }* >>> >>> >>> >>> * # Purge request check sections for hash_always_miss, purge and ban* >>> >>> * # BLOCK IF NOT IP is not in purge acl* >>> >>> * # ##########################################################* >>> >>> >>> >>> * # Enable smart refreshing using hash_always_miss* >>> >>> *if (req.http.Cache-Control ~ "no-cache") {* >>> >>> * if (client.ip ~ purge || !std.ip(req.http.X-Actual-IP, "1.2.3.4") ~ >>> purge) {* >>> >>> * set req.hash_always_miss = true;* >>> >>> * }* >>> >>> *}* >>> >>> >>> >>> *if (req.method == "PURGE") {* >>> >>> * if (!client.ip ~ purge || !std.ip(req.http.X-Actual-IP, "1.2.3.4") >>> ~ purge) {* >>> >>> * return(synth(405,"Not allowed."));* >>> >>> * }* >>> >>> * return (purge);* >>> >>> >>> >>> * }* >>> >>> *if (req.method == "BAN") {* >>> >>> * # Same ACL check as above:* >>> >>> * if (!client.ip ~ purge || !std.ip(req.http.X-Actual-IP, >>> "1.2.3.4") ~ purge) {* >>> >>> * return(synth(403, "Not allowed."));* >>> >>> * }* >>> >>> * ban("req.http.host == " + req.http.host +* >>> >>> * " && req.url == " + req.url);* >>> >>> >>> >>> * # Throw a synthetic page so the* >>> >>> * # request won't go to the backend.* >>> >>> * return(synth(200, "Ban added"));* >>> >>> *}* >>> >>> >>> >>> *# Unset cloudflare cookies* >>> >>> *# Remove has_js and CloudFlare/Google Analytics __* cookies.* >>> >>> * set req.http.Cookie = regsuball(req.http.Cookie, >>> "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");* >>> >>> * # Remove a ";" prefix, if present.* >>> >>> * set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");* >>> >>> >>> >>> * # For Testing: If you want to test with Varnish passing (not caching) >>> uncomment* >>> >>> * # return( pass );* >>> >>> >>> >>> * # FORWARD THE IP OF THE REQUEST* >>> >>> * if (req.restarts == 0) {* >>> >>> * if (req.http.x-forwarded-for) {* >>> >>> * set req.http.X-Forwarded-For =* >>> >>> * req.http.X-Forwarded-For + ", " + client.ip;* >>> >>> * } else {* >>> >>> * set req.http.X-Forwarded-For = client.ip;* >>> >>> * }* >>> >>> * }* >>> >>> >>> >>> *# DO NOT CACHE RSS FEED* >>> >>> * if (req.url ~ "/feed(/)?") {* >>> >>> * return ( pass ); * >>> >>> *}* >>> >>> >>> >>> *## Do not cache search results, comment these 3 lines if you do want to >>> cache them* >>> >>> >>> >>> *if (req.url ~ "/\?s\=") {* >>> >>> * return ( pass ); * >>> >>> *}* >>> >>> >>> >>> *# CLEAN UP THE ENCODING HEADER.* >>> >>> * # SET TO GZIP, DEFLATE, OR REMOVE ENTIRELY. WITH VARY >>> ACCEPT-ENCODING* >>> >>> * # VARNISH WILL CREATE SEPARATE CACHES FOR EACH* >>> >>> * # DO NOT ACCEPT-ENCODING IMAGES, ZIPPED FILES, AUDIO, ETC.* >>> >>> * # ##########################################################* >>> >>> * if (req.http.Accept-Encoding) {* >>> >>> * if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {* >>> >>> * # No point in compressing these* >>> >>> * unset req.http.Accept-Encoding;* >>> >>> * } elsif (req.http.Accept-Encoding ~ "gzip") {* >>> >>> * set req.http.Accept-Encoding = "gzip";* >>> >>> * } elsif (req.http.Accept-Encoding ~ "deflate") {* >>> >>> * set req.http.Accept-Encoding = "deflate";* >>> >>> * } else {* >>> >>> * # unknown algorithm* >>> >>> * unset req.http.Accept-Encoding;* >>> >>> * }* >>> >>> * }* >>> >>> >>> >>> * # PIPE ALL NON-STANDARD REQUESTS* >>> >>> * # ##########################################################* >>> >>> * if (req.method != "GET" &&* >>> >>> * req.method != "HEAD" &&* >>> >>> * req.method != "PUT" && * >>> >>> * req.method != "POST" &&* >>> >>> * req.method != "TRACE" &&* >>> >>> * req.method != "OPTIONS" &&* >>> >>> * req.method != "DELETE") {* >>> >>> * return (pipe);* >>> >>> * }* >>> >>> >>> >>> * # ONLY CACHE GET AND HEAD REQUESTS* >>> >>> * # ##########################################################* >>> >>> * if (req.method != "GET" && req.method != "HEAD") {* >>> >>> * return (pass);* >>> >>> * }* >>> >>> >>> >>> * # OPTIONAL: DO NOT CACHE LOGGED IN USERS (THIS OCCURS IN FETCH TOO, >>> EITHER* >>> >>> * # COMMENT OR UNCOMMENT BOTH* >>> >>> * # ##########################################################* >>> >>> * if ( req.http.cookie ~ "wordpress_logged_in" ) {* >>> >>> * return( pass );* >>> >>> * }* >>> >>> >>> >>> * # IF THE REQUEST IS NOT FOR A PREVIEW, WP-ADMIN OR WP-LOGIN* >>> >>> * # THEN UNSET THE COOKIES* >>> >>> * # ##########################################################* >>> >>> * if (!(req.url ~ "wp-(login|admin)") * >>> >>> * && !(req.url ~ "&preview=true" ) * >>> >>> * ){* >>> >>> * unset req.http.cookie;* >>> >>> * }* >>> >>> >>> >>> * # IF BASIC AUTH IS ON THEN DO NOT CACHE* >>> >>> * # ##########################################################* >>> >>> * if (req.http.Authorization || req.http.Cookie) {* >>> >>> * return (pass);* >>> >>> * }* >>> >>> >>> >>> * # IF YOU GET HERE THEN THIS REQUEST SHOULD BE CACHED* >>> >>> * # ##########################################################* >>> >>> * return (hash);* >>> >>> * # This is for phpmyadmin* >>> >>> *if (req.http.Host == "ki1.org <http://ki1.org>") {* >>> >>> *return (pass);* >>> >>> *}* >>> >>> >>> >>> *if (req.http.Host == "mysql.ki1.org <http://mysql.ki1.org>") {* >>> >>> *return (pass);* >>> >>> *}* >>> >>> >>> >>> *}* >>> >>> >>> >>> *# HIT FUNCTION* >>> >>> *# ##########################################################* >>> >>> *sub vcl_hit {* >>> >>> * # IF THIS IS A PURGE REQUEST THEN DO THE PURGE* >>> >>> * # ##########################################################* >>> >>> * if (req.method == "PURGE") {* >>> >>> * #* >>> >>> * # This is now handled in vcl_recv.* >>> >>> * #* >>> >>> * # purge;* >>> >>> * return (synth(200, "Purged."));* >>> >>> * }* >>> >>> * return (deliver);* >>> >>> *}* >>> >>> >>> >>> *# MISS FUNCTION* >>> >>> *# ##########################################################* >>> >>> *sub vcl_miss {* >>> >>> * if (req.method == "PURGE") {* >>> >>> * #* >>> >>> * # This is now handled in vcl_recv.* >>> >>> * #* >>> >>> * # purge;* >>> >>> * return (synth(200, "Purged."));* >>> >>> * }* >>> >>> * return (fetch);* >>> >>> *}* >>> >>> >>> >>> *# FETCH FUNCTION* >>> >>> *# ##########################################################* >>> >>> *sub vcl_backend_response {* >>> >>> * # I SET THE VARY TO ACCEPT-ENCODING, THIS OVERRIDES W3TC * >>> >>> * # TENDANCY TO SET VARY USER-AGENT. YOU MAY OR MAY NOT WANT* >>> >>> * # TO DO THIS* >>> >>> * # ##########################################################* >>> >>> * set beresp.http.Vary = "Accept-Encoding";* >>> >>> >>> >>> * # IF NOT WP-ADMIN THEN UNSET COOKIES AND SET THE AMOUNT OF * >>> >>> * # TIME THIS PAGE WILL STAY CACHED (TTL)* >>> >>> * # ##########################################################* >>> >>> * if (!(bereq.url ~ "wp-(login|admin)") && !bereq.http.cookie ~ >>> "wordpress_logged_in" ) {* >>> >>> * unset beresp.http.set-cookie;* >>> >>> * set beresp.ttl = 52w;* >>> >>> *# set beresp.grace =1w;* >>> >>> * }* >>> >>> >>> >>> * if (beresp.ttl <= 0s ||* >>> >>> * beresp.http.Set-Cookie ||* >>> >>> * beresp.http.Vary == "*") {* >>> >>> * set beresp.ttl = 120 s;* >>> >>> * # set beresp.ttl = 120s;* >>> >>> * set beresp.uncacheable = true;* >>> >>> * return (deliver);* >>> >>> * }* >>> >>> >>> >>> * return (deliver);* >>> >>> *}* >>> >>> >>> >>> *# DELIVER FUNCTION* >>> >>> *# ##########################################################* >>> >>> *sub vcl_deliver {* >>> >>> * # IF THIS PAGE IS ALREADY CACHED THEN RETURN A 'HIT' TEXT * >>> >>> * # IN THE HEADER (GREAT FOR DEBUGGING)* >>> >>> * # ##########################################################* >>> >>> * if (obj.hits > 0) {* >>> >>> * set resp.http.X-Cache = "HIT";* >>> >>> * # IF THIS IS A MISS RETURN THAT IN THE HEADER* >>> >>> * # ##########################################################* >>> >>> * } else {* >>> >>> * set resp.http.X-Cache = "MISS";* >>> >>> * }* >>> >>> *}* >>> >>> >>> >>> >>> >>> >>> >>> 2016-08-02 15:07 GMT+03:00 Stig Bakken <[email protected]>: >>> >>> Could it be that Varnish is not caching those POST requests, and that >>> they are what makes Apache consume a lot of CPU? >>> >>> >>> >>> On MPMs: with PHP prefork is the safest choice, and it will work just >>> fine as long as you have a reasonable config, especially since you have >>> Varnish in front offloading the job of shuffling bytes back to the users. >>> >>> >>> >>> If I were you I'd try to get more data on which requests take a long >>> time, some variant of "varnishncsa -F %D".. >>> >>> >>> >>> If you're into using Prometheus for metrics, you can try out this >>> exporter which will give you a lot of insight into which requests take a >>> long time to process: https://github.com/stigsb/varnish_request_exporter >>> (Prometheus is awesome!) >>> >>> >>> >>> - Stig >>> >>> >>> >>> >>> >>> On Thu, Jul 28, 2016 at 6:57 PM, Ayberk Kimsesiz < >>> [email protected]> wrote: >>> >>> Hi, >>> >>> >>> >>> *CPU Monitor: * >>> >>> >>> >>> http://i.imgur.com/5KT1xRu.jpg >>> >>> >>> >>> *Apache status:* >>> >>> >>> >>> *Srv* >>> >>> *PID* >>> >>> *Acc* >>> >>> *M* >>> >>> *CPU* >>> >>> *SS* >>> >>> *Req* >>> >>> *Conn* >>> >>> *Child* >>> >>> *Slot* >>> >>> *Client* >>> >>> *Protocol* >>> >>> *VHost* >>> >>> *Request* >>> >>> *0-0* >>> >>> - >>> >>> 0/0/9766 >>> >>> . >>> >>> 134.59 >>> >>> 37 >>> >>> 0 >>> >>> 0.0 >>> >>> 0.00 >>> >>> 64.40 >>> >>> ::1 >>> >>> http/1.1 >>> >>> ns1.***com:8080 >>> >>> OPTIONS * HTTP/1.0 >>> >>> *1-0* >>> >>> 14612 >>> >>> 0/16/9058 >>> >>> _ >>> >>> 17.83 >>> >>> 13 >>> >>> 1498 >>> >>> 0.0 >>> >>> 0.02 >>> >>> 53.29 >>> >>> 176.***.10 >>> >>> http/1.1 >>> >>> www.***.com:8080 >>> >>> POST /wp-admin/admin-ajax.php HTTP/1.1 >>> >>> *2-0* >>> >>> 10863 >>> >>> 0/179/9795 >>> >>> _ >>> >>> 185.14 >>> >>> 6 >>> >>> 1424 >>> >>> 0.0 >>> >>> 0.58 >>> >>> 60.32 >>> >>> 176.***.10 >>> >>> http/1.1 >>> >>> www.***.com:8080 >>> >>> POST /wp-admin/admin-ajax.php HTTP/1.1 >>> >>> *3-0* >>> >>> 13127 >>> >>> 0/127/9435 >>> >>> _ >>> >>> 119.80 >>> >>> 4 >>> >>> 1419 >>> >>> 0.0 >>> >>> 0.42 >>> >>> 56.51 >>> >>> 176.***.10 >>> >>> http/1.1 >>> >>> www.***.com:8080 >>> >>> POST /wp-admin/admin-ajax.php HTTP/1.1 >>> >>> *4-0* >>> >>> - >>> >>> 0/0/9187 >>> >>> . >>> >>> 0.00 >>> >>> 50 >>> >>> 0 >>> >>> 0.0 >>> >>> 0.00 >>> >>> 56.60 >>> >>> ::1 >>> >>> http/1.1 >>> >>> ns1.***.com:8080 >>> >>> OPTIONS * HTTP/1.0 >>> >>> *5-0* >>> >>> 14851 >>> >>> 0/9/8761 >>> >>> _ >>> >>> 8.95 >>> >>> 13 >>> >>> 1559 >>> >>> 0.0 >>> >>> 0.01 >>> >>> 57.90 >>> >>> 176.***.10 >>> >>> http/1.1 >>> >>> www.***.com:8080 >>> >>> POST /wp-admin/admin-ajax.php HTTP/1.1 >>> >>> *6-0* >>> >>> 14852 >>> >>> 0/6/8130 >>> >>> _ >>> >>> 6.67 >>> >>> 4 >>> >>> 1482 >>> >>> 0.0 >>> >>> 0.01 >>> >>> 51.88 >>> >>> 176.***.10 >>> >>> http/1.1 >>> >>> www.***.com:8080 >>> >>> POST /wp-admin/admin-ajax.php HTTP/1.1 >>> >>> *7-0* >>> >>> 14192 >>> >>> 11/57/8355 >>> >>> *K* >>> >>> 72.73 >>> >>> 0 >>> >>> 1363 >>> >>> 106.6 >>> >>> 0.44 >>> >>> 52.79 >>> >>> 176.***.10 >>> >>> http/1.1 >>> >>> www.***.com:8080 >>> >>> POST /wp-admin/admin-ajax.php HTTP/1.1 >>> >>> *8-0* >>> >>> 13067 >>> >>> 0/125/7795 >>> >>> _ >>> >>> 121.19 >>> >>> 13 >>> >>> 1759 >>> >>> 0.0 >>> >>> 0.65 >>> >>> 68.27 >>> >>> 176.***.10 >>> >>> http/1.1 >>> >>> www.***.com:8080 >>> >>> POST /wp-admin/admin-ajax.php HTTP/1.1 >>> >>> >>> >>> Apache error logs don't show anything about CPU. >>> >>> >>> >>> >>> >>> 2016-07-28 19:31 GMT+03:00 Guillaume Quintard < >>> [email protected]>: >>> >>> You seem to have a good hit ratio, are you seeing anything on the apache >>> logs that would explain the cpu usage? >>> >>> >>> -- >>> >>> Guillaume Quintard >>> >>> >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> [email protected] >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >>> >>> >>> >>> >>> -- >>> >>> Stig Bakken >>> CTO, Zedge.net - free your phone! >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> [email protected] >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> >
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
