Hello all, I'm pretty new to Varnish. I'm deploying it because one of our customer is going to have a special event and the website is pretty slow (I'm working for an Internet hosting company). We are expecting more than 1000 requests per seconds.
From what I read here and there, this should not be a problem for
Varnish.
My problem is that when Varnish is using cache ("deliver", as opposed to
"pass"), the CPU consumption increases drasticaly, also the RAM.
The server is a Xeon QuadCore 2.5Ghz, 8GB of RAM.
With a simple test like this (robots.txt = 300 bytes) :
ab -r -n 1000000 -c 1000 http://www.customer1.com/robots.txt
CPU consumption is fluctuating between 120% and 160%.
Second point is that Varnish consumes all the memory. Trying to limit
that, I made a tmpfs mountpoint of 3G :
mount -t tmpfs -o size=3g tmpfs /mnt/varnish/
But varnish continues to consume all the memory
My configuration is attached to this mail. Varnish is launched like
this :
/usr/sbin/varnishd -P /var/run/varnish.pid
-a :80
-f /etc/varnish/default.vcl
-T 127.0.0.1:6082
-t 120
-w 120,120,120
-u varnish -g varnish
-S /etc/varnish/secret
-s file,/mnt/varnish/varnish_storage.bin,100%
-p thread_pools 4
I also tried to launch it with parameter "-h classic"
It is installed on a Centos 5 up to date, with lastest RPMs provided by
the varnish repository.
If I put a return (pass) in vcl_fetch, everything is fine (except the
backend server, of course). It makes me think, with my little knowledges
of Varnish, that the problem is in the delivering from cache.
Output of "varnishstat -1", when running ab, is attached.
Thanks for your help.
--
Romain LE DISEZ
backend customer1 {
.host = "customer1.hoster.net";
.port = "80";
}
sub vcl_recv {
#
# Normalisation des URL
#
# Normaliser les URL envoyés par curls -X et LWP
if( req.url ~ "^http://"; ) {
set req.url = regsub(req.url, "http://[^/]*";, "");
}
# Normaliser l'hôte (domain.tldx -> www.domain.tld)
if( req.http.host == "customer1.com" || req.http.host ~
"^(www\.)?customer1\.net$" ) {
set req.http.redir = "http://www.customer1.com"; req.url;
error 750 req.http.redir;
}
#
# Configuration des sites
#
# Règles spécifiques à Customer1
if( req.http.host == "www.customer1.com" ) {
set req.backend = customer1;
# Supprimer l'entête Cookie envoyée par le navigateur
remove req.http.Cookie;
# OK pour le moment (voir quand la version mobile sera OK)
remove req.http.Accept;
remove req.http.Accept-Language;
remove req.http.Accept-Charset;
remove req.http.User-Agent;
}
#
# Règles génériques adaptées à tous les sites
#
# Période de grâce : continue de servir le contenu après son expiration du
cache
# (par exemple le temps de refaire la requête vers le backend ou de le
deplanter)
set req.grace = 3600s;
# Normaliser l'entête Accept-Encoding
if( req.http.Accept-Encoding ) {
if( req.url ~
"\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|mp4|flv)$" ) {
# Ne pas compresser les fichiers déjà compressés
remove req.http.Accept-Encoding;
} elsif( req.http.Accept-Encoding ~ "gzip" ) {
set req.http.Accept-Encoding = "gzip";
} elsif( req.http.Accept-Encoding ~ "deflate" ) {
set req.http.Accept-Encoding = "deflate";
} else {
# Supprimer les algorithmes inconnus
remove req.http.Accept-Encoding;
}
}
# Purger l'URL du cache si elle se termine par le paramètre purge
if( req.url ~ "~purge$" ) {
# Suppression du suffixe "~purge" puis purge de l'URL
set req.url = regsub(req.url, "(.*)~purge$", "\1");
purge_url( req.url );
}
}
#
# Appellé après recéption de la réponse du backend
#
sub vcl_fetch {
# Supprimer l'entête Set-Cookie envoyée par le serveur
remove beresp.http.Set-Cookie;
}
#
# Appellé avant l'envoi d'un contenu du cache
#
sub vcl_deliver {
remove resp.http.Via;
remove resp.http.X-Varnish;
remove resp.http.Server;
remove resp.http.X-Powered-By;
remove resp.http.P3P;
}
#
# "Catching" des erreurs
#
sub vcl_error {
if( obj.status == 750 ) {
set obj.http.Location = obj.response;
set obj.status = 301;
return(deliver);
}
}
client_conn 136529 117.80 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 136532 117.80 Client requests received cache_hit 136531 117.80 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 1 0.00 Cache misses backend_conn 1 0.00 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 0 0.00 Backend conn. failures backend_reuse 0 0.00 Backend conn. reuses backend_toolate 0 0.00 Backend conn. was closed backend_recycle 0 0.00 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 0 0.00 Fetch head fetch_length 1 0.00 Fetch with Length fetch_chunked 0 0.00 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 0 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed n_sess_mem 7720 . N struct sess_mem n_sess 18446744073709551606 . N struct sess n_object 1 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 481 . N struct objectcore n_objecthead 481 . N struct objecthead n_smf 3 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 1 . N large free smf n_vbe_conn 0 . N struct vbe_conn n_wrk 480 . N worker threads n_wrk_create 480 0.41 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 81 0.07 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 166 0.14 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 1 . N backends n_expired 0 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 6 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 136429 117.71 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 136534 117.80 Total Sessions s_req 136534 117.80 Total Requests s_pipe 0 0.00 Total pipe s_pass 0 0.00 Total pass s_fetch 1 0.00 Total fetch s_hdrbytes 30885129 26648.08 Total header bytes s_bodybytes 41097938 35459.83 Total body bytes sess_closed 136538 117.81 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 0 0.00 Session Linger sess_herd 0 0.00 Session herd shm_records 4233973 3653.13 SHM records shm_writes 547445 472.34 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 46002 39.69 SHM MTX contention shm_cycles 1 0.00 SHM cycles through buffer sm_nreq 2 0.00 allocator requests sm_nobj 2 . outstanding allocations sm_balloc 8192 . bytes allocated sm_bfree 2574852096 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 0 0.00 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 0 . SMS bytes allocated sms_bfree 0 . SMS bytes freed backend_req 1 0.00 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 136442 117.72 HCB Lookups without lock hcb_lock 1 0.00 HCB Lookups with lock hcb_insert 1 0.00 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 1159 1.00 Client uptime backend_retry 0 0.00 Backend conn. retry dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304)
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ varnish-misc mailing list [email protected] http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
