GRSEC and Varnish
Hi, I'am running : - varnishd (varnish-2.0.4) - linux kernel 2.6.27.10-grsec--grs-ipv4-64 and it appears that the grsec Kernel repeatedly and unexpectedly sends signal 11 to the varnishd child. .../... Feb 2 12:01:02 XX varnishd[17111]: segfault at 1000 ip 0043abf0 sp 47d89ae0 error 4 in varnishd[40+5] Feb 2 12:01:02 XX grsec: From 82.67.39.69: signal 11 sent to /usr/sbin/varnishd[varnishd:17111] uid/euid:65534/65534 gid/egid:65534/65534, parent /usr/sbin/varnishd[varnishd:28927] uid/euid:0/0 gid/egid:0/0 Feb 2 13:45:44 XX varnishd[22187]: segfault at f5000 ip 0043abf0 sp 48538ae0 error 4 in varnishd[40+5] Feb 2 13:45:44 XX grsec: From 80.13.19.228: signal 11 sent to /usr/sbin/varnishd[varnishd:22187] uid/euid:65534/65534 gid/egid:65534/65534, paren t /usr/sbin/varnishd[varnishd:28927] uid/euid:0/0 gid/egid:0/0 Feb 2 13:54:57 XX varnishd[22236]: segfault at 1000 ip 0043abf0 sp 45445ae0 error 4 in varnishd[40+5] Feb 2 13:54:57 XX grsec: From 80.13.19.228: signal 11 sent to /usr/sbin/varnishd[varnishd:22236] uid/euid:65534/65534 gid/egid:65534/65534, paren t /usr/sbin/varnishd[varnishd:28927] uid/euid:0/0 gid/egid:0/0 Feb 2 14:13:41 XX varnishd[22595]: segfault at ae000 ip 0043abf0 sp 40ff4ae0 error 4 in varnishd[40+5] Feb 2 14:13:41 XX grsec: From 83.145.80.130: signal 11 sent to /usr/sbin/varnishd[varnishd:22595] uid/euid:65534/65534 gid/egid:65534/65534, pare nt /usr/sbin/varnishd[varnishd:28927] uid/euid:0/0 gid/egid:0/0 Feb 2 14:31:08 XX varnishd[23547]: segfault at 1000 ip 0043abf0 sp 45b40ae0 error 4 in varnishd[40+5] Feb 2 14:31:08 XX grsec: From 81.49.118.48: signal 11 sent to /usr/sbin/varnishd[varnishd:23547] uid/euid:65534/65534 gid/egid:65534/65534, paren t /usr/sbin/varnishd[varnishd:28927] uid/euid:0/0 gid/egid:0/0 Feb 2 16:19:05 XX varnishd[24256]: segfault at f7000 ip 0043abf0 sp 473bcae0 error 4 in varnishd[40+5] Feb 2 16:19:05 XX grsec: From 192.196.142.20: signal 11 sent to /usr/sbin/varnishd[varnishd:24256] uid/euid:65534/65534 gid/egid:65534/65534, par ent /usr/sbin/varnishd[varnishd:28927] uid/euid:0/0 gid/egid:0/0 Then the parent varnishd process starts immediately a new child process which lasts some time. Is there any way to fix this. Remocve the GRSEC kernel ? Upgrade the kernel ? Varnish ? or whatever ? Thanks in advance. -- Bernard FRIT ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: GRSEC and Varnish
]] Bernardf FRIT | Then the parent varnishd process starts immediately a new child process | which lasts some time. | | Is there any way to fix this. Remocve the GRSEC kernel ? Upgrade the | kernel ? Varnish ? or whatever ? Work out why it thinks that varnishd is doing something wrong? It doesn't seem to say so in the log. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: GRSEC and Varnish
On Tue, Feb 2, 2010 at 11:53 PM, Tollef Fog Heen wrote: > ]] Bernardf FRIT > > | Then the parent varnishd process starts immediately a new child process > | which lasts some time. > | > | Is there any way to fix this. Remocve the GRSEC kernel ? Upgrade the > | kernel ? Varnish ? or whatever ? > > Work out why it thinks that varnishd is doing something wrong? It > doesn't seem to say so in the log. > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > grsec will often report that signals were sent, not that grsec necessarily sent that signal itself. I don't think I've ever actually seen it report itself sending a signal to a process. So varnishd could be segfaulting for some reason and grsec is just reporting this. If you're getting a core file, try loading it into gdb and using 'bt' on it, to see where it's dying. One other thing to try: As soon as it happens, try using 'dmesg' (or "dmesg -s 131072" in case you have lots of things logging to the kernel logs) and grep for PAX. It's not likely, but PAX could be killing it due to some violation. And for whatever reason, the PAX message doesn't show up in the logs, just in dmesg. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: GRSEC and Varnish
On Tue, Feb 02, 2010 at 04:44:48PM +0100, Bernardf FRIT wrote: > Hi, > > I'am running : > - varnishd (varnish-2.0.4) Why not 2.0.6? > and it appears that the grsec Kernel repeatedly and unexpectedly sends > signal 11 to the varnishd child. grsec seems to just report that a segfault occurred. SIGSEG would be a strange signal to send in any event. You want to fetch yourself a core-dump of this. However, before we get into that, I'd like to know what parameters you start Varnish with, and the general setup. VCL too, if possible. -- Kristian Lyngstøl Redpill Linpro AS Mob: +47 99014497 pgpFwMaQMo7wh.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: GRSEC and Varnish
On Thu, Feb 4, 2010 at 7:59 AM, Bernardf FRIT wrote: > Mark Moseley a écrit : >> >> grsec will often report that signals were sent, not that grsec >> necessarily sent that signal itself. I don't think I've ever actually >> seen it report itself sending a signal to a process. So varnishd could >> be segfaulting for some reason and grsec is just reporting this. If >> you're getting a core file, try loading it into gdb and using 'bt' on >> it, to see where it's dying. > > I cannot get any core file. >> >> One other thing to try: As soon as it happens, try using 'dmesg' (or >> "dmesg -s 131072" in case you have lots of things logging to the >> kernel logs) and grep for PAX. It's not likely, but PAX could be >> killing it due to some violation. And for whatever reason, the PAX >> message doesn't show up in the logs, just in dmesg. > > Nothing more in dmesg than in kern.log. ;-[ > > varnishd[27069]: segfault at 1000 ip 0043abf0 sp 45696ae0 > error 4 in varnishd[40+5] > grsec: From 80.13.9.224: signal 11 sent to > /usr/sbin/varnishd[varnishd:27069] uid/euid:65534/65534 > gid/egid:65534/65534, parent /usr/sbin/varnishd[varnishd:11327] uid/euid:0/0 > gid/egid:0/0 > varnishd[27992]: segfault at 1000 ip 0043abf0 sp 48140ae0 > error 4 in varnishd[40+5] > grsec: From 90.4.162.78: signal 11 sent to > /usr/sbin/varnishd[varnishd:27992] uid/euid:65534/65534 > gid/egid:65534/65534, parent /usr/sbin/varnishd[varnishd:11327] uid/euid:0/0 > gid/egid:0/0 > varnishd[28119]: segfault at ed000 ip 0043abf0 sp 47ceaae0 > error 4 in varnishd[40+5] > grsec: From 90.44.219.18: signal 11 sent to > /usr/sbin/varnishd[varnishd:28119] uid/euid:65534/65534 > gid/egid:65534/65534, parent /usr/sbin/varnishd[varnishd:11327] uid/euid:0/0 > gid/egid:0/0 Try putting "ulimit -c unlimited" in your varnishd rc file. I haven't needed to get a varnishd core file before, so maybe the devs might be able to advise if there's other steps necessary as well. There should also be some logs saying that it died (or at least that it restarted); dunno what your distro you're using, but in debian, those typically end up in /var/log/syslog. You could tail varnishncsa to see if there's a common request where it seems to segfault at and if there is, you could attach to varnishd with "gdb /path/to/varnishd " and try to trigger it. Then get the backtrace with 'bt'. But be aware that it'll bog it down dreadfully, so i wouldn't advise it in production. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: GRSEC and Varnish
Tollef Fog Heen a écrit : > ]] Bernardf FRIT > > | Then the parent varnishd process starts immediately a new child process > | which lasts some time. > | > | Is there any way to fix this. Remocve the GRSEC kernel ? Upgrade the > | kernel ? Varnish ? or whatever ? > > Work out why it thinks that varnishd is doing something wrong? It > doesn't seem to say so in the log. > Seems a bit complicated to add monitoring features on a running GRSEC kernel. Need to install all kernel files then configure and recompile the kernel. I will do it if we need it. The only extra feature I found relevant is CONFIG_GRKERNSEC_RESLOG wich stands for "Denied resource logging". I have monitored the following segfault with varnislog : Jan 29 19:59:07 XXX varnishd[13812]: segfault at 1000 ip 0043abf0 sp 46332ae0 error 4 in varnishd[40+5] Jan 29 19:59:07 XXX grsec: From 91.163.168.48: signal 11 sent to /usr/sbin/varnishd[varnishd:13812] uid/euid:65534/65534 gid/egid:65534/65534, pare nt /usr/sbin/varnishd[varnishd:28927] uid/euid:0/0 gid/egid:0/0 I hope that could help to figure out what happened. 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1264791539 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1264791542 1.0 9 SessionOpen c 91.163.168.48 49273 87.98.137.117:80 9 ReqStart c 91.163.168.48 49273 879416771 9 RxRequestc GET 9 RxURLc /a_liste_ville_cp.php?q=59 9 RxProtocol c HTTP/1.1 9 RxHeader c x-requested-with: XMLHttpRequest 9 RxHeader c Accept-Language: fr 9 RxHeader c Referer: http://www.your-immo.fr/recherche.php 9 RxHeader c Accept: */* 9 RxHeader c UA-CPU: x86 9 RxHeader c Accept-Encoding: gzip, deflate 9 RxHeader c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; S LCC1; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618; OfficeLiveConnector.1.3; OfficeLivePatch.0.0) 9 RxHeader c Host: www.your-immo.fr 9 RxHeader c Connection: Keep-Alive 9 RxHeader c Cookie: PHPSESSID=9d73c9bbf8944a710f3d954a2e16c838; DYNSRV=s0; __utma=235703049.2110649494.1264791338.1264791338.1264791338.1; __u tmb=235703049.6.10.1264791338; __utmc=235703049; __utmz=235703049.1264791338.1.1.utmcsr=google|utmccn=generique|utmcmd=cpc|ut 9 VCL_call c recv 9 VCL_return c lookup 9 VCL_call c hash 9 VCL_return c hash 9 VCL_call c miss 9 VCL_return c fetch 10 BackendOpen b ha_proxy 127.0.0.1 35735 127.0.0.1 80 9 Backend c 10 ha_proxy ha_proxy 10 TxRequestb GET 10 TxURLb /a_liste_ville_cp.php?q=59 10 TxProtocol b HTTP/1.1 10 TxHeader b x-requested-with: XMLHttpRequest 10 TxHeader b Accept-Language: fr 10 TxHeader b Referer: http://www.your-immo.fr/recherche.php 10 TxHeader b Accept: */* 10 TxHeader b UA-CPU: x86 10 TxHeader b User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; S LCC1; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618; OfficeLiveConnector.1.3; OfficeLivePatch.0.0) 10 TxHeader b Host: www.your-immo.fr 10 TxHeader b Cookie: PHPSESSID=9d73c9bbf8944a710f3d954a2e16c838; DYNSRV=s0; __utma=235703049.2110649494.1264791338.1264791338.1264791338.1; __u tmb=235703049.6.10.1264791338; __utmc=235703049; __utmz=235703049.1264791338.1.1.utmcsr=google|utmccn=generique|utmcmd=cpc|ut 10 TxHeader b X-Forwarded-For: 91.163.168.48 10 TxHeader b Accept-Encoding: gzip 10 TxHeader b X-Varnish: 879416771 10 TxHeader b X-Forwarded-For: 91.163.168.48 10 RxProtocol b HTTP/1.1 10 RxStatus b 200 10 RxResponse b OK 10 RxHeader b Date: Fri, 29 Jan 2010 19:05:12 GMT 10 RxHeader b Server: Apache 10 RxHeader b X-Powered-By: PHP/5.2.5-pl1-gentoo 10 RxHeader b Expires: Thu, 19 Nov 1981 08:52:00 GMT 10 RxHeader b Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 10 RxHeader b Pragma: no-cache 10 RxHeader b Connection: close 10 RxHeader b Transfer-Encoding: chunked 10 RxHeader b Content-Type: text/html 9 ObjProtocol c HTTP/1.1 9 ObjStatusc 200 9 ObjResponse c OK 9 ObjHeaderc Date: Fri, 29 Jan 2010 19:05:12 GMT 9 ObjHeaderc Server: Apache 9 ObjHeaderc X-Powered-By: PHP/5.2.5-pl1-gentoo 9 ObjHeaderc Expires: Thu, 19 Nov 1981 08:52:00 GMT 9 ObjHeaderc Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 9 ObjHeaderc Pragma: no-cache 9 ObjHeaderc Content-Type: text/html 10 BackendClose b ha_proxy 9 TTL c 879416771 RFC 0 1264791545 1264791912 375007920 0 0 9 VCL_call c fetch 9 TTL c 879416771 VCL 0
Re: GRSEC and Varnish
Kristian Lyngstol a écrit :
> On Tue, Feb 02, 2010 at 04:44:48PM +0100, Bernardf FRIT wrote:
>> Hi,
>>
>> I'am running :
>> - varnishd (varnish-2.0.4)
>
> Why not 2.0.6?
When a server is running well, I'm a bit reluctant to upgrade. Now, I'm
ok to upgrade as an attempt to fix this.
>> and it appears that the grsec Kernel repeatedly and unexpectedly sends
>> signal 11 to the varnishd child.
>
> grsec seems to just report that a segfault occurred. SIGSEG would be a
> strange signal to send in any event. You want to fetch yourself a core-dump
> of this. However, before we get into that, I'd like to know what parameters
> you start Varnish with, and the general setup. VCL too, if possible.
>
Ok, I just misunderstood the grsec report. I can't find any core dump
file in the system.
I start varnishd using /etc/init.d/varnishd with the following parameters :
# cat /etc/conf.d/varnishd
# /etc/conf.d/varnishd
# options passed to varnish on startup
# please see the varnishd man page for more options
VARNISHD_OPTS="-a 87.98.137.117:80 -f /etc/varnish/yourimmo.vcl -n
/home/varnish/yourimmo -s file,/home/varnish/cache/yourimmo,1G -T
127.0.0.1:"
# arguments passed to varnishncsa
# please see the varnishncsa man page for more options
VARNISHNCSA_ARGS="-c -a -n /home/varnish/yourimmo -w
/var/log/varnish/access.log"
---
# cat /etc/varnish/yourimmo.vcl
### define backends:
# ha proxy
backend ha_proxy {
.host = "127.0.0.1";
.port = "80";
}
acl purge {
"localhost";
"111.111.111.111";
}
### Called when a client request is received
sub vcl_recv {
### if there is a purge make sure its coming from $localhost
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
lookup;
}
# Add a unique header containing the client address
remove req.http.X-Forwarded-For;
setreq.http.X-Forwarded-For = client.ip;
# setreq.http.X-Forwarded-For = req.http.rlnclientipaddr;
# grace settings, note this is also set in vcl_fetch,
set req.grace = 600s;
if (req.http.host ~ "^(www.)?your-immo.fr$") {
set req.backend = ha_proxy;
}
### ne pas mettre en cache:
if (req.request == "GET" && req.url ~ "\.(php|html)$") {
pass;
}
if (req.request == "GET" && req.url ~ "\.(your-immo\.fr)$") {
pass;
}
### toujours mettre en cache:
if (req.request == "GET" && req.url ~ "\.(js)") {
lookup;
}
## images
if (req.request == "GET" && req.url ~
"\.(gif|jpg|jpeg|bmp|png|tiff|tif|ico|img|tga|wmf)$") {
lookup;
}
## pages statiques
if (req.request == "GET" && req.url ~ "\.(css|pdf|exe)$") {
lookup;
}
## multimedia
if (req.request == "GET" && req.url ~
"\.(svg|swf|ico|mp3|mp4|m4a|ogg|mov|avi|wmv)$") {
lookup;
}
## xml
if (req.request == "GET" && req.url ~ "\.(xml)$") {
lookup;
}
### regles a ne pas mettre en cache:
if (req.request == "GET" && req.url ~ "\/stats") {
pipe;
}
if (req.request != "GET" && req.request != "HEAD") {
pipe;
}
if (req.http.Authenticate || req.http.Authorization) {
pass;
}
### ne pas mettre en cache les sessions d'authenticfication
if (req.http.Cookie && req.http.Cookie ~ "authtoken=") {
pipe;
}
### parse accept encoding rulesets to make it look nice
if (req.http.Accept-Encoding) {
if (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}
### Modif suite a segfault
pass;
### if it passes all these tests, do a lookup anyway;
###lookup;
### end of vcl_recv
}
### Called when an object is in the cache, its a hit.
sub vcl_hit {
if (req.request == "PURGE") {
set obj.ttl = 0s;
error 200 "Purged.";
}
if (!obj.cacheable) {
pass;
}
deliver;
}
### Called when the requested object was not found in the cache
sub vcl_miss {
if (req.request == "PURGE") {
error 404 "Not in cache.";
}
}
### Called when the requested object has been retrieved from the
backend, or the request to the backend has failed
sub vcl_fetch {
## If the request to the backend returns a code other than 200, restart
the loop
## If the number of restarts reaches the value of the parameter
max_restarts,
## the request will be error'ed. max_restarts defaults to 4. This
prevents
## an eternal loop in the ev
Re: GRSEC and Varnish (FIXED)
Mark Moseley a écrit : > Try putting "ulimit -c unlimited" in your varnishd rc file. I haven't > needed to get a varnishd core file before, so maybe the devs might be > able to advise if there's other steps necessary as well. There should > also be some logs saying that it died (or at least that it restarted); > dunno what your distro you're using, but in debian, those typically > end up in /var/log/syslog. > No way to get a core file even with ulimit -c unlimited. I use Gentoo and the kernel logs are in /var/log/kern.log > You could tail varnishncsa to see if there's a common request where it > seems to segfault at and if there is, you could attach to varnishd > with "gdb /path/to/varnishd " and try to trigger it. > Then get the backtrace with 'bt'. But be aware that it'll bog it down > dreadfully, so i wouldn't advise it in production. > I could not figure out any common request. So I end up googling "gentoo varnish" and it appears that the Gentoo team had released a varnish-2.0.4-r1 package and marked unstable varnish-2.0.4 and 2.0.5 gentoo packages. So I installed the r1 package yesterday and I didn't get any more segfault since I restarted varnishd. Thanks for your help and best regards -- Bernard FRIT ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
