Thank you Anand. we have set workspace_backend=262144 and the issue is resolved.
Thanks, Mahesh. ________________________________ From: [email protected] <[email protected]> on behalf of Anand Shah <[email protected]> Sent: Monday, September 12, 2016 4:09 PM To: Mahesh Ollalwar Cc: Dag Haavi Finstad ; [email protected] Subject: Re: varnish core dump In order to reduce contention between all the threads, Varnish uses workspaces which allow each thread to use or modify memory independently of other threads, which avoids locking. Try increasing session workspace and shm_workspace. Disable any VMOD's that you use. Regards, Anand Shah From: Mahesh Ollalwar <[email protected]> Sent: Mon, 12 Sep 2016 15:16:52 To: Dag Haavi Finstad <[email protected]>, "[email protected]" <[email protected]> Subject: Re: varnish core dump Hi, Thanks for the response. We have upgraded to latest stable version 4.1.3 but still see panics. below are the logs. Sep 12 12:45:18 Prod-SHARED-CACHE6 abrt[32688]: Saved core dump of pid 24551 (/product/varnish/sbin/varnishd) to /var/spool/abrt/ccpp-2016-09-12-12:45:15-24551 (585650176 bytes) Sep 12 12:45:18 Prod-SHARED-CACHE6 abrtd: Directory 'ccpp-2016-09-12-12:45:15-24551' creation detected Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Executable '/product/varnish/sbin/varnishd' doesn't belong to any package and ProcessUnpackaged is set to 'no' Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551' exited with 1 Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551' Sep 12 12:45:19 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) not responding to CLI, killing it. Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) died signal=6 (core dumped) Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) Last panic at: Mon, 12 Sep 2016 07:15:20 GMT#012"Assert error in http_EstimateWS(), cache/cache_http.c line 793:#012 Condition((fm->hd[u].b) != 0) not true.#012thread = (cache-worker)#012version = varnish-4.1.3 revision 5e3b6d2#012ident = Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x432f13: pan_ic+0x163#012 0x42dec5: http_EstimateWS+0x435#012 0x421874: vbf_beresp2obj+0x84#012 0x422cb5: vbf_stp_error+0x235#012 0x42366b: vbf_fetch_thread+0x1db#012 0x449825: WRK_Thread+0x4a5#012 0x449c5b: pool_thread+0x2b#012 0x3c25e079d1: libpthread.so<http://libpthread.so>.0() [0x3c25e079d1]#012 0x3c25ae8b6d: libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012busyobj = 0x7f7f505208a0 {#012 ws = 0x7f7f50520960 {#012 OVERFLOWED id = \"Bo\",#012 {s,f,r,e} = {0x7f7f50522818,+57440,(nil),+57440},#012 },#012 refcnt = 2,#012 retries = 0, failed = 1, state = 1,#012 flags = {do_gzip, do_stream, is_gunzip},#012 director_req = 0x1a16130 {#012 vcl_name = farm_anon,#012 type = round-robin {#012 },#012 },#012 director_resp = 0x1a08e98 {#012 vcl_name = PROD_LB_BE_ANON_1,#012 type = backend {#012 display_name = boot.PROD_LB_BE_ANON_1,#012 ipv4 = 10.254.4.254<http://10.254.4.254>,#012 port = 8080,#012 hosthdr = 10.254.4.254<http://10.254.4.254>,#012 health=healthy, admin_health=probe, changed=1473664247.5,#012 n_conn = 2,#012 },#012 },#012 http[bereq] = 0x7f7f50520f28 {#012 ws[Bo] = 0x7f7f50520960,#012 hdrs {#012 \"GET\",#012 \"/AVS/besc?action=GetArrayContentList&channel=ANDROID&categoryId=7022\",#012 \"HTTP/1.1\",#012 \"host: www.example.com<http://www.example.com> \",#012 \"Accept-Encoding: gzip\",#012 \"Akamai-Origin-Hop: 4\",#012 \"True-Client-IP: 183.83.249.187<http://183.83.249.187>\",#012 \"User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; Le X526 Build/IIXOSOP5801607291S)\",#012 \"Via: 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 aka Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (386) said Child starts Thanks in advance, Mahesh. ________________________________ From: Dag Haavi Finstad <[email protected]> Sent: Monday, June 27, 2016 2:57 PM To: Mahesh Ollalwar Cc: [email protected] Subject: Re: varnish core dump Hi I believe this might be #1539 [1], which is fixed in 4.0.3 and newer. Please upgrade to the latest release and file a bug if you still run into this. [1]: https://www.varnish-cache.org/trac/ticket/1539 On Fri, Jun 24, 2016 at 9:06 PM, Mahesh Ollalwar <[email protected]> wrote: > Can anyone help me on below error. looks like core dump is happening and > child is getting restarted. This is happening recently without any change to > the VCL and any system configuration. > > During this time the requests are getting terminated abruptly. > > Thanks in advance. > > > Varnish version - varnish-4.0.2 > > > Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526 > (/product/varnish/sbin/varnishd) to > /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes) > Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory > 'ccpp-2016-06-24-23:48:19-26526' creation detected > Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable > '/product/varnish/sbin/varnishd' doesn't belong to any package and > ProcessUnpackaged is set to 'no' > Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on > '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1 > Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory > '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' > Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to > CLI, killing it. > Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to > CLI, killing it. > Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6 > (core dumped) > Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic > message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line > 430:#012 Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread = > (cache-worker)#012ident = > Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 > 0x4310e5: pan_ic+0xc5#012 0x434c30: cnt_purge+0x0#012 0x436461: > CNT_Request+0x241#012 0x42b73b: HTTP1_Session+0x53b#012 0x439f98: > ses_req_pool_task+0x68#012 0x43a6d4: SES_pool_accept_task+0x2a4#012 > 0x433b87: Pool_Work_Thread+0x357#012 0x4467d9: wrk_thread_real+0xb9#012 > 0x3c25e079d1: /lib64/libpthread.so<http://libpthread.so>.0() > [0x3c25e079d1]#012 0x3c25ae8b6d: > /lib64/libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012req = > 0x7f83baaf2cb0 {#012 > sp = 0x7f84dd71c430, vxid = 1108394165, step = R_STP_LOOKUP,#012 req_body > = R_BODY_NONE,#012 restarts = 0, esi_level = 0#012 sp = 0x7f84dd71c430 > {#012 fd = 64, vxid = 34652201,#012 client = > 10.254.9.78<http://10.254.9.78> 61217,#012 > step = S_STP_WORKING,#012 },#012 worker = 0x7f8574c0cc40 {#012 ws = > 0x7f8574c0ce58 {#012 id = "wrk",#012 {s,f,r,e} = > {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012 },#012 VCL::method = > 0x0,#012 VCL::return = deliver,#012 },#012 ws = 0x7f83baaf2e48 {#012 > id = "req",#012 {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012 > },#012 http[req] = {#012 ws = 0x7f83baaf2e48[req]#012 "GET",#012 > "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012 > "HTTP/1.1",#012 "host: xxx.xxx.com<http://xxx.xxx.com>",#012 > "Accept-Encoding: > gzip",#012 "Akamai-Origin-Hop: 4",#012 "True-Client-IP: > 49.15.204.238<http://49.15.204.238>",#012 "User-Agent: Dalvik/2.1.0 > (Linux; U; Android 5.1; > 1201 Build/LMY47I)",#012 "Via: 1.1 > v1-akamaitech.net<http://v1-akamaitech.net>(ghost) > (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) > (AkamaiGHost), 1.1 > v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 > akamai.net<http://akamai.net>(ghost) > (AkamaiGHost)",#012 "X-Aka > Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started > Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child > starts > > > Thanks, > > Mahesh. > > > _______________________________________________ > varnish-misc mailing list > [email protected] > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc<//prolinks.rediffmailpro.com/cgi-bin/prored.cgi?red=https%3A%2F%2Fwww%2Evarnish%2Dcache%2Eorg%2Flists%2Fmailman%2Flistinfo%2Fvarnish%2Dmisc&rediffng=0>
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
