On Tue, Nov 22, 2022 at 6:03 PM Luke Rotherfield <[email protected]> wrote: > > Hi Guys > > I am really struggling to debug this panic and wondered if you could give any > hints were I might start looking for answers. I have been trawling the > GitHub repo and have seen several people recommend upping the > thread_pool_stack value. Here is the panic and a few other bits of data that > I think are useful based on the issues I have been reading: > > varnishd[15482]: Child (15492) Panic at: Tue, 22 Nov 2022 16:10:26 GMT > > Wrong turn at cache/cache_main.c:284: > Signal 11 (Segmentation fault) received at 0xffff887fe000 si_code 2 > version = varnish-6.0.11 revision a3bc025c2df28e4a76e10c2c41217c9864e9963b, > vrt api = 7.1 > ident = > Linux,4.14.290-217.505.amzn2.aarch64,aarch64,-junix,-smalloc,-sfile,-sdefault,-hcritbit,epoll > now = 3120063.050288 (mono), 1669133426.954528 (real) > Backtrace: > 0x43f5b4: /usr/sbin/varnishd() [0x43f5b4] > 0x4a968c: /usr/sbin/varnishd(VAS_Fail+0x54) [0x4a968c] > 0x43a350: /usr/sbin/varnishd() [0x43a350] > 0xffff90fd7668: linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffff90fd7668] > 0xffff90c8f540: /lib64/libc.so.6(memset+0x100) [0xffff90c8f540] > 0x4b88f4: /usr/sbin/varnishd(deflateReset+0x48) [0x4b88f4] > 0x42e924: /usr/sbin/varnishd(VGZ_NewGzip+0x88) [0x42e924] > 0x42ebc0: /usr/sbin/varnishd() [0x42ebc0] > 0x42df28: /usr/sbin/varnishd(VFP_Open+0x98) [0x42df28] > 0x42b950: /usr/sbin/varnishd() [0x42b950] > thread = (cache-worker) > thr.req = (nil) { > }, > thr.busyobj = 0xffff3d040020 { > end = 0xffff3d050000, > retries = 0, > sp = 0xffff3c241a20 { > fd = 28, vxid = 32945, > t_open = 1669133426.953970, > t_idle = 1669133426.953970, > ws = 0xffff3c241a60 { > id = \"ses\", > {s, f, r, e} = {0xffff3c241aa0, +96, (nil), +344}, > }, > transport = HTTP/1 { > state = HTTP1::Proc > } > client = 172.31.47.149 50812 :80, > }, > worker = 0xffff832326c8 { > ws = 0xffff83232770 { > id = \"wrk\", > {s, f, r, e} = {0xffff83231e00, +0, (nil), +2040}, > }, > VCL::method = BACKEND_RESPONSE, > VCL::return = deliver, > VCL::methods = {BACKEND_FETCH, BACKEND_RESPONSE}, > }, > vfc = 0xffff3d041f30 { > failed = 0, > req = 0xffff3d040640, > resp = 0xffff3d040ab8, > wrk = 0xffff832326c8, > oc = 0xffff3b250640, > filters = { > gzip = 0xffff3d04a740 { > priv1 = (nil), > priv2 = 0, > closed = 0 > }, > V1F_STRAIGHT = 0xffff3d04a660 { > priv1 = 0xffff3d042600, > priv2 = 674132, > closed = 0 > }, > }, > obj_flags = 0x0, > }, > ws = 0xffff3d040058 { > id = \"bo\", > {s, f, r, e} = {0xffff3d041f78, +34832, (nil), +57472}, > }, > ws_bo = 0xffff3d0425e8, > http[bereq] = 0xffff3d040640 { > ws = 0xffff3d040058 { > [Already dumped, see above] > }, > hdrs { > \"GET\", > \"/build/admin/css/oro.css?v=7c08a284\", > \"HTTP/1.1\", > \"X-Forwarded-Proto: https\", > \"X-Forwarded-Port: 443\", > \"Host: london.paperstage.doverstreetmarket.com\", > \"X-Amzn-Trace-Id: Root=1-637cf472-4faa5da7246eda4f0a477811\", > \"User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36\", > \"X-Amz-Cf-Id: > WVvRZ7m_mF9MaQ88b0rJd_vEKVJo5YqQr6Povp9ODasHVw2FQop36w==\", > \"Via: 3.0 c1685d59e35fdb859ab8a1f97feb5652.cloudfront.net > (CloudFront)\", > \"Cookie: BAPID=mdmh5umu6v6l1qejvo0lbhse54; > __utma=58316727.170235450.1647004079.1669126271.1669131591.29; > __utmb=58316727.22.10.1669131591; __utmc=58316727; __utmt=1; __utmt_b=1; > __utmz=58316727.1665659005.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); > _csrf=5Inqz08yS1ACLDnJDFG8xoavis88G4BgHx4WpD6RuQM; > _ga=GA1.2.170235450.1647004079; > _ga_QNDMHZ1SBD=GS1.1.1667387720.14.1.1667387720.60.0.0; > _gcl_au=1.1.255577917.1662482672; _landing_page=%2F; _orig_referrer=; > _shopify_y=1531218a-8c0c-4153-86e1-37e06cc3a103; > _y=1531218a-8c0c-4153-86e1-37e06cc3a103; > df_preview=%7B%22preview_id%22%3A%221755%22%2C%22preview_type%22%3A%22version%22%2C%22preview_code%22%3A%22d82194c2b3c4f22164ca71d225673ff6%22%7D\", > \"Accept-Language: en-US,en;q=0.9,fr;q=0.8\", > \"Accept: text/css,*/*;q=0.1\", > \"Referer: > https://london.paperstage.doverstreetmarket.com/doverstreetadmin/pages/settings/1755/en\";, > \"Accept-Encoding: gzip, deflate, br\", > \"pragma: no-cache\", > \"cache-control: no-cache\", > \"sec-gpc: 1\", > \"sec-fetch-site: same-origin\", > \"sec-fetch-mode: no-cors\", > \"sec-fetch-dest: style\", > \"X-Forwarded-For: 32.217.249.1, 15.158.35.113, 172.31.47.149\", > \"X-Varnish: 32947\", > }, > }, > http[beresp] = 0xffff3d040ab8 { > ws = 0xffff3d040058 { > [Already dumped, see above] > }, > hdrs { > \"HTTP/1.1\", > \"200\", > \"OK\", > \"Server: nginx/1.20.0\", > \"Date: Tue, 22 Nov 2022 16:10:26 GMT\", > \"Content-Type: text/css\", > \"Content-Length: 674132\", > \"Last-Modified: Tue, 22 Nov 2022 15:28:13 GMT\", > \"Connection: keep-alive\", > \"Expires: Wed, 22 Nov 2023 16:10:26 GMT\", > \"Cache-Control: max-age=31536000, public, no-transform\", > \"Accept-Ranges: bytes\", > \"X-Url: /build/admin/css/oro.css?v=7c08a284\", > \"X-Host: london.paperstage.doverstreetmarket.com\", > }, > }, > objcore[fetch] = 0xffff3b250640 { > refcnt = 2, > flags = {busy, hfm, private}, > exp_flags = {}, > boc = 0xffff3b260160 { > refcnt = 2, > state = req_done, > vary = (nil), > stevedore_priv = (nil), > }, > exp = {1669133426.954405, -1.000000, 300.000000, 0.000000}, > objhead = 0xffff907d01d0, > stevedore = (nil), > }, > http_conn = 0xffff3d042600 { > fd = 30 (@0xffff3be704e4), > doclose = NULL, > ws = 0xffff3d040058 { > [Already dumped, see above] > }, > {rxbuf_b, rxbuf_e} = {0xffff3d042660, 0xffff3d0427a8}, > {pipeline_b, pipeline_e} = {0xffff3d0427a8, 0xffff3d04a660}, > content_length = 674132, > body_status = length, > first_byte_timeout = 180.000000, > between_bytes_timeout = 180.000000, > }, > flags = {do_gzip, do_stream, do_pass, uncacheable}, > director_req = 0xffff8ffc0190 { > vcl_name = default, > health = healthy, > admin_health = probe, changed = 1669131098.893551, > type = backend { > display_name = boot.default, > ipv4 = 127.0.0.1, > port = 8080, > hosthdr = 127.0.0.1, > n_conn = 1, > }, > }, > director_resp = director_req, > vcl = { > name = \"boot\", > busy = 5, > discard = 0, > state = auto, > temp = warm, > conf = { > syntax = \"40\", > srcname = { > \"/etc/varnish/default.vcl\", > \"Builtin\", > }, > }, > }, > }, > vmods = { > std = {Varnish 6.0.11 a3bc025c2df28e4a76e10c2c41217c9864e9963b, 0.0}, > }, > > param.show thread_pool_stack > 200 > thread_pool_stack > Value is: 128k [bytes] (default) > Minimum is: 128k > > > yum info jemalloc > Loaded plugins: extras_suggestions, langpacks, priorities, update-motd > 223 packages excluded due to repository priority protections > Installed Packages > Name : jemalloc > Arch : aarch64 > Version : 3.6.0 > Release : 1.el7 > Size : 374 k > Repo : installed > From repo : epel > Summary : General-purpose scalable concurrent malloc implementation > URL : http://www.canonware.com/jemalloc/ > License : BSD > Description : General-purpose scalable concurrent malloc(3) implementation. > : This distribution is the stand-alone "portable" implementation > of jemalloc. > > > Any insight you can offer would be greatly appreciated. I am going to try > setting the thread_pool_stack to 1024k today, however this error is pretty > intermittent so I may not know if it has resolved anything for a day or so.
Hi Luke, At first glance this does not look like something you could work around with a larger thread stack. If you have a github account, we can move the discussion there: https://github.com/varnishcache/varnish-cache/issues/3867 Best, Dridi _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
