Dead2 opened a new issue #1775: Deadlock in shm statistics URL: https://github.com/apache/incubator-pagespeed-mod/issues/1775 This has been a problem in our systems for years now, but we only recently got around to doing some debugging on it. Since the mod_pagespeed EL7 rpm does not include debug symbols, we had to compile it ourselves too, causing a delay due to the hurdle of figuring out how getting repeatedly postponed. So, after it was compiled, it was fired up on one of our production servers (very hard to reproduce). It took a couple weeks or so and bam, apache was deadlocked. We got 4 core files, probably there should have been more, not certain what happened there, as this happened during the night-watch and I was not the one that restarted apache. Due to this bug, mod_pagespeed has been disabled on nearly all our production servers for nearly a year now. It looks like 3 of the backtraces want to obtain a token to optimize a .png, a .css and a .css file respectively, one of the backtraces seem to be idle. There should be more apache processes, but no such core files, so unfortunately we don't know what they were doing at the time. The server probably is up to ~200 processes at the time of the deadlock. A tiny trickle of connections still gets served during the deadlock, probably the lucky ones that get in after a locked connection times out and then only if they try to access a vhost without pagespeed (/rewrites) enabled. This server used - Apache 2.4.26 - mod_pagespeed 1.12.34.2 - MPM-ITK 2.4.7-04 - PHP 5.6.36 - apr 1.4.8 - apr-util 1.5.2 - glibc 2.17 Mpm-itk (not a mpm) is a module for the prefork mpm that makes each fork run as root (with limits) until it receives a connection so it knows what vhost and thus what user it needs to change to. This might possibly be a factor in this bug, but we have not found any evidence suggesting so. Relevant parts of global mod_pagespeed config below. What filters were enabled for the 3 specific vhosts found in the backtraces is unknown. ```apacheconf ModPagespeedEnableCachePurge on ModPagespeedCacheFlushPollIntervalSec 5 ModPagespeedLRUCacheKbPerProcess 768 ModPagespeedDefaultSharedMemoryCacheKB 262144 ModPagespeedShmMetadataCacheCheckpointIntervalSec 1800 ModPagespeedNumRewriteThreads 3 ModPagespeedNumExpensiveRewriteThreads 1 ModPagespeedRewriteRandomDropPercentage 10 ModPagespeedRewriteDeadlinePerFlushMs 16 ModPagespeedInPlaceResourceOptimization on ModPagespeedInPlaceRewriteDeadlineMs 12 ModPagespeedEnableFilters in_place_optimize_for_browser ``` root.core ```c++ #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007fdb162d8dcb in _L_lock_812 () from /lib64/libpthread.so.0 #2 0x00007fdb162d8c98 in __GI___pthread_mutex_lock (mutex=0x7fdac05e5e80) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007fdb0e2eab6c in net_instaweb::(anonymous namespace)::PthreadSharedMemMutex::Lock (this=0x563970318368) at pagespeed/kernel/thread/pthread_shared_mem.cc:67 #4 0x00007fdb0dd05670 in net_instaweb::ScopedMutex::ScopedMutex (this=0x7fdab01eece0, mutex=0x563970318368) at ./pagespeed/kernel/base/abstract_mutex.h:50 #5 0x00007fdb0de7dc41 in net_instaweb::MutexedScalar::AddHelper (this=0x563970306ee0, delta=1) at pagespeed/kernel/base/statistics.cc:90 #6 0x00007fdb0e569dfb in net_instaweb::UpDownTemplate<net_instaweb::SharedMemVariable>::AddHelper (this=0x563970306ed8, delta=1) at ./pagespeed/kernel/base/statistics_template.h:342 #7 0x00007fdb0de09f54 in net_instaweb::UpDownCounter::Add (this=0x563970306ed8, delta=1) at ./pagespeed/kernel/base/statistics.h:94 #8 0x00007fdb0e28466e in net_instaweb::NamedLockScheduleRewriteController::LockObtained (this=0x5639703f47e8, callback=0x7fdaa800e148, key="rc:rname/ic_rtIOJOYF7lak15A4Fzbf/https://www.SITE1.no/wp-content/uploads/sites/88/2016/11/logo1.png@[email protected]_";, named_lock=0x7fdaa8003278) at pagespeed/controller/named_lock_schedule_rewrite_controller.cc:93 #9 0x00007fdb0e2880da in net_instaweb::MemberFunction3<net_instaweb::NamedLockScheduleRewriteController, net_instaweb::Function*, std::string const, net_instaweb::NamedLock*>::Run (this=0x7fdaa800eab8) at ./pagespeed/kernel/base/function.h:236 #10 0x00007fdb0dfe709a in net_instaweb::Function::CallRun (this=0x7fdaa800eab8) at pagespeed/kernel/base/function.cc:51 #11 0x00007fdb0e0166b8 in net_instaweb::SchedulerBasedAbstractLock::LockTimedWaitStealOld (this=0x7fdaa8003278, wait_ms=0, steal_ms=30000, callback=0x7fdaa800eab8) at pagespeed/kernel/thread/scheduler_based_abstract_lock.cc:204 #12 0x00007fdb0e284a10 in net_instaweb::NamedLockScheduleRewriteController::ScheduleRewrite (this=0x5639703f47e8, key="rc:rname/ic_rtIOJOYF7lak15A4Fzbf/https://www.SITE1.no/wp-content/uploads/sites/88/2016/11/logo1.png@[email protected]_";, callback=0x7fdaa800e148) at pagespeed/controller/named_lock_schedule_rewrite_controller.cc:167 #13 0x00007fdb0e282ec2 in net_instaweb::(anonymous namespace)::ScheduleRewriteContextImpl::ScheduleRewriteContextImpl (this=0x7fdaa800e118, controller=0x5639703f47e8, callback=0x7fdaa800e0c8) at pagespeed/controller/in_process_central_controller.cc:80 #14 0x00007fdb0e28329f in net_instaweb::InProcessCentralController::ScheduleRewrite (this=0x5639703f47c0, callback=0x7fdaa800e0c8) at pagespeed/controller/in_process_central_controller.cc:149 #15 0x00007fdb0de3165e in net_instaweb::RewriteContext::ObtainLockForCreation (this=0x563970452eb8, server_context=0x56396fcb4388, callback=0x7fdaa800dd38) at net/instaweb/rewriter/rewrite_context.cc:1440 #16 0x00007fdb0de313a7 in net_instaweb::RewriteContext::OutputCacheMiss (this=0x563970452eb8) at net/instaweb/rewriter/rewrite_context.cc:1408 #17 0x00007fdb0de31193 in net_instaweb::RewriteContext::OutputCacheDone (this=0x563970452eb8, cache_result=0x7fdaa80032e8) at net/instaweb/rewriter/rewrite_context.cc:1380 #18 0x00007fdb0de4618b in net_instaweb::MemberFunction1<net_instaweb::RewriteContext, net_instaweb::RewriteContext::CacheLookupResult*>::Run (this=0x7fdaa8005058) at ./pagespeed/kernel/base/function.h:171 #19 0x00007fdb0dfe709a in net_instaweb::Function::CallRun (this=0x7fdaa8005058) at pagespeed/kernel/base/function.cc:51 #20 0x00007fdb0e00bcd3 in net_instaweb::QueuedWorkerPool::Run (this=0x5639703d2d58, sequence=0x5639704124d8, worker=0x56397043d3b8) at pagespeed/kernel/thread/queued_worker_pool.cc:157 #21 0x00007fdb0e012ba6 in net_instaweb::MemberFunction2<net_instaweb::QueuedWorkerPool, net_instaweb::QueuedWorkerPool::Sequence*, net_instaweb::QueuedWorker*>::Run ( this=0x56397042fbf8) at ./pagespeed/kernel/base/function.h:202 #22 0x00007fdb0dfe709a in net_instaweb::Function::CallRun (this=0x56397042fbf8) at pagespeed/kernel/base/function.cc:51 #23 0x00007fdb0e0199d6 in net_instaweb::Worker::WorkThread::Run (this=0x56397042bef8) at pagespeed/kernel/thread/worker.cc:85 #24 0x00007fdb0e2ed33b in net_instaweb::PthreadThreadImpl::InvokeRun (self_ptr=0x56397042bfa8) at pagespeed/kernel/thread/pthread_thread_system.cc:121 #25 0x00007fdb162d6e25 in start_thread (arg=0x7fdab01f0700) at pthread_create.c:308 #26 0x00007fdb15e0034d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ``` 1.core ```c++ #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007fdb162d8dcb in _L_lock_812 () from /lib64/libpthread.so.0 #2 0x00007fdb162d8c98 in __GI___pthread_mutex_lock (mutex=0x7fdac05e5e80) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007fdb0e2eab6c in net_instaweb::(anonymous namespace)::PthreadSharedMemMutex::Lock (this=0x563970318368) at pagespeed/kernel/thread/pthread_shared_mem.cc:67 #4 0x00007fdb0dd05670 in net_instaweb::ScopedMutex::ScopedMutex (this=0x7fdab01eece0, mutex=0x563970318368) at ./pagespeed/kernel/base/abstract_mutex.h:50 #5 0x00007fdb0de7dc41 in net_instaweb::MutexedScalar::AddHelper (this=0x563970306ee0, delta=1) at pagespeed/kernel/base/statistics.cc:90 #6 0x00007fdb0e569dfb in net_instaweb::UpDownTemplate<net_instaweb::SharedMemVariable>::AddHelper (this=0x563970306ed8, delta=1) at ./pagespeed/kernel/base/statistics_template.h:342 #7 0x00007fdb0de09f54 in net_instaweb::UpDownCounter::Add (this=0x563970306ed8, delta=1) at ./pagespeed/kernel/base/statistics.h:94 #8 0x00007fdb0e28466e in net_instaweb::NamedLockScheduleRewriteController::LockObtained (this=0x5639703f47e8, callback=0x7fdaa8002c58, key="rc:rname/cf_rtIOJOYF7lak15A4Fzbf/https://www.SITE2.no/wp-content/plugins/revslider/public/assets/css/settings.css?ver=201821101921@[email protected]_";, named_lock= 0x7fdaa8005f58) at pagespeed/controller/named_lock_schedule_rewrite_controller.cc:93 #9 0x00007fdb0e2880da in net_instaweb::MemberFunction3<net_instaweb::NamedLockScheduleRewriteController, net_instaweb::Function*, std::string const, net_instaweb::NamedLock*>::Run (this=0x7fdaa8006ad8) at ./pagespeed/kernel/base/function.h:236 #10 0x00007fdb0dfe709a in net_instaweb::Function::CallRun (this=0x7fdaa8006ad8) at pagespeed/kernel/base/function.cc:51 #11 0x00007fdb0e0166ef in net_instaweb::SchedulerBasedAbstractLock::LockTimedWaitStealOld (this=0x7fdaa8005f58, wait_ms=0, steal_ms=30000, callback=0x7fdaa8006ad8) at pagespeed/kernel/thread/scheduler_based_abstract_lock.cc:209 #12 0x00007fdb0e284a10 in net_instaweb::NamedLockScheduleRewriteController::ScheduleRewrite (this=0x5639703f47e8, key="rc:rname/cf_rtIOJOYF7lak15A4Fzbf/https://www.SITE2.no/wp-content/plugins/revslider/public/assets/css/settings.css?ver=201821101921@[email protected]_";, callback=0x7fdaa8002c58) at pagespeed/controller/named_lock_schedule_rewrite_controller.cc:167 #13 0x00007fdb0e282ec2 in net_instaweb::(anonymous namespace)::ScheduleRewriteContextImpl::ScheduleRewriteContextImpl (this=0x7fdaa8004388, controller=0x5639703f47e8, callback=0x7fdaa8005578) at pagespeed/controller/in_process_central_controller.cc:80 #14 0x00007fdb0e28329f in net_instaweb::InProcessCentralController::ScheduleRewrite (this=0x5639703f47c0, callback=0x7fdaa8005578) at pagespeed/controller/in_process_central_controller.cc:149 #15 0x00007fdb0de3165e in net_instaweb::RewriteContext::ObtainLockForCreation (this=0x563970569498, server_context=0x56396fcb4388, callback=0x7fdaa8000c38) at net/instaweb/rewriter/rewrite_context.cc:1440 #16 0x00007fdb0de313a7 in net_instaweb::RewriteContext::OutputCacheMiss (this=0x563970569498) at net/instaweb/rewriter/rewrite_context.cc:1408 #17 0x00007fdb0de31193 in net_instaweb::RewriteContext::OutputCacheDone (this=0x563970569498, cache_result=0x7fdaa80008c8) at net/instaweb/rewriter/rewrite_context.cc:1380 #18 0x00007fdb0de4618b in net_instaweb::MemberFunction1<net_instaweb::RewriteContext, net_instaweb::RewriteContext::CacheLookupResult*>::Run (this=0x7fdaa8000d58) at ./pagespeed/kernel/base/function.h:171 #19 0x00007fdb0dfe709a in net_instaweb::Function::CallRun (this=0x7fdaa8000d58) at pagespeed/kernel/base/function.cc:51 #20 0x00007fdb0e00bcd3 in net_instaweb::QueuedWorkerPool::Run (this=0x5639703d2d58, sequence=0x563970416188, worker=0x5639705273c8) at pagespeed/kernel/thread/queued_worker_pool.cc:157 #21 0x00007fdb0e012ba6 in net_instaweb::MemberFunction2<net_instaweb::QueuedWorkerPool, net_instaweb::QueuedWorkerPool::Sequence*, net_instaweb::QueuedWorker*>::Run ( this=0x5639703efc18) at ./pagespeed/kernel/base/function.h:202 #22 0x00007fdb0dfe709a in net_instaweb::Function::CallRun (this=0x5639703efc18) at pagespeed/kernel/base/function.cc:51 #23 0x00007fdb0e0199d6 in net_instaweb::Worker::WorkThread::Run (this=0x56397047a628) at pagespeed/kernel/thread/worker.cc:85 #24 0x00007fdb0e2ed33b in net_instaweb::PthreadThreadImpl::InvokeRun (self_ptr=0x56397047a6d8) at pagespeed/kernel/thread/pthread_thread_system.cc:121 #25 0x00007fdb162d6e25 in start_thread (arg=0x7fdab01f0700) at pthread_create.c:308 #26 0x00007fdb15e0034d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ``` 2.core ```c++ #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fdb0e2ed7d7 in net_instaweb::PthreadCondvar::Wait (this=0x5639702f6578) at pagespeed/kernel/thread/pthread_condvar.cc:44 #2 0x00007fdb0dfe27dd in net_instaweb::CheckingThreadSystem::CheckingCondvar::Wait (this=0x5639702f6418) at pagespeed/kernel/base/checking_thread_system.cc:50 #3 0x00007fdb0e0198dc in net_instaweb::Worker::WorkThread::GetNextTask (this=0x5639702f6338) at pagespeed/kernel/thread/worker.cc:64 #4 0x00007fdb0e0199e2 in net_instaweb::Worker::WorkThread::Run (this=0x5639702f6338) at pagespeed/kernel/thread/worker.cc:82 #5 0x00007fdb0e2ed33b in net_instaweb::PthreadThreadImpl::InvokeRun (self_ptr=0x56396fd665c8) at pagespeed/kernel/thread/pthread_thread_system.cc:121 #6 0x00007fdb162d6e25 in start_thread (arg=0x7fdaaf6c8700) at pthread_create.c:308 #7 0x00007fdb15e0034d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ``` 4.core ```c++ #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007fdb162d8dcb in _L_lock_812 () from /lib64/libpthread.so.0 #2 0x00007fdb162d8c98 in __GI___pthread_mutex_lock (mutex=0x7fdac05e5e80) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007fdb0e2eab6c in net_instaweb::(anonymous namespace)::PthreadSharedMemMutex::Lock (this=0x563970318368) at pagespeed/kernel/thread/pthread_shared_mem.cc:67 #4 0x00007fdb0dd05670 in net_instaweb::ScopedMutex::ScopedMutex (this=0x7fdab01eece0, mutex=0x563970318368) at ./pagespeed/kernel/base/abstract_mutex.h:50 #5 0x00007fdb0de7dc41 in net_instaweb::MutexedScalar::AddHelper (this=0x563970306ee0, delta=1) at pagespeed/kernel/base/statistics.cc:90 #6 0x00007fdb0e569dfb in net_instaweb::UpDownTemplate<net_instaweb::SharedMemVariable>::AddHelper (this=0x563970306ed8, delta=1) at ./pagespeed/kernel/base/statistics_template.h:342 #7 0x00007fdb0de09f54 in net_instaweb::UpDownCounter::Add (this=0x563970306ed8, delta=1) at ./pagespeed/kernel/base/statistics.h:94 #8 0x00007fdb0e28466e in net_instaweb::NamedLockScheduleRewriteController::LockObtained (this=0x5639703f47e8, callback=0x7fdaa8002e78, key="rc:rname/cf_rtIOJOYF7lak15A4Fzbf/https://SITE3.no/wp-content/plugins/woocommerce/assets/css/woocommerce-layout.css?ver=201762116833@A.@amssA_";, named_lock=0x7fdaa8004488) at pagespeed/controller/named_lock_schedule_rewrite_controller.cc:93 #9 0x00007fdb0e2880da in net_instaweb::MemberFunction3<net_instaweb::NamedLockScheduleRewriteController, net_instaweb::Function*, std::string const, net_instaweb::NamedLock*>::Run (this=0x7fdaa8005078) at ./pagespeed/kernel/base/function.h:236 #10 0x00007fdb0dfe709a in net_instaweb::Function::CallRun (this=0x7fdaa8005078) at pagespeed/kernel/base/function.cc:51 #11 0x00007fdb0e0166b8 in net_instaweb::SchedulerBasedAbstractLock::LockTimedWaitStealOld (this=0x7fdaa8004488, wait_ms=0, steal_ms=30000, callback=0x7fdaa8005078) at pagespeed/kernel/thread/scheduler_based_abstract_lock.cc:204 #12 0x00007fdb0e284a10 in net_instaweb::NamedLockScheduleRewriteController::ScheduleRewrite (this=0x5639703f47e8, key="rc:rname/cf_rtIOJOYF7lak15A4Fzbf/https://SITE3.no/wp-content/plugins/woocommerce/assets/css/woocommerce-layout.css?ver=201762116833@A.@amssA_";, callback=0x7fdaa8002e78) at pagespeed/controller/named_lock_schedule_rewrite_controller.cc:167 #13 0x00007fdb0e282ec2 in net_instaweb::(anonymous namespace)::ScheduleRewriteContextImpl::ScheduleRewriteContextImpl (this=0x7fdaa8002de8, controller=0x5639703f47e8, callback=0x7fdaa80009e8) at pagespeed/controller/in_process_central_controller.cc:80 #14 0x00007fdb0e28329f in net_instaweb::InProcessCentralController::ScheduleRewrite (this=0x5639703f47c0, callback=0x7fdaa80009e8) at pagespeed/controller/in_process_central_controller.cc:149 #15 0x00007fdb0de3165e in net_instaweb::RewriteContext::ObtainLockForCreation (this=0x56397043f6f8, server_context=0x56396fcb4388, callback=0x7fdaa8000a38) at net/instaweb/rewriter/rewrite_context.cc:1440 #16 0x00007fdb0de313a7 in net_instaweb::RewriteContext::OutputCacheMiss (this=0x56397043f6f8) at net/instaweb/rewriter/rewrite_context.cc:1408 #17 0x00007fdb0de31193 in net_instaweb::RewriteContext::OutputCacheDone (this=0x56397043f6f8, cache_result=0x7fdaa80008c8) at net/instaweb/rewriter/rewrite_context.cc:1380 #18 0x00007fdb0de4618b in net_instaweb::MemberFunction1<net_instaweb::RewriteContext, net_instaweb::RewriteContext::CacheLookupResult*>::Run (this=0x7fdaa8000d58) at ./pagespeed/kernel/base/function.h:171 #19 0x00007fdb0dfe709a in net_instaweb::Function::CallRun (this=0x7fdaa8000d58) at pagespeed/kernel/base/function.cc:51 #20 0x00007fdb0e00bcd3 in net_instaweb::QueuedWorkerPool::Run (this=0x5639703d2d58, sequence=0x563970412a58, worker=0x5639704b9378) at pagespeed/kernel/thread/queued_worker_pool.cc:157 #21 0x00007fdb0e012ba6 in net_instaweb::MemberFunction2<net_instaweb::QueuedWorkerPool, net_instaweb::QueuedWorkerPool::Sequence*, net_instaweb::QueuedWorker*>::Run ( this=0x5639704311c8) at ./pagespeed/kernel/base/function.h:202 #22 0x00007fdb0dfe709a in net_instaweb::Function::CallRun (this=0x5639704311c8) at pagespeed/kernel/base/function.cc:51 #23 0x00007fdb0e0199d6 in net_instaweb::Worker::WorkThread::Run (this=0x5639704e2ac8) at pagespeed/kernel/thread/worker.cc:85 #24 0x00007fdb0e2ed33b in net_instaweb::PthreadThreadImpl::InvokeRun (self_ptr=0x5639704e2b78) at pagespeed/kernel/thread/pthread_thread_system.cc:121 #25 0x00007fdb162d6e25 in start_thread (arg=0x7fdab01f0700) at pthread_create.c:308 #26 0x00007fdb15e0034d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
![https://www.SITE1.no/wp-content/uploads/sites/88/2016/11/logo1.png@[email protected]_"](https://www.SITE1.no/wp-content/uploads/sites/88/2016/11/logo1.png@x@.mss_"){kind=link}