A lot of the Geode thread pools are defined in ClusterDistributionManager. Most of these use custom ThreadPoolExecutors like:
SerialQueuedExecutorWithDMStats PooledExecutorWithDMStats FunctionExecutionPooledExecutor These classes all extend ThreadPoolExecutor and override beforeExecute and afterExecute. These methods are currently used by helper classes to update the stats before and after a thread executes. Potentially these same methods could be used to add and remove a thread from a monitor. For example, there could be a FunctionExecutionThreadMonitor that is created as part of the FunctionExecutionPooledExecutor whose job it would be to monitor FunctionExecution threads. The beforeExecute method would add the thread to the monitor; the afterExecute would remove the thread from the monitor. I would be mindful about the performance impact of adding these monitors, though. Thanks, Barry Oglesby On Wed, Feb 21, 2018 at 11:41 AM, Gregory Vortman < gregory.vort...@amdocs.com> wrote: > That's the point exactly to have a single very thin and generic mechanism > to cover all threads/threads pool. Nothing is specific in this solution. > Regards > > > -----Original Message----- > From: Jason Huynh [jhu...@pivotal.io] > Received: Wednesday, 21 Feb 2018, 20:54 > To: d...@geode.apache.org [d...@geode.apache.org] > CC: user@geode.apache.org [user@geode.apache.org] > Subject: Re: [Proposal] Thread monitoring mechanism > > I am assuming this would be for all thread/thread pools and not specific > to Function threads. I wonder what the impact would be for put/get > operations or are we going to target specific operations. > > > > On Tue, Feb 20, 2018 at 1:04 AM Gregory Vortman < > gregory.vort...@amdocs.com<mailto:gregory.vort...@amdocs.com>> wrote: > Hello team, > One of the most severe issues hitting our real time application is thread > stuck for multiple reasons, such as long lasting locks, deadlocks, threads > which wait for reply forever in case of packet drop issue etc... > Such kind of stuck are under Radar of the existing system health check > methods. > In mission critical applications, this will be resulted as an immediate > outage. > > As a short we are implementing kind of internal watch dog mechanism for > stuck detector: > There is a registration object > Function executor having start/end hooks to > register/unregister the thread via the registration object > Customized Monitoring scheduled thread is spawned on startup. The thread > to wake up every N seconds, to scan the registration map and to detect > unregistered threads for a long time (configurable). > Once such threads has been detected, process stack is taken and thread > stack statistic metric is provided. > > This helps us to monitor, detect and take fast decision about the action > which should be taken - usually it is member bounce decision (consistency > issue is possible, in our case it is better than deny of service). > The above solution is not touching GEODE core code, but implemented in > boundaries of customized code only. > > I would like to raise a proposal to introduce a long term generic thread > monitoring mechanism, to detect threads which are stuck for any reason. > To maintain a monitoring object having a start/end methods to be invoked > similarly to FunctionStats.startFunctionExecution and FunctionStats. > endFunctionExecution. > > Your feedback would be appreciated > > Thank you for cooperation. > Best regards! > > Gregory Vortman > > This message and the information contained herein is proprietary and > confidential and subject to the Amdocs policy statement, > > you may review at https://www.amdocs.com/about/email-disclaimer < > https://www.amdocs.com/about/email-disclaimer> > This message and the information contained herein is proprietary and > confidential and subject to the Amdocs policy statement, > > you may review at https://www.amdocs.com/about/email-disclaimer < > https://www.amdocs.com/about/email-disclaimer> >
