Thanks Michi for the reply. >>> Does 'shutdown' mean calling System.exit or throwing some kind of >>> exception? Some applications use ZooKeeper embedded in their JVM, and they >>> might not like ZooKeeper calling System.exit.
Thats good point. In my mind I'm thinking of graceful shutdown of server by calling ZooKeeperServer#shutdown() api. Thanks, Rakesh -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Michi Mutsuzaki Sent: 01 April 2014 14:52 To: [email protected] Cc: [email protected] Subject: Re: Thread handling +1 for shutting down on a critical thread death. Does 'shutdown' mean calling System.exit or throwing some kind of exception? Some applications use ZooKeeper embedded in their JVM, and they might not like ZooKeeper calling System.exit. --Michi On Mon, Mar 31, 2014 at 9:03 PM, Rakesh R <[email protected]> wrote: >>>> This is how I handle the critical threads in my client apps that use >>>> Zookeeper. >>>> Keep a reference to the thread and periodically make sure it's still alive >>>> and well - respawn it if it is not. > > Thanks Greg for the inputs. Please see ZK-1907, I've included an initial > proposal patch to kick start the discussions. > Another approach is simply shutdown if a critical thread dies, so the > monitoring tool can easily detect and take necessary actions. The proposed > patch is based on this approach. > > -Rakesh > > -----Original Message----- > From: Asta, Greg [mailto:[email protected]] > Sent: 31 March 2014 23:24 > To: [email protected]; [email protected] > Subject: RE: Thread handling > > " If we have a 'DeathWatcher 'or some other mechanism in place to monitor all > the critical threads. It can take a decision like - bring down the process if > required, or shutdown the quorumpeer and go for LE again etc. > Now the monitoring or management tool will knows about the situation and can > act upon. > > Appreciate any thoughts ?" > > This is how I handle the critical threads in my client apps that use > Zookeeper. Keep a reference to the thread and periodically make sure it's > still alive and well - respawn it if it is not. > > Thanks, > Greg > > > -----Original Message----- > From: Rakesh R [mailto:[email protected]] > Sent: Thursday, March 27, 2014 10:39 AM > To: [email protected]; [email protected] > Subject: Thread handling > > Hi All, > > Server has many critical threads running and co-ordinating each other like > RequestProcessor chains et. When going through each threads, most of them > having the similar structure like: > > public void run() { > try { > while(running) > // processing logic > } > } catch (InterruptedException e) { > LOG.error("Unexpected interruption", e); > } catch (RequestProcessorException e) { > LOG.error("Unexpected exception", e); > } catch (Exception e) { > LOG.error("Unexpected exception", e); > } > LOG.info("...exited loop!"); > } > > I feel, we could improve our threads in our system. From the design I could > see, there could be a chance of silently leaving the thread in case of any > exception(abnormal or any functional issue too) If this happens in the > production, the server would get hanged forever and will not be able to > deliver its role. > > If we have a 'DeathWatcher 'or some other mechanism in place to monitor all > the critical threads. It can take a decision like - bring down the process if > required, or shutdown the quorumpeer and go for LE again etc. > Now the monitoring or management tool will knows about the situation and can > act upon. > > Appreciate any thoughts ? > > Thanks in advance, > Rakesh R
