merlimat commented on a change in pull request #13130:
URL: https://github.com/apache/pulsar/pull/13130#discussion_r762164290



##########
File path: 
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
##########
@@ -317,7 +346,33 @@ public void invalidateAll() {
      */
     protected void execute(Runnable task, CompletableFuture<?> future) {
         try {
-            executor.execute(task);
+            // Wrap the original task, so we can record the thread on which it 
is running
+            TaskWrapper taskWrapper = new TaskWrapper(task);
+            executorWatchDog.execute(() -> {

Review comment:
       I think that having a wrapper on each task is not the best option here, 
since we have a single thread where we're making all the blocking calls on the 
future. 
   
   Also, a 5 delay in a metadata operation does not necessary indicate that 
there is a deadlock (eg: a particular request could be slow while there is a 
leader election happening).
   
   Instead, we could have the watchdog thread to execute a dummy task in the 
`metadata-store` executor. If that task cannot be scheduled for, say, 30sec, 
then we can be quite confident that there is indeed a deadlock, in which case 
we should dump the stack traces and kill the broker, since there's no way for 
it to fix itself up.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to