gabrywu opened a new issue #1947: [BUG] ProcessUtils.kill blocked
URL: https://github.com/apache/incubator-dolphinscheduler/issues/1947
 
 
   ProcessUtils.kill method always invoke killYarnJob ,no matter what's the 
task type, why ?
   ```
   public static void kill(TaskInstance taskInstance) {
       try {
         int processId = taskInstance.getPid();
         if(processId == 0 ){
             logger.error("process kill failed, process id :{}, task id:{}",
                     processId, taskInstance.getId());
             return ;
         }
   
         String cmd = String.format("kill -9 %s", getPidsStr(processId));
   
         logger.info("process id:{}, cmd:{}", processId, cmd);
   
         OSUtils.exeCmd(cmd);
   
         // find log and kill yarn job
         killYarnJob(taskInstance);
   
       } catch (Exception e) {
         logger.error("kill failed : " + e.getMessage(), e);
       }
     }
   ```
   
   In our product environment,the killYarnJob blocked, so all the process 
instance cannot be killed.
    The jstack log:
   
   "Worker-Kill-Thread-Executor" #43 daemon prio=5 os_prio=0 
tid=0x00007f0ee5972800 nid=0x15084 waiting on condition [0x00007f0e2c6af000]
      java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004379b7a90> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:623)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:122)
        at 
cn.escheduler.rpc.LogViewServiceGrpc$LogViewServiceBlockingStub.viewLog(LogViewServiceGrpc.java:321)
        at cn.escheduler.server.rpc.LogClient.viewLog(LogClient.java:97)
        at 
cn.escheduler.server.utils.ProcessUtils.killYarnJob(ProcessUtils.java:303)
        at cn.escheduler.server.utils.ProcessUtils.kill(ProcessUtils.java:270)
        at cn.escheduler.server.worker.WorkerServer$3.run(WorkerServer.java:282)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to