Kislay Kumar created ZEPPELIN-3860: -------------------------------------- Summary: Zeppelin Broken pipe due to non terminating RemoteInterpreterProcess Key: ZEPPELIN-3860 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3860 Project: Zeppelin Issue Type: Bug Affects Versions: 0.7.3 Environment: HDP 2.6 Reporter: Kislay Kumar
This issue may be related to : ZEPPELIN-1832 (it had temporary fix) Broken pipe exception is repeatedly coming for Zeppelin. I starts with SocketException. Some logs are as under: WARNING: A servlet request to the URI https://xxyy:9995/api/login contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected. Exception in thread "Thread-815" Exception in thread "Thread-809" java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) at java.util.HashMap$ValueIterator.next(HashMap.java:1466) at org.apache.zeppelin.interpreter.InterpreterSetting.shutdownAndRemoveAllInterpreterGroups(InterpreterSetting.java:263) at org.apache.zeppelin.interpreter.InterpreterSettingManager$7.run(InterpreterSettingManager.java:1083) java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) at java.util.HashMap$ValueIterator.next(HashMap.java:1466) at org.apache.zeppelin.interpreter.InterpreterSetting.shutdownAndRemoveAllInterpreterGroups(InterpreterSetting.java:263) at org.apache.zeppelin.interpreter.InterpreterSettingManager$7.run(InterpreterSettingManager.java:1083) Zeppelin is restarting More logs from INFO [2018-11-09 00:00:01,334] (\{Thread-825} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process org.apache.thrift.transport.TTransportException ----------------------------------------- DEBUG [2018-11-09 00:01:31,742] (\{Scheduler-1989335500} WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@39aaeb44\{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms DEBUG [2018-11-09 00:01:31,743] (\{qtp240166646-959} AbstractConnection.java[onFillInterestedFailed]:168) - HttpConnection@6b820a35[IDLE,DecryptedEndPoint@379cfbb3\{/10.111.1.138:60502<->9995,Open,in,out,-,-,30002/30000,HttpConnection}->SelectChannelEndPoint@7b0e8d49\{/10.111.1.138:60502<->9995,Open,in,out,-,-,1/30000,SslConnection}{io=1,kio=1,kro=1}][p=HttpParser\{s=CLOSED,0 of -1},g=HttpGenerator\{s=START},c=HttpChannelOverHttp@5456b387\{r=9,c=false,a=IDLE,uri=}] onFillInterestedFailed java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms DEBUG [2018-11-09 00:01:31,743] (\{Scheduler-1989335500} WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@cbde656\{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms DEBUG [2018-11-09 00:01:31,744] (\{qtp240166646-975} AbstractConnection.java[onFillInterestedFailed]:168) - HttpConnection@671a181f[IDLE,DecryptedEndPoint@4bb91807\{/10.111.1.138:60503<->9995,Open,in,out,-,-,30001/30000,HttpConnection}->SelectChannelEndPoint@efc362c\{/10.111.1.138:60503<->9995,Open,in,out,-,-,0/30000,SslConnection}{io=1,kio=1,kro=1}][p=HttpParser\{s=CLOSED,0 of -1},g=HttpGenerator\{s=START},c=HttpChannelOverHttp@524d84e3\{r=8,c=false,a=IDLE,uri=}] onFillInterestedFailed java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms DEBUG [2018-11-09 00:01:32,477] (\{Scheduler-1989335500} WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@4c568d5d\{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms --------------------------- INFO [2018-11-09 01:17:08,547] (\{Thread-824} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process org.apache.thrift.transport.TTransportException org.apache.thrift.transport.TTransportException org.apache.thrift.transport.TTransportException ERROR [2018-11-09 01:17:08,609] (\{qtp240166646-1125 - /api/helium/suggest/2D2V1VPF3/20171128-114541_374495247} ResourcePoolUtils.java[getAllResourcesExcept]:64) - org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) Caused by: java.net.ConnectException: Connection refused (Connection refused) org.apache.thrift.transport.TTransportException DEBUG [2018-11-09 01:17:09,424] (\{qtp240166646-949} WriteFlusher.java[onFail]:460) - failed: WriteFlusher@248e2b43\{PENDING} java.nio.channels.ClosedChannelException INFO [2018-11-09 01:17:11,007] (\{Thread-1017} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process org.apache.thrift.transport.TTransportException -------------------------------------------------------------- -- This message was sent by Atlassian JIRA (v7.6.3#76005)