Kislay Kumar created ZEPPELIN-3860:
--------------------------------------
Summary: Zeppelin Broken pipe due to non terminating
RemoteInterpreterProcess
Key: ZEPPELIN-3860
URL: https://issues.apache.org/jira/browse/ZEPPELIN-3860
Project: Zeppelin
Issue Type: Bug
Affects Versions: 0.7.3
Environment: HDP 2.6
Reporter: Kislay Kumar
This issue may be related to : ZEPPELIN-1832 (it had temporary fix)
Broken pipe exception is repeatedly coming for Zeppelin. I starts with
SocketException.
Some logs are as under:
WARNING: A servlet request to the URI https://xxyy:9995/api/login contains form
parameters in the request body but the request body has been consumed by the
servlet or a servlet filter accessing the request parameters. Only resource
methods using @FormParam will work as expected. Resource methods consuming the
request body by other means will not work as expected.
Exception in thread "Thread-815" Exception in thread "Thread-809"
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
at
org.apache.zeppelin.interpreter.InterpreterSetting.shutdownAndRemoveAllInterpreterGroups(InterpreterSetting.java:263)
at
org.apache.zeppelin.interpreter.InterpreterSettingManager$7.run(InterpreterSettingManager.java:1083)
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
at
org.apache.zeppelin.interpreter.InterpreterSetting.shutdownAndRemoveAllInterpreterGroups(InterpreterSetting.java:263)
at
org.apache.zeppelin.interpreter.InterpreterSettingManager$7.run(InterpreterSettingManager.java:1083)
Zeppelin is restarting
More logs from
INFO [2018-11-09 00:00:01,334] (\{Thread-825}
RemoteInterpreterProcess.java[dereference]:133) - Exception in
RemoteInterpreterProcess while synchronized dereference, can safely ignore
exception while client.shutdown() may terminates remote process
org.apache.thrift.transport.TTransportException
-----------------------------------------
DEBUG [2018-11-09 00:01:31,742] (\{Scheduler-1989335500}
WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@39aaeb44\{IDLE}
java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
DEBUG [2018-11-09 00:01:31,743] (\{qtp240166646-959}
AbstractConnection.java[onFillInterestedFailed]:168) -
HttpConnection@6b820a35[IDLE,DecryptedEndPoint@379cfbb3\{/10.111.1.138:60502<->9995,Open,in,out,-,-,30002/30000,HttpConnection}->SelectChannelEndPoint@7b0e8d49\{/10.111.1.138:60502<->9995,Open,in,out,-,-,1/30000,SslConnection}{io=1,kio=1,kro=1}][p=HttpParser\{s=CLOSED,0
of
-1},g=HttpGenerator\{s=START},c=HttpChannelOverHttp@5456b387\{r=9,c=false,a=IDLE,uri=}]
onFillInterestedFailed java.util.concurrent.TimeoutException: Idle timeout
expired: 30001/30000 ms
DEBUG [2018-11-09 00:01:31,743] (\{Scheduler-1989335500}
WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@cbde656\{IDLE}
java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
DEBUG [2018-11-09 00:01:31,744] (\{qtp240166646-975}
AbstractConnection.java[onFillInterestedFailed]:168) -
HttpConnection@671a181f[IDLE,DecryptedEndPoint@4bb91807\{/10.111.1.138:60503<->9995,Open,in,out,-,-,30001/30000,HttpConnection}->SelectChannelEndPoint@efc362c\{/10.111.1.138:60503<->9995,Open,in,out,-,-,0/30000,SslConnection}{io=1,kio=1,kro=1}][p=HttpParser\{s=CLOSED,0
of
-1},g=HttpGenerator\{s=START},c=HttpChannelOverHttp@524d84e3\{r=8,c=false,a=IDLE,uri=}]
onFillInterestedFailed java.util.concurrent.TimeoutException: Idle timeout
expired: 30000/30000 ms
DEBUG [2018-11-09 00:01:32,477] (\{Scheduler-1989335500}
WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@4c568d5d\{IDLE}
java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
---------------------------
INFO [2018-11-09 01:17:08,547] (\{Thread-824}
RemoteInterpreterProcess.java[dereference]:133) - Exception in
RemoteInterpreterProcess while synchronized dereference, can safely ignore
exception while client.shutdown() may terminates remote process
org.apache.thrift.transport.TTransportException
org.apache.thrift.transport.TTransportException
org.apache.thrift.transport.TTransportException
ERROR [2018-11-09 01:17:08,609] (\{qtp240166646-1125 -
/api/helium/suggest/2D2V1VPF3/20171128-114541_374495247}
ResourcePoolUtils.java[getAllResourcesExcept]:64) -
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
org.apache.thrift.transport.TTransportException
DEBUG [2018-11-09 01:17:09,424] (\{qtp240166646-949}
WriteFlusher.java[onFail]:460) - failed: WriteFlusher@248e2b43\{PENDING}
java.nio.channels.ClosedChannelException
INFO [2018-11-09 01:17:11,007] (\{Thread-1017}
RemoteInterpreterProcess.java[dereference]:133) - Exception in
RemoteInterpreterProcess while synchronized dereference, can safely ignore
exception while client.shutdown() may terminates remote process
org.apache.thrift.transport.TTransportException
--------------------------------------------------------------
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
