[jira] [Created] (ZEPPELIN-4133) Idle sessions are no longer being closed even though TimeoutLifecycleManagement is configured properly

[Harel Ben Attia (JIRA)]

Harel Ben Attia created ZEPPELIN-4133:
-

 Summary: Idle sessions are no longer being closed even though 
TimeoutLifecycleManagement is configured properly
 Key: ZEPPELIN-4133
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4133
 Project: Zeppelin
  Issue Type: Bug
  Components: zeppelin-zengine
Affects Versions: 0.8.0
Reporter: Harel Ben Attia


Sometimes the timer thread which is initialized by the TimeoutLifecycleManager 
crashes. This leads to zeppelin not clearing any timedout sessions until a 
server restart is performed.

The underlying mechanism uses a java Timer (inside 
TimeoutLifecycleManager.java). We are not sure what had caused the TimerThread 
thread to crash, since it swallows exceptions/errors. However, a jstack 
comparison between the server and another zeppelin server which was working 
well had shown that there is a missing TimerThread thread in the problematic 
server (1 TimerThread thread vs. 2 TimerThreads). After restarting the zeppelin 
server, jstack started showing 2 TimerThread threads in its output.

Running on jdk 1.8.0_151, ubuntu 14.04, EC2.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ZEPPELIN-3401) Deadlock while restarting interpreter

[Harel Ben Attia (JIRA)]

Harel Ben Attia created ZEPPELIN-3401:
-

 Summary: Deadlock while restarting interpreter
 Key: ZEPPELIN-3401
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3401
 Project: Zeppelin
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

linux info: 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 
x86_64 x86_64 x86_64 GNU/Linux
Reporter: Harel Ben Attia
 Attachments: jstack.out

After running properly for ~15 days, the zeppelin UI started at some point to 
hang forever when restarting an interpreter. trying to analyze this shows that 
the server is deadlocked. The jstack output is not descriptive enough, because 
running it without -F didn't work, but shows a deadlock between two qtp threads.

The status of the blocked threads seems very similar to ZEPPELIN-1834, with 
many getParagraph/s() calls, but can't really ascertain that it's the same bug, 
especially with the fact that ZEPPELIN-1834 seems to be marked as solved for 
0.7.0.

Only a zeppelin restart solved the issue.

We're running in user-scoped sessions (spark interpreter).

We're running zeppelin-0.8.0-SNAPSHOT from the branch-0.8 branch commit 
714eb963d6b58df88239830ee487fe19613d1911 . Additional commits in that branch 
since then do not seem to be related.

 

Will be glad to provide more info if needed.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)