[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
@spektom Sorry, I didn't get your reply, I create another PR for it #1861 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
@purechoc Do you want to rebase it and continue work on it ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[purechoc]

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
LGTM. it's working nicely in my environment.  (zeppelin 0.7 + livy 2.1)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
Sorry for late response. @spektom Could you add some additional message of 
session recreation to be displayed in frontend.  Because if session is 
recreated, we may need to rerun all the paragraphs, without knowing session 
recreationusers may be confused for errors like some variables are not defined.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[gss2002]

Github user gss2002 commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
@spektom / @zjffdu @purechoc there is definitely an additional condition. 
Not sure if it's because the ConcurrentHashMaps are not being used correctly. 
But the exception doesn't get caught completely or correctly at times with the 
fix proposed here..


ERROR [2016-10-19 14:19:57,638] ({pool-2-thread-11} 
LivyHelper.java[executeHTTP]:378) - Error with 404 StatusCode: "Session '9' not 
found."
ERROR [2016-10-19 14:19:57,638] ({pool-2-thread-11} 
LivyHelper.java[interpretInput]:229) - error in interpretInput
org.apache.zeppelin.livy.LivyHelper$LivyNoSessionException: Session not 
found, Livy server would have restarted, or lost session.
at 
org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:312)
at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:241)
at 
org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189)
at 
org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106)
at 
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at 
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
 INFO [2016-10-19 14:19:57,639] ({pool-2-thread-11} 
SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1476901197622 
finished by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
 INFO [2016-10-19 14:19:57,819] ({pool-2-thread-34} 
SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1476901197819 
started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
ERROR [2016-10-19 14:19:57,835] ({pool-2-thread-34} 
LivyHelper.java[executeHTTP]:378) - Error with 404 StatusCode: "Session '9' not 
found."
ERROR [2016-10-19 14:19:57,835] ({pool-2-thread-34} 
LivyHelper.java[interpretInput]:229) - error in interpretInput
org.apache.zeppelin.livy.LivyHelper$LivyNoSessionException: Session not 
found, Livy server would have restarted, or lost session.
at 
org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:312)
at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:241)
at 
org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189)
at 
org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106)
at 
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at 
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
 INFO [2016-10-19 14:19:57,836] ({pool-2-thread-34} 
SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1476901197819 
finished by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[gss2002]

Github user gss2002 commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
@spektom  just tested against my build by catching the exception and 
rethrowing.. It definitely solves the issue.

} catch (LivyNoSessionException e) {
throw e;
} catch (Exception e) {
  LOGGER.error("error in interpretInput", e);
  return new InterpreterResult(Code.ERROR, e.getMessage());
}
  }

  public InterpreterResult interpret(String stringLines,

The code base I'm using is here with you patch and a few of @zjffdu patches 
and one of mine for NestedRuntimeException for 404's with KerberosTemplate: 
https://github.com/gss2002/zeppelin/blob/GSS_PROD_BUILD/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[gss2002]

Github user gss2002 commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
@spektom I think what happens here is this code fires.. which has nothing 
to do with the fix here.. 

in LivySparkInterpreter:
  return livyHelper.interpretInput(line, interpreterContext, 
userSessionMap, out,
  sessionId2AppIdMap.get(sessionId), 
sessionId2WebUIMap.get(sessionId), displayAppInfo);

That gets called before the NoSessionException occurs.. And then in 
LivyHelper --> public InterpreterResult interpretInput grabs the exception 
and handles it. I guess the question is can we do a rootcause on this and 
rethrow?

} catch (Exception e) {
  LOGGER.error("error in interpretInput", e);
  return new InterpreterResult(Code.ERROR, e.getMessage());
}


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[spektom]

Github user spektom commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
@gss2002 are you using latest code from the branch? I can't find 
org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189) 
invocation at 
org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106)
 as it's shown in your stack trace.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[gss2002]

Github user gss2002 commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
@spektom this fix is good. Did some extensive load testing with it this AM 
and it solves the session expiration issues. Thanks for the contribution


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[gss2002]

Github user gss2002 commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
@spektom and @zjffdu is it possible to add a null check also. I've been 
doing some debugging over the past few days and certain situations can cause 
nulls to be returned and in theory if a null is being returned the session is 
dead.

**So instead of this:**
 if (json.matches("^(\")?Session (\'[0-9]\' )?not found(.?\"?)$")) {
   throw new LivyNoSessionException();

**This:**
boolean clearSession = false;
if (json != null) {
  if (json.matches("^(\")?Session (\'[0-9]\' )?not found(.?\"?)$")) {
clearSession = true;
  }
} else {
  clearSession = true;
}
if (clearSession) {
  throw new LivyNoSessionException();
}


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[spektom]

Github user spektom commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
Of course, it would be great.
On Fri, Sep 23, 2016 at 06:28 Jeff Zhang  wrote:

> @spektom  Thanks for the contribution, I am
> working on a PR about integration test of livy interpreter, could you mind
> to wait for that PR and then add integration test to that ?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1447
  
@spektom Thanks for the contribution, I am working on a PR about 
integration test of livy interpreter, could you mind to wait for that PR and 
then add integration test to that ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---