[GitHub] [incubator-livy] yantzu commented on a change in pull request #195: [LIVY-634] To support reuse hive PasswdAuthenticationProvider implementation in ThriftServer

2019-09-04 Thread GitBox
yantzu commented on a change in pull request #195: [LIVY-634] To support reuse 
hive PasswdAuthenticationProvider implementation in ThriftServer
URL: https://github.com/apache/incubator-livy/pull/195#discussion_r320583932
 
 

 ##
 File path: thriftserver/server/pom.xml
 ##
 @@ -56,6 +56,21 @@
 
   
 
+
+  org.apache.hive
+  hive-common
+  ${hive.version}
+  
 
 Review comment:
   thanks. issue fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] jerryshao commented on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
jerryshao commented on issue #223: [LIVY-658]RSCDriver should catch exception 
if cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-527762054
 
 
   CC @yiheng @runzhiwang please help to review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] runzhiwang edited a comment on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
runzhiwang edited a comment on issue #223: [LIVY-658]RSCDriver should catch 
exception if cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-527766053
 
 
   It's easy to find the NullPointerException was caused by canceling job 
whether catch or not ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] runzhiwang commented on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
runzhiwang commented on issue #223: [LIVY-658]RSCDriver should catch exception 
if cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-527766053
 
 
   It's easy to find the exception was caused by canceling job whether catch or 
not ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yantzu commented on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
yantzu commented on issue #223: [LIVY-658]RSCDriver should catch exception if 
cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-527774511
 
 
   Without catch cancel exception, the original 
exception("scala.refrect.internal.FactalError: object Predef does not have a 
member classOf" in the screen-shoot) is concealed by NullPointerException, it 
is not easy to find application error root cause(which is FactalError in this 
case). @runzhiwang 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] runzhiwang opened a new pull request #222: [LIVY-647]Fix travis failed on batch session should not be gc-ed until application is finished

2019-09-04 Thread GitBox
runzhiwang opened a new pull request #222: [LIVY-647]Fix travis failed on batch 
session should not be gc-ed until application is finished
URL: https://github.com/apache/incubator-livy/pull/222
 
 
   ## What changes were proposed in this pull request?
   
   Fix travis failed on "batch session should not be gc-ed until application is 
finished"
   
   The cause of failed is as follows:
   
   1. When create BatchSessionManager, the GarbageCollector thread will be 
created, which collect garbage according to session.state. However the session 
was mocked, and the test thread execute doReturn(s).when(session).state several 
times, if the test thread do half of the statement 
doReturn(s).when(session).state, and GarbageCollector thread check 
session.state at this time, the exception will threw by GarbageCollector thread.
   
   2. So the fix avoid executing doReturn(s).when(session).state after session 
has been registered into SessionManager.
   
   ## How was this patch tested?
   
   Existed UT and IT.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] jerryshao closed pull request #222: [LIVY-647]Fix travis failed on batch session should not be gc-ed until application is finished

2019-09-04 Thread GitBox
jerryshao closed pull request #222: [LIVY-647]Fix travis failed on batch 
session should not be gc-ed until application is finished
URL: https://github.com/apache/incubator-livy/pull/222
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] runzhiwang edited a comment on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
runzhiwang edited a comment on issue #223: [LIVY-658]RSCDriver should catch 
exception if cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-527766053
 
 
   Is it easy to find the NullPointerException was caused by canceling job 
whether catch or not ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] runzhiwang edited a comment on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
runzhiwang edited a comment on issue #223: [LIVY-658]RSCDriver should catch 
exception if cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-52282
 
 
   @yantzu Is the original exception helpful to find the reason is fail to 
startup ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] fdeantoni commented on issue #188: LIVY-615: Set context path using property SERVER_BASE_PATH.

2019-09-04 Thread GitBox
fdeantoni commented on issue #188: LIVY-615: Set context path using property 
SERVER_BASE_PATH.
URL: https://github.com/apache/incubator-livy/pull/188#issuecomment-527779827
 
 
   Hi all, any chance this will be merged into master? We are currently using 
the forked branch but would prefer to use master...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] jerryshao closed pull request #222: [LIVY-647]Fix travis failed on batch session should not be gc-ed until application is finished

2019-09-04 Thread GitBox
jerryshao closed pull request #222: [LIVY-647]Fix travis failed on batch 
session should not be gc-ed until application is finished
URL: https://github.com/apache/incubator-livy/pull/222
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yantzu commented on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
yantzu commented on issue #223: [LIVY-658]RSCDriver should catch exception if 
cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-527770727
 
 
   Sorry @runzhiwang , I cannot catch your point, could you please explain more?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yiheng opened a new pull request #224: [LIVY-633][Server] session should not be gc-ed for long running queries

2019-09-04 Thread GitBox
yiheng opened a new pull request #224: [LIVY-633][Server] session should not be 
gc-ed for long running queries
URL: https://github.com/apache/incubator-livy/pull/224
 
 
   ## What changes were proposed in this pull request?
   Currently, Livy records the last activity time of the session before 
statement execution. If a statement runs too long, exceeding then the session 
timeout, the session will be garbage collected after the statement execution.
   
   This should not be the expected behavior. The statement execution time 
should not be count into idle. We should update the last activity time after 
the statement execution.
   
   In this patch, we add a replLastActivity field into the rscClient, which 
will be updated when the repl state changes. So when session changes its state 
from busy to idle, this field will catch the time and finally reflect on the 
session last activity.
   
   ## How was this patch tested?
   Manual test. Also, add a new unit test.
   
   Existing unit tests and integration tests.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] runzhiwang commented on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
runzhiwang commented on issue #223: [LIVY-658]RSCDriver should catch exception 
if cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-527771884
 
 
   @yantzu If do not catch the exception, it's also easy to find cancel job 
will cause exception. Is it necessary to catch ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yantzu commented on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
yantzu commented on issue #223: [LIVY-658]RSCDriver should catch exception if 
cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-527783259
 
 
   original exception is the cause of application failure. @runzhiwang 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yantzu commented on a change in pull request #207: [LIVY-645]Add Session Name, Owner, Proxy User information to Web UI

2019-09-04 Thread GitBox
yantzu commented on a change in pull request #207: [LIVY-645]Add Session Name, 
Owner, Proxy User information to Web UI
URL: https://github.com/apache/incubator-livy/pull/207#discussion_r320620874
 
 

 ##
 File path: 
server/src/main/scala/org/apache/livy/server/batch/BatchSessionServlet.scala
 ##
 @@ -73,8 +75,8 @@ class BatchSessionServlet(
   } else {
 Nil
   }
-BatchSessionView(session.id, session.name, session.state.toString, 
session.appId,
-  session.appInfo, logs)
+BatchSessionView(session.id, session.name, session.owner, 
session.proxyUser,
+  session.state.toString, session.appId, session.appInfo, logs)
 
 Review comment:
   Hi @jerryshao , sorry for late. My consideration is:
   1, I use Option[String] as proxyUser type is because BatchRecoveryMetadata's 
proxyUser is Option[String]. And name, appId are Option[String], my purpose it 
to make them consistent. Interactive session is a little different because its 
SessionInfo class is Java
   2, The only field batch session doesn't have is "kind", which is a legacy 
field, batch session doesn't have the field information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] jerryshao commented on a change in pull request #209: [LIVY-640] Add tests for ThriftServer

2019-09-04 Thread GitBox
jerryshao commented on a change in pull request #209: [LIVY-640] Add tests for 
ThriftServer
URL: https://github.com/apache/incubator-livy/pull/209#discussion_r320614092
 
 

 ##
 File path: pom.xml
 ##
 @@ -1009,6 +1007,20 @@
   
 
   
+
 
 Review comment:
   If there's no timeout currently, I would suggest to not change here and in 
travis file, let's do it in a separate PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] captainzmc commented on issue #209: [LIVY-640] Add tests for ThriftServer

2019-09-04 Thread GitBox
captainzmc commented on issue #209: [LIVY-640] Add tests for ThriftServer
URL: https://github.com/apache/incubator-livy/pull/209#issuecomment-527895051
 
 
   fix


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] shanyu commented on issue #212: [LIVY-11] Enable HA support

2019-09-04 Thread GitBox
shanyu commented on issue #212: [LIVY-11] Enable HA support
URL: https://github.com/apache/incubator-livy/pull/212#issuecomment-528010862
 
 
   > > Another question is if the current leader crash and a new leader is 
elected, should the session state in the old leader be restored in the new 
leader?
   > 
   > The code does not do the state synchronization related to the Session 
State. Is it a good choice to put it in zk?
   
   This is already implemented previously. You can set 
livy.server.recovery.state-store to "zookeeper" and 
livy.server.recovery.state-store.url to zookeeper quorum. This PR mainly 
concerns the leader election part for the active-passive Livy Server.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] captainzmc commented on a change in pull request #209: [LIVY-640] Add tests for ThriftServer

2019-09-04 Thread GitBox
captainzmc commented on a change in pull request #209: [LIVY-640] Add tests for 
ThriftServer
URL: https://github.com/apache/incubator-livy/pull/209#discussion_r320725961
 
 

 ##
 File path: pom.xml
 ##
 @@ -1009,6 +1007,20 @@
   
 
   
+
 
 Review comment:
   Now we really don't have a timeout problem after simplifying the test. I 
will restore this place. If a timeout occurs later, we can modify it in a 
separate PR


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] runzhiwang opened a new pull request #225: [LIVY-657]Fix travis failed on should not create sessions with duplicate names

2019-09-04 Thread GitBox
runzhiwang opened a new pull request #225: [LIVY-657]Fix travis failed on 
should not create sessions with duplicate names
URL: https://github.com/apache/incubator-livy/pull/225
 
 
   
   
   ## What changes were proposed in this pull request?
   
   Fix travis failed on "should not create sessions with duplicate names"
   
   The cause of failed is as follows:
   
   1. When session2 was stopped, it will call stop() method in Session.scala 
asynchronously by Scala Future. So when assert(session2.stopped), if the stop() 
method has not been called, the test will fail.
   
   ## How was this patch tested?
   
   Existed UT and IT.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] shanyu commented on a change in pull request #212: [LIVY-11] Enable HA support

2019-09-04 Thread GitBox
shanyu commented on a change in pull request #212: [LIVY-11] Enable HA support
URL: https://github.com/apache/incubator-livy/pull/212#discussion_r320942152
 
 

 ##
 File path: 
server/src/main/scala/org/apache/livy/server/CuratorElectorService.scala
 ##
 @@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.server
+
+import java.io.Closeable
+import java.io.IOException
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.AtomicInteger
+
+import org.apache.curator.framework.CuratorFramework
+import org.apache.curator.framework.CuratorFrameworkFactory
+import org.apache.curator.framework.recipes.leader.LeaderLatch
+import org.apache.curator.framework.recipes.leader.LeaderLatchListener
+import org.apache.curator.retry.RetryNTimes
+
+import org.apache.livy.{LivyConf, Logging}
+import org.apache.livy.LivyConf.Entry
+
+object CuratorElectorService {
+  val HA_KEY_PREFIX_CONF = Entry("livy.server.ha.key-prefix", "livy_ha")
+  val HA_RETRY_CONF = Entry("livy.server.ha.retry-policy", "5,100")
+}
+
+class CuratorElectorService(livyConf : LivyConf, livyServer : LivyServer)
+  extends LeaderLatchListener
+  with Logging
+{
+
+  import CuratorElectorService._
+
+  val haAddress = livyConf.get(LivyConf.HA_ZOOKEEPER_URL)
+  require(!haAddress.isEmpty, s"Please configure 
${LivyConf.HA_ZOOKEEPER_URL.key}.")
+  val haKeyPrefix = livyConf.get(HA_KEY_PREFIX_CONF)
+  val retryValue = livyConf.get(HA_RETRY_CONF)
+  // a regex to match patterns like "m, n" where m and n both are integer 
values
+  val retryPattern = """\s*(\d+)\s*,\s*(\d+)\s*""".r
+  val retryPolicy = retryValue match {
+case retryPattern(n, sleepMs) => new RetryNTimes(n.toInt, sleepMs.toInt)
+case _ => throw new IllegalArgumentException(
+  s"$HA_KEY_PREFIX_CONF contains bad value: $retryValue. " +
+"Correct format is ,. e.g. 
5,100")
+  }
+
+  val client: CuratorFramework = CuratorFrameworkFactory.newClient(haAddress, 
retryPolicy)
+  val leaderKey = s"/$haKeyPrefix/leader"
+
+  var server : LivyServer = livyServer
+
+  var leaderLatch = new LeaderLatch(client, leaderKey)
+  leaderLatch.addListener(this)
+
+  object HAState extends Enumeration{
+type HAState = Value
+val Active, Standby = Value
+  }
+  var currentState = HAState.Standby
+
+  def isLeader() {
+transitionToActive();
+  }
+
+  def notLeader(){
+transitionToStandby();
+  }
+
+  def start() : Unit = {
+transitionToStandby()
+
+client.start()
+leaderLatch.start()
+leaderLatch.await()
+
+// This instance is now the leader. Joining the webserver to the main 
thread
+info("starting join")
 
 Review comment:
   This doesn't seem right. If networking issue caused current active server to 
be standby, then transitionToStandby() will be called and server will be 
stopped, and current server process will exit.
   
   I think we should do "Thread.currentThread.join()" right after 
leaderLatch.start() to block forever, with try catch block, and if any 
exception occurs we transitionToStandby() and exit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] shanyu commented on a change in pull request #212: [LIVY-11] Enable HA support

2019-09-04 Thread GitBox
shanyu commented on a change in pull request #212: [LIVY-11] Enable HA support
URL: https://github.com/apache/incubator-livy/pull/212#discussion_r320942675
 
 

 ##
 File path: 
server/src/main/scala/org/apache/livy/server/recovery/ZooKeeperStateStore.scala
 ##
 @@ -46,7 +46,7 @@ class ZooKeeperStateStore(
   }
 
   private val zkAddress = livyConf.get(LivyConf.RECOVERY_STATE_STORE_URL)
-  require(!zkAddress.isEmpty, s"Please config 
${LivyConf.RECOVERY_STATE_STORE_URL.key}.")
+  require(!zkAddress.isEmpty, s"Please configure 
${LivyConf.RECOVERY_STATE_STORE_URL.key}.")
 
 Review comment:
   let's not make this typo change. there are other places using "config" as 
well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] jerryshao commented on issue #209: [LIVY-640] Add tests for ThriftServer

2019-09-04 Thread GitBox
jerryshao commented on issue #209: [LIVY-640] Add tests for ThriftServer
URL: https://github.com/apache/incubator-livy/pull/209#issuecomment-528187011
 
 
   +1, LGTM. @mgaido91 would you please help to take another review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] tsurushun commented on issue #119: [LIVY-527] added autocompletion api and implementation for ipython

2019-09-04 Thread GitBox
tsurushun commented on issue #119: [LIVY-527] added autocompletion api and 
implementation for ipython
URL: https://github.com/apache/incubator-livy/pull/119#issuecomment-528188936
 
 
   Hi @wangqiaoshi, you would be doing a great favor for a lot of people if you 
could merge this.
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yiheng opened a new pull request #224: [LIVY-633][Server] session should not be gc-ed for long running queries

2019-09-04 Thread GitBox
yiheng opened a new pull request #224: [LIVY-633][Server] session should not be 
gc-ed for long running queries
URL: https://github.com/apache/incubator-livy/pull/224
 
 
   ## What changes were proposed in this pull request?
   Currently, Livy records the last activity time of the session before 
statement execution. If a statement runs too long, exceeding then the session 
timeout, the session will be garbage collected after the statement execution.
   
   This should not be the expected behavior. The statement execution time 
should not be count into idle. We should update the last activity time after 
the statement execution.
   
   We cannot be updated when session changes state from busy to idle in the 
Session class. So in this patch, we add a replLastActivity field into the 
rscClient, which will be updated when the repl state changes. So when session 
changes its state from busy to idle, this field will catch the time and finally 
reflect on the session last activity.
   
   ## How was this patch tested?
   Manual test. Also, add a new unit test.
   
   Existing unit tests and integration tests.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yiheng closed pull request #224: [LIVY-633][Server] session should not be gc-ed for long running queries

2019-09-04 Thread GitBox
yiheng closed pull request #224: [LIVY-633][Server] session should not be gc-ed 
for long running queries
URL: https://github.com/apache/incubator-livy/pull/224
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yiheng commented on issue #223: [LIVY-658]RSCDriver should catch exception if cancel job failed during shutdown

2019-09-04 Thread GitBox
yiheng commented on issue #223: [LIVY-658]RSCDriver should catch exception if 
cancel job failed during shutdown
URL: https://github.com/apache/incubator-livy/pull/223#issuecomment-528184404
 
 
   can we fix the annoying `NullPointerException`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yiheng opened a new pull request #224: [LIVY-633][Server] session should not be gc-ed for long running queries

2019-09-04 Thread GitBox
yiheng opened a new pull request #224: [LIVY-633][Server] session should not be 
gc-ed for long running queries
URL: https://github.com/apache/incubator-livy/pull/224
 
 
   ## What changes were proposed in this pull request?
   Currently, Livy records the last activity time of the session before 
statement execution. If a statement runs too long, exceeding then the session 
timeout, the session will be garbage collected after the statement execution.
   
   This should not be the expected behavior. The statement execution time 
should not be count into idle. We should update the last activity time after 
the statement execution.
   
   We cannot be updated when session changes state from busy to idle in the 
Session class. So in this patch, we add a replLastActivity field into the 
rscClient, which will be updated when the repl state changes. So when session 
changes its state from busy to idle, this field will catch the time and finally 
reflect on the session last activity.
   
   ## How was this patch tested?
   Manual test. Also, add a new unit test.
   
   Existing unit tests and integration tests.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-livy] yiheng closed pull request #224: [LIVY-633][Server] session should not be gc-ed for long running queries

2019-09-04 Thread GitBox
yiheng closed pull request #224: [LIVY-633][Server] session should not be gc-ed 
for long running queries
URL: https://github.com/apache/incubator-livy/pull/224
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services