[incubator-livy] branch master updated: [LIVY-756] Add Spark 3.0 and Scala 2.12 support

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 97cf2f7  [LIVY-756] Add Spark 3.0 and Scala 2.12 support
97cf2f7 is described below

commit 97cf2f75929ef6c152afc468adbead269bd0758f
Author: jerryshao 
AuthorDate: Thu Jul 2 15:44:12 2020 +0800

[LIVY-756] Add Spark 3.0 and Scala 2.12 support

## What changes were proposed in this pull request?

This PR is based tprelle 's PR #289 , and address all the left issues in 
that PR:

1. multi-scala version support in one build (Scala 2.11 and 2.12 support).
2. make SparkR work.

Also reverts most of the unnecessary changes. Besides this PR remove the 
build below 2.4 (2.2, 2.3), since Spark 2.2 and 2.3 only ships with Scala 2.11, 
hard to maintain multiple version. But user could still use 2.2 and 2.3 without 
changes.

All credits to tprelle.

## How was this patch tested?

Run UT and IT with Spark 2.4.5 and 3.0.0 locally.

Author: jerryshao 

Closes #300 from jerryshao/LIVY-756.
---
 .gitignore |  1 +
 .rat-excludes  |  1 +
 .travis.yml| 24 +++---
 README.md  |  4 +-
 assembly/assembly.xml  |  7 ++
 assembly/pom.xml   | 23 ++
 client-common/pom.xml  |  2 +-
 .../org/apache/livy/client/common/Serializer.java  |  8 +-
 {client-common => core/scala-2.12}/pom.xml | 52 ++---
 .../org/apache/livy/LivyBaseUnitTestSuite.scala|  4 +-
 coverage/pom.xml   | 35 +
 .../org/apache/livy/examples/WordCountApp.scala|  2 +-
 integration-test/pom.xml   |  2 +-
 integration-test/src/test/resources/rtest.R|  9 +--
 .../scala/org/apache/livy/test/InteractiveIT.scala |  6 +-
 .../src/test/spark2/scala/Spark2JobApiIT.scala | 26 +--
 pom.xml| 88 +-
 repl/pom.xml   |  3 +
 repl/scala-2.11/pom.xml|  1 +
 .../org/apache/livy/repl/SparkInterpreter.scala|  5 +-
 repl/{scala-2.11 => scala-2.12}/pom.xml| 11 +--
 .../org/apache/livy/repl/SparkInterpreter.scala| 17 ++---
 .../apache/livy/repl/SparkInterpreterSpec.scala| 68 +
 .../main/scala/org/apache/livy/repl/Session.scala  |  4 +-
 .../org/apache/livy/repl/SQLInterpreterSpec.scala  |  4 +-
 rsc/pom.xml|  6 +-
 .../org/apache/livy/rsc/driver/SparkEntries.java   |  7 +-
 .../org/apache/livy/rsc/rpc/KryoMessageCodec.java  |  7 --
 {repl/scala-2.11 => scala-api/scala-2.12}/pom.xml  | 17 ++---
 scala-api/src/main/resources/build.marker  |  0
 .../org/apache/livy/scalaapi/ScalaJobHandle.scala  |  8 ++
 server/pom.xml |  9 ++-
 .../org/apache/livy/server/SessionServlet.scala|  2 +-
 .../server/interactive/InteractiveSession.scala|  6 +-
 .../org/apache/livy/utils/LivySparkUtils.scala |  4 +-
 .../apache/livy/server/BaseJsonServletSpec.scala   |  3 +-
 .../apache/livy/server/SessionServletSpec.scala|  2 +-
 .../livy/server/batch/BatchServletSpec.scala   |  2 +-
 .../livy/server/batch/BatchSessionSpec.scala   |  6 +-
 .../InteractiveSessionServletSpec.scala|  3 +-
 .../interactive/InteractiveSessionSpec.scala   |  2 +-
 .../livy/server/interactive/JobApiSpec.scala   |  2 +-
 .../server/interactive/SessionHeartbeatSpec.scala  |  2 +-
 .../server/recovery/FileSystemStateStoreSpec.scala |  2 +-
 .../livy/server/recovery/SessionStoreSpec.scala|  2 +-
 .../livy/server/recovery/StateStoreSpec.scala  |  2 -
 .../server/recovery/ZooKeeperStateStoreSpec.scala  |  2 +-
 .../apache/livy/sessions/SessionManagerSpec.scala  |  2 +-
 .../apache/livy/utils/LivySparkUtilsSuite.scala|  5 ++
 .../org/apache/livy/utils/SparkYarnAppSpec.scala   |  2 +-
 .../org/apache/livy/test/jobs/SQLGetTweets.java|  2 +-
 .../livy/thriftserver/types/DataTypeUtils.scala|  5 +-
 .../livy/thriftserver/ThriftServerSuites.scala |  3 +-
 thriftserver/session/pom.xml   | 13 
 .../thriftserver/session/ColumnBufferTest.java | 16 ++--
 55 files changed, 362 insertions(+), 189 deletions(-)

diff --git a/.gitignore b/.gitignore
index d46d49f..b1045ea 100644
--- a/.gitignore
+++ b/.gitignore
@@ -24,6 +24,7 @@ metastore_db/
 derby.log
 dependency-reduced-pom.xml
 release-staging/
+venv/
 
 # For python setup.py, which pollutes the source dirs.
 python-api/dist
diff --git a/.rat-excludes b/.rat-excludes
index ac29fe6

[incubator-livy] branch master updated: [MINOR] Modify the description of POST /sessions/{sessionId}/completion

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new ee7fdfc  [MINOR] Modify the description of POST 
/sessions/{sessionId}/completion
ee7fdfc is described below

commit ee7fdfc45d90c0478dcd446bc8a19a217eebe04d
Author: Shingo Furuyama 
AuthorDate: Thu Mar 26 14:59:21 2020 +0800

[MINOR] Modify the description of POST /sessions/{sessionId}/completion

## What changes were proposed in this pull request?

Just modified a description of POST /sessions/{sessionId}/completion in the 
api-doc.

## How was this patch tested?

Since the change is quite small, I didn't test the patch. If I have an 
instruction, I will follow it.

Author: Shingo Furuyama 

Closes #285 from marblejenka/mod-doc-completion.
---
 docs/rest-api.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/rest-api.md b/docs/rest-api.md
index cca937f..d80e77d 100644
--- a/docs/rest-api.md
+++ b/docs/rest-api.md
@@ -312,7 +312,7 @@ Cancel the specified statement in this session.
 
 ### POST /sessions/{sessionId}/completion
 
-Runs a statement in a session.
+Returns code completion candidates for the specified code in the session.
 
  Request Body
 

[incubator-livy] branch master updated: [LIVY-751] Livy server should allow to customize LIVY_CLASSPATH

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new e39d8fe  [LIVY-751] Livy server should allow to customize 
LIVY_CLASSPATH
e39d8fe is described below

commit e39d8fee43adbddf88acb2e04b470aa14b713785
Author: Shingo Furuyama 
AuthorDate: Thu Mar 26 14:07:42 2020 +0800

[LIVY-751] Livy server should allow to customize LIVY_CLASSPATH

## What changes were proposed in this pull request?

The purpose and background is https://issues.apache.org/jira/browse/LIVY-751

## How was this patch tested?

I tested the following two manually.

1. To confirm there is no degradation, I run 0.7.0-incubating livy server 
with sources in this PR. I also run an example jobs, and it completed without 
error.
2.  To confirm our workaround works, I build 0.7.0-incubating branch with 
specifying `-Dhadoop.scope=provided` and sources with this PR. After that, I 
added `export LIVY_CLASSPATH="$LIVY_HOME/jars/*:$(hadoop classpath)"` in 
conf/livy-env.sh and boot livy server.  I also run an example jobs, and it 
completed without error.

Author: Shingo Furuyama 
Author: Shingo Furuyama 

Closes #282 from marblejenka/livy-classpath.
---
 bin/livy-server   | 2 +-
 conf/livy-env.sh.template | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/bin/livy-server b/bin/livy-server
index 8d27d4e..a0e2fb7 100755
--- a/bin/livy-server
+++ b/bin/livy-server
@@ -90,7 +90,7 @@ start_livy_server() {
 fi
   fi
 
-  LIVY_CLASSPATH="$LIBDIR/*:$LIVY_CONF_DIR"
+  LIVY_CLASSPATH="${LIVY_CLASSPATH:-${LIBDIR}/*:${LIVY_CONF_DIR}}"
 
   if [ -n "$SPARK_CONF_DIR" ]; then
 LIVY_CLASSPATH="$LIVY_CLASSPATH:$SPARK_CONF_DIR"
diff --git a/conf/livy-env.sh.template b/conf/livy-env.sh.template
index 7cba5c3..14f22c3 100644
--- a/conf/livy-env.sh.template
+++ b/conf/livy-env.sh.template
@@ -30,3 +30,4 @@
 # names. (Default: name of the user starting Livy).
 # - LIVY_MAX_LOG_FILES Max number of log file to keep in the log directory. 
(Default: 5.)
 # - LIVY_NICENESS   Niceness of the Livy server process when running in the 
background. (Default: 0.)
+# - LIVY_CLASSPATH  Override if the additional classpath is required.

[incubator-livy] branch master updated: [MINOR] Add description of POST /batches

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new d07d103  [MINOR] Add description of POST /batches
d07d103 is described below

commit d07d103f22941525d3cfa2f07f647e310ffb34a1
Author: Shingo Furuyama 
AuthorDate: Thu Mar 26 13:55:51 2020 +0800

[MINOR] Add description of POST /batches

## What changes were proposed in this pull request?

Just added a description of POST /batches in the api-doc.

## How was this patch tested?

Since the change is quite small, I didn't test the patch. If I have an 
instruction, I will follow it.

Author: Shingo Furuyama 

Closes #283 from marblejenka/add-description.
---
 docs/rest-api.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/docs/rest-api.md b/docs/rest-api.md
index f1ff9b4..cca937f 100644
--- a/docs/rest-api.md
+++ b/docs/rest-api.md
@@ -389,6 +389,8 @@ Returns all the active batch sessions.
 
 ### POST /batches
 
+Creates a new batch session.   
+
  Request Body
 
 

[incubator-livy] branch master updated (3a26856 -> 06a8d4f)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from 3a26856  [LIVY-745] Ensure that a single RSCClientFactory gets loaded.
 add 06a8d4f  [LIVY-748] Add support for running Livy Integration tests 
against secure external clusters

No new revisions were added by this update.

Summary of changes:
 .../apache/livy/client/http/LivyConnection.java|   5 +
 integration-test/pom.xml   |   4 +-
 .../test/framework/BaseIntegrationTestSuite.scala  |  57 ++-
 .../org/apache/livy/test/framework/Cluster.scala   |  44 +++-
 .../livy/test/framework/ExternalCluster.scala  | 103 +++
 .../livy/test/framework/LivyRestClient.scala   | 113 +
 .../apache/livy/test/framework/MiniCluster.scala   |  60 +++
 .../resources/{rtest.R => cluster.spec.template}   |  36 ---
 .../src/test/resources/test_python_api.py  |  34 +--
 .../test/scala/org/apache/livy/test/BatchIT.scala  |   2 +-
 .../scala/org/apache/livy/test/InteractiveIT.scala |   8 +-
 .../test/scala/org/apache/livy/test/JobApiIT.scala |  21 +++-
 .../src/test/spark2/scala/Spark2JobApiIT.scala |  17 +++-
 pom.xml|   6 +-
 14 files changed, 401 insertions(+), 109 deletions(-)
 create mode 100644 
integration-test/src/main/scala/org/apache/livy/test/framework/ExternalCluster.scala
 copy integration-test/src/test/resources/{rtest.R => cluster.spec.template} 
(52%)

[incubator-livy] 01/01: [BUILD] Update version for 0.7.1-incubating-SNAPSHOT

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch branch-0.7
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git

commit c563bd403896743006f5d350b0ee3b7a0698b8f3
Author: jerryshao 
AuthorDate: Tue Jan 7 20:40:02 2020 +0800

[BUILD] Update version for 0.7.1-incubating-SNAPSHOT
---
 api/pom.xml  | 4 ++--
 assembly/pom.xml | 4 ++--
 client-common/pom.xml| 4 ++--
 client-http/pom.xml  | 4 ++--
 core/pom.xml | 4 ++--
 core/scala-2.11/pom.xml  | 4 ++--
 coverage/pom.xml | 4 ++--
 docs/_data/project.yml   | 2 +-
 examples/pom.xml | 4 ++--
 integration-test/pom.xml | 4 ++--
 pom.xml  | 2 +-
 python-api/pom.xml   | 4 ++--
 python-api/setup.py  | 2 +-
 repl/pom.xml | 4 ++--
 repl/scala-2.11/pom.xml  | 4 ++--
 rsc/pom.xml  | 2 +-
 scala-api/pom.xml| 4 ++--
 scala-api/scala-2.11/pom.xml | 4 ++--
 scala/pom.xml| 4 ++--
 server/pom.xml   | 4 ++--
 test-lib/pom.xml | 4 ++--
 thriftserver/client/pom.xml  | 2 +-
 thriftserver/server/pom.xml  | 2 +-
 thriftserver/session/pom.xml | 2 +-
 24 files changed, 41 insertions(+), 41 deletions(-)

diff --git a/api/pom.xml b/api/pom.xml
index dc3a5af..66f175c 100644
--- a/api/pom.xml
+++ b/api/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-api
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 41cca2b..b94f0da 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../pom.xml
   
 
   livy-assembly
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   pom
 
   
diff --git a/client-common/pom.xml b/client-common/pom.xml
index 8bc52b9..dac522c 100644
--- a/client-common/pom.xml
+++ b/client-common/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-client-common
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/client-http/pom.xml b/client-http/pom.xml
index b3d5848..ad31b41 100644
--- a/client-http/pom.xml
+++ b/client-http/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-client-http
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/core/pom.xml b/core/pom.xml
index a367bc7..5623220 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -22,12 +22,12 @@
   
 org.apache.livy
 multi-scala-project-root
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../scala/pom.xml
   
 
   livy-core-parent
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   pom
 
   
diff --git a/core/scala-2.11/pom.xml b/core/scala-2.11/pom.xml
index 6e2062b..041f9c1 100644
--- a/core/scala-2.11/pom.xml
+++ b/core/scala-2.11/pom.xml
@@ -19,13 +19,13 @@
   4.0.0
   org.apache.livy
   livy-core_2.11
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
 org.apache.livy
 livy-core-parent
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../pom.xml
   
 
diff --git a/coverage/pom.xml b/coverage/pom.xml
index e4c508a..6419bc4 100644
--- a/coverage/pom.xml
+++ b/coverage/pom.xml
@@ -23,11 +23,11 @@
 org.apache.livy
 livy-main
 ../pom.xml
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   livy-coverage-report
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   pom
 
   
diff --git a/docs/_data/project.yml b/docs/_data/project.yml
index 3b4bbd4..3228413 100644
--- a/docs/_data/project.yml
+++ b/docs/_data/project.yml
@@ -16,6 +16,6 @@
 # Apache Project configurations
 #
 name: Apache Livy
-version: 0.7.0-incubating
+version: 0.7.1-incubating-SNAPSHOT
 
 podling: true
diff --git a/examples/pom.xml b/examples/pom.xml
index 9782e08..1f4aa32 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -23,13 +23,13 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../pom.xml
   
 
   org.apache.livy
   livy-examples
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/integration-test/pom.xml b/integration-test/pom.xml
index f652c64..9fa230b 100644
--- a/integration-test/pom.xml
+++ b/integration-test/pom.xml
@@ -23,11 +23,11 @@
 org.apache.livy
 livy-main
 ../pom.xml
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   livy-integration-test
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/pom.xml b/pom.xml
index 6b4d5a4..938bdbf 100644
--- a/pom.xml
+++ b/pom.xml
@@ -22,7 +22,7 @@
 
   org.apache.livy
   livy-main
-  0.7.0-incubating

[incubator-livy] branch branch-0.7 updated: [MINOR] Update NOTICE file to reflect the correct date

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch branch-0.7
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/branch-0.7 by this push:
 new a4baa62  [MINOR] Update NOTICE file to reflect the correct date
a4baa62 is described below

commit a4baa62c7e02ec34919a432d72780c20e3e88c58
Author: jerryshao 
AuthorDate: Mon Jan 6 15:52:59 2020 +0800

[MINOR] Update NOTICE file to reflect the correct date

Author: jerryshao 

Closes #271 from jerryshao/notice-fix.

(cherry picked from commit 277b0c5af019bf944a17a7f870d4998d948d7917)
Signed-off-by: jerryshao 
---
 NOTICE | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/NOTICE b/NOTICE
index 7eeebe8..9d023c2 100644
--- a/NOTICE
+++ b/NOTICE
@@ -1,5 +1,5 @@
 Apache Livy
-Copyright 2018 The Apache Software Foundation
+Copyright 2018 and onwards The Apache Software Foundation
 
 This product includes software developed at
 The Apache Software Foundation (http://www.apache.org/).

[incubator-livy] branch master updated: [LIVY-727] Fix session state always be idle though the yarn application has been killed after restart livy

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 25542e4  [LIVY-727] Fix session state always be idle though the yarn 
application has been killed after restart livy
25542e4 is described below

commit 25542e4e78b39a3c9b9426a70a92ca7c183daea3
Author: runzhiwang 
AuthorDate: Thu Dec 19 14:29:01 2019 +0800

[LIVY-727] Fix session state always be idle though the yarn application has 
been killed after restart livy

## What changes were proposed in this pull request?

[LIVY-727] Fix session state always be idle though the yarn application has 
been killed after restart livy.

Follows are steps to reproduce the problem:
1. Set livy.server.recovery.mode=recovery, and create a session in 
yarn-cluster
2. Restart livy
3. Kill the yarn application of the session.
4. The session state will always be idle and never change to killed or 
dead. Just as the image, livy-session-16 has been killed in yarn, but the state 
is still idle.

![image](https://user-images.githubusercontent.com/51938049/70371088-92695c80-1909-11ea-875c-73696db693ce.png)

The cause of the problem are as follows:
1. Because when recover session, livy will not startDriver again, so the 
driverProcess is None.
2. SparkYarnApp will not be created in `driverProcess.map { _ => 
SparkApp.create(appTag, appId, driverProcess, livyConf, Some(this)) }` when 
driverProcess is None.
3. So yarnAppMonitorThread of the session will never start, and the session 
state will never change.

How to fix the bug:
1. If livy run in yarn, SparkApp will create even though the driverProcess 
is None
2. If not run in yarn, SparkApp will not create, because the code require 
driverProcess is not None at 
https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/utils/SparkApp.scala#L93,
  and I don't want to change the behavior.

## How was this patch tested?

1. Set livy.server.recovery.mode=recovery, and create a session in 
yarn-cluster
2. Restart livy
3. Kill the yarn application of the session.
4. The session state will change to killed

Author: runzhiwang 

Closes #265 from runzhiwang/session-state.
---
 .../org/apache/livy/server/interactive/InteractiveSession.scala| 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
index 4b318b8..790bd5a 100644
--- 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
+++ 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
@@ -399,7 +399,12 @@ class InteractiveSession(
 app = mockApp.orElse {
   val driverProcess = client.flatMap { c => Option(c.getDriverProcess) }
 .map(new LineBufferedProcess(_, 
livyConf.getInt(LivyConf.SPARK_LOGS_SIZE)))
-  driverProcess.map { _ => SparkApp.create(appTag, appId, driverProcess, 
livyConf, Some(this)) }
+
+  if (livyConf.isRunningOnYarn() || driverProcess.isDefined) {
+Some(SparkApp.create(appTag, appId, driverProcess, livyConf, 
Some(this)))
+  } else {
+None
+  }
 }
 
 if (client.isEmpty) {

svn commit: r37274 - /dev/incubator/livy/0.7.0-incubating-rc3/

[jshao]

Author: jshao
Date: Wed Dec 18 02:36:42 2019
New Revision: 37274

Log:
Apache Livy 0.7.0-incubating-rc3

Added:
dev/incubator/livy/0.7.0-incubating-rc3/

dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip   
(with props)

dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip.asc
   (with props)

dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip.sha512

dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip   
(with props)

dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip.asc
   (with props)

dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip.sha512

Added: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip
==
Binary file - no diff available.

Propchange: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip
--
svn:mime-type = application/zip

Added: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip.asc
==
Binary file - no diff available.

Propchange: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip.asc
--
svn:mime-type = application/pgp-signature

Added: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip.sha512
==
--- 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip.sha512
 (added)
+++ 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-bin.zip.sha512
 Wed Dec 18 02:36:42 2019
@@ -0,0 +1,4 @@
+apache-livy-0.7.0-incubating-bin.zip: 570FBBEA D736C61A AAA4268D 2EF9C441
+  176E76E9 B86A69CF BF5DC3E0 40AEC460
+  C913ACBC C23A6B59 27E7717C 3F7E8CB6
+  F52E6CF2 73811F74 01163956 8D099036

Added: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip
==
Binary file - no diff available.

Propchange: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip
--
svn:mime-type = application/zip

Added: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip.asc
==
Binary file - no diff available.

Propchange: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip.asc
--
svn:mime-type = application/pgp-signature

Added: 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip.sha512
==
--- 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip.sha512
 (added)
+++ 
dev/incubator/livy/0.7.0-incubating-rc3/apache-livy-0.7.0-incubating-src.zip.sha512
 Wed Dec 18 02:36:42 2019
@@ -0,0 +1,4 @@
+apache-livy-0.7.0-incubating-src.zip: AD6CEE1C 97E39A7A DFC51215 E9E3DE86
+  41ED8EE0 DB2F1D7B BA909D08 34962DF1
+  AE0D0D70 443F0A7D 186B41D7 C1AB66FC
+  E07280B5 67B27086 0E28576C 5EB81356

[incubator-livy] 01/01: [BUILD] Update version for 0.7.1-incubating-SNAPSHOT

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch branch-0.7
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git

commit c3e5188799e361abd5b8342ad86259725a0a7706
Author: jerryshao 
AuthorDate: Wed Dec 18 10:06:47 2019 +0800

[BUILD] Update version for 0.7.1-incubating-SNAPSHOT
---
 api/pom.xml  | 4 ++--
 assembly/pom.xml | 4 ++--
 client-common/pom.xml| 4 ++--
 client-http/pom.xml  | 4 ++--
 core/pom.xml | 4 ++--
 core/scala-2.11/pom.xml  | 4 ++--
 coverage/pom.xml | 4 ++--
 docs/_data/project.yml   | 2 +-
 examples/pom.xml | 4 ++--
 integration-test/pom.xml | 4 ++--
 pom.xml  | 2 +-
 python-api/pom.xml   | 4 ++--
 python-api/setup.py  | 2 +-
 repl/pom.xml | 4 ++--
 repl/scala-2.11/pom.xml  | 4 ++--
 rsc/pom.xml  | 2 +-
 scala-api/pom.xml| 4 ++--
 scala-api/scala-2.11/pom.xml | 4 ++--
 scala/pom.xml| 4 ++--
 server/pom.xml   | 4 ++--
 test-lib/pom.xml | 4 ++--
 thriftserver/client/pom.xml  | 2 +-
 thriftserver/server/pom.xml  | 2 +-
 thriftserver/session/pom.xml | 2 +-
 24 files changed, 41 insertions(+), 41 deletions(-)

diff --git a/api/pom.xml b/api/pom.xml
index dc3a5af..66f175c 100644
--- a/api/pom.xml
+++ b/api/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-api
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 41cca2b..b94f0da 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../pom.xml
   
 
   livy-assembly
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   pom
 
   
diff --git a/client-common/pom.xml b/client-common/pom.xml
index 8bc52b9..dac522c 100644
--- a/client-common/pom.xml
+++ b/client-common/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-client-common
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/client-http/pom.xml b/client-http/pom.xml
index b3d5848..ad31b41 100644
--- a/client-http/pom.xml
+++ b/client-http/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-client-http
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/core/pom.xml b/core/pom.xml
index a367bc7..5623220 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -22,12 +22,12 @@
   
 org.apache.livy
 multi-scala-project-root
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../scala/pom.xml
   
 
   livy-core-parent
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   pom
 
   
diff --git a/core/scala-2.11/pom.xml b/core/scala-2.11/pom.xml
index 6e2062b..041f9c1 100644
--- a/core/scala-2.11/pom.xml
+++ b/core/scala-2.11/pom.xml
@@ -19,13 +19,13 @@
   4.0.0
   org.apache.livy
   livy-core_2.11
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
 org.apache.livy
 livy-core-parent
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../pom.xml
   
 
diff --git a/coverage/pom.xml b/coverage/pom.xml
index e4c508a..6419bc4 100644
--- a/coverage/pom.xml
+++ b/coverage/pom.xml
@@ -23,11 +23,11 @@
 org.apache.livy
 livy-main
 ../pom.xml
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   livy-coverage-report
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   pom
 
   
diff --git a/docs/_data/project.yml b/docs/_data/project.yml
index 3b4bbd4..3228413 100644
--- a/docs/_data/project.yml
+++ b/docs/_data/project.yml
@@ -16,6 +16,6 @@
 # Apache Project configurations
 #
 name: Apache Livy
-version: 0.7.0-incubating
+version: 0.7.1-incubating-SNAPSHOT
 
 podling: true
diff --git a/examples/pom.xml b/examples/pom.xml
index 9782e08..1f4aa32 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -23,13 +23,13 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../pom.xml
   
 
   org.apache.livy
   livy-examples
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/integration-test/pom.xml b/integration-test/pom.xml
index f652c64..9fa230b 100644
--- a/integration-test/pom.xml
+++ b/integration-test/pom.xml
@@ -23,11 +23,11 @@
 org.apache.livy
 livy-main
 ../pom.xml
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   livy-integration-test
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/pom.xml b/pom.xml
index 1d9d0f1..1322aeb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -22,7 +22,7 @@
 
   org.apache.livy
   livy-main
-  0.7.0-incubating

[incubator-livy] branch branch-0.7 updated (3ae0e65 -> 9c638c4)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch branch-0.7
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from 3ae0e65  [Minor] Revert unnecessary local changes
 add eebb6ec  [BUILD] Update version for 0.7.0-incubating
 new 9c638c4  [BUILD] Update version for 0.7.1-incubating-SNAPSHOT

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

[incubator-livy] 01/01: [BUILD] Update version for 0.7.0-incubating

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to tag v0.7.0-incubating-rc2
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git

commit eebb6ecfe5ec021244c69096fa240e84f933b2ca
Author: jerryshao 
AuthorDate: Mon Dec 16 14:43:12 2019 +0800

[BUILD] Update version for 0.7.0-incubating
---
 api/pom.xml  | 4 ++--
 assembly/pom.xml | 4 ++--
 client-common/pom.xml| 4 ++--
 client-http/pom.xml  | 4 ++--
 core/pom.xml | 4 ++--
 core/scala-2.11/pom.xml  | 4 ++--
 coverage/pom.xml | 4 ++--
 docs/_data/project.yml   | 2 +-
 examples/pom.xml | 4 ++--
 integration-test/pom.xml | 4 ++--
 pom.xml  | 2 +-
 python-api/pom.xml   | 4 ++--
 python-api/setup.py  | 2 +-
 repl/pom.xml | 4 ++--
 repl/scala-2.11/pom.xml  | 4 ++--
 rsc/pom.xml  | 2 +-
 scala-api/pom.xml| 4 ++--
 scala-api/scala-2.11/pom.xml | 4 ++--
 scala/pom.xml| 4 ++--
 server/pom.xml   | 4 ++--
 test-lib/pom.xml | 4 ++--
 thriftserver/client/pom.xml  | 2 +-
 thriftserver/server/pom.xml  | 2 +-
 thriftserver/session/pom.xml | 2 +-
 24 files changed, 41 insertions(+), 41 deletions(-)

diff --git a/api/pom.xml b/api/pom.xml
index 66f175c..dc3a5af 100644
--- a/api/pom.xml
+++ b/api/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.1-incubating-SNAPSHOT
+0.7.0-incubating
   
 
   org.apache.livy
   livy-api
-  0.7.1-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
diff --git a/assembly/pom.xml b/assembly/pom.xml
index b94f0da..41cca2b 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.1-incubating-SNAPSHOT
+0.7.0-incubating
 ../pom.xml
   
 
   livy-assembly
-  0.7.1-incubating-SNAPSHOT
+  0.7.0-incubating
   pom
 
   
diff --git a/client-common/pom.xml b/client-common/pom.xml
index dac522c..8bc52b9 100644
--- a/client-common/pom.xml
+++ b/client-common/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.1-incubating-SNAPSHOT
+0.7.0-incubating
   
 
   org.apache.livy
   livy-client-common
-  0.7.1-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
diff --git a/client-http/pom.xml b/client-http/pom.xml
index ad31b41..b3d5848 100644
--- a/client-http/pom.xml
+++ b/client-http/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.1-incubating-SNAPSHOT
+0.7.0-incubating
   
 
   org.apache.livy
   livy-client-http
-  0.7.1-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
diff --git a/core/pom.xml b/core/pom.xml
index 5623220..a367bc7 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -22,12 +22,12 @@
   
 org.apache.livy
 multi-scala-project-root
-0.7.1-incubating-SNAPSHOT
+0.7.0-incubating
 ../scala/pom.xml
   
 
   livy-core-parent
-  0.7.1-incubating-SNAPSHOT
+  0.7.0-incubating
   pom
 
   
diff --git a/core/scala-2.11/pom.xml b/core/scala-2.11/pom.xml
index 041f9c1..6e2062b 100644
--- a/core/scala-2.11/pom.xml
+++ b/core/scala-2.11/pom.xml
@@ -19,13 +19,13 @@
   4.0.0
   org.apache.livy
   livy-core_2.11
-  0.7.1-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
 org.apache.livy
 livy-core-parent
-0.7.1-incubating-SNAPSHOT
+0.7.0-incubating
 ../pom.xml
   
 
diff --git a/coverage/pom.xml b/coverage/pom.xml
index 6419bc4..e4c508a 100644
--- a/coverage/pom.xml
+++ b/coverage/pom.xml
@@ -23,11 +23,11 @@
 org.apache.livy
 livy-main
 ../pom.xml
-0.7.1-incubating-SNAPSHOT
+0.7.0-incubating
   
 
   livy-coverage-report
-  0.7.1-incubating-SNAPSHOT
+  0.7.0-incubating
   pom
 
   
diff --git a/docs/_data/project.yml b/docs/_data/project.yml
index 3228413..3b4bbd4 100644
--- a/docs/_data/project.yml
+++ b/docs/_data/project.yml
@@ -16,6 +16,6 @@
 # Apache Project configurations
 #
 name: Apache Livy
-version: 0.7.1-incubating-SNAPSHOT
+version: 0.7.0-incubating
 
 podling: true
diff --git a/examples/pom.xml b/examples/pom.xml
index 1f4aa32..9782e08 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -23,13 +23,13 @@
   
 org.apache.livy
 livy-main
-0.7.1-incubating-SNAPSHOT
+0.7.0-incubating
 ../pom.xml
   
 
   org.apache.livy
   livy-examples
-  0.7.1-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
diff --git a/integration-test/pom.xml b/integration-test/pom.xml
index 9fa230b..f652c64 100644
--- a/integration-test/pom.xml
+++ b/integration-test/pom.xml
@@ -23,11 +23,11 @@
 org.apache.livy
 livy-main
 ../pom.xml
-0.7.1-incubating-SNAPSHOT
+0.7.0-incubating
   
 
   livy-integration-test
-  0.7.1-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
diff --git a/pom.xml b/pom.xml
index 1322aeb..1d9d0f1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -22,7 +22,7 @@
 
   org.apache.livy
   livy-main
-  0.7.1-incubating

[incubator-livy] tag v0.7.0-incubating-rc2 created (now eebb6ec)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to tag v0.7.0-incubating-rc2
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


  at eebb6ec  (commit)
This tag includes the following new commits:

 new eebb6ec  [BUILD] Update version for 0.7.0-incubating

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-livy] branch branch-0.7 updated: [Minor] Revert unnecessary local changes

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch branch-0.7
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/branch-0.7 by this push:
 new 3ae0e65  [Minor] Revert unnecessary local changes
3ae0e65 is described below

commit 3ae0e6575b0c5858e081e72f69ff23e042424fcf
Author: jerryshao 
AuthorDate: Mon Dec 16 14:38:39 2019 +0800

[Minor] Revert unnecessary local changes
---
 dev/release-build.sh | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/dev/release-build.sh b/dev/release-build.sh
index a1a30a5..5c3bbc2 100755
--- a/dev/release-build.sh
+++ b/dev/release-build.sh
@@ -122,8 +122,8 @@ if [[ "$1" == "package" ]]; then
   echo "Packaging release tarballs"
   cp -r incubator-livy $ARCHIVE_NAME_PREFIX
   zip -r $SRC_ARCHIVE $ARCHIVE_NAME_PREFIX
-  echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --armour --output 
$SRC_ARCHIVE.asc --detach-sig $SRC_ARCHIVE
-  echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --print-md SHA512 $SRC_ARCHIVE 
> $SRC_ARCHIVE.sha512
+  echo "" | $GPG --passphrase-fd 0 --armour --output $SRC_ARCHIVE.asc 
--detach-sig $SRC_ARCHIVE
+  echo "" | $GPG --passphrase-fd 0 --print-md SHA512 $SRC_ARCHIVE > 
$SRC_ARCHIVE.sha512
   rm -rf $ARCHIVE_NAME_PREFIX
 
   # Updated for binary build
@@ -135,8 +135,8 @@ if [[ "$1" == "package" ]]; then
 
 echo "Copying and signing regular binary distribution"
 cp assembly/target/$BIN_ARCHIVE .
-echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --armour --output 
$BIN_ARCHIVE.asc --detach-sig $BIN_ARCHIVE
-echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --print-md SHA512 
$BIN_ARCHIVE > $BIN_ARCHIVE.sha512
+echo "" | $GPG --passphrase-fd 0 --armour --output $BIN_ARCHIVE.asc 
--detach-sig $BIN_ARCHIVE
+echo "" | $GPG --passphrase-fd 0 --print-md SHA512 $BIN_ARCHIVE > 
$BIN_ARCHIVE.sha512
 
 cp $BIN_ARCHIVE* ../
 cd ..
@@ -192,7 +192,7 @@ if [[ "$1" == "publish-release" ]]; then
   echo "Creating hash and signature files"
   for file in $(find . -type f)
   do
-echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --output $file.asc \
+echo "" | $GPG --passphrase-fd 0 --output $file.asc \
   --detach-sig --armour $file;
 if [ $(command -v md5) ]; then
   # Available on OS X; -q to keep only hash

[incubator-livy] branch master updated: [MINOR][BUILD] Update docs version for 0.8.0-incubating-SNAPSHOT

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 393c162  [MINOR][BUILD] Update docs version for 
0.8.0-incubating-SNAPSHOT
393c162 is described below

commit 393c16201a0594defa0e04e1c1b526096212921a
Author: jerryshao 
AuthorDate: Mon Dec 16 14:26:26 2019 +0800

[MINOR][BUILD] Update docs version for 0.8.0-incubating-SNAPSHOT
---
 docs/_data/project.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/_data/project.yml b/docs/_data/project.yml
index 7af6fdd..8568f32 100644
--- a/docs/_data/project.yml
+++ b/docs/_data/project.yml
@@ -16,6 +16,6 @@
 # Apache Project configurations
 #
 name: Apache Livy
-version: 0.7.0-incubating-SNAPSHOT
+version: 0.8.0-incubating-SNAPSHOT
 
-podling: true
\ No newline at end of file
+podling: true

[incubator-livy] 01/01: [BUILD] Update version for 0.7.1-incubating-SNAPSHOT

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch branch-0.7
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git

commit 909b99035ee3bff5f6b807039518470c2c8bef82
Author: jerryshao 
AuthorDate: Mon Dec 16 14:23:23 2019 +0800

[BUILD] Update version for 0.7.1-incubating-SNAPSHOT
---
 api/pom.xml  | 4 ++--
 assembly/pom.xml | 4 ++--
 client-common/pom.xml| 4 ++--
 client-http/pom.xml  | 4 ++--
 core/pom.xml | 4 ++--
 core/scala-2.11/pom.xml  | 4 ++--
 coverage/pom.xml | 4 ++--
 docs/_data/project.yml   | 2 +-
 examples/pom.xml | 4 ++--
 integration-test/pom.xml | 4 ++--
 pom.xml  | 2 +-
 python-api/pom.xml   | 4 ++--
 python-api/setup.py  | 2 +-
 repl/pom.xml | 4 ++--
 repl/scala-2.11/pom.xml  | 4 ++--
 rsc/pom.xml  | 2 +-
 scala-api/pom.xml| 4 ++--
 scala-api/scala-2.11/pom.xml | 4 ++--
 scala/pom.xml| 4 ++--
 server/pom.xml   | 4 ++--
 test-lib/pom.xml | 4 ++--
 thriftserver/client/pom.xml  | 2 +-
 thriftserver/server/pom.xml  | 2 +-
 thriftserver/session/pom.xml | 2 +-
 24 files changed, 41 insertions(+), 41 deletions(-)

diff --git a/api/pom.xml b/api/pom.xml
index dc3a5af..66f175c 100644
--- a/api/pom.xml
+++ b/api/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-api
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 41cca2b..b94f0da 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../pom.xml
   
 
   livy-assembly
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   pom
 
   
diff --git a/client-common/pom.xml b/client-common/pom.xml
index 8bc52b9..dac522c 100644
--- a/client-common/pom.xml
+++ b/client-common/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-client-common
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/client-http/pom.xml b/client-http/pom.xml
index b3d5848..ad31b41 100644
--- a/client-http/pom.xml
+++ b/client-http/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-client-http
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/core/pom.xml b/core/pom.xml
index a367bc7..5623220 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -22,12 +22,12 @@
   
 org.apache.livy
 multi-scala-project-root
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../scala/pom.xml
   
 
   livy-core-parent
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   pom
 
   
diff --git a/core/scala-2.11/pom.xml b/core/scala-2.11/pom.xml
index 6e2062b..041f9c1 100644
--- a/core/scala-2.11/pom.xml
+++ b/core/scala-2.11/pom.xml
@@ -19,13 +19,13 @@
   4.0.0
   org.apache.livy
   livy-core_2.11
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
 org.apache.livy
 livy-core-parent
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../pom.xml
   
 
diff --git a/coverage/pom.xml b/coverage/pom.xml
index e4c508a..6419bc4 100644
--- a/coverage/pom.xml
+++ b/coverage/pom.xml
@@ -23,11 +23,11 @@
 org.apache.livy
 livy-main
 ../pom.xml
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   livy-coverage-report
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   pom
 
   
diff --git a/docs/_data/project.yml b/docs/_data/project.yml
index 3b4bbd4..3228413 100644
--- a/docs/_data/project.yml
+++ b/docs/_data/project.yml
@@ -16,6 +16,6 @@
 # Apache Project configurations
 #
 name: Apache Livy
-version: 0.7.0-incubating
+version: 0.7.1-incubating-SNAPSHOT
 
 podling: true
diff --git a/examples/pom.xml b/examples/pom.xml
index 9782e08..1f4aa32 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -23,13 +23,13 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
 ../pom.xml
   
 
   org.apache.livy
   livy-examples
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/integration-test/pom.xml b/integration-test/pom.xml
index f652c64..9fa230b 100644
--- a/integration-test/pom.xml
+++ b/integration-test/pom.xml
@@ -23,11 +23,11 @@
 org.apache.livy
 livy-main
 ../pom.xml
-0.7.0-incubating
+0.7.1-incubating-SNAPSHOT
   
 
   livy-integration-test
-  0.7.0-incubating
+  0.7.1-incubating-SNAPSHOT
   jar
 
   
diff --git a/pom.xml b/pom.xml
index 1d9d0f1..1322aeb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -22,7 +22,7 @@
 
   org.apache.livy
   livy-main
-  0.7.0-incubating

[incubator-livy] branch branch-0.7 updated (0a527e8 -> 909b990)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch branch-0.7
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from 0a527e8  [LIVY-717] introduce maven property to set ZooKeeper version
 add 1918f6a  [BUILD] Update version for 0.7.0-incubating
 new 909b990  [BUILD] Update version for 0.7.1-incubating-SNAPSHOT

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 api/pom.xml  |  4 ++--
 assembly/pom.xml |  4 ++--
 client-common/pom.xml|  4 ++--
 client-http/pom.xml  |  4 ++--
 core/pom.xml |  4 ++--
 core/scala-2.11/pom.xml  |  4 ++--
 coverage/pom.xml |  4 ++--
 dev/release-build.sh | 10 +-
 docs/_data/project.yml   |  4 ++--
 examples/pom.xml |  4 ++--
 integration-test/pom.xml |  4 ++--
 pom.xml  |  2 +-
 python-api/pom.xml   |  4 ++--
 python-api/setup.py  |  2 +-
 repl/pom.xml |  4 ++--
 repl/scala-2.11/pom.xml  |  4 ++--
 rsc/pom.xml  |  2 +-
 scala-api/pom.xml|  4 ++--
 scala-api/scala-2.11/pom.xml |  4 ++--
 scala/pom.xml|  4 ++--
 server/pom.xml   |  4 ++--
 test-lib/pom.xml |  4 ++--
 thriftserver/client/pom.xml  |  2 +-
 thriftserver/server/pom.xml  |  2 +-
 thriftserver/session/pom.xml |  2 +-
 25 files changed, 47 insertions(+), 47 deletions(-)

[incubator-livy] 01/01: [BUILD] Update version for 0.7.0-incubating

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to tag v0.7.0-incubating-rc1
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git

commit 1918f6a510805eb4422f2d23075c4825ce9d23a1
Author: jerryshao 
AuthorDate: Mon Dec 16 14:18:46 2019 +0800

[BUILD] Update version for 0.7.0-incubating
---
 api/pom.xml  |  4 ++--
 assembly/pom.xml |  4 ++--
 client-common/pom.xml|  4 ++--
 client-http/pom.xml  |  4 ++--
 core/pom.xml |  4 ++--
 core/scala-2.11/pom.xml  |  4 ++--
 coverage/pom.xml |  4 ++--
 dev/release-build.sh | 10 +-
 docs/_data/project.yml   |  4 ++--
 examples/pom.xml |  4 ++--
 integration-test/pom.xml |  4 ++--
 pom.xml  |  2 +-
 python-api/pom.xml   |  4 ++--
 python-api/setup.py  |  2 +-
 repl/pom.xml |  4 ++--
 repl/scala-2.11/pom.xml  |  4 ++--
 rsc/pom.xml  |  2 +-
 scala-api/pom.xml|  4 ++--
 scala-api/scala-2.11/pom.xml |  4 ++--
 scala/pom.xml|  4 ++--
 server/pom.xml   |  4 ++--
 test-lib/pom.xml |  4 ++--
 thriftserver/client/pom.xml  |  2 +-
 thriftserver/server/pom.xml  |  2 +-
 thriftserver/session/pom.xml |  2 +-
 25 files changed, 47 insertions(+), 47 deletions(-)

diff --git a/api/pom.xml b/api/pom.xml
index f66e82a..dc3a5af 100644
--- a/api/pom.xml
+++ b/api/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating-SNAPSHOT
+0.7.0-incubating
   
 
   org.apache.livy
   livy-api
-  0.7.0-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
diff --git a/assembly/pom.xml b/assembly/pom.xml
index c0d0b26..41cca2b 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating-SNAPSHOT
+0.7.0-incubating
 ../pom.xml
   
 
   livy-assembly
-  0.7.0-incubating-SNAPSHOT
+  0.7.0-incubating
   pom
 
   
diff --git a/client-common/pom.xml b/client-common/pom.xml
index 0d9a27e..8bc52b9 100644
--- a/client-common/pom.xml
+++ b/client-common/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating-SNAPSHOT
+0.7.0-incubating
   
 
   org.apache.livy
   livy-client-common
-  0.7.0-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
diff --git a/client-http/pom.xml b/client-http/pom.xml
index 8401950..b3d5848 100644
--- a/client-http/pom.xml
+++ b/client-http/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.7.0-incubating-SNAPSHOT
+0.7.0-incubating
   
 
   org.apache.livy
   livy-client-http
-  0.7.0-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
diff --git a/core/pom.xml b/core/pom.xml
index 6179a7a..a367bc7 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -22,12 +22,12 @@
   
 org.apache.livy
 multi-scala-project-root
-0.7.0-incubating-SNAPSHOT
+0.7.0-incubating
 ../scala/pom.xml
   
 
   livy-core-parent
-  0.7.0-incubating-SNAPSHOT
+  0.7.0-incubating
   pom
 
   
diff --git a/core/scala-2.11/pom.xml b/core/scala-2.11/pom.xml
index 417f2e2..6e2062b 100644
--- a/core/scala-2.11/pom.xml
+++ b/core/scala-2.11/pom.xml
@@ -19,13 +19,13 @@
   4.0.0
   org.apache.livy
   livy-core_2.11
-  0.7.0-incubating-SNAPSHOT
+  0.7.0-incubating
   jar
 
   
 org.apache.livy
 livy-core-parent
-0.7.0-incubating-SNAPSHOT
+0.7.0-incubating
 ../pom.xml
   
 
diff --git a/coverage/pom.xml b/coverage/pom.xml
index 3887965..e4c508a 100644
--- a/coverage/pom.xml
+++ b/coverage/pom.xml
@@ -23,11 +23,11 @@
 org.apache.livy
 livy-main
 ../pom.xml
-0.7.0-incubating-SNAPSHOT
+0.7.0-incubating
   
 
   livy-coverage-report
-  0.7.0-incubating-SNAPSHOT
+  0.7.0-incubating
   pom
 
   
diff --git a/dev/release-build.sh b/dev/release-build.sh
index 5c3bbc2..a1a30a5 100755
--- a/dev/release-build.sh
+++ b/dev/release-build.sh
@@ -122,8 +122,8 @@ if [[ "$1" == "package" ]]; then
   echo "Packaging release tarballs"
   cp -r incubator-livy $ARCHIVE_NAME_PREFIX
   zip -r $SRC_ARCHIVE $ARCHIVE_NAME_PREFIX
-  echo "" | $GPG --passphrase-fd 0 --armour --output $SRC_ARCHIVE.asc 
--detach-sig $SRC_ARCHIVE
-  echo "" | $GPG --passphrase-fd 0 --print-md SHA512 $SRC_ARCHIVE > 
$SRC_ARCHIVE.sha512
+  echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --armour --output 
$SRC_ARCHIVE.asc --detach-sig $SRC_ARCHIVE
+  echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --print-md SHA512 $SRC_ARCHIVE 
> $SRC_ARCHIVE.sha512
   rm -rf $ARCHIVE_NAME_PREFIX
 
   # Updated for binary build
@@ -135,8 +135,8 @@ if [[ "$1" == "package" ]]; then
 
 echo "Copying and signing regular binary distribution"
 cp assembly/target/$BIN_ARCHIVE .
-echo "" | $GPG --passphrase-fd 0 --armour --output $BIN_ARCHIVE.asc

[incubator-livy] tag v0.7.0-incubating-rc1 created (now 1918f6a)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to tag v0.7.0-incubating-rc1
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


  at 1918f6a  (commit)
This tag includes the following new commits:

 new 1918f6a  [BUILD] Update version for 0.7.0-incubating

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-livy] branch master updated (0a527e8 -> 319386a)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from 0a527e8  [LIVY-717] introduce maven property to set ZooKeeper version
 add 319386a  [MINOR] Bump version to 0.8.0-incubating-SNAPSHOT for master 
branch

No new revisions were added by this update.

Summary of changes:
 api/pom.xml  | 4 ++--
 assembly/pom.xml | 4 ++--
 client-common/pom.xml| 4 ++--
 client-http/pom.xml  | 4 ++--
 core/pom.xml | 4 ++--
 core/scala-2.11/pom.xml  | 4 ++--
 coverage/pom.xml | 4 ++--
 examples/pom.xml | 4 ++--
 integration-test/pom.xml | 4 ++--
 pom.xml  | 2 +-
 python-api/pom.xml   | 4 ++--
 python-api/setup.py  | 2 +-
 repl/pom.xml | 4 ++--
 repl/scala-2.11/pom.xml  | 4 ++--
 rsc/pom.xml  | 2 +-
 scala-api/pom.xml| 4 ++--
 scala-api/scala-2.11/pom.xml | 4 ++--
 scala/pom.xml| 4 ++--
 server/pom.xml   | 4 ++--
 test-lib/pom.xml | 4 ++--
 thriftserver/client/pom.xml  | 2 +-
 thriftserver/server/pom.xml  | 2 +-
 thriftserver/session/pom.xml | 2 +-
 23 files changed, 40 insertions(+), 40 deletions(-)

[incubator-livy] branch branch-0.7 created (now 0a527e8)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch branch-0.7
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


  at 0a527e8  [LIVY-717] introduce maven property to set ZooKeeper version

No new revisions were added by this update.

[incubator-livy] branch master updated (cccba94 -> 0a527e8)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from cccba94  [LIVY-714][SERVER] Fix cannot remove the app in leakedAppTags 
when timeout
 add 0a527e8  [LIVY-717] introduce maven property to set ZooKeeper version

No new revisions were added by this update.

Summary of changes:
 integration-test/pom.xml | 37 +
 pom.xml  |  2 ++
 server/pom.xml   | 28 
 3 files changed, 67 insertions(+)

[incubator-livy] branch master updated: [LIVY-714][SERVER] Fix cannot remove the app in leakedAppTags when timeout

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new cccba94  [LIVY-714][SERVER] Fix cannot remove the app in leakedAppTags 
when timeout
cccba94 is described below

commit cccba9480e2db821d6cc67f580eeb67f2fac4e95
Author: runzhiwang 
AuthorDate: Mon Nov 25 11:44:37 2019 +0800

[LIVY-714][SERVER] Fix cannot remove the app in leakedAppTags when timeout

## What changes were proposed in this pull request?
1.`var isRemoved = false` should be in `while(iter.hasNext),` otherwise if 
there are two apps, the first app will be killApplication and the second app 
will timeout in this loop, and after removing the first app,` isRemoved = 
true`, and the second app cannot pass the` if(!isRemoved)` and only will be 
deleted in the next loop.

2.`entry.getValue - now` is negative, and never greater than 
`sessionLeakageCheckTimeout`.


![image](https://user-images.githubusercontent.com/51938049/69202431-99a81080-0b7c-11ea-8084-9801af5a75bd.png)

## How was this patch tested?

Existed IT and UT.

Author: runzhiwang 

Closes #259 from runzhiwang/leakapp.
---
 .../scala/org/apache/livy/utils/SparkYarnApp.scala | 27 --
 .../org/apache/livy/utils/SparkYarnAppSpec.scala   | 21 +
 2 files changed, 41 insertions(+), 7 deletions(-)

diff --git a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala 
b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
index 14af9fa..a245823 100644
--- a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
+++ b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
@@ -37,7 +37,8 @@ import org.apache.livy.{LivyConf, Logging, Utils}
 
 object SparkYarnApp extends Logging {
 
-  def init(livyConf: LivyConf): Unit = {
+  def init(livyConf: LivyConf, client: Option[YarnClient] = None): Unit = {
+mockYarnClient = client
 sessionLeakageCheckInterval = 
livyConf.getTimeAsMs(LivyConf.YARN_APP_LEAKAGE_CHECK_INTERVAL)
 sessionLeakageCheckTimeout = 
livyConf.getTimeAsMs(LivyConf.YARN_APP_LEAKAGE_CHECK_TIMEOUT)
 leakedAppsGCThread.setDaemon(true)
@@ -45,6 +46,8 @@ object SparkYarnApp extends Logging {
 leakedAppsGCThread.start()
   }
 
+  private var mockYarnClient: Option[YarnClient] = None
+
   // YarnClient is thread safe. Create once, share it across threads.
   lazy val yarnClient = {
 val c = YarnClient.createYarnClient()
@@ -59,9 +62,9 @@ object SparkYarnApp extends Logging {
   private def getYarnPollInterval(livyConf: LivyConf): FiniteDuration =
 livyConf.getTimeAsMs(LivyConf.YARN_POLL_INTERVAL) milliseconds
 
-  private val appType = Set("SPARK").asJava
+  private[utils] val appType = Set("SPARK").asJava
 
-  private val leakedAppTags = new 
java.util.concurrent.ConcurrentHashMap[String, Long]()
+  private[utils] val leakedAppTags = new 
java.util.concurrent.ConcurrentHashMap[String, Long]()
 
   private var sessionLeakageCheckTimeout: Long = _
 
@@ -69,24 +72,34 @@ object SparkYarnApp extends Logging {
 
   private val leakedAppsGCThread = new Thread() {
 override def run(): Unit = {
+  val client = {
+mockYarnClient match {
+  case Some(client) => client
+  case None => yarnClient
+}
+  }
+
   while (true) {
 if (!leakedAppTags.isEmpty) {
   // kill the app if found it and remove it if exceeding a threshold
   val iter = leakedAppTags.entrySet().iterator()
-  var isRemoved = false
   val now = System.currentTimeMillis()
-  val apps = yarnClient.getApplications(appType).asScala
+  val apps = client.getApplications(appType).asScala
+
   while(iter.hasNext) {
+var isRemoved = false
 val entry = iter.next()
+
 apps.find(_.getApplicationTags.contains(entry.getKey))
   .foreach({ e =>
 info(s"Kill leaked app ${e.getApplicationId}")
-yarnClient.killApplication(e.getApplicationId)
+client.killApplication(e.getApplicationId)
 iter.remove()
 isRemoved = true
   })
+
 if (!isRemoved) {
-  if ((entry.getValue - now) > sessionLeakageCheckTimeout) {
+  if ((now - entry.getValue) > sessionLeakageCheckTimeout) {
 iter.remove()
 info(s"Remove leaked yarn app tag ${entry.getKey}")
   }
diff --git a/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala 
b/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala
index 064bb77..d43125d 100644
--- a/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala
+++ b/server/src/test/scala/org/

[incubator-livy] branch master updated: [LIVY-715][DOC] The configuration in the template is inconsistent with LivyConf.scala

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 8f1e898  [LIVY-715][DOC] The configuration in the template is 
inconsistent with LivyConf.scala
8f1e898 is described below

commit 8f1e8986b2fa8a5d4047c900c224478bb1829489
Author: captainzmc 
AuthorDate: Fri Nov 22 10:39:20 2019 +0800

[LIVY-715][DOC] The configuration in the template is inconsistent with 
LivyConf.scala

## What changes were proposed in this pull request?

When I test livy impersonation found that, in livy.conf.template the 
value of livy.impersonation.enabled is true. So I thought impersonation was 
enabled by default.
However, impersonation was not turned on when we test. I found that the 
real configuration in LivyConf. scala is false.
This can mislead users.

## How was this patch tested?

no need

Author: captainzmc 

Closes #261 from captainzmc/apache-livy-master.
---
 conf/livy.conf.template | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/conf/livy.conf.template b/conf/livy.conf.template
index de7c248..1fe6047 100644
--- a/conf/livy.conf.template
+++ b/conf/livy.conf.template
@@ -62,7 +62,7 @@
 # livy.server.session.state-retain.sec = 600s
 
 # If livy should impersonate the requesting users when creating a new session.
-# livy.impersonation.enabled = true
+# livy.impersonation.enabled = false
 
 # Logs size livy can cache for each session/batch. 0 means don't cache the 
logs.
 # livy.cache-log.size = 200

[incubator-livy] branch master updated: [LIVY-707] Add audit log for SqlJobs from ThriftServer

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 6261c57  [LIVY-707] Add audit log for SqlJobs from ThriftServer
6261c57 is described below

commit 6261c57be8df66d5f3fc3ccdaa15f8c4e1989d1d
Author: BoneAn 
AuthorDate: Thu Nov 14 09:51:48 2019 +0800

[LIVY-707] Add audit log for SqlJobs from ThriftServer

## What changes were proposed in this pull request?

We should add audit logs in thriftServer for admin to easily to manage 
operations,

## How was this patch tested?

An audit log example showed below,

```
19/11/06 16:38:30 INFO ThriftServerAudit$: user: test ipAddress: 
10.25.22.46 query: select count(*) from test1 beforeExecute: 1573029416951 
afterExecute: 1573029510972 time spent: 94021
```

Author: BoneAn 

Closes #255 from huianyi/LIVY-707.
---
 .../LivyExecuteStatementOperation.scala|  6 
 .../livy/thriftserver/ThriftServerAudit.scala  | 36 ++
 2 files changed, 42 insertions(+)

diff --git 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyExecuteStatementOperation.scala
 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyExecuteStatementOperation.scala
index ebb8e1d..f7d6c16 100644
--- 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyExecuteStatementOperation.scala
+++ 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyExecuteStatementOperation.scala
@@ -137,6 +137,8 @@ class LivyExecuteStatementOperation(
 }
 setState(OperationState.RUNNING)
 
+val before = System.currentTimeMillis()
+
 try {
   rpcClient.executeSql(sessionHandle, statementId, statement).get()
 } catch {
@@ -147,6 +149,10 @@ class LivyExecuteStatementOperation(
 throw new HiveSQLException(e)
 }
 setState(OperationState.FINISHED)
+
+val sessionInfo = sessionManager.getSessionInfo(sessionHandle)
+val after = System.currentTimeMillis()
+ThriftServerAudit.audit(sessionInfo.username, sessionInfo.ipAddress, 
statement, before, after)
   }
 
   def close(): Unit = {
diff --git 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/ThriftServerAudit.scala
 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/ThriftServerAudit.scala
new file mode 100644
index 000..5bf7760
--- /dev/null
+++ 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/ThriftServerAudit.scala
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver
+
+import org.apache.livy.Logging
+
+object ThriftServerAudit extends Logging {
+
+  def audit(
+  user: String,
+  ipAddress: String,
+  query: String,
+  startTime: Long,
+  endTime: Long): Unit = {
+info(
+  s"user: $user ipAddress: $ipAddress query: ${query.replace('\n', ' ')} " 
+
+s"start time: ${startTime} end time: ${endTime} " +
+s"time spent: ${Math.round((endTime - startTime) / 1000)}s")
+  }
+
+}

[incubator-livy] branch master updated: [LIVY-711][TEST] Fix Travis fails to build on Ubuntu16.04

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 6c7df77  [LIVY-711][TEST] Fix Travis fails to build on Ubuntu16.04
6c7df77 is described below

commit 6c7df77204a5a7bfb04beb9789253120d8d7db6c
Author: runzhiwang 
AuthorDate: Tue Nov 12 15:35:34 2019 +0800

[LIVY-711][TEST] Fix Travis fails to build on Ubuntu16.04

## What changes were proposed in this pull request?

Fix Travis fails to build on Ubuntu16.04

## How was this patch tested?

1. Previously the `dist `of .travis.yml is `xenial ` which points to Ubuntu 
14.04.5 LTS. Travis build successfully.

![image](https://user-images.githubusercontent.com/51938049/68647534-10c81e00-0559-11ea-9152-362711b30946.png)


![image](https://user-images.githubusercontent.com/51938049/68647646-669cc600-0559-11ea-9413-9c29860d63f5.png)

2. However, recently `xenial ` points Ubuntu 16.04.6 LTS which needs jdk > 
8, but 8 is needed by Livy, so travis build failed.

![image](https://user-images.githubusercontent.com/51938049/68647880-edea3980-0559-11ea-8135-f5a68b3d303d.png)

![image](https://user-images.githubusercontent.com/51938049/68648187-baf47580-055a-11ea-90cf-8592e628a32c.png)

![image](https://user-images.githubusercontent.com/51938049/68647919-0e19f880-055a-11ea-94b9-4f19099654cb.png)

3. So I change the dist to `trusty` which points to Ubuntu 14.04.5 LTS 
according to the travis doc, and travis build successfully.

![image](https://user-images.githubusercontent.com/51938049/68648018-4d484980-055a-11ea-8358-c4234d7cf56c.png)

![image](https://user-images.githubusercontent.com/51938049/68648028-5507ee00-055a-11ea-8f6e-7c7efcbc6390.png)

Author: runzhiwang 

Closes #257 from runzhiwang/travis-build-error.
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index bc26f82..c2c0ffd 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -16,7 +16,7 @@
 #
 
 sudo: required
-dist: xenial
+dist: trusty
 language: scala
   - 2.11.12
 

[incubator-livy] branch master updated: [LIVY-708][SERVER] Align curator jars version

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 1f64264  [LIVY-708][SERVER] Align curator jars version
1f64264 is described below

commit 1f64264135409f4a9c7f094660c7d4d6e4130a34
Author: yihengwang 
AuthorDate: Tue Nov 12 09:53:44 2019 +0800

[LIVY-708][SERVER] Align curator jars version

## What changes were proposed in this pull request?
Livy server has dependency of Apache Curator through hadoop client. 
However, the versions of the curator jars are not aligned. Here're the curator 
jars after build

* curator-client-2.7.1.jar
* curator-framework-2.7.1.jar
* curator-recipes-2.6.0.jar
This will cause Method not found issue in some case:
```
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)V
```
This patch specify the version of curator-recipes to 2.7.1.

## How was this patch tested?
Manually test in the env where no such method exception are thrown.
Existing UT/IT

Author: yihengwang 

Closes #256 from yiheng/fix_708.
---
 pom.xml| 1 +
 server/pom.xml | 6 ++
 2 files changed, 7 insertions(+)

diff --git a/pom.xml b/pom.xml
index 5bcf1a3..2f6dd09 100644
--- a/pom.xml
+++ b/pom.xml
@@ -124,6 +124,7 @@
 
 2.0.0-M21
 1.0.0-M33
+2.7.1
 
 

[incubator-livy] branch master updated: [MINOR][DOC] Add missing session kind in UI

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new c7313c5  [MINOR][DOC] Add missing session kind in UI
c7313c5 is described below

commit c7313c5648c63089413464b47f33c9749dbabb20
Author: Yishuang Lu 
AuthorDate: Thu Oct 31 10:47:46 2019 +0800

[MINOR][DOC] Add missing session kind in UI

## What changes were proposed in this pull request?

Add missing session kind in UI

## How was this patch tested?

No need to test

Author: Yishuang Lu 

Closes #248 from lys0716/dev-1.
---
 .../org/apache/livy/server/ui/static/html/sessions-table.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git 
a/server/src/main/resources/org/apache/livy/server/ui/static/html/sessions-table.html
 
b/server/src/main/resources/org/apache/livy/server/ui/static/html/sessions-table.html
index d832e07..6818f76 100644
--- 
a/server/src/main/resources/org/apache/livy/server/ui/static/html/sessions-table.html
+++ 
b/server/src/main/resources/org/apache/livy/server/ui/static/html/sessions-table.html
@@ -52,7 +52,7 @@
 
 
   
+title="Session kind (spark, pyspark, pyspark3, sparkr or sql)">
 Session Kind
   
 
@@ -72,4 +72,4 @@
   
   
   
-
\ No newline at end of file
+

[incubator-livy] branch master updated (85837e3 -> ba12b51)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from 85837e3  [LIVY-695] Upgrade JQuery to 3.4.1 and Bootstrap to 3.4.1
 add ba12b51  [LIVY-697] Rsc client cannot resolve the hostname of driver 
in yarn-cluster mode

No new revisions were added by this update.

Summary of changes:
 rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

[incubator-livy] branch master updated: [LIVY-690][THRIFT] Exclude curator in thrift server pom to avoid conflict jars

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 0804c8e  [LIVY-690][THRIFT] Exclude curator in thrift server pom to 
avoid conflict jars
0804c8e is described below

commit 0804c8ea8ece67d01ababec616c9ad8e3b15dc9f
Author: Yiheng Wang 
AuthorDate: Sun Sep 29 16:24:30 2019 +0800

[LIVY-690][THRIFT] Exclude curator in thrift server pom to avoid conflict 
jars

## What changes were proposed in this pull request?
Currently, thrift server has a dependency of curator-client:2.12.0 through 
the hive service. After the build, a `curator-client-2.12.0.jar` file will be 
generated in the `jars` folder. It is conflicted with the 
`curator-client-2.7.1.jar` file, which is used by livy server.

We observed that in some JDK, the `curator-client-2.12.0.jar` is loaded 
before the `curator-client-2.7.1.jar`, and will crash the recovery enabled livy 
server.

In this patch, we exclude the `org.apache.curator` modules from the hive 
dependency.

## How was this patch tested?
Manual test and existing UT/ITs

Author: Yiheng Wang 

Closes #239 from yiheng/exclude_curator.
---
 thriftserver/server/pom.xml | 4 
 1 file changed, 4 insertions(+)

diff --git a/thriftserver/server/pom.xml b/thriftserver/server/pom.xml
index fe17f96..86f0b86 100644
--- a/thriftserver/server/pom.xml
+++ b/thriftserver/server/pom.xml
@@ -58,6 +58,10 @@
   org.eclipse.jetty
   *
 
+
+  org.apache.curator
+  *
+
   
 
 

[incubator-livy] branch master updated: [LIVY-688] Error message of BypassJobStatus should contains cause information of Exception

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 0ea4a14  [LIVY-688] Error message of BypassJobStatus should contains 
cause information of Exception
0ea4a14 is described below

commit 0ea4a14fcd77a954898f58b4a624d0d515b52eda
Author: weiwenda 
AuthorDate: Sun Sep 29 16:19:41 2019 +0800

[LIVY-688] Error message of BypassJobStatus should contains cause 
information of Exception

## What changes were proposed in this pull request?
Change the implement of org.apache.livy.rsc.Utils.stackTraceAsString to 
guava Throwables.getStackTraceAsString, so that user can receive details of 
error message by calling org.apache.livy.client.http.JobHandleImpl.get.

https://issues.apache.org/jira/browse/LIVY-688

Author: weiwenda 

Closes #237 from WeiWenda/livy-err-clear.
---
 rsc/src/main/java/org/apache/livy/rsc/Utils.java | 12 +---
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/rsc/src/main/java/org/apache/livy/rsc/Utils.java 
b/rsc/src/main/java/org/apache/livy/rsc/Utils.java
index d2c0059..3c8a5e6 100644
--- a/rsc/src/main/java/org/apache/livy/rsc/Utils.java
+++ b/rsc/src/main/java/org/apache/livy/rsc/Utils.java
@@ -17,6 +17,8 @@
 
 package org.apache.livy.rsc;
 
+import java.io.PrintWriter;
+import java.io.StringWriter;
 import java.util.concurrent.ThreadFactory;
 import java.util.concurrent.atomic.AtomicInteger;
 
@@ -91,13 +93,9 @@ public class Utils {
   }
 
   public static String stackTraceAsString(Throwable t) {
-StringBuilder sb = new StringBuilder();
-sb.append(t.getClass().getName()).append(": ").append(t.getMessage());
-for (StackTraceElement e : t.getStackTrace()) {
-  sb.append("\n");
-  sb.append(e.toString());
-}
-return sb.toString();
+StringWriter stringWriter = new StringWriter();
+t.printStackTrace(new PrintWriter(stringWriter));
+return stringWriter.toString();
   }
 
   public static  void addListener(Future future, final FutureListener 
lsnr) {

[incubator-livy] branch master updated: [LIVY-658] RSCDriver should catch exception if cancel job failed during shutdown

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 6c53d2b  [LIVY-658] RSCDriver should catch exception if cancel job 
failed during shutdown
6c53d2b is described below

commit 6c53d2b41975f5b0171ce45320deb4b69f8ddea7
Author: Jeffrey(Xilang) Yan <7855100+yan...@users.noreply.github.com>
AuthorDate: Sun Sep 29 10:09:10 2019 +0800

[LIVY-658] RSCDriver should catch exception if cancel job failed during 
shutdown

## What changes were proposed in this pull request?

Currently, if startup meet exception, exception will trigger spark to 
shutdown, then trigger cancel job, but cancel job will throw another exception 
due to spark is not initialized. The new exception will swallow the old 
exception.

https://issues.apache.org/jira/browse/LIVY-658

Before changes:
![cancel job 
exception](https://user-images.githubusercontent.com/7855100/64118287-f0961900-cdc9-11e9-9b72-d051fb4bdbdf.jpg)

After changes:
![cancel job exception after 
fix](https://user-images.githubusercontent.com/7855100/64118295-f4c23680-cdc9-11e9-9a2d-38efa0770a99.jpg)

## How was this patch tested?

Tested manually， and  add unit test.

Please review https://livy.incubator.apache.org/community/ before opening a 
pull request.

Author: Jeffrey(Xilang) Yan <7855100+yan...@users.noreply.github.com>

Closes #223 from yantzu/initialize_exception_swallow_by_shutdown_exception.
---
 .../java/org/apache/livy/rsc/driver/RSCDriver.java |  6 +++-
 .../org/apache/livy/rsc/driver/TestRSCDriver.java  | 36 ++
 2 files changed, 41 insertions(+), 1 deletion(-)

diff --git a/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java 
b/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
index eeba300..0d8eec5 100644
--- a/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
+++ b/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
@@ -124,7 +124,11 @@ public class RSCDriver extends BaseProtocol {
 
 // Cancel any pending jobs.
 for (JobWrapper job : activeJobs.values()) {
-  job.cancel();
+  try {
+job.cancel();
+  } catch (Exception e) {
+LOG.warn("Error during cancel job.", e);
+  }
 }
 
 try {
diff --git a/rsc/src/test/java/org/apache/livy/rsc/driver/TestRSCDriver.java 
b/rsc/src/test/java/org/apache/livy/rsc/driver/TestRSCDriver.java
new file mode 100644
index 000..df6ccea
--- /dev/null
+++ b/rsc/src/test/java/org/apache/livy/rsc/driver/TestRSCDriver.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.rsc.driver;
+
+import org.apache.spark.SparkConf;
+import org.junit.Test;
+
+import org.apache.livy.rsc.BaseProtocol;
+import org.apache.livy.rsc.RSCConf;
+
+public class TestRSCDriver {
+  @Test(expected = IllegalArgumentException.class)
+  public void testCancelJobAfterInitializeFailed()
+  throws Exception {
+//use empty Conf to trigger initialize throw IllegalArgumentException
+RSCDriver rscDriver = new RSCDriver(new SparkConf(), new RSCConf());
+//add asynchronous dummy job request to trigger cancel job failure
+rscDriver.handle(null, new BaseProtocol.BypassJobRequest("RequestId", 
null, null, false));
+rscDriver.run();
+  }
+}

[incubator-livy] branch master updated: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new b8251eb  [LIVY-644][TEST] Flaky test: Failed to execute goal 
org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
project livy-coverage-report
b8251eb is described below

commit b8251eb9b3d63c77c61a5950dee5958b654d9633
Author: yihengwang 
AuthorDate: Thu Sep 19 10:35:26 2019 +0800

[LIVY-644][TEST] Flaky test: Failed to execute goal 
org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
project livy-coverage-report

## What changes were proposed in this pull request?
This patch fixes the flaky test: Failed to execute goal 
org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
project livy-coverage-report.

When JVM shutdown no gracefully in a test, the code coverage data file 
generated by jacoco may be corrupt. Jacoco will throw an exception when 
generate code coverage report.

In Livy integration test, two test cases shut down no gracefully(one of 
them uses System.exit). We can find random failure when jacoco process code 
coverage data file generated by that test case.

In this patch, we turn off the code coverage analysis on these two test 
cases.

## How was this patch tested?
Compare the jacoco data file generated in the integration test. Before the 
fix, there're 18 files, and after the fix there're 16 files, which means the 
fix works.

Run 10 builds on Travis each before and after the fix:
1. Before the fix: 3 builds failed due to the jacoco code coverage exception
2. After the fix: No build failed

Existing UTs and ITs.

Author: yihengwang 

Closes #229 from yiheng/fix_644.
---
 .../src/test/scala/org/apache/livy/test/InteractiveIT.scala | 6 --
 rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java  | 2 +-
 rsc/src/main/java/org/apache/livy/rsc/RSCConf.java  | 1 +
 3 files changed, 6 insertions(+), 3 deletions(-)

diff --git 
a/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala 
b/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
index 0613bf3..0c3d632 100644
--- a/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
+++ b/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
@@ -114,14 +114,16 @@ class InteractiveIT extends BaseIntegrationTestSuite {
   }
 
   test("application kills session") {
-withNewSession(Spark) { s =>
+val noCodeCoverageConf = 
s"${RSCConf.Entry.TEST_NO_CODE_COVERAGE_ANALYSIS.key()}"
+withNewSession(Spark, Map(noCodeCoverageConf -> "true")) { s =>
   s.runFatalStatement("System.exit(0)")
 }
   }
 
   test("should kill RSCDriver if it doesn't respond to end session") {
 val testConfName = 
s"${RSCConf.LIVY_SPARK_PREFIX}${RSCConf.Entry.TEST_STUCK_END_SESSION.key()}"
-withNewSession(Spark, Map(testConfName -> "true")) { s =>
+val noCodeCoverageConf = 
s"${RSCConf.Entry.TEST_NO_CODE_COVERAGE_ANALYSIS.key()}"
+withNewSession(Spark, Map(testConfName -> "true", noCodeCoverageConf -> 
"true")) { s =>
   val appId = s.appId()
   s.stop()
   val appReport = cluster.yarnClient.getApplicationReport(appId)
diff --git a/rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java 
b/rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java
index 5a819d5..d67b78a 100644
--- a/rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java
+++ b/rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java
@@ -207,7 +207,7 @@ class ContextLauncher {
 if (!conf.getBoolean(CLIENT_IN_PROCESS) &&
 // For tests which doesn't shutdown RscDriver gracefully, JaCoCo exec 
isn't dumped properly.
 // Disable JaCoCo for this case.
-!conf.getBoolean(TEST_STUCK_END_SESSION)) {
+!conf.getBoolean(TEST_NO_CODE_COVERAGE_ANALYSIS)) {
   // For testing; propagate jacoco settings so that we also do coverage 
analysis
   // on the launched driver. We replace the name of the main file 
("main.exec")
   // so that we don't end up fighting with the main test launcher.
diff --git a/rsc/src/main/java/org/apache/livy/rsc/RSCConf.java 
b/rsc/src/main/java/org/apache/livy/rsc/RSCConf.java
index d2496b5..4c45956 100644
--- a/rsc/src/main/java/org/apache/livy/rsc/RSCConf.java
+++ b/rsc/src/main/java/org/apache/livy/rsc/RSCConf.java
@@ -71,6 +71,7 @@ public class RSCConf extends ClientConf {
 SASL_MECHANISMS("rpc.sasl.mechanisms", "DIGEST-MD5"),
 SASL_QOP("rpc.sasl.qop", null),
 
+
TEST_NO_CODE_COVERAGE_ANALYSIS("test.d

[incubator-livy] branch master updated (145cc2b -> e2e966b)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from 145cc2b  [LIVY-633][SERVER] session should not be gc-ed for long 
running queries
 add e2e966b  [LIVY-657][TEST] Fix travis failed on should not create 
sessions with duplicate names

No new revisions were added by this update.

Summary of changes:
 .../test/scala/org/apache/livy/sessions/SessionManagerSpec.scala  | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

[incubator-livy] branch master updated: [LIVY-633][SERVER] session should not be gc-ed for long running queries

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 145cc2b  [LIVY-633][SERVER] session should not be gc-ed for long 
running queries
145cc2b is described below

commit 145cc2b77db4c7d7bdd8f953dc3a16856d9fcf0f
Author: yihengwang 
AuthorDate: Tue Sep 17 17:21:28 2019 +0800

[LIVY-633][SERVER] session should not be gc-ed for long running queries

## What changes were proposed in this pull request?
Currently, Livy records the last activity time of the session before 
statement execution. If a statement runs too long, exceeding then the session 
timeout, the session will be garbage collected after the statement execution.

This should not be the expected behavior. The statement execution time 
should not be count into idle. We should update the last activity time after 
the statement execution.

We cannot be updated when session changes state from busy to idle in the 
Session class. So in this patch, we add a replLastActivity field into the 
rscClient, which will be updated when the repl state changes. So when session 
changes its state from busy to idle, this field will catch the time and finally 
reflect on the session last activity.

## How was this patch tested?
Manual test. Also, add a new unit test.

Existing unit tests and integration tests.

Author: yihengwang 
Author: Yiheng Wang 

Closes #224 from yiheng/fix_633.
---
 rsc/pom.xml|  7 +++
 rsc/src/main/java/org/apache/livy/rsc/RSCClient.java   | 18 ++
 .../livy/server/interactive/InteractiveSession.scala   | 12 
 .../server/interactive/InteractiveSessionSpec.scala| 15 +++
 4 files changed, 52 insertions(+)

diff --git a/rsc/pom.xml b/rsc/pom.xml
index dcb58a6..1f3d6a3 100644
--- a/rsc/pom.xml
+++ b/rsc/pom.xml
@@ -49,6 +49,13 @@
   ${project.version}
   test
 
+
+  org.apache.livy
+  livy-core_${scala.binary.version}
+  ${project.version}
+  
+  provided
+
 
 
   com.esotericsoftware.kryo
diff --git a/rsc/src/main/java/org/apache/livy/rsc/RSCClient.java 
b/rsc/src/main/java/org/apache/livy/rsc/RSCClient.java
index f2879b8..c1c9534 100644
--- a/rsc/src/main/java/org/apache/livy/rsc/RSCClient.java
+++ b/rsc/src/main/java/org/apache/livy/rsc/RSCClient.java
@@ -44,6 +44,7 @@ import org.apache.livy.client.common.BufferUtils;
 import org.apache.livy.rsc.driver.AddFileJob;
 import org.apache.livy.rsc.driver.AddJarJob;
 import org.apache.livy.rsc.rpc.Rpc;
+import org.apache.livy.sessions.SessionState;
 
 import static org.apache.livy.rsc.RSCConf.Entry.*;
 
@@ -64,6 +65,8 @@ public class RSCClient implements LivyClient {
   private Process driverProcess;
   private volatile boolean isAlive;
   private volatile String replState;
+  // Record the last activity timestamp of the repl
+  private volatile long replLastActivity = System.nanoTime();
 
   RSCClient(RSCConf conf, Promise ctx, Process driverProcess) 
throws IOException {
 this.conf = conf;
@@ -315,6 +318,16 @@ public class RSCClient implements LivyClient {
 return replState;
   }
 
+  /**
+   * Get the timestamp of the last activity of the repl. It will be updated 
when the repl state
+   * changed from busy to idle
+   *
+   * @return last activity timestamp
+   */
+  public long getReplLastActivity() {
+return replLastActivity;
+  }
+
   private class ClientProtocol extends BaseProtocol {
 
  JobHandleImpl submit(Job job) {
@@ -411,6 +424,11 @@ public class RSCClient implements LivyClient {
 
 private void handle(ChannelHandlerContext ctx, ReplState msg) {
   LOG.trace("Received repl state for {}", msg.state);
+  // Update last activity timestamp when state change is from busy to idle.
+  if (SessionState.Busy$.MODULE$.state().equals(replState) && msg != null 
&&
+SessionState.Idle$.MODULE$.state().equals(msg.state)) {
+replLastActivity = System.nanoTime();
+  }
   replState = msg.state;
 }
   }
diff --git 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
index bccdb4d..cdeddda 100644
--- 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
+++ 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
@@ -626,4 +626,16 @@ class InteractiveSession(
   }
 
   override def infoChanged(appInfo: AppInfo): Unit = { this.appInfo = appInfo }
+
+  override def lastActivity: Long = {
+val serverSideLastActivity = super.lastActivity
+if (serverSideState == SessionState.Running) {
+  // If the rsc client is running, w

[incubator-livy] branch master updated (3cdb7b2 -> 3c4eab9)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from 3cdb7b2  [LIVY-640] Add tests for ThriftServer
 add 3c4eab9  [LIVY-659][TEST] Fix travis failed on can kill spark-submit 
while it's running

No new revisions were added by this update.

Summary of changes:
 .../org/apache/livy/utils/SparkYarnAppSpec.scala   | 22 ++
 1 file changed, 18 insertions(+), 4 deletions(-)

[incubator-livy] branch master updated: [LIVY-645] Add Session Name, Owner, Proxy User information to Web UI

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new c4bb55b  [LIVY-645] Add Session Name, Owner, Proxy User information to 
Web UI
c4bb55b is described below

commit c4bb55b12f5978ba151d92e1f1e655f91233138b
Author: Jeffrey(Xilang) Yan <7855100+yan...@users.noreply.github.com>
AuthorDate: Thu Sep 5 14:14:17 2019 +0800

[LIVY-645] Add Session Name, Owner, Proxy User information to Web UI

## What changes were proposed in this pull request?

1, Web UI, add Session Name to Interactive Sessions list
2, Web UI, add Session Name, Owner, Proxy User to Batch Sessions list
3,~~fix thrift server session doesn't set Session Name issue.~~ Move to PR 
#218

## How was this patch tested?

Update existing unit test, and have tested manually.


![AddName](https://user-images.githubusercontent.com/7855100/63513238-3de7d000-c518-11e9-8b18-613874ed635a.jpg)

![AddName2](https://user-images.githubusercontent.com/7855100/63513246-42ac8400-c518-11e9-9019-679bb82e4bb0.jpg)

Author: Jeffrey(Xilang) Yan <7855100+yan...@users.noreply.github.com>

Closes #207 from yantzu/add_more_info_to_ui.
---
 .../apache/livy/server/ui/static/html/batches-table.html | 16 
 .../livy/server/ui/static/html/sessions-table.html   |  6 ++
 .../org/apache/livy/server/ui/static/js/all-sessions.js  |  4 
 .../org/apache/livy/server/ui/static/js/session.js   |  1 +
 .../apache/livy/server/batch/BatchSessionServlet.scala   |  6 --
 .../org/apache/livy/server/batch/BatchServletSpec.scala  |  7 ++-
 6 files changed, 37 insertions(+), 3 deletions(-)

diff --git 
a/server/src/main/resources/org/apache/livy/server/ui/static/html/batches-table.html
 
b/server/src/main/resources/org/apache/livy/server/ui/static/html/batches-table.html
index aeab23b..b198e98 100644
--- 
a/server/src/main/resources/org/apache/livy/server/ui/static/html/batches-table.html
+++ 
b/server/src/main/resources/org/apache/livy/server/ui/static/html/batches-table.html
@@ -35,6 +35,22 @@
 
 
   
+Name
+  
+
+
+  
+Owner
+  
+
+
+  
+Proxy User
+  
+
+
+  
 State
diff --git 
a/server/src/main/resources/org/apache/livy/server/ui/static/html/sessions-table.html
 
b/server/src/main/resources/org/apache/livy/server/ui/static/html/sessions-table.html
index 8c64100..d832e07 100644
--- 
a/server/src/main/resources/org/apache/livy/server/ui/static/html/sessions-table.html
+++ 
b/server/src/main/resources/org/apache/livy/server/ui/static/html/sessions-table.html
@@ -35,6 +35,12 @@
   
 
 
+  
+Name
+  
+
+
   
 Owner
   
diff --git 
a/server/src/main/resources/org/apache/livy/server/ui/static/js/all-sessions.js 
b/server/src/main/resources/org/apache/livy/server/ui/static/js/all-sessions.js
index 90de331..6e35702 100644
--- 
a/server/src/main/resources/org/apache/livy/server/ui/static/js/all-sessions.js
+++ 
b/server/src/main/resources/org/apache/livy/server/ui/static/js/all-sessions.js
@@ -21,6 +21,7 @@ function loadSessionsTable(sessions) {
   "" +
 tdWrap(uiLink("session/" + session.id, session.id)) +
 tdWrap(appIdLink(session)) +
+tdWrap(session.name) +
 tdWrap(session.owner) +
 tdWrap(session.proxyUser) +
 tdWrap(session.kind) +
@@ -37,6 +38,9 @@ function loadBatchesTable(sessions) {
   "" +
 tdWrap(session.id) +
 tdWrap(appIdLink(session)) +
+tdWrap(session.name) +
+tdWrap(session.owner) +
+tdWrap(session.proxyUser) +
 tdWrap(session.state) +
 tdWrap(logLinks(session, "batch")) +
 ""
diff --git 
a/server/src/main/resources/org/apache/livy/server/ui/static/js/session.js 
b/server/src/main/resources/org/apache/livy/server/ui/static/js/session.js
index d49e0ec..c87e5ca 100644
--- a/server/src/main/resources/org/apache/livy/server/ui/static/js/session.js
+++ b/server/src/main/resources/org/apache/livy/server/ui/static/js/session.js
@@ -88,6 +88,7 @@ function appendSummary(session) {
 "Session " + session.id + "" +
 "" +
   sumWrap("Application Id", appIdLink(session)) +
+  sumWrap("Name", session.name) +
   sumWrap("Owner", session.owner) +
   sumWrap("Proxy User", session.proxyUser) +
   sumWrap("Session Kind", session.kind) +
diff --git 
a/server/src/main/scala/org/apache/livy/server/batch/BatchSessionServlet.scala 
b/server/src/main/scala/org/apache/livy/server/batch/BatchSessionServlet.scala
index 0bf6799..d14e649 100644
--- 
a/server/src/main/scala/org/a

[incubator-livy] branch master updated: [LIVY-652] Thrifserver doesn't set session name correctly

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new e6b6ec0  [LIVY-652] Thrifserver doesn't set session name correctly
e6b6ec0 is described below

commit e6b6ec060d18d33a300ffa7d766ad46d6c4c70cf
Author: Jeffrey(Xilang) Yan <7855100+yan...@users.noreply.github.com>
AuthorDate: Wed Sep 4 11:08:58 2019 +0800

[LIVY-652] Thrifserver doesn't set session name correctly

## What changes were proposed in this pull request?

Thriftserver should set session name as the value passed in 
livy.session.name , but it current always set it NONE

## How was this patch tested?

add IT

https://issues.apache.org/jira/browse/LIVY-652

Author: Jeffrey(Xilang) Yan <7855100+yan...@users.noreply.github.com>

Closes #218 from yantzu/thrift_session_has_no_name.
---
 .../livy/thriftserver/LivyThriftSessionManager.scala|  2 +-
 .../apache/livy/thriftserver/ThriftServerSuites.scala   | 17 +
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyThriftSessionManager.scala
 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyThriftSessionManager.scala
index bc62084..67c8265 100644
--- 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyThriftSessionManager.scala
+++ 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyThriftSessionManager.scala
@@ -228,7 +228,7 @@ class LivyThriftSessionManager(val server: 
LivyThriftServer, val livyConf: LivyC
   createInteractiveRequest.kind = Spark
   val newSession = InteractiveSession.create(
 server.livySessionManager.nextId(),
-None,
+createInteractiveRequest.name,
 username,
 None,
 server.livyConf,
diff --git 
a/thriftserver/server/src/test/scala/org/apache/livy/thriftserver/ThriftServerSuites.scala
 
b/thriftserver/server/src/test/scala/org/apache/livy/thriftserver/ThriftServerSuites.scala
index 438d86c..48750da 100644
--- 
a/thriftserver/server/src/test/scala/org/apache/livy/thriftserver/ThriftServerSuites.scala
+++ 
b/thriftserver/server/src/test/scala/org/apache/livy/thriftserver/ThriftServerSuites.scala
@@ -396,6 +396,23 @@ class BinaryThriftServerSuite extends ThriftServerBaseTest 
with CommonThriftTest
   getTypeInfoTest(c)
 }
   }
+
+  test("LIVY-652: should set session name correctly") {
+val livySessionManager = 
LivyThriftServer.getInstance.get.livySessionManager
+val testSessionName = "MySessionName"
+assert(livySessionManager.get(testSessionName).isEmpty)
+withJdbcConnection("default", 
Seq(s"livy.session.name=${testSessionName}")) { c =>
+  // execute a statement and block until session is ready
+  val statement = c.createStatement()
+  try {
+statement.executeQuery("select current_database()")
+  } finally {
+statement.close()
+  }
+
+  assert(livySessionManager.get(testSessionName).get.name.get == 
testSessionName)
+}
+  }
 }
 
 class HttpThriftServerSuite extends ThriftServerBaseTest with 
CommonThriftTests {

[incubator-livy] branch master updated: [LIVY-519][TEST] Fix travis failed on should kill yarn app

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 830d740  [LIVY-519][TEST] Fix travis failed on should kill yarn app
830d740 is described below

commit 830d740db5193314f469fa7bcbd4f6f93cbfc08b
Author: runzhiwang 
AuthorDate: Tue Sep 3 12:58:51 2019 +0800

[LIVY-519][TEST] Fix travis failed on should kill yarn app

## What changes were proposed in this pull request?

Fix travis failed on "should kill yarn app"

The cause of failed is as follows:
1. When create SparkYarnApp, the yarnAppMonitorThread will be created, 
which change app state to Failed. Because before recent commit 
https://github.com/apache/incubator-livy/commit/a90f4fac8be27a38cc961c24043a494a739ff188,
 the pair  which was mocked in test, but was not defined in 
mapYarnState, so the state of app will be changed to failed.

2. Then the test kills app, which will call killApplication when the app is 
running. However the app has been changed to failed in step 1, so 
killApplication won't be called, and 
verify(mockYarnClient).killApplication(appId) failed.

3. So if yarnAppMonitorThread changes app state before main thread kills 
app, the test will failed. If not, the test will succeed.

4. Though the recent commit 
https://github.com/apache/incubator-livy/commit/a90f4fac8be27a38cc961c24043a494a739ff188
 fixed the bug accidentally, it is necessary to ensure the app is running 
before kill app.

## How was this patch tested?

Existed UT and IT.

Author: runzhiwang 

Closes #221 from runzhiwang/LIVY-519.
---
 server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala | 3 ++-
 server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala 
b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
index d51af62..14af9fa 100644
--- a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
+++ b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
@@ -209,7 +209,8 @@ class SparkYarnApp private[utils] (
   .getOrElse(IndexedSeq.empty)
   }
 
-  private def isRunning: Boolean = {
+  // Exposed for unit test.
+  private[utils] def isRunning: Boolean = {
 state != SparkApp.State.FAILED && state != SparkApp.State.FINISHED &&
   state != SparkApp.State.KILLED
   }
diff --git a/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala 
b/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala
index 823ae72..672444f 100644
--- a/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala
+++ b/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala
@@ -35,7 +35,7 @@ import org.mockito.stubbing.Answer
 import org.scalatest.FunSpec
 import org.scalatest.mock.MockitoSugar.mock
 
-import org.apache.livy.{LivyBaseUnitTestSuite, LivyConf}
+import org.apache.livy.{LivyBaseUnitTestSuite, LivyConf, Utils}
 import org.apache.livy.utils.SparkApp._
 
 class SparkYarnAppSpec extends FunSpec with LivyBaseUnitTestSuite {
@@ -145,6 +145,7 @@ class SparkYarnAppSpec extends FunSpec with 
LivyBaseUnitTestSuite {
 
when(mockYarnClient.getApplicationReport(appId)).thenReturn(mockAppReport)
 
 val app = new SparkYarnApp(appTag, appIdOption, None, None, livyConf, 
mockYarnClient)
+Utils.waitUntil({ () => app.isRunning }, Duration(10, 
TimeUnit.SECONDS))
 cleanupThread(app.yarnAppMonitorThread) {
   app.kill()
   appKilled = true

[incubator-livy] branch master updated (a90f4fa -> 4521ef9)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from a90f4fa  [LIVY-642] Fix a rare status happened in yarn cause SparkApp 
change into error state
 add 4521ef9  [LIVY-586] Fix batch state from starting to dead when startup 
fail

No new revisions were added by this update.

Summary of changes:
 .../scala/org/apache/livy/utils/SparkYarnApp.scala | 10 -
 .../org/apache/livy/utils/SparkYarnAppSpec.scala   | 24 ++
 2 files changed, 33 insertions(+), 1 deletion(-)

[incubator-livy] branch master updated: [LIVY-617] Livy session leak on Yarn when creating session duplicated names

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 4ec3b9b  [LIVY-617] Livy session leak on Yarn when creating session 
duplicated names
4ec3b9b is described below

commit 4ec3b9b47b556390b2f738df62b5b277fa02f6ef
Author: Shanyu Zhao 
AuthorDate: Wed Aug 28 19:04:57 2019 +0800

[LIVY-617] Livy session leak on Yarn when creating session duplicated names

## What changes were proposed in this pull request?
When creating a session with duplicated name, instead of throw exception in 
SessionManager.register() method, we should stop the session. Otherwise the 
session driver process will keep running and end up creating a leaked Yarn 
application.

https://issues.apache.org/jira/browse/LIVY-617

## How was this patch tested?

This is just a simple fix and verified with manual end to end test.

Author: Shanyu Zhao 

Closes #187 from shanyu/shanyu.
---
 .../src/main/scala/org/apache/livy/sessions/SessionManager.scala | 9 -
 server/src/test/scala/org/apache/livy/sessions/MockSession.scala | 5 -
 .../test/scala/org/apache/livy/sessions/SessionManagerSpec.scala | 2 ++
 3 files changed, 14 insertions(+), 2 deletions(-)

diff --git 
a/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala 
b/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala
index f8f98a2..f2548ac 100644
--- a/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala
+++ b/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala
@@ -98,7 +98,10 @@ class SessionManager[S <: Session, R <: RecoveryMetadata : 
ClassTag](
 synchronized {
   session.name.foreach { sessionName =>
 if (sessionsByName.contains(sessionName)) {
-  throw new IllegalArgumentException(s"Duplicate session name: 
${session.name}")
+  val errMsg = s"Duplicate session name: ${session.name} for session 
${session.id}"
+  error(errMsg)
+  session.stop()
+  throw new IllegalArgumentException(errMsg)
 } else {
   sessionsByName.put(sessionName, session)
 }
@@ -106,6 +109,7 @@ class SessionManager[S <: Session, R <: RecoveryMetadata : 
ClassTag](
   sessions.put(session.id, session)
   session.start()
 }
+info(s"Registered new session ${session.id}")
 session
   }
 
@@ -122,6 +126,7 @@ class SessionManager[S <: Session, R <: RecoveryMetadata : 
ClassTag](
   }
 
   def delete(session: S): Future[Unit] = {
+info(s"Deleting session ${session.id}")
 session.stop().map { case _ =>
   try {
 sessionStore.remove(sessionType, session.id)
@@ -133,6 +138,8 @@ class SessionManager[S <: Session, R <: RecoveryMetadata : 
ClassTag](
 case NonFatal(e) =>
   error("Exception was thrown during stop session:", e)
   throw e
+  } finally {
+info(s"Deleted session ${session.id}")
   }
 }
   }
diff --git a/server/src/test/scala/org/apache/livy/sessions/MockSession.scala 
b/server/src/test/scala/org/apache/livy/sessions/MockSession.scala
index ddcbd4b..f9609b1 100644
--- a/server/src/test/scala/org/apache/livy/sessions/MockSession.scala
+++ b/server/src/test/scala/org/apache/livy/sessions/MockSession.scala
@@ -27,7 +27,10 @@ class MockSession(id: Int, owner: String, conf: LivyConf, 
name: Option[String] =
 
   override def start(): Unit = ()
 
-  override protected def stopSession(): Unit = ()
+  var stopped = false
+  override protected def stopSession(): Unit = {
+stopped = true
+  }
 
   override def logLines(): IndexedSeq[String] = IndexedSeq()
 
diff --git 
a/server/src/test/scala/org/apache/livy/sessions/SessionManagerSpec.scala 
b/server/src/test/scala/org/apache/livy/sessions/SessionManagerSpec.scala
index a5e9ffa..100c756 100644
--- a/server/src/test/scala/org/apache/livy/sessions/SessionManagerSpec.scala
+++ b/server/src/test/scala/org/apache/livy/sessions/SessionManagerSpec.scala
@@ -92,6 +92,8 @@ class SessionManagerSpec extends FunSpec with Matchers with 
LivyBaseUnitTestSuit
   an[IllegalArgumentException] should be thrownBy 
manager.register(session2)
   manager.get(session1.id).isDefined should be(true)
   manager.get(session2.id).isDefined should be(false)
+  assert(!session1.stopped)
+  assert(session2.stopped)
   manager.shutdown()
 }
 

[incubator-livy] branch master updated (3626382 -> 256702e)

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from 3626382  [LIVY-574][TESTS][THRIFT] Add tests for metadata operations
 add 256702e  [LIVY-639][REPL] add start time and completion time and 
duration to the statements web ui

No new revisions were added by this update.

Summary of changes:
 docs/rest-api.md   | 15 +
 .../main/scala/org/apache/livy/repl/Session.scala  |  3 ++
 .../java/org/apache/livy/rsc/driver/Statement.java |  2 ++
 .../server/ui/static/html/statements-table.html| 18 ++
 .../org/apache/livy/server/ui/static/js/session.js | 38 ++
 5 files changed, 76 insertions(+)

[incubator-livy] branch master updated: [LIVY-622][LIVY-623][LIVY-624][LIVY-625][THRIFT] Support GetFunctions, GetSchemas, GetTables, GetColumns in Livy thrift server

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new cae9d97  [LIVY-622][LIVY-623][LIVY-624][LIVY-625][THRIFT] Support 
GetFunctions, GetSchemas, GetTables, GetColumns in Livy thrift server
cae9d97 is described below

commit cae9d97185bf371912dcd863dff5babfd9cb704a
Author: yihengwang 
AuthorDate: Fri Aug 16 10:32:11 2019 +0800

[LIVY-622][LIVY-623][LIVY-624][LIVY-625][THRIFT] Support GetFunctions, 
GetSchemas, GetTables, GetColumns in Livy thrift server

## What changes were proposed in this pull request?
In this patch, we add the implementations of GetSchemas, GetFunctions, 
GetTables, and GetColumns in Livy Thrift server.

https://issues.apache.org/jira/browse/LIVY-622
https://issues.apache.org/jira/browse/LIVY-623
https://issues.apache.org/jira/browse/LIVY-624
https://issues.apache.org/jira/browse/LIVY-625

## How was this patch tested?
Add new unit tests and integration test. Run them with existing tests.

Author: yihengwang 

Closes #194 from yiheng/fix_575.
---
 .../apache/livy/thriftserver/LivyCLIService.scala  |  16 +--
 .../livy/thriftserver/LivyOperationManager.scala   |  63 
 .../livy/thriftserver/cli/ThriftCLIService.scala   |  17 ++-
 .../operation/GetColumnsOperation.scala| 102 +
 .../operation/GetFunctionsOperation.scala  |  94 
 .../operation/GetSchemasOperation.scala|  63 
 .../operation/GetTablesOperation.scala |  73 ++
 .../thriftserver/operation/MetadataOperation.scala |   6 +
 .../operation/SparkCatalogOperation.scala  | 119 
 .../livy/thriftserver/ThriftServerSuites.scala | 158 -
 .../livy/thriftserver/session/CatalogJobState.java |  28 
 .../session/CleanupCatalogResultJob.java   |  37 +
 .../livy/thriftserver/session/ColumnBuffer.java|  36 +
 .../session/FetchCatalogResultJob.java |  51 +++
 .../livy/thriftserver/session/GetColumnsJob.java   |  93 
 .../livy/thriftserver/session/GetFunctionsJob.java |  67 +
 .../livy/thriftserver/session/GetSchemasJob.java   |  47 ++
 .../livy/thriftserver/session/GetTablesJob.java|  92 
 .../livy/thriftserver/session/SparkCatalogJob.java |  50 +++
 .../livy/thriftserver/session/SparkUtils.java  | 113 +++
 .../thriftserver/session/ThriftSessionState.java   |  32 +
 .../thriftserver/session/ThriftSessionTest.java|  53 ++-
 22 files changed, 1395 insertions(+), 15 deletions(-)

diff --git 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyCLIService.scala
 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyCLIService.scala
index 725bdc8..3c84b4a 100644
--- 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyCLIService.scala
+++ 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyCLIService.scala
@@ -215,8 +215,8 @@ class LivyCLIService(server: LivyThriftServer)
   sessionHandle: SessionHandle,
   catalogName: String,
   schemaName: String): OperationHandle = {
-// TODO
-throw new HiveSQLException("Operation GET_SCHEMAS is not yet supported")
+sessionManager.operationManager.getSchemas(
+  sessionHandle, catalogName, schemaName)
   }
 
   @throws[HiveSQLException]
@@ -226,8 +226,8 @@ class LivyCLIService(server: LivyThriftServer)
   schemaName: String,
   tableName: String,
   tableTypes: util.List[String]): OperationHandle = {
-// TODO
-throw new HiveSQLException("Operation GET_TABLES is not yet supported")
+sessionManager.operationManager.getTables(
+  sessionHandle, catalogName, schemaName, tableName, tableTypes)
   }
 
   @throws[HiveSQLException]
@@ -243,8 +243,8 @@ class LivyCLIService(server: LivyThriftServer)
   schemaName: String,
   tableName: String,
   columnName: String): OperationHandle = {
-// TODO
-throw new HiveSQLException("Operation GET_COLUMNS is not yet supported")
+sessionManager.operationManager.getColumns(
+  sessionHandle, catalogName, schemaName, tableName, columnName)
   }
 
   @throws[HiveSQLException]
@@ -253,8 +253,8 @@ class LivyCLIService(server: LivyThriftServer)
   catalogName: String,
   schemaName: String,
   functionName: String): OperationHandle = {
-// TODO
-throw new HiveSQLException("Operation GET_FUNCTIONS is not yet supported")
+sessionManager.operationManager.getFunctions(
+  sessionHandle, catalogName, schemaName, functionName)
   }
 
   @throws[HiveSQLException]
diff --git 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyOperationManager.scala
 

[incubator-livy] branch master updated: [LIVY-547][SERVER] Livy kills session after livy.server.session.timeout even if the session is active

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new e7f23e0  [LIVY-547][SERVER] Livy kills session after 
livy.server.session.timeout even if the session is active
e7f23e0 is described below

commit e7f23e06606ff79aec1c22e79a96f959cb89a8be
Author: Shanyu Zhao 
AuthorDate: Fri Aug 9 14:51:08 2019 +0800

[LIVY-547][SERVER] Livy kills session after livy.server.session.timeout 
even if the session is active

## What changes were proposed in this pull request?
Add a new configuration:
  livy.server.session.timeout-check.skip-busy
To indicate whether or not to skip timeout check for a busy session. It 
defaults to false for
backward compatibility.

https://issues.apache.org/jira/browse/LIVY-547

## How was this patch tested?

Manually tested the configuration.

Author: Shanyu Zhao 

Closes #190 from shanyu/shanyu-547.
---
 conf/livy.conf.template  |  6 +-
 server/src/main/scala/org/apache/livy/LivyConf.scala |  3 +++
 .../org/apache/livy/sessions/SessionManager.scala|  4 
 .../scala/org/apache/livy/sessions/MockSession.scala |  3 ++-
 .../apache/livy/sessions/SessionManagerSpec.scala| 20 ++--
 5 files changed, 32 insertions(+), 4 deletions(-)

diff --git a/conf/livy.conf.template b/conf/livy.conf.template
index 2590e87..de7c248 100644
--- a/conf/livy.conf.template
+++ b/conf/livy.conf.template
@@ -50,8 +50,12 @@
 
 # Enabled to check whether timeout Livy sessions should be stopped.
 # livy.server.session.timeout-check = true
+#
+# Whether or not to skip timeout check for a busy session
+# livy.server.session.timeout-check.skip-busy = false
 
-# Time in milliseconds on how long Livy will wait before timing out an idle 
session.
+# Time in milliseconds on how long Livy will wait before timing out an 
inactive session.
+# Note that the inactive session could be busy running jobs.
 # livy.server.session.timeout = 1h
 #
 # How long a finished session state should be kept in LivyServer for query.
diff --git a/server/src/main/scala/org/apache/livy/LivyConf.scala 
b/server/src/main/scala/org/apache/livy/LivyConf.scala
index 32b3522..dec8e4a 100644
--- a/server/src/main/scala/org/apache/livy/LivyConf.scala
+++ b/server/src/main/scala/org/apache/livy/LivyConf.scala
@@ -216,6 +216,9 @@ object LivyConf {
   // Whether session timeout should be checked, by default it will be checked, 
which means inactive
   // session will be stopped after "livy.server.session.timeout"
   val SESSION_TIMEOUT_CHECK = Entry("livy.server.session.timeout-check", true)
+  // Whether session timeout check should skip busy sessions, if set to true, 
then busy sessions
+  // that have jobs running will never timeout.
+  val SESSION_TIMEOUT_CHECK_SKIP_BUSY = 
Entry("livy.server.session.timeout-check.skip-busy", false)
   // How long will an inactive session be gc-ed.
   val SESSION_TIMEOUT = Entry("livy.server.session.timeout", "1h")
   // How long a finished session state will be kept in memory
diff --git 
a/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala 
b/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala
index a63cab3..f8f98a2 100644
--- a/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala
+++ b/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala
@@ -77,6 +77,8 @@ class SessionManager[S <: Session, R <: RecoveryMetadata : 
ClassTag](
 
 
   private[this] final val sessionTimeoutCheck = 
livyConf.getBoolean(LivyConf.SESSION_TIMEOUT_CHECK)
+  private[this] final val sessionTimeoutCheckSkipBusy =
+livyConf.getBoolean(LivyConf.SESSION_TIMEOUT_CHECK_SKIP_BUSY)
   private[this] final val sessionTimeout =
 
TimeUnit.MILLISECONDS.toNanos(livyConf.getTimeAsMs(LivyConf.SESSION_TIMEOUT))
   private[this] final val sessionStateRetainedInSec =
@@ -153,6 +155,8 @@ class SessionManager[S <: Session, R <: RecoveryMetadata : 
ClassTag](
 case _ =>
   if (!sessionTimeoutCheck) {
 false
+  } else if (session.state == SessionState.Busy && 
sessionTimeoutCheckSkipBusy) {
+false
   } else if (session.isInstanceOf[BatchSession]) {
 false
   } else {
diff --git a/server/src/test/scala/org/apache/livy/sessions/MockSession.scala 
b/server/src/test/scala/org/apache/livy/sessions/MockSession.scala
index 3d0cc26..ddcbd4b 100644
--- a/server/src/test/scala/org/apache/livy/sessions/MockSession.scala
+++ b/server/src/test/scala/org/apache/livy/sessions/MockSession.scala
@@ -31,7 +31,8 @@ class MockSession(id: Int, owner: String, conf: LivyConf, 
name: Option[String] =
 
   override def logLines(): IndexedSeq[String] = Inde

[incubator-livy] branch master updated: [MINOR] Remove unused guava import

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 788767e  [MINOR] Remove unused guava import
788767e is described below

commit 788767e4cced78f18798f73a651f3ec2d087f938
Author: jerryshao 
AuthorDate: Thu Aug 1 19:41:23 2019 +0800

[MINOR] Remove unused guava import

## What changes were proposed in this pull request?

PR #181 removed guava dependency in LivyServer, but it still left unused 
guava import. Here in this minor fix, removed this unused import.

## How was this patch tested?

Existing UTs.

Author: jerryshao 

Closes #191 from jerryshao/remove-guava-minor.
---
 server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala | 2 --
 1 file changed, 2 deletions(-)

diff --git 
a/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala 
b/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
index c792b61..69e8d19 100644
--- a/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
+++ b/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
@@ -23,8 +23,6 @@ import java.util.concurrent.locks.ReentrantLock
 
 import scala.io.Source
 
-import com.google.common.collect.EvictingQueue
-
 import org.apache.livy.Logging
 
 class CircularQueue[T](var capacity: Int) extends util.LinkedList[T] {

[incubator-livy] branch master updated: [LIVY-587] Remove unused guava dependency

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 1ce266d  [LIVY-587] Remove unused guava dependency
1ce266d is described below

commit 1ce266d78fb3475af6154262629ff92f5d342ae6
Author: runzhiwang <1769910...@qq.com>
AuthorDate: Mon Jul 29 16:29:23 2019 +0800

[LIVY-587] Remove unused guava dependency

## What changes were proposed in this pull request?

Guava was unused any more, and it's too heavy to include, so remove the 
guava dependency.

## How was this patch tested?

Existing unit tests.

Author: runzhiwang <1769910...@qq.com>

Closes #181 from runzhiwang/livy-587.
---
 pom.xml|  7 ---
 server/pom.xml |  5 -
 .../apache/livy/server/interactive/InteractiveSession.scala|  2 --
 .../main/scala/org/apache/livy/utils/LineBufferedStream.scala  | 10 +-
 4 files changed, 9 insertions(+), 15 deletions(-)

diff --git a/pom.xml b/pom.xml
index 23909d1..f98071c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -84,7 +84,6 @@
 ${spark.scala-2.11.version}
 3.0.0
 1.9
-15.0
 4.5.3
 4.4.4
 2.9.9
@@ -284,12 +283,6 @@
   
 
   
-com.google.guava
-guava
-${guava.version}
-  
-
-  
 io.dropwizard.metrics
 metrics-core
 ${metrics.version}
diff --git a/server/pom.xml b/server/pom.xml
index d043dfe..e708964 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -75,11 +75,6 @@
 
 
 
-  com.google.guava
-  guava
-
-
-
   io.dropwizard.metrics
   metrics-core
 
diff --git 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
index 9529ed3..bccdb4d 100644
--- 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
+++ 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
@@ -31,7 +31,6 @@ import scala.concurrent.duration.{Duration, FiniteDuration}
 import scala.util.{Random, Try}
 
 import com.fasterxml.jackson.annotation.JsonIgnoreProperties
-import com.google.common.annotations.VisibleForTesting
 import org.apache.hadoop.fs.Path
 import org.apache.spark.launcher.SparkLauncher
 
@@ -155,7 +154,6 @@ object InteractiveSession extends Logging {
   mockApp)
   }
 
-  @VisibleForTesting
   private[interactive] def prepareBuilderProp(
 conf: Map[String, String],
 kind: Kind,
diff --git 
a/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala 
b/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
index 6896d2f..c792b61 100644
--- a/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
+++ b/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
@@ -18,6 +18,7 @@
 package org.apache.livy.utils
 
 import java.io.InputStream
+import java.util
 import java.util.concurrent.locks.ReentrantLock
 
 import scala.io.Source
@@ -26,9 +27,16 @@ import com.google.common.collect.EvictingQueue
 
 import org.apache.livy.Logging
 
+class CircularQueue[T](var capacity: Int) extends util.LinkedList[T] {
+  override def add(t: T): Boolean = {
+if (size >= capacity) removeFirst
+super.add(t)
+  }
+}
+
 class LineBufferedStream(inputStream: InputStream, logSize: Int) extends 
Logging {
 
-  private[this] val _lines: EvictingQueue[String] = 
EvictingQueue.create[String](logSize)
+  private[this] val _lines: CircularQueue[String] = new 
CircularQueue[String](logSize)
 
   private[this] val _lock = new ReentrantLock()
   private[this] val _condition = _lock.newCondition()

[incubator-livy] branch master updated: [LIVY-613][REPL] Livy can't handle the java.sql.Date type correctly.

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 92062e1  [LIVY-613][REPL] Livy can't handle the java.sql.Date type 
correctly.
92062e1 is described below

commit 92062e1659db2af85711b1f35c50ff4050fec675
Author: yangping.wyp 
AuthorDate: Fri Jul 26 19:34:42 2019 +0800

[LIVY-613][REPL] Livy can't handle the java.sql.Date type correctly.

## What changes were proposed in this pull request?

When Spark table has `java.sql.Date` type column, Livy can't handle the 
`java.sql.Date` type correctly. e.g
```
create table test(
name string,
birthday date
);

insert into test values ('Livy', '2019-07-24')

curl -H "Content-Type:application/json" -X POST -d '{"code":"select * from 
test", "kind":"sql"}' 192.168.1.6:8998/sessions/48/statements
{"id":1,"code":"select * from 
test","state":"waiting","output":null,"progress":0.0}

curl 192.168.1.6:8998/sessions/48/statements/1
{"id":1,"code":"select * from 
test","state":"available","output":{"status":"ok","execution_count":1,"data":{"application/json":{"schema":{"type":"struct","fields":[{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"birthday","type":"date","nullable":true,"metadata":{}}]},"data":[["Livy",{}]]}}},"progress":1.0}
```
as you can see, the output of `select * from test` is `["Livy",{}]` , 
birthday column's value isn't handle  correctly.

The reason is that json4s can't handle `java.sql.Date`, so we should define 
the `CustomSerializer` for `java.sql.Date`.

This PR  add a `DateSerializer` to support `java.sql.Date` parser.

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review https://livy.incubator.apache.org/community/ before opening a 
pull request.

Author: yangping.wyp 

Closes #186 from 397090770/master.
---
 .../org/apache/livy/repl/SQLInterpreter.scala  | 11 +++-
 .../org/apache/livy/repl/SQLInterpreterSpec.scala  | 69 +-
 2 files changed, 78 insertions(+), 2 deletions(-)

diff --git a/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala 
b/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
index 5a7b606..9abbf2c 100644
--- a/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
+++ b/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
@@ -18,6 +18,7 @@
 package org.apache.livy.repl
 
 import java.lang.reflect.InvocationTargetException
+import java.sql.Date
 
 import scala.util.control.NonFatal
 
@@ -25,6 +26,7 @@ import org.apache.spark.SparkConf
 import org.apache.spark.sql.Row
 import org.apache.spark.sql.SparkSession
 import org.json4s._
+import org.json4s.JsonAST.{JNull, JString}
 import org.json4s.JsonDSL._
 import org.json4s.jackson.JsonMethods._
 
@@ -66,7 +68,14 @@ class SQLInterpreter(
 rscConf: RSCConf,
 sparkEntries: SparkEntries) extends Interpreter with Logging {
 
-  private implicit def formats = DefaultFormats
+  case object DateSerializer extends CustomSerializer[Date](_ => ( {
+case JString(s) => Date.valueOf(s)
+case JNull => null
+  }, {
+case d: Date => JString(d.toString)
+  }))
+
+  private implicit def formats: Formats = DefaultFormats + DateSerializer
 
   private var spark: SparkSession = null
 
diff --git a/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala 
b/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
index 37c9594..781ed72 100644
--- a/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
+++ b/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
@@ -17,17 +17,20 @@
 
 package org.apache.livy.repl
 
+import java.sql.Date
+
 import scala.util.Try
 
 import org.apache.spark.SparkConf
 import org.json4s.{DefaultFormats, JValue}
-import org.json4s.JsonAST.JArray
+import org.json4s.JsonAST.{JArray, JNull}
 import org.json4s.JsonDSL._
 
 import org.apache.livy.rsc.RSCConf
 import org.apache.livy.rsc.driver.SparkEntries
 
 case class People(name: String, age: Int)
+case class Person(name: String, birthday: Date)
 
 class SQLInterpreterSpec extends BaseInterpreterSpec {
 
@@ -43,6 +46,7

[incubator-livy] branch master updated: [LIVY-582][TESTS] Hostname in python-api test should be lower case to avoid test failures

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new f796192  [LIVY-582][TESTS] Hostname in python-api test should be lower 
case to avoid test failures
f796192 is described below

commit f7961924e4bed4d1257817cd871bfe0555d1cca0
Author: Yiheng Wang 
AuthorDate: Tue Jul 16 10:22:11 2019 +0800

[LIVY-582][TESTS] Hostname in python-api test should be lower case to avoid 
test failures

## What changes were proposed in this pull request?
In the python-API test code, when returning a mocked response, the mock lib 
will compare the URL with the predefined URLs case sensitive. However, the 
`Request` lib used in the Livy python API will change the URL to lower case. 
This will cause test failures on a machine with an upper case hostname.

This patch turns the hostname in python-API test code into the lower case 
to avoid such test failures.

## How was this patch tested?
Existing test. Run test specifically on a machine with an upper case 
hostname.

Author: Yiheng Wang 

Closes #180 from yiheng/fix_582.
---
 python-api/src/test/python/livy-tests/client_test.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/python-api/src/test/python/livy-tests/client_test.py 
b/python-api/src/test/python/livy-tests/client_test.py
index 5e47cb0..b6426ae 100644
--- a/python-api/src/test/python/livy-tests/client_test.py
+++ b/python-api/src/test/python/livy-tests/client_test.py
@@ -26,7 +26,8 @@ from livy.client import HttpClient
 
 session_id = 0
 job_id = 1
-base_uri = 'http://{0}:{1}'.format(socket.gethostname(), 8998)
+# Make sure host name is lower case. See LIVY-582
+base_uri = 'http://{0}:{1}'.format(socket.gethostname().lower(), 8998)
 client_test = None
 invoked_queued_callback = False
 invoked_running_callback = False

[incubator-livy] branch master updated: [LIVY-603][BUILD] Upgrade build spark version to 2.4.3

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new 6ad9ae9  [LIVY-603][BUILD] Upgrade build spark version to 2.4.3
6ad9ae9 is described below

commit 6ad9ae9607e8369d5787d0d0c07eb3bd7b060bfc
Author: yihengwang 
AuthorDate: Fri Jul 12 10:11:25 2019 +0800

[LIVY-603][BUILD] Upgrade build spark version to 2.4.3

## What changes were proposed in this pull request?
Bump Spark 2.4 minor version to 2.4.3.

## How was this patch tested?
Existing test

Author: yihengwang 

Closes #179 from yiheng/bump_spark_2_4_3.
---
 pom.xml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/pom.xml b/pom.xml
index fcc5b87..6bfdcc6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1050,15 +1050,15 @@
 
   
   
-2.4.0
+2.4.3
 ${spark.scala-2.11.version}
 4.1.17.Final
 1.8
 0.10.7
 
-  
https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
+  
https://archive.apache.org/dist/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz
 
-spark-2.4.0-bin-hadoop2.7
+spark-2.4.3-bin-hadoop2.7
   
 
 

[incubator-livy] branch master updated: [MINOR] Fix spark 2.4.0 tgz file dead download link in pom.xml

[jshao]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new ee94065  [MINOR] Fix spark 2.4.0 tgz file dead download link in pom.xml
ee94065 is described below

commit ee94065a630fa932488229365de68ab5e053d3a0
Author: yihengwang 
AuthorDate: Tue Jul 9 19:16:53 2019 +0800

[MINOR] Fix spark 2.4.0 tgz file dead download link in pom.xml

## What changes were proposed in this pull request?
The download link for spark 2.4.0 tgz file in the pom.xml file is a dead 
link. This patch change all the spark tgz download links to the official spark 
release archive site, which should be more stable.

## How was this patch tested?
Existing test.

Author: yihengwang 

Closes #178 from yiheng/fix_travis.
---
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pom.xml b/pom.xml
index 43e698d..fcc5b87 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1036,7 +1036,7 @@
 ${spark.scala-2.11.version}
 4.1.17.Final
 
-  
http://mirrors.advancedhosters.com/apache/spark/spark-2.3.3/spark-2.3.3-bin-hadoop2.7.tgz
+  
https://archive.apache.org/dist/spark/spark-2.3.3/spark-2.3.3-bin-hadoop2.7.tgz
 
 spark-2.3.3-bin-hadoop2.7
   
@@ -1056,7 +1056,7 @@
 1.8
 0.10.7
 
-  
http://mirrors.advancedhosters.com/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
+  
https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
 
 spark-2.4.0-bin-hadoop2.7
   

incubator-livy git commit: [LIVY-526] Upgrade jetty version

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master d87a34872 -> a068363a0


[LIVY-526] Upgrade jetty version

## What changes were proposed in this pull request?

Upgrade the jetty patch version to a more recent version that has fixes for a 
few security issues.

## How was this patch tested?

Existing unit tests

Author: Arun Mahadevan 

Closes #120 from arunmahadevan/jetty-upgrade.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/a068363a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/a068363a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/a068363a

Branch: refs/heads/master
Commit: a068363a0c097ab167e8fb6de1015930e71b4cb8
Parents: d87a348
Author: Arun Mahadevan 
Authored: Sat Oct 13 15:47:18 2018 +0800
Committer: jerryshao 
Committed: Sat Oct 13 15:47:18 2018 +0800

--
 .../scala/org/apache/livy/client/http/LivyConnectionSpec.scala | 2 +-
 pom.xml| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/a068363a/client-http/src/test/scala/org/apache/livy/client/http/LivyConnectionSpec.scala
--
diff --git 
a/client-http/src/test/scala/org/apache/livy/client/http/LivyConnectionSpec.scala
 
b/client-http/src/test/scala/org/apache/livy/client/http/LivyConnectionSpec.scala
index 110eb35..01db5d5 100644
--- 
a/client-http/src/test/scala/org/apache/livy/client/http/LivyConnectionSpec.scala
+++ 
b/client-http/src/test/scala/org/apache/livy/client/http/LivyConnectionSpec.scala
@@ -104,7 +104,7 @@ class LivyConnectionSpec extends FunSpecLike with 
BeforeAndAfterAll with LivyBas
 .set(LivyConf.RESPONSE_HEADER_SIZE, 1024)
   val pwd = "test-password" * 100
   val exception = intercept[IOException](test(pwd, livyConf))
-  exception.getMessage.contains("Request Entity Too Large") should be(true)
+  exception.getMessage.contains("Request Header Fields Too Large") should 
be(true)
 }
 
 it("should be succeeded with configured header size") {

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/a068363a/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 32a6b89..d9adacb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -89,7 +89,7 @@
 4.4.4
 2.9.5
 3.1.0
-9.3.8.v20160314
+9.3.24.v20180605
 3.2.11
 4.11
 0.9.3

incubator-livy git commit: [LIVY-510] Remove support for JDK7.

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master f99cc32df -> d39ab356b


[LIVY-510] Remove support for JDK7.

With this change Livy will only build and run with JDK8. While this
doesn't necessarily preclude using Java 7 for running Spark itself,
that is not encouraged due to possible issues with serialization.

There are a few small changes elsewhere that I made so tests would pass,
but aren't related to the java 8 change:

- a small tweak to the fallback code that initializes the python
  interpreter, which was failing with a weird reflection error.

- expose the user's home directory in the MiniYarnCluster, to allow
  the tests to see pip packages installed in the user's home.

- tweak the python API tests so that the pytest cache does not pollute
  the source directory.

- changed Travis to use Oracle's JDK, since the Open JDK path was
  mixing Java 8 and 9 and causing errors.

- updated jacoco to latest stable.

Author: Marcelo Vanzin 

Closes #111 from vanzin/LIVY-510.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/d39ab356
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/d39ab356
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/d39ab356

Branch: refs/heads/master
Commit: d39ab356b4e36d8780bd44b9d5020bfeec90cb2f
Parents: f99cc32
Author: Marcelo Vanzin 
Authored: Thu Sep 20 11:49:22 2018 +0800
Committer: jerryshao 
Committed: Thu Sep 20 11:49:22 2018 +0800

--
 .travis.yml |  6 +-
 README.md   |  6 +-
 .../livy/test/framework/MiniCluster.scala   |  7 +-
 .../src/test/resources/test_python_api.py   |  3 +-
 .../scala/org/apache/livy/test/JobApiIT.scala   | 16 ++--
 pom.xml | 82 ++--
 python-api/setup.cfg|  1 +
 .../apache/livy/repl/PythonInterpreter.scala|  8 +-
 8 files changed, 34 insertions(+), 95 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/d39ab356/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 6c5eee3..d38cf4f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -33,17 +33,13 @@ matrix:
   # Spark 2.2+ will only be verified using JDK8
   # Thriftserver requires JDK8
 - env: MVN_FLAG='-Pthriftserver -Pspark-2.2-it -DskipTests'
-  jdk: oraclejdk8
 - env: MVN_FLAG='-Pthriftserver -Pspark-2.2 -DskipITs'
-  jdk: oraclejdk8
 - env: MVN_FLAG='-Pthriftserver -Pspark-2.3-it -DskipTests'
-  jdk: oraclejdk8
 - env: MVN_FLAG='-Pthriftserver -Pspark-2.3 -DskipITs'
-  jdk: oraclejdk8
 
 
 jdk:
-  - openjdk7
+  - oraclejdk8
 
 addons:
   apt:

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/d39ab356/README.md
--
diff --git a/README.md b/README.md
index 2dc33c2..ff6ae16 100644
--- a/README.md
+++ b/README.md
@@ -28,19 +28,19 @@ To build Livy, you will need:
 
 Debian/Ubuntu:
   * mvn (from ``maven`` package or maven3 tarball)
-  * openjdk-7-jdk (or Oracle Java7 jdk)
+  * openjdk-8-jdk (or Oracle JDK 8)
   * Python 2.6+
   * R 3.x
 
 Redhat/CentOS:
   * mvn (from ``maven`` package or maven3 tarball)
-  * java-1.7.0-openjdk (or Oracle Java7 jdk)
+  * java-1.8.0-openjdk (or Oracle JDK 8)
   * Python 2.6+
   * R 3.x
 
 MacOS:
   * Xcode command line tools
-  * Oracle's JDK 1.7+
+  * Oracle's JDK 1.8
   * Maven (Homebrew)
   * Python 2.6+
   * R 3.x

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/d39ab356/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
--
diff --git 
a/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
 
b/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
index c4d4eb5..b58abe2 100644
--- 
a/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
+++ 
b/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
@@ -101,6 +101,10 @@ object MiniYarnMain extends MiniClusterBase {
   config.localDirCount, config.logDirCount)
 yarnCluster.init(baseConfig)
 
+// This allows applications run by YARN during the integration tests to 
find PIP modules
+// installed in the user's home directory (instead of just the global 
ones).
+baseConfig.set(YarnConfiguration.NM_USER_HOME_DIR, sys.env("HOME"))
+
 // Install a shutdown hook for stop the service and kill all running 
applications.
 Runtime.getRuntime().addShutdownHook(new Thread() {
   override def run(): Unit = yarnCluster.stop()
@@ -283,8 +287,7 @@ 

[4/4] incubator-livy git commit: [LIVY-491][LIVY-492][LIVY-493] Add Thriftserver module and implementation

[jshao]

[LIVY-491][LIVY-492][LIVY-493] Add Thriftserver module and implementation

## What changes were proposed in this pull request?

The PR contains an implementation of a JDBC API for Livy server based on the 
Hive Thriftserver. The implementation is based on the version 3.0 of Hive 
Thriftserver.

This initial PR contains the thriftserver module added to Livy and its 
implementation. It doesn't contain any binding for starting it, this will be 
added later.

Some Hive classes have been ported here because they needed come modifications 
in order to works properly in Livy. Long term solution is to get of them all by 
re-implementing the needed parts in the Livy thriftserver itself, without 
relying on Hive code (other than the PRC interface) anymore. Those 
classes/changes can be summarized in three categories:

 1. Changes to make the Hive classes easy to extend: for instance, some 
visibility modifiers were changes (moving from `private` or `package private` 
to `protected` or `public`);
 2. Changes in order to reduce the dependencies on Hive modules/classes: for 
instance all the classes in the `operation` package were modified in order to 
get rid of the usage of `HiveSession` and the `HiveServer2` was changed in 
order not to use `CLIService`.
 3. The UGI management which is currently performed in Livy is definitely very 
different from the Hive one, this required changes to the `HiveAuthFactory` in 
order not to interfere with the existing codebase.

## How was this patch tested?

added integration tests as UTs

Author: Marco Gaido 

Closes #104 from mgaido91/LIVY-491.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/7d8fa69f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/7d8fa69f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/7d8fa69f

Branch: refs/heads/master
Commit: 7d8fa69f1804025d95971bcfb389951075dc6b98
Parents: 8027ca7
Author: Marco Gaido 
Authored: Mon Sep 10 11:42:10 2018 +0800
Committer: jerryshao 
Committed: Mon Sep 10 11:42:10 2018 +0800

--
 .travis.yml |   9 +-
 checkstyle-suppressions.xml |   1 +
 pom.xml |  38 +-
 .../main/scala/org/apache/livy/LivyConf.scala   |   4 +
 .../server/interactive/InteractiveSession.scala |   2 +-
 thriftserver/server/pom.xml | 234 +
 .../hive/service/auth/HiveAuthFactory.java  | 320 +++
 .../apache/hive/service/cli/ColumnBasedSet.java | 205 +
 .../hive/service/cli/JobProgressUpdate.java |  55 ++
 .../hive/service/cli/OperationStatus.java   |  75 ++
 .../cli/operation/GetCatalogsOperation.java |  80 ++
 .../cli/operation/GetTableTypesOperation.java   | 101 +++
 .../cli/operation/GetTypeInfoOperation.java | 147 
 .../cli/operation/HiveTableTypeMapping.java |  61 ++
 .../cli/operation/MetadataOperation.java| 133 +++
 .../hive/service/cli/operation/Operation.java   | 290 ++
 .../cli/thrift/ThriftBinaryCLIService.java  | 167 
 .../service/cli/thrift/ThriftCLIService.java| 871 +++
 .../cli/thrift/ThriftHttpCLIService.java| 234 +
 .../service/cli/thrift/ThriftHttpServlet.java   | 592 +
 .../apache/hive/service/server/HiveServer2.java | 316 +++
 .../livy/thriftserver/LivyCLIService.scala  | 432 +
 .../LivyExecuteStatementOperation.scala | 209 +
 .../thriftserver/LivyOperationManager.scala | 244 ++
 .../livy/thriftserver/LivyThriftServer.scala| 122 +++
 .../thriftserver/LivyThriftSessionManager.scala | 635 ++
 .../apache/livy/thriftserver/SessionInfo.scala  |  90 ++
 .../livy/thriftserver/SessionStates.scala   |  26 +
 .../livy/thriftserver/rpc/RpcClient.scala   | 289 ++
 .../livy/thriftserver/serde/ColumnBuffer.scala  | 186 
 .../serde/ColumnOrientedResultSet.scala |  35 +
 .../livy/thriftserver/types/DataType.scala  |  38 +
 .../livy/thriftserver/types/DataTypeUtils.scala | 178 
 .../thriftserver/ThriftServerBaseTest.scala | 124 +++
 .../livy/thriftserver/ThriftServerSuites.scala  | 149 
 35 files changed, 6686 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/7d8fa69f/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index e62a8d9..6c5eee3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -31,13 +31,14 @@ env:
 matrix:
   include:
   # Spark 2.2+ will only be verified using JDK8
-- env: MVN_FLAG='-Pspark-2.2-it -DskipTests'
+  # Thriftserver requires JDK8
+- env: MVN_FLAG='-Pthriftserver -Pspark-2.2-it -DskipTests'
   jdk: oraclejdk8
-- env: MVN_FLAG='-Pspark-2.2 

[1/4] incubator-livy git commit: [LIVY-491][LIVY-492][LIVY-493] Add Thriftserver module and implementation

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 8027ca708 -> 7d8fa69f1


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/7d8fa69f/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/serde/ColumnBuffer.scala
--
diff --git 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/serde/ColumnBuffer.scala
 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/serde/ColumnBuffer.scala
new file mode 100644
index 000..248a77d
--- /dev/null
+++ 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/serde/ColumnBuffer.scala
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver.serde
+
+import java.nio.ByteBuffer
+import java.util
+
+import scala.collection.mutable
+
+import org.apache.livy.thriftserver.types.{DataType, DataTypeUtils}
+
+object ColumnBuffer {
+  private val DEFAULT_SIZE = 100
+  private val EMPTY_BINARY = ByteBuffer.allocate(0)
+  private val EMPTY_STRING = ""
+  private val HANDLED_TYPES =
+Set("boolean", "byte", "short", "integer", "long", "float", "double", 
"binary")
+}
+
+class ColumnBuffer(val dataType: DataType) {
+  private val nulls = new mutable.BitSet()
+  private var currentSize = 0
+  private var boolVars: Array[Boolean] = _
+  private var byteVars: Array[Byte] = _
+  private var shortVars: Array[Short] = _
+  private var intVars: Array[Int] = _
+  private var longVars: Array[Long] = _
+  private var doubleVars: Array[Double] = _
+  private var stringVars: util.List[String] = _
+  private var binaryVars: util.List[ByteBuffer] = _
+
+  dataType.name match {
+case "boolean" =>
+  boolVars = new Array[Boolean](ColumnBuffer.DEFAULT_SIZE)
+case "byte" =>
+  byteVars = new Array[Byte](ColumnBuffer.DEFAULT_SIZE)
+case "short" =>
+  shortVars = new Array[Short](ColumnBuffer.DEFAULT_SIZE)
+case "integer" =>
+  intVars = new Array[Int](ColumnBuffer.DEFAULT_SIZE)
+case "long" =>
+  longVars = new Array[Long](ColumnBuffer.DEFAULT_SIZE)
+case "float" | "double" =>
+  doubleVars = new Array[Double](ColumnBuffer.DEFAULT_SIZE)
+case "binary" =>
+  binaryVars = new util.ArrayList[ByteBuffer]
+case "void" => // all NULLs, nothing to do
+case _ =>
+  stringVars = new util.ArrayList[String]
+  }
+
+  def get(index: Int): Any = {
+if (this.nulls(index)) {
+  null
+} else {
+  dataType.name match {
+case "boolean" =>
+  boolVars(index)
+case "byte" =>
+  byteVars(index)
+case "short" =>
+  shortVars(index)
+case "integer" =>
+  intVars(index)
+case "long" =>
+  longVars(index)
+case "float" | "double" =>
+  doubleVars(index)
+case "binary" =>
+  binaryVars.get(index).array()
+case _ =>
+  stringVars.get(index)
+  }
+}
+  }
+
+  def size: Int = currentSize
+
+  def addValue(field: Any): Unit = {
+if (field == null) {
+  nulls += currentSize
+  if (!ColumnBuffer.HANDLED_TYPES.contains(dataType.name)) {
+stringVars.add(ColumnBuffer.EMPTY_STRING)
+  } else if (dataType.name == "binary") {
+binaryVars.add(ColumnBuffer.EMPTY_BINARY)
+  }
+} else {
+  dataType.name match {
+case "boolean" =>
+  ensureBoolVarsSize()
+  boolVars(currentSize) = field.asInstanceOf[Boolean]
+case "byte" =>
+  ensureByteVarsSize()
+  byteVars(currentSize) = field.asInstanceOf[Byte]
+case "short" =>
+  ensureShortVarsSize()
+  shortVars(currentSize) = field.asInstanceOf[Short]
+case "integer" =>
+  ensureIntVarsSize()
+  intVars(currentSize) = field.asInstanceOf[Int]
+case "long" =>
+  ensureLongVarsSize()
+  longVars(currentSize) = field.asInstanceOf[Long]
+case "float" =>
+  ensureDoubleVarsSize()
+  // We need to convert the float to string and then to double in 
order to avoid precision
+  // issues caused by the poor precision of Float
+  doubleVars(currentSize) = 

[2/4] incubator-livy git commit: [LIVY-491][LIVY-492][LIVY-493] Add Thriftserver module and implementation

[jshao]

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/7d8fa69f/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyCLIService.scala
--
diff --git 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyCLIService.scala
 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyCLIService.scala
new file mode 100644
index 000..5289354
--- /dev/null
+++ 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyCLIService.scala
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.thriftserver
+
+import java.io.IOException
+import java.util
+import java.util.concurrent.{CancellationException, ExecutionException, 
TimeoutException, TimeUnit}
+import javax.security.auth.login.LoginException
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.hive.common.log.ProgressMonitor
+import org.apache.hadoop.hive.conf.HiveConf
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars
+import org.apache.hadoop.hive.ql.parse.ParseUtils
+import org.apache.hadoop.hive.shims.Utils
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hive.service.{CompositeService, ServiceException}
+import org.apache.hive.service.auth.HiveAuthFactory
+import org.apache.hive.service.cli._
+import org.apache.hive.service.cli.operation.Operation
+import org.apache.hive.service.rpc.thrift.{TOperationHandle, TProtocolVersion}
+
+import org.apache.livy.{LIVY_VERSION, Logging}
+
+class LivyCLIService(server: LivyThriftServer)
+  extends CompositeService(classOf[LivyCLIService].getName) with ICLIService 
with Logging {
+  import LivyCLIService._
+
+  private var sessionManager: LivyThriftSessionManager = _
+  private var defaultFetchRows: Int = _
+  private var serviceUGI: UserGroupInformation = _
+  private var httpUGI: UserGroupInformation = _
+
+  override def init(hiveConf: HiveConf): Unit = {
+sessionManager = new LivyThriftSessionManager(server, hiveConf)
+addService(sessionManager)
+defaultFetchRows =
+  
hiveConf.getIntVar(ConfVars.HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE)
+//  If the hadoop cluster is secure, do a kerberos login for the service 
from the keytab
+if (UserGroupInformation.isSecurityEnabled) {
+  try {
+serviceUGI = Utils.getUGI
+  } catch {
+case e: IOException =>
+  throw new ServiceException("Unable to login to kerberos with given 
principal/keytab", e)
+case e: LoginException =>
+  throw new ServiceException("Unable to login to kerberos with given 
principal/keytab", e)
+  }
+  // Also try creating a UGI object for the SPNego principal
+  val principal = hiveConf.getVar(ConfVars.HIVE_SERVER2_SPNEGO_PRINCIPAL)
+  val keyTabFile = hiveConf.getVar(ConfVars.HIVE_SERVER2_SPNEGO_KEYTAB)
+  if (principal.isEmpty || keyTabFile.isEmpty) {
+info(s"SPNego httpUGI not created, SPNegoPrincipal: $principal, 
ketabFile: $keyTabFile")
+  } else try {
+httpUGI = HiveAuthFactory.loginFromSpnegoKeytabAndReturnUGI(hiveConf)
+info("SPNego httpUGI successfully created.")
+  } catch {
+case e: IOException =>
+  warn("SPNego httpUGI creation failed: ", e)
+  }
+}
+super.init(hiveConf)
+  }
+
+  def getServiceUGI: UserGroupInformation = this.serviceUGI
+
+  def getHttpUGI: UserGroupInformation = this.httpUGI
+
+  def getSessionManager: LivyThriftSessionManager = sessionManager
+
+  @throws[HiveSQLException]
+  override def getInfo(sessionHandle: SessionHandle, getInfoType: 
GetInfoType): GetInfoValue = {
+getInfoType match {
+  case GetInfoType.CLI_SERVER_NAME => new GetInfoValue("Livy JDBC")
+  case GetInfoType.CLI_DBMS_NAME => new GetInfoValue("Livy JDBC")
+  case GetInfoType.CLI_DBMS_VER => new GetInfoValue(LIVY_VERSION)
+  // below values are copied from Hive
+  case GetInfoType.CLI_MAX_COLUMN_NAME_LEN => new GetInfoValue(128)
+  case GetInfoType.CLI_MAX_SCHEMA_NAME_LEN => new GetInfoValue(128)
+  case GetInfoType.CLI_MAX_TABLE_NAME_LEN => new GetInfoValue(128)
+  case 

[3/4] incubator-livy git commit: [LIVY-491][LIVY-492][LIVY-493] Add Thriftserver module and implementation

[jshao]

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/7d8fa69f/thriftserver/server/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
--
diff --git 
a/thriftserver/server/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
 
b/thriftserver/server/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
new file mode 100644
index 000..e16c6de
--- /dev/null
+++ 
b/thriftserver/server/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
@@ -0,0 +1,871 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.service.cli.thrift;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import org.apache.hive.service.rpc.thrift.TSetClientInfoReq;
+import org.apache.hive.service.rpc.thrift.TSetClientInfoResp;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.security.auth.login.LoginException;
+import org.apache.hadoop.hive.common.ServerUtils;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.shims.HadoopShims.KerberosNameShim;
+import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hive.service.AbstractService;
+import org.apache.hive.service.ServiceException;
+import org.apache.hive.service.ServiceUtils;
+import org.apache.hive.service.auth.HiveAuthConstants;
+import org.apache.hive.service.auth.HiveAuthFactory;
+import org.apache.hive.service.auth.TSetIpAddressProcessor;
+import org.apache.hive.service.cli.FetchOrientation;
+import org.apache.hive.service.cli.FetchType;
+import org.apache.hive.service.cli.GetInfoType;
+import org.apache.hive.service.cli.GetInfoValue;
+import org.apache.hive.service.cli.HiveSQLException;
+import org.apache.hive.service.cli.ICLIService;
+import org.apache.hive.service.cli.JobProgressUpdate;
+import org.apache.hive.service.cli.OperationHandle;
+import org.apache.hive.service.cli.OperationStatus;
+import org.apache.hive.service.cli.OperationType;
+import org.apache.hive.service.cli.ProgressMonitorStatusMapper;
+import org.apache.hive.service.cli.RowSet;
+import org.apache.hive.service.cli.SessionHandle;
+import org.apache.hive.service.cli.TableSchema;
+import org.apache.hive.service.rpc.thrift.TCLIService;
+import org.apache.hive.service.rpc.thrift.TCancelDelegationTokenReq;
+import org.apache.hive.service.rpc.thrift.TCancelDelegationTokenResp;
+import org.apache.hive.service.rpc.thrift.TCancelOperationReq;
+import org.apache.hive.service.rpc.thrift.TCancelOperationResp;
+import org.apache.hive.service.rpc.thrift.TCloseOperationReq;
+import org.apache.hive.service.rpc.thrift.TCloseOperationResp;
+import org.apache.hive.service.rpc.thrift.TCloseSessionReq;
+import org.apache.hive.service.rpc.thrift.TCloseSessionResp;
+import org.apache.hive.service.rpc.thrift.TExecuteStatementReq;
+import org.apache.hive.service.rpc.thrift.TExecuteStatementResp;
+import org.apache.hive.service.rpc.thrift.TFetchResultsReq;
+import org.apache.hive.service.rpc.thrift.TFetchResultsResp;
+import org.apache.hive.service.rpc.thrift.TGetCatalogsReq;
+import org.apache.hive.service.rpc.thrift.TGetCatalogsResp;
+import org.apache.hive.service.rpc.thrift.TGetColumnsReq;
+import org.apache.hive.service.rpc.thrift.TGetColumnsResp;
+import org.apache.hive.service.rpc.thrift.TGetCrossReferenceReq;
+import org.apache.hive.service.rpc.thrift.TGetCrossReferenceResp;
+import org.apache.hive.service.rpc.thrift.TGetDelegationTokenReq;
+import org.apache.hive.service.rpc.thrift.TGetDelegationTokenResp;
+import org.apache.hive.service.rpc.thrift.TGetFunctionsReq;
+import org.apache.hive.service.rpc.thrift.TGetFunctionsResp;
+import org.apache.hive.service.rpc.thrift.TGetInfoReq;
+import org.apache.hive.service.rpc.thrift.TGetInfoResp;
+import org.apache.hive.service.rpc.thrift.TGetOperationStatusReq;
+import org.apache.hive.service.rpc.thrift.TGetOperationStatusResp;
+import org.apache.hive.service.rpc.thrift.TGetPrimaryKeysReq;
+import 

incubator-livy git commit: [LIVY-498][REPL] Fix Windows CRLF line ending issue in SparkR interpreter

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 551dd5c27 -> 8027ca708


[LIVY-498][REPL] Fix Windows CRLF line ending issue in SparkR interpreter

## What changes were proposed in this pull request?

If the issued query contains CRLF EOL, it will be failed to execute on *nix 
machine. This happens when submitting queries from Windows machine and 
executing on Linux machine.

So here propose to convert statement to match system's EOL.

## How was this patch tested?

New UT added.

Author: jerryshao 

Closes #105 from jerryshao/LIVY-498.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/8027ca70
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/8027ca70
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/8027ca70

Branch: refs/heads/master
Commit: 8027ca708fdc3df9a5b08d2d33d0436018154bcc
Parents: 551dd5c
Author: jerryshao 
Authored: Wed Aug 22 10:48:33 2018 +0800
Committer: jerryshao 
Committed: Wed Aug 22 10:48:33 2018 +0800

--
 .../main/scala/org/apache/livy/EOLUtils.scala   | 101 +++
 .../scala/org/apache/livy/EOLUtilsSuite.scala   |  57 +++
 .../org/apache/livy/test/InteractiveIT.scala|   2 +
 .../scala/org/apache/livy/repl/ReplDriver.scala |   6 +-
 4 files changed, 163 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/8027ca70/core/src/main/scala/org/apache/livy/EOLUtils.scala
--
diff --git a/core/src/main/scala/org/apache/livy/EOLUtils.scala 
b/core/src/main/scala/org/apache/livy/EOLUtils.scala
new file mode 100644
index 000..10f2c62
--- /dev/null
+++ b/core/src/main/scala/org/apache/livy/EOLUtils.scala
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy
+
+/**
+ * Helper class to deal with end-of-line markers in text files.
+ */
+object EOLUtils {
+  /** Unix-style end-of-line marker (LF) */
+  private val EOL_UNIX: String = "\n"
+
+  /** Windows-style end-of-line marker (CRLF) */
+  private val EOL_WINDOWS: String = "\r\n"
+
+  /** "Old Mac"-style end-of-line marker (CR) */
+  private val EOL_OLD_MAC: String = "\r"
+
+  /** Default end-of-line marker on current syste */
+  private val EOL_SYSTEM_DEFAULT: String = System.getProperty("line.separator")
+
+  object Mode extends Enumeration {
+type Mode = Value
+
+val LF, CRLF, CR = Value
+
+lazy val SYSTEM_DEFAULT: Mode = {
+  val tmp = if (EOL_SYSTEM_DEFAULT == EOL_UNIX) {
+LF
+  } else if (EOL_SYSTEM_DEFAULT == EOL_WINDOWS) {
+CRLF
+  } else if (EOL_SYSTEM_DEFAULT == EOL_OLD_MAC) {
+CR
+  } else {
+null
+  }
+
+  if (tmp == null) {
+throw new IllegalStateException("Could not determine system default 
end-of-line marker")
+  }
+  tmp
+}
+
+private def determineEOL(s: String): Mode = {
+  val charArray = s.toCharArray
+
+  var prev: Char = null.asInstanceOf[Char]
+  for (ch <- charArray) {
+if (ch == '\n') {
+  if (prev == '\r') {
+return CRLF
+  } else {
+return LF
+  }
+} else if (prev == '\r') {
+  return CR
+}
+
+prev = ch
+  }
+
+  null
+}
+
+def hasWindowsEOL(s: String): Boolean = determineEOL(s) == CRLF
+
+def hasUnixEOL(s: String): Boolean = determineEOL(s) == LF
+
+def hasOldMacEOL(s: String): Boolean = determineEOL(s) == CR
+
+def hasSystemDefaultEOL(s: String): Boolean = determineEOL(s) == 
SYSTEM_DEFAULT
+  }
+
+  def convertToSystemEOL(s: String): String = convertLineEndings(s, 
EOL_SYSTEM_DEFAULT)
+
+  private def convertLineEndings(s: String, eol: String): String = {
+if (Mode.hasWindowsEOL(s)) {
+  s.replaceAll(EOL_WINDOWS, eol)
+} else if (Mode.hasUnixEOL(s)) {
+  s.replaceAll(EOL_UNIX, eol)
+} else if (Mode.hasOldMacEOL(s)) {
+  s.replaceAll(EOL_OLD_MAC, eol)
+} else {
+  s
+}
+  }

incubator-livy git commit: [LIVY-498][REPL] Fix Windows CRLF line ending issue in SparkR interpreter

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.5 916255a9f -> d6467fd0c


[LIVY-498][REPL] Fix Windows CRLF line ending issue in SparkR interpreter

## What changes were proposed in this pull request?

If the issued query contains CRLF EOL, it will be failed to execute on *nix 
machine. This happens when submitting queries from Windows machine and 
executing on Linux machine.

So here propose to convert statement to match system's EOL.

## How was this patch tested?

New UT added.

Author: jerryshao 

Closes #105 from jerryshao/LIVY-498.

(cherry picked from commit 8027ca708fdc3df9a5b08d2d33d0436018154bcc)
Signed-off-by: jerryshao 


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/d6467fd0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/d6467fd0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/d6467fd0

Branch: refs/heads/branch-0.5
Commit: d6467fd0c5be465c7f86fcaa820b6bb8dbfebd2f
Parents: 916255a
Author: jerryshao 
Authored: Wed Aug 22 10:48:33 2018 +0800
Committer: jerryshao 
Committed: Wed Aug 22 10:48:48 2018 +0800

--
 .../main/scala/org/apache/livy/EOLUtils.scala   | 101 +++
 .../scala/org/apache/livy/EOLUtilsSuite.scala   |  57 +++
 .../org/apache/livy/test/InteractiveIT.scala|   2 +
 .../scala/org/apache/livy/repl/ReplDriver.scala |   6 +-
 4 files changed, 163 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/d6467fd0/core/src/main/scala/org/apache/livy/EOLUtils.scala
--
diff --git a/core/src/main/scala/org/apache/livy/EOLUtils.scala 
b/core/src/main/scala/org/apache/livy/EOLUtils.scala
new file mode 100644
index 000..10f2c62
--- /dev/null
+++ b/core/src/main/scala/org/apache/livy/EOLUtils.scala
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy
+
+/**
+ * Helper class to deal with end-of-line markers in text files.
+ */
+object EOLUtils {
+  /** Unix-style end-of-line marker (LF) */
+  private val EOL_UNIX: String = "\n"
+
+  /** Windows-style end-of-line marker (CRLF) */
+  private val EOL_WINDOWS: String = "\r\n"
+
+  /** "Old Mac"-style end-of-line marker (CR) */
+  private val EOL_OLD_MAC: String = "\r"
+
+  /** Default end-of-line marker on current syste */
+  private val EOL_SYSTEM_DEFAULT: String = System.getProperty("line.separator")
+
+  object Mode extends Enumeration {
+type Mode = Value
+
+val LF, CRLF, CR = Value
+
+lazy val SYSTEM_DEFAULT: Mode = {
+  val tmp = if (EOL_SYSTEM_DEFAULT == EOL_UNIX) {
+LF
+  } else if (EOL_SYSTEM_DEFAULT == EOL_WINDOWS) {
+CRLF
+  } else if (EOL_SYSTEM_DEFAULT == EOL_OLD_MAC) {
+CR
+  } else {
+null
+  }
+
+  if (tmp == null) {
+throw new IllegalStateException("Could not determine system default 
end-of-line marker")
+  }
+  tmp
+}
+
+private def determineEOL(s: String): Mode = {
+  val charArray = s.toCharArray
+
+  var prev: Char = null.asInstanceOf[Char]
+  for (ch <- charArray) {
+if (ch == '\n') {
+  if (prev == '\r') {
+return CRLF
+  } else {
+return LF
+  }
+} else if (prev == '\r') {
+  return CR
+}
+
+prev = ch
+  }
+
+  null
+}
+
+def hasWindowsEOL(s: String): Boolean = determineEOL(s) == CRLF
+
+def hasUnixEOL(s: String): Boolean = determineEOL(s) == LF
+
+def hasOldMacEOL(s: String): Boolean = determineEOL(s) == CR
+
+def hasSystemDefaultEOL(s: String): Boolean = determineEOL(s) == 
SYSTEM_DEFAULT
+  }
+
+  def convertToSystemEOL(s: String): String = convertLineEndings(s, 
EOL_SYSTEM_DEFAULT)
+
+  private def convertLineEndings(s: String, eol: String): String = {
+if (Mode.hasWindowsEOL(s)) {
+  s.replaceAll(EOL_WINDOWS, eol)
+} else if (Mode.hasUnixEOL(s)) {
+  s.replaceAll(EOL_UNIX, eol)
+} 

incubator-livy-website git commit: Publishing from 293dcc865131d00fb66d7ed283544758a92653aa

[jshao]

Repository: incubator-livy-website
Updated Branches:
  refs/heads/asf-site 579916d45 -> 281e56f2f


Publishing from 293dcc865131d00fb66d7ed283544758a92653aa


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy-website/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/commit/281e56f2
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/tree/281e56f2
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/diff/281e56f2

Branch: refs/heads/asf-site
Commit: 281e56f2f8e934b192992714667e09a5802293d8
Parents: 579916d
Author: jerryshao 
Authored: Fri Jun 15 11:10:31 2018 +0800
Committer: jerryshao 
Committed: Fri Jun 15 11:10:31 2018 +0800

--
 content/feed.xml| 8 
 content/get-started/index.html  | 2 +-
 content/third-party-projects/index.html | 7 ++-
 3 files changed, 11 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/281e56f2/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index e144f45..99a259a 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -5,8 +5,8 @@
 
 /
 
-Wed, 02 May 2018 12:48:58 -0700
-Wed, 02 May 2018 12:48:58 -0700
+Fri, 15 Jun 2018 11:10:28 +0800
+Fri, 15 Jun 2018 11:10:28 +0800
 Jekyll v3.4.5
 
   
@@ -44,7 +44,7 @@ Support multiple languages in a single Session/p
   /li
 /ul
 
-Mon, 05 Feb 2018 04:00:00 -0800
+Mon, 05 Feb 2018 20:00:00 +0800
 /news/2018/02/05/release-0.5.0/
 /news/2018/02/05/release-0.5.0/
 
@@ -74,7 +74,7 @@ Create a Livy Web UI to monitor sessions/li
 Improve Livy’s ACLs/li
 /ul
 
-Fri, 01 Sep 2017 05:00:00 -0700
+Fri, 01 Sep 2017 20:00:00 +0800
 /news/2017/09/01/release-0.4.0/
 /news/2017/09/01/release-0.4.0/
 

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/281e56f2/content/get-started/index.html
--
diff --git a/content/get-started/index.html b/content/get-started/index.html
index f2b712f..01b17fe 100644
--- a/content/get-started/index.html
+++ b/content/get-started/index.html
@@ -168,7 +168,7 @@ builds of Spark. To run Livy with local sessions, first 
export these variables:<
 
 Then start the server with:
 
-./bin/livy-server
+./bin/livy-server start
 
 Livy uses the Spark configuration under SPARK_HOME by default. You can override the 
Spark configuration by setting the
 SPARK_CONF_DIR environment variable 
before starting Livy.

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/281e56f2/content/third-party-projects/index.html
--
diff --git a/content/third-party-projects/index.html 
b/content/third-party-projects/index.html
index b0d1d1b..a9be477 100644
--- a/content/third-party-projects/index.html
+++ b/content/third-party-projects/index.html
@@ -160,7 +160,12 @@ Please open a pull request https://github.com/apache/incubator-livy-web
 Clients
 
 
-  https://www.npmjs.com/package/node-livy-client;>node-livy-client - 
Livy client written in NodeJS
+  
+https://www.npmjs.com/package/node-livy-client;>node-livy-client - 
Livy client written in NodeJS
+  
+  
+https://github.com/maropu/spark-sql-server;>spark-sql-server - A 
Spark SQL server based on the PostgreSQL V3 protocol
+  
 
 
 

incubator-livy-website git commit: Add a new link in third-party-projects.md

[jshao]

Repository: incubator-livy-website
Updated Branches:
  refs/heads/master fc83685b3 -> 293dcc865


Add a new link in third-party-projects.md

This pr added a new link in `third-party-projects.md`. I'm working on a Spark 
SQL server based on the PostgreSQL V3 protocol; it uses `Livy` for managing 
multiple `SparkContext` (See for details: 
https://github.com/maropu/spark-sql-server#execution-modes)

Author: Takeshi Yamamuro 

Closes #23 from maropu/AddLink.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy-website/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/commit/293dcc86
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/tree/293dcc86
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/diff/293dcc86

Branch: refs/heads/master
Commit: 293dcc865131d00fb66d7ed283544758a92653aa
Parents: fc83685
Author: Takeshi Yamamuro 
Authored: Fri Jun 15 11:02:19 2018 +0800
Committer: jerryshao 
Committed: Fri Jun 15 11:02:19 2018 +0800

--
 site/third-party-projects.md | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/293dcc86/site/third-party-projects.md
--
diff --git a/site/third-party-projects.md b/site/third-party-projects.md
index 96511c8..9bd6474 100644
--- a/site/third-party-projects.md
+++ b/site/third-party-projects.md
@@ -34,3 +34,5 @@ Please open a pull request [here]({{ 
site.data.project.website_repository_mirror
 
 * [node-livy-client](https://www.npmjs.com/package/node-livy-client) - Livy 
client written in NodeJS
 
+* [spark-sql-server](https://github.com/maropu/spark-sql-server) - A Spark SQL 
server based on the PostgreSQL V3 protocol
+

incubator-livy git commit: [HOTFIX] Fix Base64 not existed in jdk7 issue

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 219630273 -> 551dd5c27


[HOTFIX] Fix Base64 not existed in jdk7 issue


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/551dd5c2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/551dd5c2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/551dd5c2

Branch: refs/heads/master
Commit: 551dd5c27f05c35c75015c7dae9153e2a7f23089
Parents: 2196302
Author: jerryshao 
Authored: Wed Jun 13 11:42:40 2018 +0800
Committer: jerryshao 
Committed: Wed Jun 13 11:42:40 2018 +0800

--
 core/pom.xml| 5 +
 core/src/main/scala/org/apache/livy/Utils.scala | 7 +--
 2 files changed, 10 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/551dd5c2/core/pom.xml
--
diff --git a/core/pom.xml b/core/pom.xml
index 9113cad..d06e633 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -61,6 +61,11 @@
   org.slf4j
   slf4j-api
 
+
+
+  commons-codec
+  commons-codec
+
   
 
 

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/551dd5c2/core/src/main/scala/org/apache/livy/Utils.scala
--
diff --git a/core/src/main/scala/org/apache/livy/Utils.scala 
b/core/src/main/scala/org/apache/livy/Utils.scala
index 3855679..51020e2 100644
--- a/core/src/main/scala/org/apache/livy/Utils.scala
+++ b/core/src/main/scala/org/apache/livy/Utils.scala
@@ -21,13 +21,15 @@ import java.io.{Closeable, File, InputStreamReader}
 import java.net.URL
 import java.nio.charset.StandardCharsets.UTF_8
 import java.security.SecureRandom
-import java.util.{Base64, Properties}
+import java.util.Properties
 
 import scala.annotation.tailrec
 import scala.collection.JavaConverters._
 import scala.concurrent.TimeoutException
 import scala.concurrent.duration.Duration
 
+import org.apache.commons.codec.binary.Base64
+
 object Utils {
   def getPropertiesFromFile(file: File): Map[String, String] = {
 loadProperties(file.toURI().toURL())
@@ -111,6 +113,7 @@ object Utils {
 val rnd = new SecureRandom()
 val secretBytes = new Array[Byte](secretBitLength / java.lang.Byte.SIZE)
 rnd.nextBytes(secretBytes)
-Base64.getEncoder.encodeToString(secretBytes)
+
+Base64.encodeBase64String(secretBytes)
   }
 }

incubator-livy git commit: [Security] Update to support pyspark and sparkr changes in Spark 2.3.1

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master fe0283f86 -> 219630273


[Security] Update to support pyspark and sparkr changes in Spark 2.3.1


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/21963027
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/21963027
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/21963027

Branch: refs/heads/master
Commit: 2196302731590def9a8f8a25628dd302eac06260
Parents: fe0283f
Author: jerryshao 
Authored: Tue Jun 12 20:06:16 2018 +0800
Committer: jerryshao 
Committed: Wed Jun 13 10:00:30 2018 +0800

--
 core/src/main/scala/org/apache/livy/Utils.scala | 11 +-
 pom.xml |  8 ++--
 repl/src/main/resources/fake_shell.py   | 23 ---
 .../apache/livy/repl/PythonInterpreter.scala| 30 ++-
 .../apache/livy/repl/SparkRInterpreter.scala| 40 
 5 files changed, 91 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/21963027/core/src/main/scala/org/apache/livy/Utils.scala
--
diff --git a/core/src/main/scala/org/apache/livy/Utils.scala 
b/core/src/main/scala/org/apache/livy/Utils.scala
index c1cffe4..3855679 100644
--- a/core/src/main/scala/org/apache/livy/Utils.scala
+++ b/core/src/main/scala/org/apache/livy/Utils.scala
@@ -17,10 +17,11 @@
 
 package org.apache.livy
 
-import java.io.{Closeable, File, FileInputStream, InputStreamReader}
+import java.io.{Closeable, File, InputStreamReader}
 import java.net.URL
 import java.nio.charset.StandardCharsets.UTF_8
-import java.util.Properties
+import java.security.SecureRandom
+import java.util.{Base64, Properties}
 
 import scala.annotation.tailrec
 import scala.collection.JavaConverters._
@@ -106,4 +107,10 @@ object Utils {
 }
   }
 
+  def createSecret(secretBitLength: Int): String = {
+val rnd = new SecureRandom()
+val secretBytes = new Array[Byte](secretBitLength / java.lang.Byte.SIZE)
+rnd.nextBytes(secretBytes)
+Base64.getEncoder.encodeToString(secretBytes)
+  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/21963027/pom.xml
--
diff --git a/pom.xml b/pom.xml
index bcb2dfb..85adaa1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1196,13 +1196,13 @@
 
   
   
-2.3.0
+2.3.1
 2.2.0
 ${spark.scala-2.11.version}
 4.1.17.Final
 4.0.37.Final
 1.8
-0.10.4
+0.10.7
 3.2.11
   
 
@@ -1216,9 +1216,9 @@
   
   
 
-  
http://apache.mirrors.ionfish.org/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz
+  
http://mirrors.advancedhosters.com/apache/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz
 
-spark-2.3.0-bin-hadoop2.7
+spark-2.3.1-bin-hadoop2.7
   
 
 

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/21963027/repl/src/main/resources/fake_shell.py
--
diff --git a/repl/src/main/resources/fake_shell.py 
b/repl/src/main/resources/fake_shell.py
index ee86978..2a99185 100644
--- a/repl/src/main/resources/fake_shell.py
+++ b/repl/src/main/resources/fake_shell.py
@@ -569,7 +569,13 @@ def main():
 from pyspark.sql import SQLContext, HiveContext, Row
 # Connect to the gateway
 gateway_port = int(os.environ["PYSPARK_GATEWAY_PORT"])
-gateway = JavaGateway(GatewayClient(port=gateway_port), 
auto_convert=True)
+try:
+from py4j.java_gateway import GatewayParameters
+gateway_secret = os.environ["PYSPARK_GATEWAY_SECRET"]
+gateway = JavaGateway(gateway_parameters=GatewayParameters(
+port=gateway_port, auth_token=gateway_secret, 
auto_convert=True))
+except:
+gateway = JavaGateway(GatewayClient(port=gateway_port), 
auto_convert=True)
 
 # Import the classes used by PySpark
 java_import(gateway.jvm, "org.apache.spark.SparkConf")
@@ -613,12 +619,17 @@ def main():
 
 #Start py4j callback server
 from py4j.protocol import ENTRY_POINT_OBJECT_ID
-from py4j.java_gateway import JavaGateway, GatewayClient, 
CallbackServerParameters
+from py4j.java_gateway import CallbackServerParameters
+
+try:
+gateway_secret = os.environ["PYSPARK_GATEWAY_SECRET"]
+gateway.start_callback_server(
+callback_server_parameters=CallbackServerParameters(
+port=0, 

incubator-livy git commit: [LIVY-477][BUILD] Upgrade Scala version to 2.11.12

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 02c5915f3 -> fe0283f86


[LIVY-477][BUILD] Upgrade Scala version to 2.11.12

## What changes were proposed in this pull request?

Scala version below 2.11.12 has CVE 
(https://scala-lang.org/news/security-update-nov17.html), and Spark will also 
upgrade its supported version to 2.11.12.

So here upgrading Livy's Scala version also.

## How was this patch tested?

Existing tests.

Author: jerryshao 

Closes #100 from jerryshao/LIVY-477.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/fe0283f8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/fe0283f8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/fe0283f8

Branch: refs/heads/master
Commit: fe0283f864b850973276f3c857908ccb2e6370f2
Parents: 02c5915
Author: jerryshao 
Authored: Tue Jun 12 20:52:52 2018 +0800
Committer: jerryshao 
Committed: Tue Jun 12 20:52:52 2018 +0800

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/fe0283f8/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 0bdf6f9..bcb2dfb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -101,7 +101,7 @@
 UTF-8
 0.9
 2.10.4
-2.11.8
+2.11.12
 2.11
 ${scala-2.11.version}
 2.2.4

incubator-livy git commit: [LIVY-475] Support of Hadoop CredentialProvider API

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 8b3fcbc8d -> 02c5915f3


[LIVY-475] Support of Hadoop CredentialProvider API

In this PR I've added following option to livy.conf:
```
```
to allow to specify path to Hadoop Credential Provider, which than used in 
`WebServer.scala` to set Keystore password and Key password to JKS that used to 
enable SSL encryption.

Also before trying to use Hadoop CredentialProvider API I'm checking if it 
available (as it was not available in Hadoop < 2.6) using the same [method that 
used in 
Oozie](https://github.com/apache/oozie/commit/6a731f9926158da38d1e3b518671ada95a544fe8#diff-800f95e605f21c5aaf5edef13039c9b9R124).

To use this, you will need to generate Credential Provider containing 
"livy.keystore.password" and/or "livy.key-password" in the common way:
```bash
hadoop credential create "livy.keystore.password" -value "keystore_secret" 
-provider jceks://hdfsnn1.example.com/my/path/livy_creds.jceks
hadoop credential create "livy.key-password" -value "key_secret" -provider 
jceks://hdfsnn1.example.com/my/path/livy_creds.jceks
```

Author: Ivan Dzikovsky 

Closes #99 from idzikovsky/LIVY-475.

Change-Id: Iae60550900067e61a37a6f74acb5e299005f3397


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/02c5915f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/02c5915f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/02c5915f

Branch: refs/heads/master
Commit: 02c5915f38a0cd19acfc833cba37d73393f9
Parents: 8b3fcbc
Author: Ivan Dzikovsky 
Authored: Thu May 31 19:53:25 2018 +0800
Committer: jerryshao 
Committed: Thu May 31 19:53:25 2018 +0800

--
 conf/livy.conf.template |  5 
 .../main/scala/org/apache/livy/LivyConf.scala   |  2 ++
 .../org/apache/livy/server/WebServer.scala  | 24 
 3 files changed, 27 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/02c5915f/conf/livy.conf.template
--
diff --git a/conf/livy.conf.template b/conf/livy.conf.template
index 6f50e2f..fede70a 100644
--- a/conf/livy.conf.template
+++ b/conf/livy.conf.template
@@ -23,6 +23,11 @@
 # Specify the key password.
 # livy.key-password =
 
+# Hadoop Credential Provider Path to get "livy.keystore.password" and 
"livy.key-password".
+# Credential Provider can be created using command as follow:
+# hadoop credential create "livy.keystore.password" -value "secret" -provider 
jceks://hdfs/path/to/livy.jceks
+# livy.hadoop.security.credential.provider.path =
+
 # What host address to start the server on. By default, Livy will bind to all 
network interfaces.
 # livy.server.host = 0.0.0.0
 

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/02c5915f/server/src/main/scala/org/apache/livy/LivyConf.scala
--
diff --git a/server/src/main/scala/org/apache/livy/LivyConf.scala 
b/server/src/main/scala/org/apache/livy/LivyConf.scala
index 3f6a86c..edee79a 100644
--- a/server/src/main/scala/org/apache/livy/LivyConf.scala
+++ b/server/src/main/scala/org/apache/livy/LivyConf.scala
@@ -85,6 +85,8 @@ object LivyConf {
   val SSL_KEYSTORE_PASSWORD = Entry("livy.keystore.password", null)
   val SSL_KEY_PASSWORD = Entry("livy.key-password", null)
 
+  val HADOOP_CREDENTIAL_PROVIDER_PATH = 
Entry("livy.hadoop.security.credential.provider.path", null)
+
   val AUTH_TYPE = Entry("livy.server.auth.type", null)
   val AUTH_KERBEROS_PRINCIPAL = Entry("livy.server.auth.kerberos.principal", 
null)
   val AUTH_KERBEROS_KEYTAB = Entry("livy.server.auth.kerberos.keytab", null)

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/02c5915f/server/src/main/scala/org/apache/livy/server/WebServer.scala
--
diff --git a/server/src/main/scala/org/apache/livy/server/WebServer.scala 
b/server/src/main/scala/org/apache/livy/server/WebServer.scala
index 549d6ab..1955974 100644
--- a/server/src/main/scala/org/apache/livy/server/WebServer.scala
+++ b/server/src/main/scala/org/apache/livy/server/WebServer.scala
@@ -20,6 +20,7 @@ package org.apache.livy.server
 import java.net.InetAddress
 import javax.servlet.ServletContextListener
 
+import org.apache.hadoop.conf.Configuration
 import org.eclipse.jetty.server._
 import org.eclipse.jetty.server.handler.{HandlerCollection, RequestLogHandler}
 import org.eclipse.jetty.servlet.{DefaultServlet, ServletContextHandler}
@@ -48,10 +49,25 @@ class WebServer(livyConf: LivyConf, var host: String, var 
port: Int) extends Log
 
   val sslContextFactory = new SslContextFactory()
   sslContextFactory.setKeyStorePath(keystore)
-

incubator-livy git commit: [LIVY-473][TEST] Minor refactor of integration test to remove legacy code

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master ca4cad229 -> 8b3fcbc8d


[LIVY-473][TEST] Minor refactor of integration test to remove legacy code

Integration test has some legacy codes related to different cluster type (mini 
or real). Since now we already use real Spark package to do test and partially 
removed that code, we should refactor to remove all the unused legacy code.

Existing tests.

Author: jerryshao 

Closes #97 from jerryshao/LIVY-473.

Change-Id: I7ca6c69884f7ed11a34cde5a6e1fc9bfcd141cbb


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/8b3fcbc8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/8b3fcbc8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/8b3fcbc8

Branch: refs/heads/master
Commit: 8b3fcbc8d52a6e6e1628cf139c0b0f49be797cec
Parents: ca4cad2
Author: jerryshao 
Authored: Tue May 29 13:51:20 2018 +0800
Committer: jerryshao 
Committed: Tue May 29 13:51:20 2018 +0800

--
 .../apache/livy/test/framework/Cluster.scala| 38 +--
 .../livy/test/framework/MiniCluster.scala   | 22 +--
 .../livy/test/framework/MiniClusterUtils.scala  | 68 
 3 files changed, 71 insertions(+), 57 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/8b3fcbc8/integration-test/src/main/scala/org/apache/livy/test/framework/Cluster.scala
--
diff --git 
a/integration-test/src/main/scala/org/apache/livy/test/framework/Cluster.scala 
b/integration-test/src/main/scala/org/apache/livy/test/framework/Cluster.scala
index 26b08a9..907a6bf 100644
--- 
a/integration-test/src/main/scala/org/apache/livy/test/framework/Cluster.scala
+++ 
b/integration-test/src/main/scala/org/apache/livy/test/framework/Cluster.scala
@@ -75,8 +75,6 @@ trait Cluster {
 }
 
 object Cluster extends Logging {
-  private val CLUSTER_TYPE = "cluster.type"
-
   private lazy val config = {
 sys.props.get("cluster.spec")
   .filter { path => path.nonEmpty && path != "default" }
@@ -92,16 +90,13 @@ object Cluster extends Logging {
 }
 p.asScala.toMap
   }
-  .getOrElse(Map(CLUSTER_TYPE -> "mini"))
+  .getOrElse(Map.empty)
   }
 
   private lazy val cluster = {
 var _cluster: Cluster = null
 try {
-  _cluster = config.get(CLUSTER_TYPE) match {
-case Some("mini") => new MiniCluster(config)
-case t => throw new Exception(s"Unknown or unset cluster.type $t")
-  }
+  _cluster = new MiniCluster(config)
   Runtime.getRuntime.addShutdownHook(new Thread {
 override def run(): Unit = {
   info("Shutting down cluster pool.")
@@ -126,32 +121,3 @@ object Cluster extends Logging {
 
   def isRunningOnTravis: Boolean = sys.env.contains("TRAVIS")
 }
-
-trait ClusterUtils {
-
-  protected def saveProperties(props: Map[String, String], dest: File): Unit = 
{
-val jprops = new Properties()
-props.foreach { case (k, v) => jprops.put(k, v) }
-
-val tempFile = new File(dest.getAbsolutePath() + ".tmp")
-val out = new OutputStreamWriter(new FileOutputStream(tempFile), UTF_8)
-try {
-  jprops.store(out, "Configuration")
-} finally {
-  out.close()
-}
-tempFile.renameTo(dest)
-  }
-
-  protected def loadProperties(file: File): Map[String, String] = {
-val in = new InputStreamReader(new FileInputStream(file), UTF_8)
-val props = new Properties()
-try {
-  props.load(in)
-} finally {
-  in.close()
-}
-props.asScala.toMap
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/8b3fcbc8/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
--
diff --git 
a/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
 
b/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
index 3cfc772..c4d4eb5 100644
--- 
a/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
+++ 
b/integration-test/src/main/scala/org/apache/livy/test/framework/MiniCluster.scala
@@ -18,8 +18,6 @@
 package org.apache.livy.test.framework
 
 import java.io._
-import java.nio.charset.Charset
-import java.nio.file.{Files, Paths}
 import javax.servlet.http.HttpServletResponse
 
 import scala.concurrent.duration._
@@ -52,24 +50,6 @@ private class MiniClusterConfig(val config: Map[String, 
String]) {
 
 }
 
-sealed trait MiniClusterUtils extends ClusterUtils {
-
-  protected def saveConfig(conf: Configuration, dest: File): Unit = {
-val redacted = new Configuration(conf)
-// This setting references a test class that is not available when using a 
real Spark
-// 

incubator-livy git commit: [LIVY-472][SERVER] Improve the logs for fail-to-create session

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 6f1756cff -> ca4cad229


[LIVY-472][SERVER] Improve the logs for fail-to-create session

## What changes were proposed in this pull request?

Livy currently doesn't give a very clear log about the fail-to-create session, 
it only says that session related app tag cannot be found in RM, but doesn't 
tell user how to search and get the true root cause. So here change the logs to 
make it more clear.

## How was this patch tested?

Local verification.

Author: jerryshao 

Closes #96 from jerryshao/LIVY-472.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/ca4cad22
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/ca4cad22
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/ca4cad22

Branch: refs/heads/master
Commit: ca4cad22968e1a2f88fa0ec262c1088812e3d251
Parents: 6f1756c
Author: jerryshao 
Authored: Fri May 25 09:36:03 2018 +0800
Committer: jerryshao 
Committed: Fri May 25 09:36:03 2018 +0800

--
 .../interactive/InteractiveSessionServlet.scala  |  2 +-
 .../org/apache/livy/utils/SparkYarnApp.scala | 19 +++
 2 files changed, 12 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/ca4cad22/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
--
diff --git 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
index 54046a1..4d614f4 100644
--- 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
+++ 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
@@ -70,7 +70,7 @@ class InteractiveSessionServlet(
 Option(session.logLines())
   .map { lines =>
 val size = 10
-var from = math.max(0, lines.length - size)
+val from = math.max(0, lines.length - size)
 val until = from + size
 
 lines.view(from, until)

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/ca4cad22/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
--
diff --git a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala 
b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
index 91c70ca..d255796 100644
--- a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
+++ b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
@@ -25,6 +25,7 @@ import scala.concurrent._
 import scala.concurrent.duration._
 import scala.language.postfixOps
 import scala.util.Try
+import scala.util.control.NonFatal
 
 import org.apache.hadoop.yarn.api.records.{ApplicationId, ApplicationReport, 
FinalApplicationStatus, YarnApplicationState}
 import org.apache.hadoop.yarn.client.api.YarnClient
@@ -70,7 +71,7 @@ object SparkYarnApp extends Logging {
 override def run(): Unit = {
   while (true) {
 if (!leakedAppTags.isEmpty) {
-  // kill the app if found it and remove it if exceeding a threashold
+  // kill the app if found it and remove it if exceeding a threshold
   val iter = leakedAppTags.entrySet().iterator()
   var isRemoved = false
   val now = System.currentTimeMillis()
@@ -179,9 +180,11 @@ class SparkYarnApp private[utils] (
 if (deadline.isOverdue) {
   process.foreach(_.destroy())
   leakedAppTags.put(appTag, System.currentTimeMillis())
-  throw new Exception(s"No YARN application is found with tag 
$appTagLowerCase in " +
-livyConf.getTimeAsMs(LivyConf.YARN_APP_LOOKUP_TIMEOUT)/1000 + " 
seconds. " +
-"Please check your cluster status, it is may be very busy.")
+  throw new IllegalStateException(s"No YARN application is found with 
tag" +
+s" $appTagLowerCase in 
${livyConf.getTimeAsMs(LivyConf.YARN_APP_LOOKUP_TIMEOUT)/1000}" +
+" seconds. This may be because 1) spark-submit fail to submit 
application to YARN; " +
+"or 2) YARN cluster doesn't have enough resources to start the 
application in time. " +
+"Please check Livy log and YARN log to know the details.")
 } else {
   Clock.sleep(pollInterval.toMillis)
   getAppIdFromTag(appTagLowerCase, pollInterval, deadline)
@@ -290,12 +293,12 @@ class SparkYarnApp private[utils] (
 
   debug(s"$appId $state ${yarnDiagnostics.mkString(" ")}")
 } catch {
-  case e: 

incubator-livy git commit: [LIVY-472][SERVER] Improve the logs for fail-to-create session

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.5 8c9698113 -> 916255a9f


[LIVY-472][SERVER] Improve the logs for fail-to-create session

## What changes were proposed in this pull request?

Livy currently doesn't give a very clear log about the fail-to-create session, 
it only says that session related app tag cannot be found in RM, but doesn't 
tell user how to search and get the true root cause. So here change the logs to 
make it more clear.

## How was this patch tested?

Local verification.

Author: jerryshao 

Closes #96 from jerryshao/LIVY-472.

(cherry picked from commit ca4cad22968e1a2f88fa0ec262c1088812e3d251)
Signed-off-by: jerryshao 


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/916255a9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/916255a9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/916255a9

Branch: refs/heads/branch-0.5
Commit: 916255a9f3587cf30ed0d35d0a3e58aeb12d1dba
Parents: 8c96981
Author: jerryshao 
Authored: Fri May 25 09:36:03 2018 +0800
Committer: jerryshao 
Committed: Fri May 25 09:36:13 2018 +0800

--
 .../interactive/InteractiveSessionServlet.scala  |  2 +-
 .../org/apache/livy/utils/SparkYarnApp.scala | 19 +++
 2 files changed, 12 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/916255a9/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
--
diff --git 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
index 54046a1..4d614f4 100644
--- 
a/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
+++ 
b/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
@@ -70,7 +70,7 @@ class InteractiveSessionServlet(
 Option(session.logLines())
   .map { lines =>
 val size = 10
-var from = math.max(0, lines.length - size)
+val from = math.max(0, lines.length - size)
 val until = from + size
 
 lines.view(from, until)

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/916255a9/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
--
diff --git a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala 
b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
index 91c70ca..d255796 100644
--- a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
+++ b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
@@ -25,6 +25,7 @@ import scala.concurrent._
 import scala.concurrent.duration._
 import scala.language.postfixOps
 import scala.util.Try
+import scala.util.control.NonFatal
 
 import org.apache.hadoop.yarn.api.records.{ApplicationId, ApplicationReport, 
FinalApplicationStatus, YarnApplicationState}
 import org.apache.hadoop.yarn.client.api.YarnClient
@@ -70,7 +71,7 @@ object SparkYarnApp extends Logging {
 override def run(): Unit = {
   while (true) {
 if (!leakedAppTags.isEmpty) {
-  // kill the app if found it and remove it if exceeding a threashold
+  // kill the app if found it and remove it if exceeding a threshold
   val iter = leakedAppTags.entrySet().iterator()
   var isRemoved = false
   val now = System.currentTimeMillis()
@@ -179,9 +180,11 @@ class SparkYarnApp private[utils] (
 if (deadline.isOverdue) {
   process.foreach(_.destroy())
   leakedAppTags.put(appTag, System.currentTimeMillis())
-  throw new Exception(s"No YARN application is found with tag 
$appTagLowerCase in " +
-livyConf.getTimeAsMs(LivyConf.YARN_APP_LOOKUP_TIMEOUT)/1000 + " 
seconds. " +
-"Please check your cluster status, it is may be very busy.")
+  throw new IllegalStateException(s"No YARN application is found with 
tag" +
+s" $appTagLowerCase in 
${livyConf.getTimeAsMs(LivyConf.YARN_APP_LOOKUP_TIMEOUT)/1000}" +
+" seconds. This may be because 1) spark-submit fail to submit 
application to YARN; " +
+"or 2) YARN cluster doesn't have enough resources to start the 
application in time. " +
+"Please check Livy log and YARN log to know the details.")
 } else {
   Clock.sleep(pollInterval.toMillis)
   getAppIdFromTag(appTagLowerCase, pollInterval, deadline)
@@ -290,12 +293,12 @@ class SparkYarnApp 

incubator-livy git commit: [LIVY-452] Differentiate FAILED and KILLED states

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master e3f45a057 -> 551cc5309


[LIVY-452] Differentiate FAILED and KILLED states

Currently, it's not possible to distinguish between two states - 
SparkApp.State.KILLED and SparkApp.State.FAILED. In both cases the session 
state will be SessionState.Dead(). This patch adds new SessionState.Killed() 
which will be used when job was actually killed by user.

https://issues.apache.org/jira/browse/LIVY-452

- Corresponding unit test was adjusted
- Tested manually on YARN cluster with `livy.spark.deploy-mode=cluster`

Author: Alexey Romanenko 

Closes #92 from aromanenko-dev/LIVY-452-SessionState-Killed.

Change-Id: Ia18808d7954e20653e152d8bf6748f43c2707b18


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/551cc530
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/551cc530
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/551cc530

Branch: refs/heads/master
Commit: 551cc53095f0a4b5382602ba0c296f8cf8932e44
Parents: e3f45a0
Author: Alexey Romanenko 
Authored: Mon May 21 09:25:04 2018 +0800
Committer: jerryshao 
Committed: Mon May 21 09:25:04 2018 +0800

--
 .../scala/org/apache/livy/sessions/SessionState.scala  |  4 
 docs/rest-api.md   |  4 
 .../apache/livy/test/framework/LivyRestClient.scala|  5 +
 .../src/test/scala/org/apache/livy/test/BatchIT.scala  |  2 +-
 .../org/apache/livy/server/batch/BatchSession.scala|  4 ++--
 .../livy/server/interactive/InteractiveSession.scala   |  3 ++-
 .../scala/org/apache/livy/utils/SparkProcApp.scala | 13 +++--
 7 files changed, 29 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/551cc530/core/src/main/scala/org/apache/livy/sessions/SessionState.scala
--
diff --git a/core/src/main/scala/org/apache/livy/sessions/SessionState.scala 
b/core/src/main/scala/org/apache/livy/sessions/SessionState.scala
index e56cfbb..d731c9b 100644
--- a/core/src/main/scala/org/apache/livy/sessions/SessionState.scala
+++ b/core/src/main/scala/org/apache/livy/sessions/SessionState.scala
@@ -39,6 +39,7 @@ object SessionState {
 case "shutting_down" => ShuttingDown
 case "error" => Error()
 case "dead" => Dead()
+case "killed" => Killed()
 case "success" => Success()
 case _ => throw new IllegalArgumentException(s"Illegal session state: $s")
   }
@@ -57,6 +58,9 @@ object SessionState {
 
   object ShuttingDown extends SessionState("shutting_down", false)
 
+  case class Killed(override val time: Long = System.nanoTime()) extends
+FinishedSessionState("killed", false, time)
+
   case class Error(override val time: Long = System.nanoTime()) extends
 FinishedSessionState("error", true, time)
 

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/551cc530/docs/rest-api.md
--
diff --git a/docs/rest-api.md b/docs/rest-api.md
index 8c928a8..5949b93 100644
--- a/docs/rest-api.md
+++ b/docs/rest-api.md
@@ -644,6 +644,10 @@ A session represents an interactive shell.
 Session has exited
   
   
+killed
+Session has been killed
+  
+  
 success
 Session is successfully stopped
   

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/551cc530/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
--
diff --git 
a/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
 
b/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
index 5300848..1087559 100644
--- 
a/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
+++ 
b/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
@@ -112,6 +112,7 @@ class LivyRestClient(val httpClient: AsyncHttpClient, val 
livyEndpoint: String)
 
   class BatchSession(id: Int) extends Session(id, BATCH_TYPE) {
 def verifySessionDead(): Unit = verifySessionState(SessionState.Dead())
+def verifySessionKilled(): Unit = verifySessionState(SessionState.Killed())
 def verifySessionRunning(): Unit = verifySessionState(SessionState.Running)
 def verifySessionSuccess(): Unit = 
verifySessionState(SessionState.Success())
   }
@@ -241,6 +242,10 @@ class LivyRestClient(val httpClient: AsyncHttpClient, val 
livyEndpoint: String)
 def verifySessionIdle(): Unit = {
   verifySessionState(SessionState.Idle)
 }
+
+def verifySessionKilled(): Unit = {
+  

incubator-livy git commit: [LIVY-466][RSC] Fix RSCDriver exception during RPC shutdown

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 9d381bdf0 -> e3f45a057


[LIVY-466][RSC] Fix RSCDriver exception during RPC shutdown

## What changes were proposed in this pull request?

During RSCDriver's shutdown, it will first shutdown RPC server, and then all 
the RPC clients. When RPC client is closed, it will register a timeout to avoid 
orphaned RSCDriver, but this is not necessary during RSCDriver's shutdown, so 
here fixing this issue. The details can be seen in 
[JIRA](https://issues.apache.org/jira/browse/LIVY-466).

## How was this patch tested?

Local verification.

Author: jerryshao 

Closes #90 from jerryshao/LIVY-466.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/e3f45a05
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/e3f45a05
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/e3f45a05

Branch: refs/heads/master
Commit: e3f45a057cc45bca5bceb04af8ea9218b35fa621
Parents: 9d381bd
Author: jerryshao 
Authored: Thu May 3 14:37:07 2018 +0800
Committer: jerryshao 
Committed: Thu May 3 14:37:07 2018 +0800

--
 rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/e3f45a05/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
--
diff --git a/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java 
b/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
index f727570..eeba300 100644
--- a/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
+++ b/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
@@ -37,6 +37,7 @@ import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicReference;
 
 import io.netty.channel.ChannelHandler.Sharable;
@@ -92,6 +93,7 @@ public class RSCDriver extends BaseProtocol {
   protected final RSCConf livyConf;
 
   private final AtomicReference idleTimeout;
+  private final AtomicBoolean inShutdown;
 
   public RSCDriver(SparkConf conf, RSCConf livyConf) throws Exception {
 Set perms = 
PosixFilePermissions.fromString("rwx--");
@@ -110,6 +112,7 @@ public class RSCDriver extends BaseProtocol {
 this.activeJobs = new ConcurrentHashMap<>();
 this.bypassJobs = new ConcurrentLinkedDeque<>();
 this.idleTimeout = new AtomicReference<>();
+this.inShutdown = new AtomicBoolean(false);
   }
 
   private synchronized void shutdown() {
@@ -217,7 +220,9 @@ public class RSCDriver extends BaseProtocol {
   @Override
   public void onSuccess(Void unused) {
 clients.remove(client);
-setupIdleTimeout();
+if (!inShutdown.get()) {
+  setupIdleTimeout();
+}
   }
 });
 LOG.debug("Registered new connection from {}.", client.getChannel());
@@ -304,6 +309,7 @@ public class RSCDriver extends BaseProtocol {
   }
 
   private void shutdownServer() {
+inShutdown.compareAndSet(false, true);
 if (server != null) {
   server.close();
 }

incubator-livy git commit: [LIVY-466][RSC] Fix RSCDriver exception during RPC shutdown

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.5 bdb954798 -> 8c9698113


[LIVY-466][RSC] Fix RSCDriver exception during RPC shutdown

## What changes were proposed in this pull request?

During RSCDriver's shutdown, it will first shutdown RPC server, and then all 
the RPC clients. When RPC client is closed, it will register a timeout to avoid 
orphaned RSCDriver, but this is not necessary during RSCDriver's shutdown, so 
here fixing this issue. The details can be seen in 
[JIRA](https://issues.apache.org/jira/browse/LIVY-466).

## How was this patch tested?

Local verification.

Author: jerryshao 

Closes #90 from jerryshao/LIVY-466.

(cherry picked from commit e3f45a057cc45bca5bceb04af8ea9218b35fa621)
Signed-off-by: jerryshao 


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/8c969811
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/8c969811
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/8c969811

Branch: refs/heads/branch-0.5
Commit: 8c96981134a862353b751a29398ae81d1a3a4450
Parents: bdb9547
Author: jerryshao 
Authored: Thu May 3 14:37:07 2018 +0800
Committer: jerryshao 
Committed: Thu May 3 14:37:20 2018 +0800

--
 rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/8c969811/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
--
diff --git a/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java 
b/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
index f727570..eeba300 100644
--- a/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
+++ b/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
@@ -37,6 +37,7 @@ import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicReference;
 
 import io.netty.channel.ChannelHandler.Sharable;
@@ -92,6 +93,7 @@ public class RSCDriver extends BaseProtocol {
   protected final RSCConf livyConf;
 
   private final AtomicReference idleTimeout;
+  private final AtomicBoolean inShutdown;
 
   public RSCDriver(SparkConf conf, RSCConf livyConf) throws Exception {
 Set perms = 
PosixFilePermissions.fromString("rwx--");
@@ -110,6 +112,7 @@ public class RSCDriver extends BaseProtocol {
 this.activeJobs = new ConcurrentHashMap<>();
 this.bypassJobs = new ConcurrentLinkedDeque<>();
 this.idleTimeout = new AtomicReference<>();
+this.inShutdown = new AtomicBoolean(false);
   }
 
   private synchronized void shutdown() {
@@ -217,7 +220,9 @@ public class RSCDriver extends BaseProtocol {
   @Override
   public void onSuccess(Void unused) {
 clients.remove(client);
-setupIdleTimeout();
+if (!inShutdown.get()) {
+  setupIdleTimeout();
+}
   }
 });
 LOG.debug("Registered new connection from {}.", client.getChannel());
@@ -304,6 +309,7 @@ public class RSCDriver extends BaseProtocol {
   }
 
   private void shutdownServer() {
+inShutdown.compareAndSet(false, true);
 if (server != null) {
   server.close();
 }

incubator-livy git commit: [LIVY-460][SERVER] Remove misleading log prefix

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 71c3feed9 -> 9d381bdf0


[LIVY-460][SERVER] Remove misleading log prefix

## What changes were proposed in this pull request?

Removing misleading log prefix "stdout: " for the captured stdout/stderr of 
spark-submit process

## How was this patch tested?

N/A

Author: jerryshao 

Closes #89 from jerryshao/LIVY-460.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/9d381bdf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/9d381bdf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/9d381bdf

Branch: refs/heads/master
Commit: 9d381bdf04704b596006f51de56dc3827054f96c
Parents: 71c3fee
Author: jerryshao 
Authored: Thu Apr 26 09:36:47 2018 +0800
Committer: jerryshao 
Committed: Thu Apr 26 09:36:47 2018 +0800

--
 .../src/main/scala/org/apache/livy/utils/LineBufferedStream.scala  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/9d381bdf/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
--
diff --git 
a/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala 
b/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
index 3298c9d..6896d2f 100644
--- a/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
+++ b/server/src/main/scala/org/apache/livy/utils/LineBufferedStream.scala
@@ -38,7 +38,7 @@ class LineBufferedStream(inputStream: InputStream, logSize: 
Int) extends Logging
 override def run() = {
   val lines = Source.fromInputStream(inputStream).getLines()
   for (line <- lines) {
-info(s"stdout: $line")
+info(line)
 _lock.lock()
 try {
   _lines.add(line)

incubator-livy git commit: [LIVY-458][BUILD] Upgrade jackson version to 2.9.5

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master cd8b11261 -> 71c3feed9


[LIVY-458][BUILD] Upgrade jackson version to 2.9.5

## What changes were proposed in this pull request?

Due to several security issues of jackson databind module (CVE-2018-5968, 
CVE-2017-17485, CVE-2018-7489), here propose to upgrade jackson version 2.9.5.

## How was this patch tested?

Existing UTs.

Author: jerryshao 

Closes #87 from jerryshao/LIVY-458.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/71c3feed
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/71c3feed
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/71c3feed

Branch: refs/heads/master
Commit: 71c3feed9cbafef40805975ea7ce6fa2498bab72
Parents: cd8b112
Author: jerryshao 
Authored: Wed Apr 18 16:51:36 2018 +0800
Committer: jerryshao 
Committed: Wed Apr 18 16:51:36 2018 +0800

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/71c3feed/pom.xml
--
diff --git a/pom.xml b/pom.xml
index e026a91..0bdf6f9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -87,7 +87,7 @@
 15.0
 4.5.3
 4.4.4
-2.9.2
+2.9.5
 3.1.0
 9.2.16.v20160414
 3.2.10

incubator-livy git commit: [LIVY-457][REPL] Fix SQLContext is not initialized correctly issue

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.5 88d66a95f -> bdb954798


[LIVY-457][REPL] Fix SQLContext is not initialized correctly issue

## What changes were proposed in this pull request?

The signature of SQLContext's constructor is changed in Spark2, but we're still 
using the Spark1's signature, which will throw an exception when using this 
object.

## How was this patch tested?

UT and local verification.

Author: jerryshao 

Closes #86 from jerryshao/LIVY-457.

(cherry picked from commit cd8b11261811abb4fe3afe0f19540cd4bdaeb5f7)
Signed-off-by: jerryshao 


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/bdb95479
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/bdb95479
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/bdb95479

Branch: refs/heads/branch-0.5
Commit: bdb954798969359ac1e84b279d626129ea2a8591
Parents: 88d66a9
Author: jerryshao 
Authored: Wed Apr 18 10:29:34 2018 +0800
Committer: jerryshao 
Committed: Wed Apr 18 10:29:50 2018 +0800

--
 .travis.yml   | 5 ++---
 .../src/test/scala/org/apache/livy/test/InteractiveIT.scala   | 4 
 repl/src/main/resources/fake_shell.py | 7 +--
 3 files changed, 11 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/bdb95479/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 02578e8..8cfc513 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -51,14 +51,13 @@ cache:
   pip: true
   directories:
 - $HOME/.m2
-- $(npm config get prefix)
 
 before_install:
   - sudo apt-get -y install python3-pip python-dev
   - sudo apt-get -y install libkrb5-dev
   - sudo apt-get -y remove python-setuptools
-  - sudo pip2 install --upgrade pip "setuptools < 36"
-  - sudo python3 -m pip install --upgrade pip "setuptools < 36"
+  - sudo pip2 install --upgrade "pip < 10.0.0" "setuptools < 36"
+  - sudo python3 -m pip install --upgrade "pip < 10.0.0" "setuptools < 36"
   - sudo pip2 install codecov cloudpickle
   - sudo python3 -m pip install cloudpickle
 

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/bdb95479/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
--
diff --git 
a/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala 
b/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
index 3777fdd..ff29d95 100644
--- a/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
+++ b/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
@@ -90,6 +90,10 @@ class InteractiveIT extends BaseIntegrationTestSuite {
   s.run("from pyspark.sql.types import Row").verifyResult("")
   s.run("x = [Row(age=1, name=u'a'), Row(age=2, name=u'b'), Row(age=3, 
name=u'c')]")
 .verifyResult("")
+  // Check if we're running with Spark2.
+  if (s.run("spark").result().isLeft) {
+
s.run("sqlContext.sparkSession").verifyResult(".*pyspark\\.sql\\.session\\.SparkSession.*")
+  }
   s.run("%table x").verifyResult(".*headers.*type.*name.*data.*")
   s.run("abcde").verifyError(ename = "NameError", evalue = "name 'abcde' 
is not defined")
   s.run("raise KeyError, 'foo'").verifyError(ename = "KeyError", evalue = 
"'foo'")

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/bdb95479/repl/src/main/resources/fake_shell.py
--
diff --git a/repl/src/main/resources/fake_shell.py 
b/repl/src/main/resources/fake_shell.py
index 56a0ac4..ee86978 100644
--- a/repl/src/main/resources/fake_shell.py
+++ b/repl/src/main/resources/fake_shell.py
@@ -588,14 +588,17 @@ def main():
 conf = SparkConf(_jvm = gateway.jvm, _jconf = jconf)
 sc = SparkContext(jsc=jsc, gateway=gateway, conf=conf)
 global_dict['sc'] = sc
-sqlc = SQLContext(sc, jsqlc)
-global_dict['sqlContext'] = sqlc
 
 if spark_major_version >= "2":
 from pyspark.sql import SparkSession
 spark_session = SparkSession(sc, 
gateway.entry_point.sparkSession())
+sqlc = SQLContext(sc, spark_session, jsqlc)
+global_dict['sqlContext'] = sqlc
 global_dict['spark'] = spark_session
 else:
+sqlc = SQLContext(sc, jsqlc)
+global_dict['sqlContext'] = sqlc
+
 # LIVY-294, need to check whether HiveContext can work 

incubator-livy git commit: [LIVY-457][REPL] Fix SQLContext is not initialized correctly issue

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 7e4bb3bd6 -> cd8b11261


[LIVY-457][REPL] Fix SQLContext is not initialized correctly issue

## What changes were proposed in this pull request?

The signature of SQLContext's constructor is changed in Spark2, but we're still 
using the Spark1's signature, which will throw an exception when using this 
object.

## How was this patch tested?

UT and local verification.

Author: jerryshao 

Closes #86 from jerryshao/LIVY-457.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/cd8b1126
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/cd8b1126
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/cd8b1126

Branch: refs/heads/master
Commit: cd8b11261811abb4fe3afe0f19540cd4bdaeb5f7
Parents: 7e4bb3b
Author: jerryshao 
Authored: Wed Apr 18 10:29:34 2018 +0800
Committer: jerryshao 
Committed: Wed Apr 18 10:29:34 2018 +0800

--
 .travis.yml   | 5 ++---
 .../src/test/scala/org/apache/livy/test/InteractiveIT.scala   | 4 
 repl/src/main/resources/fake_shell.py | 7 +--
 3 files changed, 11 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/cd8b1126/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index e9ae42e..e62a8d9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -55,14 +55,13 @@ cache:
   pip: true
   directories:
 - $HOME/.m2
-- $(npm config get prefix)
 
 before_install:
   - sudo apt-get -y install python3-pip python-dev
   - sudo apt-get -y install libkrb5-dev
   - sudo apt-get -y remove python-setuptools
-  - sudo pip2 install --upgrade pip "setuptools < 36"
-  - sudo python3 -m pip install --upgrade pip "setuptools < 36"
+  - sudo pip2 install --upgrade "pip < 10.0.0" "setuptools < 36"
+  - sudo python3 -m pip install --upgrade "pip < 10.0.0" "setuptools < 36"
   - sudo pip2 install codecov cloudpickle
   - sudo python3 -m pip install cloudpickle
 

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/cd8b1126/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
--
diff --git 
a/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala 
b/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
index 3777fdd..ff29d95 100644
--- a/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
+++ b/integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
@@ -90,6 +90,10 @@ class InteractiveIT extends BaseIntegrationTestSuite {
   s.run("from pyspark.sql.types import Row").verifyResult("")
   s.run("x = [Row(age=1, name=u'a'), Row(age=2, name=u'b'), Row(age=3, 
name=u'c')]")
 .verifyResult("")
+  // Check if we're running with Spark2.
+  if (s.run("spark").result().isLeft) {
+
s.run("sqlContext.sparkSession").verifyResult(".*pyspark\\.sql\\.session\\.SparkSession.*")
+  }
   s.run("%table x").verifyResult(".*headers.*type.*name.*data.*")
   s.run("abcde").verifyError(ename = "NameError", evalue = "name 'abcde' 
is not defined")
   s.run("raise KeyError, 'foo'").verifyError(ename = "KeyError", evalue = 
"'foo'")

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/cd8b1126/repl/src/main/resources/fake_shell.py
--
diff --git a/repl/src/main/resources/fake_shell.py 
b/repl/src/main/resources/fake_shell.py
index 56a0ac4..ee86978 100644
--- a/repl/src/main/resources/fake_shell.py
+++ b/repl/src/main/resources/fake_shell.py
@@ -588,14 +588,17 @@ def main():
 conf = SparkConf(_jvm = gateway.jvm, _jconf = jconf)
 sc = SparkContext(jsc=jsc, gateway=gateway, conf=conf)
 global_dict['sc'] = sc
-sqlc = SQLContext(sc, jsqlc)
-global_dict['sqlContext'] = sqlc
 
 if spark_major_version >= "2":
 from pyspark.sql import SparkSession
 spark_session = SparkSession(sc, 
gateway.entry_point.sparkSession())
+sqlc = SQLContext(sc, spark_session, jsqlc)
+global_dict['sqlContext'] = sqlc
 global_dict['spark'] = spark_session
 else:
+sqlc = SQLContext(sc, jsqlc)
+global_dict['sqlContext'] = sqlc
+
 # LIVY-294, need to check whether HiveContext can work 
properly,
 # fallback to SQLContext if HiveContext can not be initialized 
successfully.
 # Only for 

incubator-livy git commit: [LIVY-455][REPL] Fix json4s doesn't support java.math.BigDecimal issue

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.5 4783fbc25 -> 88d66a95f


[LIVY-455][REPL] Fix json4s doesn't support java.math.BigDecimal issue

## What changes were proposed in this pull request?

Livy's SQLInterpreter will throw exception when rows contain 
java.math.BigDecimal data. This is because current version of json4s doesn't 
treat java.math.BigDecimal type as primitive type. On the contrary, json4s 
supports Scala BigDecimal as primitive type. So the fix is to convert java 
BigDecimal to Scala BigDecimal.

## How was this patch tested?

Unit test is added.

Author: jerryshao 

Closes #85 from jerryshao/LIVY-455.

(cherry picked from commit 7e4bb3bd6888742e43aed7396ed490d343a063a2)
Signed-off-by: jerryshao 


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/88d66a95
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/88d66a95
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/88d66a95

Branch: refs/heads/branch-0.5
Commit: 88d66a95fb6737a1e3bdeed34138e16f64fc23af
Parents: 4783fbc
Author: jerryshao 
Authored: Wed Apr 11 10:24:20 2018 +0800
Committer: jerryshao 
Committed: Wed Apr 11 10:24:31 2018 +0800

--
 .../org/apache/livy/repl/SQLInterpreter.scala   | 10 +-
 .../apache/livy/repl/SQLInterpreterSpec.scala   | 36 
 2 files changed, 45 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/88d66a95/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
--
diff --git a/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala 
b/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
index a98223e..afef09b 100644
--- a/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
+++ b/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
@@ -97,7 +97,15 @@ class SQLInterpreter(
   val rows = result.getClass.getMethod("take", classOf[Int])
 .invoke(result, maxResult: java.lang.Integer)
 .asInstanceOf[Array[Row]]
-  .map(_.toSeq)
+.map {
+  _.toSeq.map {
+// Convert java BigDecimal type to Scala BigDecimal, because 
current version of
+// Json4s doesn't support java BigDecimal as a primitive type 
(LIVY-455).
+case i: java.math.BigDecimal => BigDecimal(i)
+case e => e
+  }
+}
+
   val jRows = Extraction.decompose(rows)
 
   Interpreter.ExecuteSuccess(

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/88d66a95/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
--
diff --git a/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala 
b/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
index 58e89ee..d2c12eb 100644
--- a/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
+++ b/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
@@ -95,6 +95,42 @@ class SQLInterpreterSpec extends BaseInterpreterSpec {
 ))
   }
 
+  it should "handle java BigDecimal" in withInterpreter { interpreter =>
+val rdd = sparkEntries.sc().parallelize(Seq(
+  ("1", new java.math.BigDecimal(1.0)),
+  ("2", new java.math.BigDecimal(2.0
+val df = sparkEntries.sqlctx().createDataFrame(rdd).selectExpr("_1 as 
col1", "_2 as col2")
+df.registerTempTable("test")
+
+val resp1 = interpreter.execute(
+  """
+|SELECT * FROM test
+  """.stripMargin)
+
+val expectedResult = (nullable: Boolean) => {
+  Interpreter.ExecuteSuccess(
+APPLICATION_JSON -> (("schema" ->
+  (("type" -> "struct") ~
+("fields" -> List(
+  ("name" -> "col1") ~ ("type" -> "string") ~ ("nullable" -> true) 
~
+("metadata" -> List()),
+  ("name" -> "col2") ~ ("type" -> "decimal(38,18)") ~ ("nullable" 
-> nullable) ~
+("metadata" -> List())
+ ~
+  ("data" -> List(
+List[JValue]("1", 1.0d),
+List[JValue]("2", 2.0d)
+  )))
+  )
+}
+
+val result = Try { resp1 should equal(expectedResult(false))}
+  .orElse(Try { resp1 should equal(expectedResult(true)) })
+if (result.isFailure) {
+  fail(s"$resp1 doesn't equal to expected result")
+}
+  }
+
   it should "throw exception for illegal query" in withInterpreter { 
interpreter =>
 val resp = interpreter.execute(
   """

incubator-livy git commit: [LIVY-455][REPL] Fix json4s doesn't support java.math.BigDecimal issue

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 06cfa7bce -> 7e4bb3bd6


[LIVY-455][REPL] Fix json4s doesn't support java.math.BigDecimal issue

## What changes were proposed in this pull request?

Livy's SQLInterpreter will throw exception when rows contain 
java.math.BigDecimal data. This is because current version of json4s doesn't 
treat java.math.BigDecimal type as primitive type. On the contrary, json4s 
supports Scala BigDecimal as primitive type. So the fix is to convert java 
BigDecimal to Scala BigDecimal.

## How was this patch tested?

Unit test is added.

Author: jerryshao 

Closes #85 from jerryshao/LIVY-455.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/7e4bb3bd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/7e4bb3bd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/7e4bb3bd

Branch: refs/heads/master
Commit: 7e4bb3bd6888742e43aed7396ed490d343a063a2
Parents: 06cfa7b
Author: jerryshao 
Authored: Wed Apr 11 10:24:20 2018 +0800
Committer: jerryshao 
Committed: Wed Apr 11 10:24:20 2018 +0800

--
 .../org/apache/livy/repl/SQLInterpreter.scala   | 10 +-
 .../apache/livy/repl/SQLInterpreterSpec.scala   | 36 
 2 files changed, 45 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/7e4bb3bd/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
--
diff --git a/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala 
b/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
index a98223e..afef09b 100644
--- a/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
+++ b/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala
@@ -97,7 +97,15 @@ class SQLInterpreter(
   val rows = result.getClass.getMethod("take", classOf[Int])
 .invoke(result, maxResult: java.lang.Integer)
 .asInstanceOf[Array[Row]]
-  .map(_.toSeq)
+.map {
+  _.toSeq.map {
+// Convert java BigDecimal type to Scala BigDecimal, because 
current version of
+// Json4s doesn't support java BigDecimal as a primitive type 
(LIVY-455).
+case i: java.math.BigDecimal => BigDecimal(i)
+case e => e
+  }
+}
+
   val jRows = Extraction.decompose(rows)
 
   Interpreter.ExecuteSuccess(

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/7e4bb3bd/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
--
diff --git a/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala 
b/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
index 58e89ee..d2c12eb 100644
--- a/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
+++ b/repl/src/test/scala/org/apache/livy/repl/SQLInterpreterSpec.scala
@@ -95,6 +95,42 @@ class SQLInterpreterSpec extends BaseInterpreterSpec {
 ))
   }
 
+  it should "handle java BigDecimal" in withInterpreter { interpreter =>
+val rdd = sparkEntries.sc().parallelize(Seq(
+  ("1", new java.math.BigDecimal(1.0)),
+  ("2", new java.math.BigDecimal(2.0
+val df = sparkEntries.sqlctx().createDataFrame(rdd).selectExpr("_1 as 
col1", "_2 as col2")
+df.registerTempTable("test")
+
+val resp1 = interpreter.execute(
+  """
+|SELECT * FROM test
+  """.stripMargin)
+
+val expectedResult = (nullable: Boolean) => {
+  Interpreter.ExecuteSuccess(
+APPLICATION_JSON -> (("schema" ->
+  (("type" -> "struct") ~
+("fields" -> List(
+  ("name" -> "col1") ~ ("type" -> "string") ~ ("nullable" -> true) 
~
+("metadata" -> List()),
+  ("name" -> "col2") ~ ("type" -> "decimal(38,18)") ~ ("nullable" 
-> nullable) ~
+("metadata" -> List())
+ ~
+  ("data" -> List(
+List[JValue]("1", 1.0d),
+List[JValue]("2", 2.0d)
+  )))
+  )
+}
+
+val result = Try { resp1 should equal(expectedResult(false))}
+  .orElse(Try { resp1 should equal(expectedResult(true)) })
+if (result.isFailure) {
+  fail(s"$resp1 doesn't equal to expected result")
+}
+  }
+
   it should "throw exception for illegal query" in withInterpreter { 
interpreter =>
 val resp = interpreter.execute(
   """

incubator-livy git commit: [MINOR] Fix travis builds

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.5 1f5841ee7 -> 4783fbc25


[MINOR] Fix travis builds

Each individual commit has a more detailed description of what's being changed 
and why.

At the moment, Travis builds don't work:

* https://travis-ci.org/mineo/incubator-livy/builds/359324523 - the `sudo pip3 
install --upgrade pip "setuptools < 36"` command fails with `sudo: pip3: 
command not found`.
* fixing that, the `failing the org.apache.rat:apache-rat-plugin:0.12:check` 
maven goal fails because it sees `.pytest_cache` folders that it doesn't know 
about (https://github.com/pytest-dev/pytest/issues/3286, failure in 
https://travis-ci.org/mineo/incubator-livy/jobs/359326261, check the raw log).

This pull request works around the pip3 failures by just using pip as a 
callable module (possible since Python 3.4) and adding `.pytest_cache` to 
raw-excludes, as well as showing the contents of `rat.txt` files in the 
`after_failure` step. I concede that the pip change is more of a workaround, 
but I don't know how else to fix it and unblock testing.

Running the tests on travis.

Please review https://livy.incubator.apache.org/community/ before opening a 
pull request.

Author: Wieland Hoffmann 

Closes #84 from mineo/rat.

(cherry picked from commit 06cfa7bce2cde205895be31fa83adc006e7d4ec2)
Signed-off-by: jerryshao 


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/4783fbc2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/4783fbc2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/4783fbc2

Branch: refs/heads/branch-0.5
Commit: 4783fbc2539f45f8e62c54eb17c5b5ff33955fd3
Parents: 1f5841e
Author: Wieland Hoffmann 
Authored: Mon Apr 9 13:47:36 2018 +0800
Committer: jerryshao 
Committed: Mon Apr 9 13:49:48 2018 +0800

--
 .rat-excludes | 1 +
 .travis.yml   | 6 +++---
 2 files changed, 4 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/4783fbc2/.rat-excludes
--
diff --git a/.rat-excludes b/.rat-excludes
index 0829aaa..d0b3a9b 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -23,6 +23,7 @@ logs/*
 **/derby.log
 **/metastore_db/**
 **/.eggs/**
+**/.pytest_cache/**
 **/Gemfile.lock
 **/jquery-2.1.1.min.js
 docs/**/*.html

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/4783fbc2/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index c31114a..02578e8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -58,9 +58,9 @@ before_install:
   - sudo apt-get -y install libkrb5-dev
   - sudo apt-get -y remove python-setuptools
   - sudo pip2 install --upgrade pip "setuptools < 36"
-  - sudo pip3 install --upgrade pip "setuptools < 36"
+  - sudo python3 -m pip install --upgrade pip "setuptools < 36"
   - sudo pip2 install codecov cloudpickle
-  - sudo pip3 install cloudpickle
+  - sudo python3 -m pip install cloudpickle
 
 install:
   - mvn $MVN_FLAG install -Dskip -DskipTests -DskipITs 
-Dmaven.javadoc.skip=true -B -V
@@ -76,4 +76,4 @@ after_success:
   - bash <(curl -s https://codecov.io/bash)
 
 after_failure:
-  - for log in `find * -name "*.log" -o -name "stderr" -o -name "stdout"`; do 
echo "=$log="; cat $log; done
+  - for log in `find * -name "*.log" -o -name "stderr" -o -name "stdout" -o 
-name "rat.txt"`; do echo "=$log="; cat $log; done

incubator-livy git commit: [MINOR] Fix travis builds

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 10373b6e2 -> 06cfa7bce


[MINOR] Fix travis builds

Each individual commit has a more detailed description of what's being changed 
and why.

At the moment, Travis builds don't work:

* https://travis-ci.org/mineo/incubator-livy/builds/359324523 - the `sudo pip3 
install --upgrade pip "setuptools < 36"` command fails with `sudo: pip3: 
command not found`.
* fixing that, the `failing the org.apache.rat:apache-rat-plugin:0.12:check` 
maven goal fails because it sees `.pytest_cache` folders that it doesn't know 
about (https://github.com/pytest-dev/pytest/issues/3286, failure in 
https://travis-ci.org/mineo/incubator-livy/jobs/359326261, check the raw log).

This pull request works around the pip3 failures by just using pip as a 
callable module (possible since Python 3.4) and adding `.pytest_cache` to 
raw-excludes, as well as showing the contents of `rat.txt` files in the 
`after_failure` step. I concede that the pip change is more of a workaround, 
but I don't know how else to fix it and unblock testing.

Running the tests on travis.

Please review https://livy.incubator.apache.org/community/ before opening a 
pull request.

Author: Wieland Hoffmann 

Closes #84 from mineo/rat.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/06cfa7bc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/06cfa7bc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/06cfa7bc

Branch: refs/heads/master
Commit: 06cfa7bce2cde205895be31fa83adc006e7d4ec2
Parents: 10373b6
Author: Wieland Hoffmann 
Authored: Mon Apr 9 13:47:36 2018 +0800
Committer: jerryshao 
Committed: Mon Apr 9 13:48:18 2018 +0800

--
 .rat-excludes | 1 +
 .travis.yml   | 6 +++---
 2 files changed, 4 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/06cfa7bc/.rat-excludes
--
diff --git a/.rat-excludes b/.rat-excludes
index 0829aaa..d0b3a9b 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -23,6 +23,7 @@ logs/*
 **/derby.log
 **/metastore_db/**
 **/.eggs/**
+**/.pytest_cache/**
 **/Gemfile.lock
 **/jquery-2.1.1.min.js
 docs/**/*.html

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/06cfa7bc/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index f1247f1..e9ae42e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -62,9 +62,9 @@ before_install:
   - sudo apt-get -y install libkrb5-dev
   - sudo apt-get -y remove python-setuptools
   - sudo pip2 install --upgrade pip "setuptools < 36"
-  - sudo pip3 install --upgrade pip "setuptools < 36"
+  - sudo python3 -m pip install --upgrade pip "setuptools < 36"
   - sudo pip2 install codecov cloudpickle
-  - sudo pip3 install cloudpickle
+  - sudo python3 -m pip install cloudpickle
 
 install:
   - mvn $MVN_FLAG install -Dskip -DskipTests -DskipITs 
-Dmaven.javadoc.skip=true -B -V
@@ -80,4 +80,4 @@ after_success:
   - bash <(curl -s https://codecov.io/bash)
 
 after_failure:
-  - for log in `find * -name "*.log" -o -name "stderr" -o -name "stdout"`; do 
echo "=$log="; cat $log; done
+  - for log in `find * -name "*.log" -o -name "stderr" -o -name "stdout" -o 
-name "rat.txt"`; do echo "=$log="; cat $log; done

incubator-livy git commit: [LIVY-446][BUILD] Livy to Support Spark 2.3

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master b9c2f1019 -> 10373b6e2


[LIVY-446][BUILD] Livy to Support Spark 2.3

## What changes were proposed in this pull request?

This PR aims to support Spark 2.3 in Livy, several changes due to internal 
changes of Spark 2.3

## How was this patch tested?

Existing UTs.

Author: jerryshao 

Closes #81 from jerryshao/LIVY-446.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/10373b6e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/10373b6e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/10373b6e

Branch: refs/heads/master
Commit: 10373b6e2edc1fa7af6ec363491eb25ff48d730f
Parents: b9c2f10
Author: jerryshao 
Authored: Mon Mar 12 10:37:28 2018 +0800
Committer: jerryshao 
Committed: Mon Mar 12 10:37:28 2018 +0800

--
 .travis.yml |  14 ++-
 .../scala/org/apache/livy/test/JobApiIT.scala   |   2 +-
 pom.xml | 110 +++
 repl/scala-2.10/pom.xml |   2 +
 repl/scala-2.11/pom.xml |   2 +
 .../apache/livy/repl/PythonInterpreter.scala|   8 +-
 .../apache/livy/repl/SparkRInterpreter.scala|   4 +-
 .../org/apache/livy/rsc/TestSparkClient.java|   1 +
 .../livy/rsc/rpc/TestKryoMessageCodec.java  |   2 +-
 .../java/org/apache/livy/rsc/rpc/TestRpc.java   |   4 +-
 scala-api/pom.xml   |  11 ++
 scala-api/scala-2.10/pom.xml|   2 +
 scala-api/scala-2.11/pom.xml|   2 +
 .../apache/livy/scalaapi/ScalaClientTest.scala  |   1 +
 .../org/apache/livy/utils/LivySparkUtils.scala  |   4 +-
 .../interactive/InteractiveSessionSpec.scala|   2 +-
 .../org/apache/livy/test/jobs/SQLGetTweets.java |   2 +-
 17 files changed, 133 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/10373b6e/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index c31114a..f1247f1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -21,20 +21,24 @@ language: scala
 
 env:
   matrix:
-- MVN_FLAG='-Pspark-1.6 -DskipTests'
-- MVN_FLAG='-Pspark-2.0 -DskipTests'
-- MVN_FLAG='-Pspark-2.1 -DskipTests'
+- MVN_FLAG='-DskipTests'
+- MVN_FLAG='-Pspark-2.0-it -DskipTests'
+- MVN_FLAG='-Pspark-2.1-it -DskipTests'
 - MVN_FLAG='-Pspark-1.6 -DskipITs'
 - MVN_FLAG='-Pspark-2.0 -DskipITs'
 - MVN_FLAG='-Pspark-2.1 -DskipITs'
 
 matrix:
   include:
-  # Spark 2.2 will only be verified using JDK8
-- env: MVN_FLAG='-Pspark-2.2 -DskipTests'
+  # Spark 2.2+ will only be verified using JDK8
+- env: MVN_FLAG='-Pspark-2.2-it -DskipTests'
   jdk: oraclejdk8
 - env: MVN_FLAG='-Pspark-2.2 -DskipITs'
   jdk: oraclejdk8
+- env: MVN_FLAG='-Pspark-2.3-it -DskipTests'
+  jdk: oraclejdk8
+- env: MVN_FLAG='-Pspark-2.3 -DskipITs'
+  jdk: oraclejdk8
 
 
 jdk:

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/10373b6e/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala
--
diff --git 
a/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala 
b/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala
index b10e8f6..7c0e560 100644
--- a/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala
+++ b/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala
@@ -133,7 +133,7 @@ class JobApiIT extends BaseIntegrationTestSuite with 
BeforeAndAfterAll with Logg
 assert(result === 100)
   }
 
-  test("run spark sql job") {
+  ignore("run spark sql job") {
 assume(client != null, "Client not active.")
 val result = waitFor(client.submit(new SQLGetTweets(false)))
 assert(result.size() > 0)

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/10373b6e/pom.xml
--
diff --git a/pom.xml b/pom.xml
index a8d41ed..e026a91 100644
--- a/pom.xml
+++ b/pom.xml
@@ -80,7 +80,9 @@
   
 2.7.3
 compile
-1.6.2
+1.6.2
+1.6.2
+${spark.scala-2.11.version}
 1.9
 15.0
 4.5.3
@@ -93,7 +95,9 @@
 2.22
 3.1.0
 1.9.5
-4.0.37.Final
+4.0.37.Final
+4.0.37.Final
+${netty.spark-2.11.version}
 UTF-8
 0.9
 2.10.4
@@ -184,25 +188,6 @@
   
 
 
-  cdh.repo
-  https://repository.cloudera.com/artifactory/cloudera-repos
-  Cloudera Repositories
-  
-false
-  
-
-
-  cdh.snapshots.repo
-  

incubator-livy git commit: [LIVY-448] Assembly format is duplicated in pom.xml and assembly.xml

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 6d2ffdd54 -> b9c2f1019


[LIVY-448] Assembly format is duplicated in pom.xml and assembly.xml

## What changes were proposed in this pull request?
The assembly format for Livy is specified twice:
1 in `assembly/pom.xml`:
```xml
  
${assembly.format}
  
```
2 in `assembly/assembly.xml`
```
  
zip
  
```
But only one is enough. Also in `assembly/assembly.xml`, the format is hard 
coded as zip, but is should be based

## How was this patch tested?
All the unit tests and ITs pass as usual. Moreover, I tested that  `targz` 
profile builds a tar.gz archive and the default profile builds a zip archive.

For example,
this builds the zip archive:
```
mvn -am -pl assembly package
```
And this builds the tar.gz artifact:
```
mvn -Ptargz -am -pl assembly package
```

Task-url: https://issues.apache.org/jira/browse/LIVY-448

Author: Meisam 

Closes #83 from meisam/LIVY-448.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/b9c2f101
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/b9c2f101
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/b9c2f101

Branch: refs/heads/master
Commit: b9c2f10194f32532009690fe89a11959c4c083c0
Parents: 6d2ffdd
Author: Meisam 
Authored: Thu Mar 8 08:56:01 2018 +0800
Committer: jerryshao 
Committed: Thu Mar 8 08:56:01 2018 +0800

--
 assembly/assembly.xml | 2 +-
 assembly/pom.xml  | 3 ---
 2 files changed, 1 insertion(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/b9c2f101/assembly/assembly.xml
--
diff --git a/assembly/assembly.xml b/assembly/assembly.xml
index b9d9416..e1dcdff 100644
--- a/assembly/assembly.xml
+++ b/assembly/assembly.xml
@@ -19,7 +19,7 @@
   
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0
 http://maven.apache.org/xsd/assembly-1.1.0.xsd;>
   livy-server
   
-zip
+${assembly.format}
   
   false
   

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/b9c2f101/assembly/pom.xml
--
diff --git a/assembly/pom.xml b/assembly/pom.xml
index c16a184..1aeb908 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -77,9 +77,6 @@
   
 assembly.xml
   
-  
-${assembly.format}
-  
 
   
 

incubator-livy git commit: [LIVY-429] preserve url fragment in spark.yarn.dist.archives

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master e54318314 -> 85e8fba3c


[LIVY-429] preserve url fragment in spark.yarn.dist.archives

## What changes were proposed in this pull request?

Preserve url fragment identifier in spark.yarn.dist.archives setting when there 
is no schema specified. Current code truncates url fragment, as a result Spark 
just copies archives to job local dir without extracting it.

## How was this patch tested?

Added case to existing unit test

Author: Artem Plotnikov 

Closes #71 from broartem/LIVY-429.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/85e8fba3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/85e8fba3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/85e8fba3

Branch: refs/heads/master
Commit: 85e8fba3ce5d0e180655fd9ee239cb320d550780
Parents: e543183
Author: Artem Plotnikov 
Authored: Thu Jan 4 10:22:40 2018 +0800
Committer: jerryshao 
Committed: Thu Jan 4 10:22:40 2018 +0800

--
 server/src/main/scala/org/apache/livy/sessions/Session.scala  | 7 +--
 .../src/test/scala/org/apache/livy/sessions/SessionSpec.scala | 4 ++--
 2 files changed, 7 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/85e8fba3/server/src/main/scala/org/apache/livy/sessions/Session.scala
--
diff --git a/server/src/main/scala/org/apache/livy/sessions/Session.scala 
b/server/src/main/scala/org/apache/livy/sessions/Session.scala
index d467076..0b54779 100644
--- a/server/src/main/scala/org/apache/livy/sessions/Session.scala
+++ b/server/src/main/scala/org/apache/livy/sessions/Session.scala
@@ -116,8 +116,11 @@ object Session {
 val defaultFS = livyConf.hadoopConf.get("fs.defaultFS").stripSuffix("/")
 val resolved =
   if (uri.getScheme() == null) {
-require(uri.getPath().startsWith("/"), s"Path '${uri.getPath()}' is 
not absolute.")
-new URI(defaultFS + uri.getPath())
+val pathWithSegment =
+  if (uri.getFragment() != null) uri.getPath() + '#' + 
uri.getFragment() else uri.getPath()
+
+require(pathWithSegment.startsWith("/"), s"Path '${uri.getPath()}' is 
not absolute.")
+new URI(defaultFS + pathWithSegment)
   } else {
 uri
   }

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/85e8fba3/server/src/test/scala/org/apache/livy/sessions/SessionSpec.scala
--
diff --git a/server/src/test/scala/org/apache/livy/sessions/SessionSpec.scala 
b/server/src/test/scala/org/apache/livy/sessions/SessionSpec.scala
index a7bfaaa..05b41bb 100644
--- a/server/src/test/scala/org/apache/livy/sessions/SessionSpec.scala
+++ b/server/src/test/scala/org/apache/livy/sessions/SessionSpec.scala
@@ -29,8 +29,8 @@ class SessionSpec extends FunSuite with LivyBaseUnitTestSuite 
{
 val conf = new LivyConf(false)
 conf.hadoopConf.set("fs.defaultFS", "dummy:///")
 
-val uris = Seq("http://example.com/foo;, "hdfs:/bar", "/baz")
-val expected = Seq(uris(0), uris(1), "dummy:///baz")
+val uris = Seq("http://example.com/foo;, "hdfs:/bar", "/baz", "/foo#bar")
+val expected = Seq(uris(0), uris(1), "dummy:///baz", "dummy:///foo#bar")
 assert(Session.resolveURIs(uris, conf) === expected)
 
 intercept[IllegalArgumentException] {

incubator-livy git commit: [LIVY-19][REPL] Add SQL interpreter for Livy interactive session

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 1efc80180 -> e54318314


[LIVY-19][REPL] Add SQL interpreter for Livy interactive session

## What changes were proposed in this pull request?

With the requirement to support executing SQL query in Livy interactive 
session. Here propose to add this new interpreter for SQL specific usage.

This interpreter will honor a new code kind "SQL", and pass query to 
SparkSession/SQLContext to get executed. The returned format is json format 
with schema included, each type is properly mapped a related json type.

## How was this patch tested?

Add new UTs and ITs.

Author: jerryshao 

Closes #68 from jerryshao/LIVY-19.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/e5431831
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/e5431831
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/e5431831

Branch: refs/heads/master
Commit: e54318314c59580d40eb6944ed780d8762e97cf6
Parents: 1efc801
Author: jerryshao 
Authored: Tue Dec 5 09:29:57 2017 +0800
Committer: jerryshao 
Committed: Tue Dec 5 09:29:57 2017 +0800

--
 .../main/scala/org/apache/livy/Logging.scala|   4 +
 .../scala/org/apache/livy/sessions/Kind.scala   |   3 +
 docs/rest-api.md|  15 ++-
 .../livy/test/framework/LivyRestClient.scala|  15 ++-
 .../org/apache/livy/test/InteractiveIT.scala|  13 ++
 pom.xml |   3 +
 .../org/apache/livy/repl/SQLInterpreter.scala   | 118 ++
 .../scala/org/apache/livy/repl/Session.scala|  20 ++--
 .../apache/livy/repl/SQLInterpreterSpec.scala   | 120 +++
 .../main/java/org/apache/livy/rsc/RSCConf.java  |   7 +-
 10 files changed, 299 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/e5431831/core/src/main/scala/org/apache/livy/Logging.scala
--
diff --git a/core/src/main/scala/org/apache/livy/Logging.scala 
b/core/src/main/scala/org/apache/livy/Logging.scala
index 73ff7df..29572ff 100644
--- a/core/src/main/scala/org/apache/livy/Logging.scala
+++ b/core/src/main/scala/org/apache/livy/Logging.scala
@@ -44,6 +44,10 @@ trait Logging {
 logger.warn(message.toString)
   }
 
+  def warn(message: => Any, t: Throwable): Unit = {
+logger.warn(message.toString, t)
+  }
+
   def error(message: => Any, t: Throwable): Unit = {
 logger.error(message.toString, t)
   }

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/e5431831/core/src/main/scala/org/apache/livy/sessions/Kind.scala
--
diff --git a/core/src/main/scala/org/apache/livy/sessions/Kind.scala 
b/core/src/main/scala/org/apache/livy/sessions/Kind.scala
index 4779822..0a05c8f 100644
--- a/core/src/main/scala/org/apache/livy/sessions/Kind.scala
+++ b/core/src/main/scala/org/apache/livy/sessions/Kind.scala
@@ -33,6 +33,8 @@ object SparkR extends Kind("sparkr")
 
 object Shared extends Kind("shared")
 
+object SQL extends Kind("sql")
+
 object Kind {
 
   def apply(kind: String): Kind = kind match {
@@ -40,6 +42,7 @@ object Kind {
 case "pyspark" | "python" => PySpark
 case "sparkr" | "r" => SparkR
 case "shared" => Shared
+case "sql" => SQL
 case other => throw new IllegalArgumentException(s"Invalid kind: $other")
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/e5431831/docs/rest-api.md
--
diff --git a/docs/rest-api.md b/docs/rest-api.md
index 6248b2a..8c928a8 100644
--- a/docs/rest-api.md
+++ b/docs/rest-api.md
@@ -665,18 +665,23 @@ A session represents an interactive shell.
 sparkr
 Interactive R Spark session
   
-
+  
+sql
+Interactive SQL Spark session
+  
+  
 
-Starting with version 0.5.0-incubating, each session can support all three 
Scala, Python and R
-interpreters. The ``kind`` field in session creation is no longer required, 
instead users should
-specify code kind (spark, pyspark or sparkr) during statement submission.
+Starting with version 0.5.0-incubating, each session can support all four 
Scala, Python and R
+interpreters with newly added SQL interpreter. The ``kind`` field in session 
creation 
+is no longer required, instead users should specify code kind (spark, pyspark, 
sparkr or sql) 
+during statement submission.
 
 To be compatible with previous versions, users can still specify ``kind`` in 
session creation,
 while ignoring ``kind`` in statement submission. Livy will then use this 
session
 ``kind`` as default kind for all the submitted 

incubator-livy git commit: [LIVY-415][CORE][FOLLOWUP] Fix static object creation issue for SessionState

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master cd777e209 -> 1efc80180


[LIVY-415][CORE][FOLLOWUP] Fix static object creation issue for SessionState

## What changes were proposed in this pull request?

In LIVY-415, we changed `SessionState` to static object instead of case class. 
This is OK for `SessionState`, but for `FinishedSessionState`, it requires 
`time` field to be set during object creation. Using static object will never 
reflect the actual `time` when object is created. So here propose to fix it.

CC aa8y please take a review, this is introduced in your changes.

## How was this patch tested?

Existing tests.

Author: jerryshao 

Closes #67 from jerryshao/LIVY-415-followup.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/1efc8018
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/1efc8018
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/1efc8018

Branch: refs/heads/master
Commit: 1efc801806619d7741f453eff07afac33e841d7a
Parents: cd777e2
Author: jerryshao 
Authored: Wed Nov 29 12:49:39 2017 +0800
Committer: jerryshao 
Committed: Wed Nov 29 12:49:39 2017 +0800

--
 .../org/apache/livy/sessions/SessionState.scala  | 15 ++-
 .../apache/livy/test/framework/LivyRestClient.scala  |  6 +++---
 .../main/scala/org/apache/livy/repl/Session.scala|  4 ++--
 .../org/apache/livy/server/batch/BatchSession.scala  |  4 ++--
 .../livy/server/interactive/InteractiveSession.scala |  8 
 .../apache/livy/sessions/SessionManagerSpec.scala|  6 +++---
 6 files changed, 20 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/1efc8018/core/src/main/scala/org/apache/livy/sessions/SessionState.scala
--
diff --git a/core/src/main/scala/org/apache/livy/sessions/SessionState.scala 
b/core/src/main/scala/org/apache/livy/sessions/SessionState.scala
index 577a27a..e56cfbb 100644
--- a/core/src/main/scala/org/apache/livy/sessions/SessionState.scala
+++ b/core/src/main/scala/org/apache/livy/sessions/SessionState.scala
@@ -37,9 +37,9 @@ object SessionState {
 case "running" => Running
 case "busy" => Busy
 case "shutting_down" => ShuttingDown
-case "error" => Error
-case "dead" => Dead
-case "success" => Success
+case "error" => Error()
+case "dead" => Dead()
+case "success" => Success()
 case _ => throw new IllegalArgumentException(s"Illegal session state: $s")
   }
 
@@ -57,15 +57,12 @@ object SessionState {
 
   object ShuttingDown extends SessionState("shutting_down", false)
 
-  case class Error(override val time: Long) extends
+  case class Error(override val time: Long = System.nanoTime()) extends
 FinishedSessionState("error", true, time)
-  object Error extends Error(System.nanoTime)
 
-  case class Dead(override val time: Long) extends
+  case class Dead(override val time: Long = System.nanoTime()) extends
 FinishedSessionState("dead", false, time)
-  object Dead extends Dead(System.nanoTime)
 
-  case class Success(override val time: Long) extends
+  case class Success(override val time: Long = System.nanoTime()) extends
 FinishedSessionState("success", false, time)
-  object Success extends Success(System.nanoTime)
 }

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/1efc8018/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
--
diff --git 
a/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
 
b/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
index 941a650..f1f 100644
--- 
a/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
+++ 
b/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
@@ -111,9 +111,9 @@ class LivyRestClient(val httpClient: AsyncHttpClient, val 
livyEndpoint: String)
   }
 
   class BatchSession(id: Int) extends Session(id, BATCH_TYPE) {
-def verifySessionDead(): Unit = verifySessionState(SessionState.Dead)
+def verifySessionDead(): Unit = verifySessionState(SessionState.Dead())
 def verifySessionRunning(): Unit = verifySessionState(SessionState.Running)
-def verifySessionSuccess(): Unit = verifySessionState(SessionState.Success)
+def verifySessionSuccess(): Unit = 
verifySessionState(SessionState.Success())
   }
 
   class InteractiveSession(id: Int) extends Session(id, INTERACTIVE_TYPE) {
@@ -226,7 +226,7 @@ class LivyRestClient(val httpClient: AsyncHttpClient, val 
livyEndpoint: String)
 

incubator-livy git commit: [LIVY-399][TEST] Enable real test for PySpark and SparkR interpreters

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master f893d1991 -> cd777e209


[LIVY-399][TEST] Enable real test for PySpark and SparkR interpreters

Currently because we lack pyspark and sparkr dependencies in our environment, 
so we neglect the pyspark and sparkr integration tests, here propose a way to 
enable these integration tests. The solution is mainly borrowed from Zeppelin - 
to download Spark binary before integration tests, so that we will have a 
complete Spark environment to do the test. Because of this several modules like 
minicluster-dependencies are not required, so here removed such modules.

Besides, we proposed to remove real test implementations, since we never used 
real test before and it mainly did what scripts should do with Scala code, 
which makes it hard to maintain. Also for now mini-cluster test is quite close 
to real test, so it is not so useful to still keep real test.

Author: jerryshao 

Closes #49 from jerryshao/LIVY-399.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/cd777e20
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/cd777e20
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/cd777e20

Branch: refs/heads/master
Commit: cd777e209ae26d9f66814fed6565b029da3b2479
Parents: f893d19
Author: jerryshao 
Authored: Mon Nov 27 16:25:52 2017 +0800
Committer: jerryshao 
Committed: Mon Nov 27 16:25:52 2017 +0800

--
 .travis.yml |  15 +-
 dev/travis-upload-logs.sh   |  39 ---
 .../minicluster-dependencies/pom.xml| 135 -
 .../minicluster-dependencies/scala-2.10/pom.xml |  36 ---
 .../minicluster-dependencies/scala-2.11/pom.xml |  36 ---
 .../src/main/resources/build.marker |   0
 integration-test/pom.xml|  97 ++-
 .../framework/BaseIntegrationTestSuite.scala|  18 --
 .../apache/livy/test/framework/Cluster.scala|   3 -
 .../livy/test/framework/MiniCluster.scala   |  41 +--
 .../livy/test/framework/RealCluster.scala   | 277 ---
 .../src/test/resources/test_python_api.py   |   8 +-
 .../scala/org/apache/livy/test/BatchIT.scala|   6 +-
 .../org/apache/livy/test/InteractiveIT.scala|   7 +-
 .../scala/org/apache/livy/test/JobApiIT.scala   |  13 +-
 pom.xml |  28 +-
 .../apache/livy/repl/PythonInterpreter.scala|   2 +-
 17 files changed, 76 insertions(+), 685 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/cd777e20/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 8de73e3..c31114a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -57,17 +57,17 @@ before_install:
   - sudo apt-get -y install python3-pip python-dev
   - sudo apt-get -y install libkrb5-dev
   - sudo apt-get -y remove python-setuptools
-  - pip2 install --user --upgrade pip "setuptools < 36"
-  - pip3 install --user --upgrade pip "setuptools < 36"
-  - pip2 install --user codecov cloudpickle
-  - pip3 install --user cloudpickle
+  - sudo pip2 install --upgrade pip "setuptools < 36"
+  - sudo pip3 install --upgrade pip "setuptools < 36"
+  - sudo pip2 install codecov cloudpickle
+  - sudo pip3 install cloudpickle
 
 install:
   - mvn $MVN_FLAG install -Dskip -DskipTests -DskipITs 
-Dmaven.javadoc.skip=true -B -V
 
 before_script:
-  - pip2 install --user "requests >= 2.10.0" pytest flaky flake8 
requests-kerberos
-  - pip3 install --user "requests >= 2.10.0" pytest flaky requests-kerberos
+  - sudo pip2 install "requests >= 2.10.0" pytest flaky flake8 
requests-kerberos
+  - sudo pip3 install "requests >= 2.10.0" pytest flaky requests-kerberos
 
 script:
   - mvn $MVN_FLAG verify -e
@@ -76,5 +76,4 @@ after_success:
   - bash <(curl -s https://codecov.io/bash)
 
 after_failure:
-  - npm list -g azure-cli --depth=0 || npm install -g azure-cli
-  - ./dev/travis-upload-logs.sh
+  - for log in `find * -name "*.log" -o -name "stderr" -o -name "stdout"`; do 
echo "=$log="; cat $log; done

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/cd777e20/dev/travis-upload-logs.sh
--
diff --git a/dev/travis-upload-logs.sh b/dev/travis-upload-logs.sh
deleted file mode 100755
index 97333d1..000
--- a/dev/travis-upload-logs.sh
+++ /dev/null
@@ -1,39 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file 

incubator-livy git commit: [LIVY-416] Upgrade the version of Jackson from 2.4.4 to 2.9.2

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 64d71aa75 -> 1f59e102b


[LIVY-416] Upgrade the version of Jackson from 2.4.4 to 2.9.2

## What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/LIVY-416

`com.fasterxml.jackson.core.JsonGenerationException` is sometimes thrown. The 
full stack trace is show on 
[JsonGenerationException.txt](https://github.com/apache/incubator-livy/files/1482566/JsonGenerationException.txt).

This is because of the Jackson's bug 
(https://github.com/FasterXML/jackson-core/issues/307) which was fixed at 
Jackson 2.7.7.

To fix this issue, the version of Jackson should be updated from 2.4.4 to the 
latest one (2.9.2).

In addition, `com.google.guava:guava:15.0` was added to the dependencies 
because it was removed from the dependencies of 
`com.fasterxml.jackson.module:jackson-module-scala_{2.10,2.11}:2.9.2`.

* 
[com.fasterxml.jackson.module:jackson-module-scala_2.10:2.4.4](https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-scala_2.10/2.4.4)
* 
[com.fasterxml.jackson.module:jackson-module-scala_2.11:2.4.4](https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-scala_2.11/2.4.4)
* 
[com.fasterxml.jackson.module:jackson-module-scala_2.10:2.9.2](https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-scala_2.10/2.9.2)
* 
[com.fasterxml.jackson.module:jackson-module-scala_2.11:2.9.2](https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-scala_2.11/2.9.2)

## How was this patch tested?
By executing `mvn clean package`.

Author: Keiji Yoshida 

Closes #64 from kjmrknsn/LIVY-416.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/1f59e102
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/1f59e102
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/1f59e102

Branch: refs/heads/master
Commit: 1f59e102b65c1c18775fe9d286d22b9228497807
Parents: 64d71aa
Author: Keiji Yoshida 
Authored: Thu Nov 23 13:19:53 2017 +0800
Committer: jerryshao 
Committed: Thu Nov 23 13:19:53 2017 +0800

--
 pom.xml| 9 -
 server/pom.xml | 5 +
 2 files changed, 13 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/1f59e102/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 44b33fe..5fb2a7a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -82,9 +82,10 @@
 compile
 1.6.2
 1.9
+15.0
 4.5.3
 4.4.4
-2.4.4
+2.9.2
 3.1.0
 9.2.16.v20160414
 3.2.10
@@ -310,6 +311,12 @@
   
 
   
+com.google.guava
+guava
+${guava.version}
+  
+
+  
 io.dropwizard.metrics
 metrics-core
 ${metrics.version}

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/1f59e102/server/pom.xml
--
diff --git a/server/pom.xml b/server/pom.xml
index 973bcce..256bb45 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -75,6 +75,11 @@
 
 
 
+  com.google.guava
+  guava
+
+
+
   io.dropwizard.metrics
   metrics-core
 

incubator-livy git commit: [LIVY-418] Respect spark.pyspark.python for PythonInterpreter

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 54715447e -> 64d71aa75


[LIVY-418] Respect spark.pyspark.python for PythonInterpreter

## What changes were proposed in this pull request?

`spark.pyspark.python` is introduced in spark 2.1, livy also need to update its 
code for choosing python version

## How was this patch tested?

Manually verified.

Author: Jeff Zhang 

Closes #65 from zjffdu/LIVY-418.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/64d71aa7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/64d71aa7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/64d71aa7

Branch: refs/heads/master
Commit: 64d71aa757a93518427f315e908dcf54373948b2
Parents: 5471544
Author: Jeff Zhang 
Authored: Thu Nov 23 10:14:15 2017 +0800
Committer: jerryshao 
Committed: Thu Nov 23 10:14:15 2017 +0800

--
 repl/src/main/scala/org/apache/livy/repl/PythonInterpreter.scala | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/64d71aa7/repl/src/main/scala/org/apache/livy/repl/PythonInterpreter.scala
--
diff --git a/repl/src/main/scala/org/apache/livy/repl/PythonInterpreter.scala 
b/repl/src/main/scala/org/apache/livy/repl/PythonInterpreter.scala
index fa9be2d..f4f5e41 100644
--- a/repl/src/main/scala/org/apache/livy/repl/PythonInterpreter.scala
+++ b/repl/src/main/scala/org/apache/livy/repl/PythonInterpreter.scala
@@ -44,7 +44,8 @@ import org.apache.livy.sessions._
 object PythonInterpreter extends Logging {
 
   def apply(conf: SparkConf, sparkEntries: SparkEntries): Interpreter = {
-val pythonExec = sys.env.get("PYSPARK_PYTHON")
+val pythonExec = conf.getOption("spark.pyspark.python")
+  .orElse(sys.env.get("PYSPARK_PYTHON"))
   .orElse(sys.props.get("pyspark.python")) // This java property is only 
used for internal UT.
   .getOrElse("python")
 

incubator-livy git commit: [MINOR][HOTFIX] Fix python code style issue

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.4 939c97673 -> 7556d5f6a


[MINOR][HOTFIX] Fix python code style issue

## What changes were proposed in this pull request?

Travis complains "E722 do not use bare except" in python-client style check. 
Minor changes to fix it.

Author: jerryshao 

Closes #60 from jerryshao/python-style-fix.

(cherry picked from commit 20cbdd57e48a37bdff3e7a383d6e94108105a274)
Signed-off-by: jerryshao 


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/7556d5f6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/7556d5f6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/7556d5f6

Branch: refs/heads/branch-0.4
Commit: 7556d5f6ad95a58a1aa47ae26ad3c61fc70e68a8
Parents: 939c976
Author: jerryshao 
Authored: Thu Oct 26 11:41:34 2017 +0800
Committer: jerryshao 
Committed: Thu Oct 26 11:41:51 2017 +0800

--
 python-api/src/main/python/livy/client.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/7556d5f6/python-api/src/main/python/livy/client.py
--
diff --git a/python-api/src/main/python/livy/client.py 
b/python-api/src/main/python/livy/client.py
index 4a5a985..e19d78a 100644
--- a/python-api/src/main/python/livy/client.py
+++ b/python-api/src/main/python/livy/client.py
@@ -334,7 +334,7 @@ class HttpClient(object):
 headers = {'X-Requested-By': 'livy'}
 self._conn.send_request("DELETE", session_uri,
 headers=headers)
-except:
+except Exception:
 raise Exception(traceback.format_exc())
 self._stopped = True
 

incubator-livy git commit: [LIVY-245][RSC] Add support shared variables across Jobs

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 1bbefe601 -> bef5b4f6d


[LIVY-245][RSC] Add support shared variables across Jobs

Currently we cannot share variables across different Jobs in Livy, so here 
propose to add a cache layer in RSC to store shared objects. This cache 
followed LRU, the least not used will be removed when exceeding limits.

This work is based on alex-the-man 's work.

Author: jerryshao 

Closes #19 from jerryshao/LIVY-245.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/bef5b4f6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/bef5b4f6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/bef5b4f6

Branch: refs/heads/master
Commit: bef5b4f6d8ac931613027cf50cbc44be3be2e107
Parents: 1bbefe6
Author: jerryshao 
Authored: Fri Oct 20 14:58:22 2017 +0800
Committer: jerryshao 
Committed: Fri Oct 20 14:58:22 2017 +0800

--
 .../main/java/org/apache/livy/JobContext.java   | 13 +++-
 .../scala/org/apache/livy/test/JobApiIT.scala   | 11 
 repl/src/main/resources/fake_shell.py   | 28 +
 .../scala/org/apache/livy/repl/Session.scala|  2 +-
 .../org/apache/livy/repl/SessionSpec.scala  |  2 +-
 .../main/java/org/apache/livy/rsc/RSCConf.java  |  7 ++-
 .../apache/livy/rsc/driver/JobContextImpl.java  | 51 
 .../org/apache/livy/rsc/TestSparkClient.java| 64 +---
 .../apache/livy/scalaapi/ScalaJobContext.scala  |  9 +++
 .../livy/test/jobs/SharedVariableCounter.java   | 48 +++
 .../test/jobs/ScalaSharedVariableCounter.scala  | 38 
 11 files changed, 258 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/bef5b4f6/api/src/main/java/org/apache/livy/JobContext.java
--
diff --git a/api/src/main/java/org/apache/livy/JobContext.java 
b/api/src/main/java/org/apache/livy/JobContext.java
index 47027b7..84663b1 100644
--- a/api/src/main/java/org/apache/livy/JobContext.java
+++ b/api/src/main/java/org/apache/livy/JobContext.java
@@ -18,9 +18,7 @@
 package org.apache.livy;
 
 import java.io.File;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.NoSuchElementException;
 
 import org.apache.spark.api.java.JavaSparkContext;
 import org.apache.spark.sql.SQLContext;
@@ -47,6 +45,15 @@ public interface JobContext {
   /** Returns the JavaStreamingContext which has already been created. */
   JavaStreamingContext streamingctx();
 
+  /** Get shared object */
+   E getSharedObject(String name) throws NoSuchElementException;
+
+  /** Set shared object, it will replace the old one if already existed */
+   void setSharedObject(String name, E object);
+
+  /** Remove shared object from cache */
+   E removeSharedObject(String name);
+
   /**
* Creates the SparkStreaming context.
*

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/bef5b4f6/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala
--
diff --git 
a/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala 
b/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala
index 880ad42..8fe9dc8 100644
--- a/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala
+++ b/integration-test/src/test/scala/org/apache/livy/test/JobApiIT.scala
@@ -157,6 +157,12 @@ class JobApiIT extends BaseIntegrationTestSuite with 
BeforeAndAfterAll with Logg
 assert(result === "hello")
   }
 
+  test("share variables across jobs") {
+assume(client2 != null, "Client not active.")
+waitFor(client2.submit(new SharedVariableCounter("x"))) shouldBe 0
+waitFor(client2.submit(new SharedVariableCounter("x"))) shouldBe 1
+  }
+
   scalaTest("run scala jobs") {
 assume(client2 != null, "Client not active.")
 
@@ -173,6 +179,11 @@ class JobApiIT extends BaseIntegrationTestSuite with 
BeforeAndAfterAll with Logg
   val result = waitFor(client2.submit(job))
   assert(result === job.value)
 }
+
+(0 until 2).foreach { i =>
+  val result = waitFor(client2.submit(new 
ScalaSharedVariableCounter("test")))
+  assert(i === result)
+}
   }
 
   protected def scalaTest(desc: String)(testFn: => Unit): Unit = {

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/bef5b4f6/repl/src/main/resources/fake_shell.py
--
diff --git a/repl/src/main/resources/fake_shell.py 
b/repl/src/main/resources/fake_shell.py
index 523f7a0..56a0ac4 100644
--- a/repl/src/main/resources/fake_shell.py
+++ 

incubator-livy git commit: [LIVY-7] added autocompletion api and implementation for scala

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master ec069e32f -> 8056e4682


[LIVY-7] added autocompletion api and implementation for scala

I started an implementation of the very old feature request (LIVY-7) for code 
autocompletion.
This implementation works with scala 2.11 and scala 2.10.

I'd be happy if somebody could review and comment it.

As for the API: I chose a synchronous call because resolving the code options 
shouldn't be a very long process (and if it were it wouldn't make sense anyway).

Author: Pascal Pellmont 

Closes #51 from pellmont/master.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/8056e468
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/8056e468
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/8056e468

Branch: refs/heads/master
Commit: 8056e4682531a5a4b717a0b13d992a46637fd412
Parents: ec069e3
Author: Pascal Pellmont 
Authored: Tue Oct 10 08:39:00 2017 +0800
Committer: jerryshao 
Committed: Tue Oct 10 08:39:00 2017 +0800

--
 core/src/main/scala/org/apache/livy/msgs.scala  |  4 +++
 docs/rest-api.md| 36 
 .../livy/test/framework/LivyRestClient.scala| 30 
 .../org/apache/livy/test/InteractiveIT.scala|  3 ++
 .../org/apache/livy/repl/SparkInterpreter.scala |  6 
 .../org/apache/livy/repl/SparkInterpreter.scala | 16 +
 .../livy/repl/AbstractSparkInterpreter.scala|  6 
 .../org/apache/livy/repl/Interpreter.scala  |  4 +++
 .../scala/org/apache/livy/repl/ReplDriver.scala |  4 +++
 .../scala/org/apache/livy/repl/Session.scala|  6 
 .../apache/livy/repl/ScalaInterpreterSpec.scala | 10 ++
 .../java/org/apache/livy/rsc/BaseProtocol.java  | 16 +
 .../java/org/apache/livy/rsc/RSCClient.java |  6 
 .../server/interactive/InteractiveSession.scala |  9 +
 .../interactive/InteractiveSessionServlet.scala |  9 -
 15 files changed, 164 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/8056e468/core/src/main/scala/org/apache/livy/msgs.scala
--
diff --git a/core/src/main/scala/org/apache/livy/msgs.scala 
b/core/src/main/scala/org/apache/livy/msgs.scala
index 048aa7f..bb89a29 100644
--- a/core/src/main/scala/org/apache/livy/msgs.scala
+++ b/core/src/main/scala/org/apache/livy/msgs.scala
@@ -60,4 +60,8 @@ case class ExecuteReplyError(execution_count: Int,
 
 case class ExecuteResponse(id: Int, input: Seq[String], output: Seq[String])
 
+case class CompletionRequest(code: String, kind: String, cursor: Int) extends 
Content
+
+case class CompletionResponse(candidates: List[String]) extends Content
+
 case class ShutdownRequest() extends Content

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/8056e468/docs/rest-api.md
--
diff --git a/docs/rest-api.md b/docs/rest-api.md
index 3bd8eab..f76ce09 100644
--- a/docs/rest-api.md
+++ b/docs/rest-api.md
@@ -310,6 +310,42 @@ Cancel the specified statement in this session.
   
 
 
+### POST /sessions/{sessionId}/completion
+
+Runs a statement in a session.
+
+ Request Body
+
+
+  NameDescriptionType
+  
+code
+The code for which completion proposals are requested
+string
+  
+  
+kind
+The kind of code to execute[2]
+code kind
+  
+  
+cursor
+cursor position to get proposals
+string
+  
+
+
+ Response Body
+
+
+  NameDescriptionType
+  
+candidates
+Code completions proposals
+array[string]
+  
+
+
 ### GET /batches
 
 Returns all the active batch sessions.

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/8056e468/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
--
diff --git 
a/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
 
b/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
index 6d319c7..eaa023a 100644
--- 
a/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
+++ 
b/integration-test/src/main/scala/org/apache/livy/test/framework/LivyRestClient.scala
@@ -46,6 +46,8 @@ object LivyRestClient {
   @JsonIgnoreProperties(ignoreUnknown = true)
   private case class StatementResult(id: Int, state: String, output: 
Map[String, Any])
 
+  private case class CompletionResult(candidates: Seq[String])
+
   @JsonIgnoreProperties(ignoreUnknown = true)
   case class StatementError(ename: String, evalue: String, stackTrace: 
Seq[String])
 
@@ -188,8 +190,36 @@ 

incubator-livy git commit: [LIVY-408][RSC] Update Netty version to 4.0.37.Final

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 0ca16d5d2 -> ff6c2ebbb


[LIVY-408][RSC] Update Netty version to 4.0.37.Final

Netty version below 4.0.37.Final has some potential security issue 
(https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-4970), it is fixed in 
this version. So propose to upgrade Netty to avoid such issue.

Author: jerryshao 

Closes #53 from jerryshao/LIVY-408.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/ff6c2ebb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/ff6c2ebb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/ff6c2ebb

Branch: refs/heads/master
Commit: ff6c2e367a9718e3efab4f460a8225c245ed
Parents: 0ca16d5
Author: jerryshao 
Authored: Fri Sep 29 10:05:58 2017 +0800
Committer: jerryshao 
Committed: Fri Sep 29 10:05:58 2017 +0800

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/ff6c2ebb/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 5864cf0..44b33fe 100644
--- a/pom.xml
+++ b/pom.xml
@@ -92,7 +92,7 @@
 2.22
 3.1.0
 1.9.5
-4.0.29.Final
+4.0.37.Final
 UTF-8
 0.9
 2.10.4

incubator-livy git commit: [LIVY-406][LIVY-406][FOLLOWUP][SERVER] Fix UT failure in branch 0.4 and apply to Master branch for consistency

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.4 3d70104cf -> 939c97673


[LIVY-406][LIVY-406][FOLLOWUP][SERVER] Fix UT failure in branch 0.4 and apply 
to Master branch for consistency

Because we changed the AppState logic in #39 , so the UT which passed in master 
branch will be failed in branch 0.4, here to fix this issue.

Also apply this PR to master branch for the consistency, it should also be 
worked in master branch.

Author: jerryshao 

Closes #52 from jerryshao/LIVY-406-followup.

(cherry picked from commit 0ca16d5d2fb9bed6c66a6a787afb5a7f025c577f)
Signed-off-by: jerryshao 


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/939c9767
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/939c9767
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/939c9767

Branch: refs/heads/branch-0.4
Commit: 939c97673b1349ee71935ff68bb96b3ae6722746
Parents: 3d70104
Author: jerryshao 
Authored: Tue Sep 26 21:07:39 2017 +0800
Committer: jerryshao 
Committed: Tue Sep 26 21:07:53 2017 +0800

--
 server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/939c9767/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala
--
diff --git a/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala 
b/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala
index 27afd85..964bc7b 100644
--- a/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala
+++ b/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala
@@ -227,7 +227,7 @@ class SparkYarnAppSpec extends FunSpec with 
LivyBaseUnitTestSuite {
 
when(mockAppReport.getApplicationTags).thenReturn(Set(appTag.toLowerCase).asJava)
 when(mockAppReport.getApplicationId).thenReturn(appId)
 
when(mockAppReport.getFinalApplicationStatus).thenReturn(FinalApplicationStatus.SUCCEEDED)
-when(mockAppReport.getYarnApplicationState).thenReturn(RUNNING)
+
when(mockAppReport.getYarnApplicationState).thenReturn(YarnApplicationState.FINISHED)
 
when(mockYarnClient.getApplicationReport(appId)).thenReturn(mockAppReport)
 when(mockYarnClient.getApplications(Set("SPARK").asJava))
   .thenReturn(List(mockAppReport).asJava)

incubator-livy git commit: [LIVY-406][SERVER] Fix Livy cannot app id in yarn client mode issue

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master 1bd92b9f4 -> 4a537e24d


[LIVY-406][SERVER] Fix Livy cannot app id in yarn client mode issue

In our `SparkYarnApp` logic, we have to wait for process to exit before to 
query app id. Since the process is never exited in yarn client mode, so it will 
block follow-up logic to get app id and other yarn application information.

So here propose to remove this logic, also because now we will query app id 
ever since app is launched, so we should increase the look-up time to avoid 
timeout.

Author: jerryshao 

Closes #50 from jerryshao/LIVY-406.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/4a537e24
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/4a537e24
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/4a537e24

Branch: refs/heads/master
Commit: 4a537e24d605766f901232760f39b44adae817a2
Parents: 1bd92b9
Author: jerryshao 
Authored: Tue Sep 26 13:25:11 2017 +0800
Committer: jerryshao 
Committed: Tue Sep 26 13:25:11 2017 +0800

--
 conf/livy.conf.template |  4 +--
 .../main/scala/org/apache/livy/LivyConf.scala   |  2 +-
 .../org/apache/livy/utils/SparkYarnApp.scala|  9 --
 .../apache/livy/utils/SparkYarnAppSpec.scala| 30 
 4 files changed, 33 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/4a537e24/conf/livy.conf.template
--
diff --git a/conf/livy.conf.template b/conf/livy.conf.template
index 2425059..86ca9ab 100644
--- a/conf/livy.conf.template
+++ b/conf/livy.conf.template
@@ -107,7 +107,7 @@
 # livy.server.recovery.state-store.url =
 
 # If Livy can't find the yarn app within this time, consider it lost.
-# livy.server.yarn.app-lookup-timeout = 60s
+# livy.server.yarn.app-lookup-timeout = 120s
 # When the cluster is busy, we may fail to launch yarn app in 
app-lookup-timeout, then it would
 # cause session leakage, so we need to check session leakage.
 # How long to check livy session leakage
@@ -116,7 +116,7 @@
 # livy.server.yarn.app-leakage.check-interval = 60s
 
 # How often Livy polls YARN to refresh YARN app state.
-# livy.server.yarn.poll-interval = 1s
+# livy.server.yarn.poll-interval = 5s
 #
 # Days to keep Livy server request logs.
 # livy.server.request-log-retain.days = 5

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/4a537e24/server/src/main/scala/org/apache/livy/LivyConf.scala
--
diff --git a/server/src/main/scala/org/apache/livy/LivyConf.scala 
b/server/src/main/scala/org/apache/livy/LivyConf.scala
index 46af8e8..0cfc8f3 100644
--- a/server/src/main/scala/org/apache/livy/LivyConf.scala
+++ b/server/src/main/scala/org/apache/livy/LivyConf.scala
@@ -123,7 +123,7 @@ object LivyConf {
   val SPARK_LOGS_SIZE = Entry("livy.cache-log.size", 200)
 
   // If Livy can't find the yarn app within this time, consider it lost.
-  val YARN_APP_LOOKUP_TIMEOUT = Entry("livy.server.yarn.app-lookup-timeout", 
"60s")
+  val YARN_APP_LOOKUP_TIMEOUT = Entry("livy.server.yarn.app-lookup-timeout", 
"120s")
 
   // How often Livy polls YARN to refresh YARN app state.
   val YARN_POLL_INTERVAL = Entry("livy.server.yarn.poll-interval", "5s")

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/4a537e24/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
--
diff --git a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala 
b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
index 804be54..91c70ca 100644
--- a/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
+++ b/server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala
@@ -231,15 +231,6 @@ class SparkYarnApp private[utils] (
   // batch YARN queries.
   private[utils] val yarnAppMonitorThread = 
Utils.startDaemonThread(s"yarnAppMonitorThread-$this") {
 try {
-  // Wait for spark-submit to finish submitting the app to YARN.
-  process.foreach { p =>
-val exitCode = p.waitFor()
-if (exitCode != 0) {
-  throw new Exception(s"spark-submit exited with code $exitCode}.\n" +
-s"${process.get.inputLines.mkString("\n")}")
-}
-  }
-
   // If appId is not known, query YARN by appTag to get it.
   val appId = try {
 appIdOption.map(ConverterUtils.toApplicationId).getOrElse {

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/4a537e24/server/src/test/scala/org/apache/livy/utils/SparkYarnAppSpec.scala

incubator-livy git commit: [MINOR] Update Travis Python Requests version to fix build error

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.4 118d10d77 -> 844275ccb


[MINOR] Update Travis Python Requests version to fix build error

The issue is from #39 , seems like `requests` version doesn't meet, so here 
update `requests` version to fix it.

Author: jerryshao 

Closes #46 from jerryshao/travis-python-fix.

Change-Id: Id4713eaa3d72a68c776181542b0f9506924b6578
(cherry picked from commit 52c89a9c4a8d2d58bf51ecd9b0e049191ed7a0b6)
Signed-off-by: jerryshao 


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/844275cc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/844275cc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/844275cc

Branch: refs/heads/branch-0.4
Commit: 844275ccba90a03a4c138c089b2a9e683ec27311
Parents: 118d10d
Author: jerryshao 
Authored: Fri Sep 15 14:51:26 2017 +0800
Committer: jerryshao 
Committed: Fri Sep 15 14:51:47 2017 +0800

--
 .travis.yml | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/844275cc/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index f935c35..8de73e3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -57,17 +57,17 @@ before_install:
   - sudo apt-get -y install python3-pip python-dev
   - sudo apt-get -y install libkrb5-dev
   - sudo apt-get -y remove python-setuptools
-  - pip install --user --upgrade pip "setuptools < 36"
+  - pip2 install --user --upgrade pip "setuptools < 36"
   - pip3 install --user --upgrade pip "setuptools < 36"
-  - pip install --user codecov cloudpickle
+  - pip2 install --user codecov cloudpickle
   - pip3 install --user cloudpickle
 
 install:
   - mvn $MVN_FLAG install -Dskip -DskipTests -DskipITs 
-Dmaven.javadoc.skip=true -B -V
 
 before_script:
-  - pip install --user requests pytest flaky flake8 requests-kerberos
-  - pip3 install --user requests pytest flaky requests-kerberos
+  - pip2 install --user "requests >= 2.10.0" pytest flaky flake8 
requests-kerberos
+  - pip3 install --user "requests >= 2.10.0" pytest flaky requests-kerberos
 
 script:
   - mvn $MVN_FLAG verify -e

incubator-livy-website git commit: [MINOR] Updated tar.gz to zip on download page

[jshao]

Repository: incubator-livy-website
Updated Branches:
  refs/heads/master 46e39ee66 -> e3317a823


[MINOR] Updated tar.gz to zip on download page

Followup to #8

Accidentally left the tar.gz naming after switching the file type to zip in the 
links.

Author: Alex Bozarth 

Closes #9 from ajbozarth/release_fix.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy-website/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/commit/e3317a82
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/tree/e3317a82
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/diff/e3317a82

Branch: refs/heads/master
Commit: e3317a8234644b914297a6d61d97c9318cc3a145
Parents: 46e39ee
Author: Alex Bozarth 
Authored: Tue Sep 5 09:14:48 2017 +0800
Committer: jerryshao 
Committed: Tue Sep 5 09:14:48 2017 +0800

--
 site/download.md | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/e3317a82/site/download.md
--
diff --git a/site/download.md b/site/download.md
index 625be8a..0a45728 100644
--- a/site/download.md
+++ b/site/download.md
@@ -42,15 +42,15 @@ limitations under the License.
 
 
 
-{{ site.data.project.name }} 
{{site.data.project.latest_release}} (tar.gz)
-http://www.apache.org/dyn/closer.lua/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-bin.zip;>tar.gz
+{{ site.data.project.name }} 
{{site.data.project.latest_release}} (zip)
+http://www.apache.org/dyn/closer.lua/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-bin.zip;>zip
 http://www.apache.org/dist/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-bin.zip.md5;>MD5
 http://www.apache.org/dist/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-bin.zip.sha512;>SHA-512
 http://www.apache.org/dist/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-bin.zip.asc;>ASC
 
 
-{{ site.data.project.name }} 
{{site.data.project.latest_release}} (source tar.gz)
-http://www.apache.org/dyn/closer.lua/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-src.zip;>tar.gz
+{{ site.data.project.name }} 
{{site.data.project.latest_release}} (source zip)
+http://www.apache.org/dyn/closer.lua/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-src.zip;>zip
 http://www.apache.org/dist/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-src.zip.md5;>MD5
 http://www.apache.org/dist/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-src.zip.sha512;>SHA-512
 http://www.apache.org/dist/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-src.zip.asc;>ASC

incubator-livy-website git commit: [LIVY-389][RELEASE] 0.4.0-incubating release update

[jshao]

Repository: incubator-livy-website
Updated Branches:
  refs/heads/master 5a044aefa -> 46e39ee66


[LIVY-389][RELEASE] 0.4.0-incubating release update

Website updates for 0.4.0-incubating, including download page, history page and 
0.4.0-incubating release post.

Author: Alex Bozarth 

Closes #8 from ajbozarth/release.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy-website/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/commit/46e39ee6
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/tree/46e39ee6
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/diff/46e39ee6

Branch: refs/heads/master
Commit: 46e39ee6647b145aa29d14e339c0fee49a66a4e4
Parents: 5a044ae
Author: Alex Bozarth 
Authored: Mon Sep 4 09:09:27 2017 +0800
Committer: jerryshao 
Committed: Mon Sep 4 09:09:27 2017 +0800

--
 site/_data/project.yml|  6 +-
 site/_includes/themes/apache/_navigation.html |  2 +-
 site/_posts/2017-09-01-release-0.4.0.md   | 38 ++
 site/download.md  | 81 --
 site/history.md   | 24 +++
 5 files changed, 96 insertions(+), 55 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/46e39ee6/site/_data/project.yml
--
diff --git a/site/_data/project.yml b/site/_data/project.yml
index ed64dd8..923bfb2 100644
--- a/site/_data/project.yml
+++ b/site/_data/project.yml
@@ -22,8 +22,8 @@ incubator_name: incubator-livy
 incubator_slash_name: incubator/livy
 description: An REST Service for Apache Spark
 
-# uncomment after first release
-# download: download
+download: download
+latest_release: 0.4.0-incubating
 
 dev_list: d...@livy.incubator.apache.org
 dev_list_subscribe: dev-subscr...@livy.incubator.apache.org
@@ -48,6 +48,8 @@ commits_list_archive_markmail:
 
 jira: LIVY
 
+github_project_name: incubator-livy
+
 source_repository: https://git-wip-us.apache.org/repos/asf/incubator-livy.git
 source_repository_mirror: https://github.com/apache/incubator-livy
 

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/46e39ee6/site/_includes/themes/apache/_navigation.html
--
diff --git a/site/_includes/themes/apache/_navigation.html 
b/site/_includes/themes/apache/_navigation.html
index f837a5f..aa5ab97 100644
--- a/site/_includes/themes/apache/_navigation.html
+++ b/site/_includes/themes/apache/_navigation.html
@@ -48,7 +48,7 @@
 {{ site.data.project.description }}
 
 {% if site.data.project.download %}
-   Get {{ site.data.project.short_name 
}}
+   Get {{ site.data.project.short_name }} {{ 
site.data.project.latest_release }}
 {% endif %}
   
 

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/46e39ee6/site/_posts/2017-09-01-release-0.4.0.md
--
diff --git a/site/_posts/2017-09-01-release-0.4.0.md 
b/site/_posts/2017-09-01-release-0.4.0.md
new file mode 100644
index 000..ccf2516
--- /dev/null
+++ b/site/_posts/2017-09-01-release-0.4.0.md
@@ -0,0 +1,38 @@
+---
+layout: post
+date: "2017-09-01 12:00:00 +"
+author: ajbozarth
+version: 0.4.0-incubating
+categories: [release]
+tag: v0-4-0-incubating
+sha: e36b6f5
+---
+
+
+This is our first Apache release!
+
+We're excited to be in the Apache Incubator.
+
+New features
+
+* [https://issues.apache.org/jira/browse/LIVY-87;>LIVY-87]
+  Create a Livy Web UI to monitor sessions
+* [https://issues.apache.org/jira/browse/LIVY-348;>LIVY-348]
+  Improve Livy's ACLs

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/46e39ee6/site/download.md
--
diff --git a/site/download.md b/site/download.md
index 2aa9022..625be8a 100644
--- a/site/download.md
+++ b/site/download.md
@@ -30,40 +30,49 @@ limitations under the License.
 
 ### Source releases
 
-There is no official release of Apache Livy yet, try checking out the
-[GitHub repo](https://github.com/apache/{{ site.data.project.incubator_name 
}}) and building the source code.
-
-
+
+
+
+Name
+Archive
+MD5
+SHA-512
+signature
+
+
+
+
+{{ site.data.project.name }} 
{{site.data.project.latest_release}} (tar.gz)
+http://www.apache.org/dyn/closer.lua/incubator/{{site.data.project.unix_name}}/{{site.data.project.latest_release}}/{{site.data.project.unix_name}}-{{site.data.project.latest_release}}-bin.zip;>tar.gz
+

[1/2] incubator-livy-website git commit: Publishing from 5a044aefa4de8b31a1d94f65afdd3eaaef70911d

[jshao]

Repository: incubator-livy-website
Updated Branches:
  refs/heads/asf-site 0ac7686f6 -> 98e1c540b


http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/98e1c540/content/docs/latest/rest-api.html
--
diff --git a/content/docs/latest/rest-api.html 
b/content/docs/latest/rest-api.html
new file mode 100644
index 000..43f2dfa
--- /dev/null
+++ b/content/docs/latest/rest-api.html
@@ -0,0 +1,880 @@
+
+
+
+
+  
+
+Livy Docs - REST API
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+
+  
+
+
+  
+
+  
+Toggle navigation
+
+
+
+  
+  
+  
+  Apache Livy
+0.4.0-incubating
+  
+
+
+  
+  
+  
+  
+  
+  Documentation
+  
+  
+  
+  REST 
API
+  
+  
+  Programmatic API
+  
+  
+  
+  
+  
+  
+  
+  Apache
+  
+  
+  
+  http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
+  
+  
+  http://www.apache.org/licenses/; 
target="_blank">Apache License
+  
+  
+  http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
+  
+  
+  http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks
+  
+  
+  
+  
+  
+  
+  
+
+  
+
+
+
+
+  
+
+
+
+
+  
+
+
+REST API
+
+GET /sessions
+
+Returns all the active interactive sessions.
+
+Request Parameters
+
+
+  NameDescriptionType
+  
+from
+The start index to fetch sessions
+int
+  
+  
+size
+Number of sessions to fetch
+int
+  
+
+
+Response Body
+
+
+  NameDescriptionType
+  
+from
+The start index to fetch sessions
+int
+  
+  
+size
+Number of sessions to fetch
+int
+  
+  
+sessions
+Session list
+list
+  
+
+
+POST /sessions
+
+Creates a new interactive Scala, Python, or R shell in the cluster.
+
+Request Body
+
+
+  NameDescriptionType
+  
+kind
+The session kind (required)
+session kind
+  
+  
+proxyUser
+User to impersonate when starting the session
+string
+  
+  
+jars
+jars to be used in this session
+List of string
+  
+  
+pyFiles
+Python files to be used in this session
+List of string
+  
+  
+files
+files to be used in this session
+List of string
+  
+  
+driverMemory
+Amount of memory to use for the driver process
+string
+  
+  
+driverCores
+Number of cores to use for the driver process
+int
+  
+  
+executorMemory
+Amount of memory to use per executor process
+string
+  
+  
+executorCores
+Number of cores to use for each executor
+int
+  
+  
+numExecutors
+Number of executors to launch for this session
+int
+  
+  
+archives
+Archives to be used in this session
+List of string
+  
+  
+queue
+The name of the YARN queue to which submitted
+string
+  
+  
+name
+The name of this session
+string
+  
+  
+conf
+Spark configuration properties
+Map of key=val
+  
+  
+heartbeatTimeoutInSecond
+Timeout in second to which session be orphaned
+int
+  
+
+
+Response Body
+
+The created Session.
+
+GET /sessions/{sessionId}
+
+Returns the session information.
+
+Response Body
+
+The Session.
+
+GET /sessions/{sessionId}/state
+
+Returns the state of session
+
+Response
+
+
+  NameDescriptionType
+  
+id
+Session id
+int
+  
+  
+state
+The current state of session
+string
+  
+
+
+DELETE /sessions/{sessionId}
+
+Kills the Session job.
+
+GET /sessions/{sessionId}/log
+
+Gets the log lines from this session.
+
+Request Parameters
+
+
+  NameDescriptionType
+  
+from
+Offset
+int
+  
+  
+size
+Max number of log lines to return
+int
+  
+
+
+Response Body
+
+
+  NameDescriptionType
+  
+id
+The session id
+int
+  
+  
+from
+Offset from start of log
+int
+  
+  
+size
+Max number of log lines
+int
+  
+  
+log
+The log lines
+list of strings
+  
+
+
+GET 

[2/2] incubator-livy-website git commit: Publishing from 5a044aefa4de8b31a1d94f65afdd3eaaef70911d

[jshao]

Publishing from 5a044aefa4de8b31a1d94f65afdd3eaaef70911d


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy-website/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/commit/98e1c540
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/tree/98e1c540
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/diff/98e1c540

Branch: refs/heads/asf-site
Commit: 98e1c540b62f251edd805fe69b326de33c355d2f
Parents: 0ac7686
Author: jerryshao 
Authored: Mon Sep 4 08:56:45 2017 +0800
Committer: jerryshao 
Committed: Mon Sep 4 08:56:45 2017 +0800

--
 content/assets/themes/apache/css/style.css  |   5 +
 content/community-members/index.html|   4 +-
 content/community/index.html|   4 +-
 content/docs/0.4.0-incubating/index.html| 169 
 .../docs/0.4.0-incubating/programmatic-api.html | 253 ++
 content/docs/0.4.0-incubating/rest-api.html | 880 +++
 content/docs/latest/index.html  | 169 
 content/docs/latest/programmatic-api.html   | 253 ++
 content/docs/latest/rest-api.html   | 880 +++
 content/download/index.html |   4 +-
 content/examples/index.html |   4 +-
 content/feed.xml|   4 +-
 content/get-started/index.html  |   4 +-
 content/history/index.html  |   4 +-
 content/index.html  |   4 +-
 15 files changed, 2625 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/98e1c540/content/assets/themes/apache/css/style.css
--
diff --git a/content/assets/themes/apache/css/style.css 
b/content/assets/themes/apache/css/style.css
index b18a7aa..a0af144 100644
--- a/content/assets/themes/apache/css/style.css
+++ b/content/assets/themes/apache/css/style.css
@@ -15,6 +15,11 @@ body {
   padding-bottom: 10px;
 }
 
+.navbar-brand .version {
+  font-size: 12px;
+  font-weight: bold;
+}
+
 .navbar {
  background-color:#33;
  border-bottom:0px;

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/98e1c540/content/community-members/index.html
--
diff --git a/content/community-members/index.html 
b/content/community-members/index.html
index d7296a6..74371ae 100644
--- a/content/community-members/index.html
+++ b/content/community-members/index.html
@@ -63,10 +63,10 @@
   
   
   
-  https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md#rest-api;
 target="_blank">REST API
+  REST API
   
   
-  https://github.com/apache/incubator-livy/blob/master/docs/programmatic-api.md#using-the-programmatic-api;
 target="_blank">Programmatic API
+  Programmatic API
   
   
   Examples

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/98e1c540/content/community/index.html
--
diff --git a/content/community/index.html b/content/community/index.html
index efdb2e6..48efb36 100644
--- a/content/community/index.html
+++ b/content/community/index.html
@@ -63,10 +63,10 @@
   
   
   
-  https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md#rest-api;
 target="_blank">REST API
+  REST API
   
   
-  https://github.com/apache/incubator-livy/blob/master/docs/programmatic-api.md#using-the-programmatic-api;
 target="_blank">Programmatic API
+  Programmatic API
   
   
   Examples

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/98e1c540/content/docs/0.4.0-incubating/index.html
--
diff --git a/content/docs/0.4.0-incubating/index.html 
b/content/docs/0.4.0-incubating/index.html
new file mode 100644
index 000..53b2a13
--- /dev/null
+++ b/content/docs/0.4.0-incubating/index.html
@@ -0,0 +1,169 @@
+
+
+
+
+  
+
+Livy Docs
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+
+  
+
+
+  
+
+  
+Toggle navigation
+
+  

incubator-livy-website git commit: [LIVY-384][DOCS] Include local build of Docs

[jshao]

Repository: incubator-livy-website
Updated Branches:
  refs/heads/master 315b22db2 -> 5a044aefa


[LIVY-384][DOCS] Include local build of Docs

[LIVY-384](https://issues.cloudera.org/browse/LIVY-384)

Included a copy of the 0.4.0-incubating build of the Livy Docs in the website. 
The process for updating the Docs can be found in the Docs README in the main 
repo.

Paired with apache/incubator-livy#26

Author: Alex Bozarth 

Closes #7 from ajbozarth/docs.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy-website/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/commit/5a044aef
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/tree/5a044aef
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-livy-website/diff/5a044aef

Branch: refs/heads/master
Commit: 5a044aefa4de8b31a1d94f65afdd3eaaef70911d
Parents: 315b22d
Author: Alex Bozarth 
Authored: Mon Sep 4 08:44:17 2017 +0800
Committer: jerryshao 
Committed: Mon Sep 4 08:44:17 2017 +0800

--
 site/_data/navigation.yml   |   4 +-
 site/assets/themes/apache/css/style.css |   5 +
 site/docs/0.4.0-incubating/index.html   | 169 
 .../docs/0.4.0-incubating/programmatic-api.html | 253 ++
 site/docs/0.4.0-incubating/rest-api.html| 880 +++
 site/docs/latest|   1 +
 6 files changed, 1310 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/5a044aef/site/_data/navigation.yml
--
diff --git a/site/_data/navigation.yml b/site/_data/navigation.yml
index cbac78a..446f114 100644
--- a/site/_data/navigation.yml
+++ b/site/_data/navigation.yml
@@ -21,9 +21,9 @@ topnav:
 - title: Documentation
   subcategories:
 - title: REST API
-  url: 
https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md#rest-api
+  url: /docs/latest/rest-api.html
 - title: Programmatic API
-  url: 
https://github.com/apache/incubator-livy/blob/master/docs/programmatic-api.md#using-the-programmatic-api
+  url: /docs/latest/programmatic-api.html
 - title: Examples
   url: /examples
 

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/5a044aef/site/assets/themes/apache/css/style.css
--
diff --git a/site/assets/themes/apache/css/style.css 
b/site/assets/themes/apache/css/style.css
index b18a7aa..a0af144 100644
--- a/site/assets/themes/apache/css/style.css
+++ b/site/assets/themes/apache/css/style.css
@@ -15,6 +15,11 @@ body {
   padding-bottom: 10px;
 }
 
+.navbar-brand .version {
+  font-size: 12px;
+  font-weight: bold;
+}
+
 .navbar {
  background-color:#33;
  border-bottom:0px;

http://git-wip-us.apache.org/repos/asf/incubator-livy-website/blob/5a044aef/site/docs/0.4.0-incubating/index.html
--
diff --git a/site/docs/0.4.0-incubating/index.html 
b/site/docs/0.4.0-incubating/index.html
new file mode 100644
index 000..53b2a13
--- /dev/null
+++ b/site/docs/0.4.0-incubating/index.html
@@ -0,0 +1,169 @@
+
+
+
+
+  
+
+Livy Docs
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+
+  
+
+
+  
+
+  
+Toggle navigation
+
+
+
+  
+  
+  
+  Apache Livy
+0.4.0-incubating
+  
+
+
+  
+  
+  
+  
+  
+  Documentation
+  
+  
+  
+  REST 
API
+  
+  
+  Programmatic API
+  
+  
+  
+  
+  
+  
+  
+  Apache
+  
+  
+  
+  http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
+  
+  
+  http://www.apache.org/licenses/; 
target="_blank">Apache License
+  
+  
+  http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
+  
+  
+  http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks
+ 

incubator-livy git commit: [LIVY-394][BUILD] Update branch 0.4 to version 0.4.1-incubating-SNAPSHOT

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/branch-0.4 0e30800a4 -> 5fa53c3d5


[LIVY-394][BUILD] Update branch 0.4 to version 0.4.1-incubating-SNAPSHOT

This PR updates branch-0.4 Livy version to 0.4.1-incubating-SNAPSHOT.

Author: jerryshao 

Closes #41 from jerryshao/branch-0.4-version-update.

Change-Id: I6ec1ff925875552b384b28b7ec2aa04358031ba2


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/5fa53c3d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/5fa53c3d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/5fa53c3d

Branch: refs/heads/branch-0.4
Commit: 5fa53c3d50f7cf1bae4e2f7120782244c339864d
Parents: 0e30800
Author: jerryshao 
Authored: Fri Sep 1 14:22:51 2017 +0800
Committer: jerryshao 
Committed: Fri Sep 1 14:22:51 2017 +0800

--
 api/pom.xml  | 4 ++--
 assembly/pom.xml | 6 +++---
 client-common/pom.xml| 4 ++--
 client-http/pom.xml  | 4 ++--
 core/pom.xml | 4 ++--
 core/scala-2.10/pom.xml  | 4 ++--
 core/scala-2.11/pom.xml  | 4 ++--
 coverage/pom.xml | 4 ++--
 examples/pom.xml | 4 ++--
 integration-test/minicluster-dependencies/pom.xml| 4 ++--
 integration-test/minicluster-dependencies/scala-2.10/pom.xml | 4 ++--
 integration-test/minicluster-dependencies/scala-2.11/pom.xml | 4 ++--
 integration-test/pom.xml | 4 ++--
 pom.xml  | 2 +-
 python-api/pom.xml   | 4 ++--
 python-api/setup.py  | 2 +-
 repl/pom.xml | 4 ++--
 repl/scala-2.10/pom.xml  | 4 ++--
 repl/scala-2.11/pom.xml  | 4 ++--
 rsc/pom.xml  | 4 ++--
 scala-api/pom.xml| 4 ++--
 scala-api/scala-2.10/pom.xml | 4 ++--
 scala-api/scala-2.11/pom.xml | 4 ++--
 scala/pom.xml| 4 ++--
 server/pom.xml   | 4 ++--
 test-lib/pom.xml | 4 ++--
 26 files changed, 51 insertions(+), 51 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/5fa53c3d/api/pom.xml
--
diff --git a/api/pom.xml b/api/pom.xml
index 0b1411e..76f9f8e 100644
--- a/api/pom.xml
+++ b/api/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.4.0-incubating-SNAPSHOT
+0.4.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-api
-  0.4.0-incubating-SNAPSHOT
+  0.4.1-incubating-SNAPSHOT
   jar
 
   

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/5fa53c3d/assembly/pom.xml
--
diff --git a/assembly/pom.xml b/assembly/pom.xml
index abe9545..f5c6450 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -20,16 +20,16 @@
   
 org.apache.livy
 livy-main
-0.4.0-incubating-SNAPSHOT
+0.4.1-incubating-SNAPSHOT
 ../pom.xml
   
 
   livy-assembly
-  0.4.0-incubating-SNAPSHOT
+  0.4.1-incubating-SNAPSHOT
   pom
 
   
-livy-server-${project.version}
+livy-${project.version}-bin
 zip
 true
   

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/5fa53c3d/client-common/pom.xml
--
diff --git a/client-common/pom.xml b/client-common/pom.xml
index 6f40e0d..9ec769f 100644
--- a/client-common/pom.xml
+++ b/client-common/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.4.0-incubating-SNAPSHOT
+0.4.1-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-client-common
-  0.4.0-incubating-SNAPSHOT
+  0.4.1-incubating-SNAPSHOT
   jar
 
   

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/5fa53c3d/client-http/pom.xml
--
diff --git a/client-http/pom.xml b/client-http/pom.xml
index b70a844..db99332 100644
--- a/client-http/pom.xml
+++ b/client-http/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-

incubator-livy git commit: [LIVY-394][BUILD] Update Livy master to 0.5.0-incubating-SNAPSHOT

[jshao]

Repository: incubator-livy
Updated Branches:
  refs/heads/master c1aafeb6c -> a23969721


[LIVY-394][BUILD] Update Livy master to 0.5.0-incubating-SNAPSHOT

This PR updates Livy master branch version to 0.5.0-incubating-SNAPSHOT, also 
changes the Livy package name to livy--bin.zip.

Author: jerryshao 

Closes #40 from jerryshao/master-version-update.

Change-Id: If76e823a740e5178bb1e0efb4d39c3ba95dba9de


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/a2396972
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/a2396972
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/a2396972

Branch: refs/heads/master
Commit: a23969721ff5378a6206ace483c47e8062e21f88
Parents: c1aafeb
Author: jerryshao 
Authored: Fri Sep 1 14:05:18 2017 +0800
Committer: jerryshao 
Committed: Fri Sep 1 14:05:18 2017 +0800

--
 api/pom.xml  | 4 ++--
 assembly/pom.xml | 6 +++---
 client-common/pom.xml| 4 ++--
 client-http/pom.xml  | 4 ++--
 core/pom.xml | 4 ++--
 core/scala-2.10/pom.xml  | 4 ++--
 core/scala-2.11/pom.xml  | 4 ++--
 coverage/pom.xml | 4 ++--
 examples/pom.xml | 4 ++--
 integration-test/minicluster-dependencies/pom.xml| 4 ++--
 integration-test/minicluster-dependencies/scala-2.10/pom.xml | 4 ++--
 integration-test/minicluster-dependencies/scala-2.11/pom.xml | 4 ++--
 integration-test/pom.xml | 4 ++--
 pom.xml  | 2 +-
 python-api/pom.xml   | 4 ++--
 python-api/setup.py  | 2 +-
 repl/pom.xml | 4 ++--
 repl/scala-2.10/pom.xml  | 4 ++--
 repl/scala-2.11/pom.xml  | 4 ++--
 rsc/pom.xml  | 4 ++--
 scala-api/pom.xml| 4 ++--
 scala-api/scala-2.10/pom.xml | 4 ++--
 scala-api/scala-2.11/pom.xml | 4 ++--
 scala/pom.xml| 4 ++--
 server/pom.xml   | 4 ++--
 test-lib/pom.xml | 4 ++--
 26 files changed, 51 insertions(+), 51 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/a2396972/api/pom.xml
--
diff --git a/api/pom.xml b/api/pom.xml
index 0b1411e..4f67ac1 100644
--- a/api/pom.xml
+++ b/api/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.4.0-incubating-SNAPSHOT
+0.5.0-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-api
-  0.4.0-incubating-SNAPSHOT
+  0.5.0-incubating-SNAPSHOT
   jar
 
   

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/a2396972/assembly/pom.xml
--
diff --git a/assembly/pom.xml b/assembly/pom.xml
index abe9545..3a2ab15 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -20,16 +20,16 @@
   
 org.apache.livy
 livy-main
-0.4.0-incubating-SNAPSHOT
+0.5.0-incubating-SNAPSHOT
 ../pom.xml
   
 
   livy-assembly
-  0.4.0-incubating-SNAPSHOT
+  0.5.0-incubating-SNAPSHOT
   pom
 
   
-livy-server-${project.version}
+livy-${project.version}-bin
 zip
 true
   

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/a2396972/client-common/pom.xml
--
diff --git a/client-common/pom.xml b/client-common/pom.xml
index 6f40e0d..c646ab9 100644
--- a/client-common/pom.xml
+++ b/client-common/pom.xml
@@ -20,12 +20,12 @@
   
 org.apache.livy
 livy-main
-0.4.0-incubating-SNAPSHOT
+0.5.0-incubating-SNAPSHOT
   
 
   org.apache.livy
   livy-client-common
-  0.4.0-incubating-SNAPSHOT
+  0.5.0-incubating-SNAPSHOT
   jar
 
   

http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/a2396972/client-http/pom.xml
--
diff --git a/client-http/pom.xml b/client-http/pom.xml
index b70a844..7515b39 100644
--- a/client-http/pom.xml
+++ b/client-http/pom.xml
@@ -20,12 +20,12 @@