This is an automated email from the ASF dual-hosted git repository.
jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git
The following commit(s) were added to refs/heads/master by this push:
new 97cf2f7 [LIVY-756] Add Spark 3.0 and Scala 2.12 support
97cf2f7 is described below
commit 97cf2f75929ef6c152afc468adbead269bd0758f
Author: jerryshao
AuthorDate: Thu Jul 2 15:44:12 2020 +0800
[LIVY-756] Add Spark 3.0 and Scala 2.12 support
## What changes were proposed in this pull request?
This PR is based tprelle 's PR #289 , and address all the left issues in
that PR:
1. multi-scala version support in one build (Scala 2.11 and 2.12 support).
2. make SparkR work.
Also reverts most of the unnecessary changes. Besides this PR remove the
build below 2.4 (2.2, 2.3), since Spark 2.2 and 2.3 only ships with Scala 2.11,
hard to maintain multiple version. But user could still use 2.2 and 2.3 without
changes.
All credits to tprelle.
## How was this patch tested?
Run UT and IT with Spark 2.4.5 and 3.0.0 locally.
Author: jerryshao
Closes #300 from jerryshao/LIVY-756.
---
.gitignore | 1 +
.rat-excludes | 1 +
.travis.yml| 24 +++---
README.md | 4 +-
assembly/assembly.xml | 7 ++
assembly/pom.xml | 23 ++
client-common/pom.xml | 2 +-
.../org/apache/livy/client/common/Serializer.java | 8 +-
{client-common => core/scala-2.12}/pom.xml | 52 ++---
.../org/apache/livy/LivyBaseUnitTestSuite.scala| 4 +-
coverage/pom.xml | 35 +
.../org/apache/livy/examples/WordCountApp.scala| 2 +-
integration-test/pom.xml | 2 +-
integration-test/src/test/resources/rtest.R| 9 +--
.../scala/org/apache/livy/test/InteractiveIT.scala | 6 +-
.../src/test/spark2/scala/Spark2JobApiIT.scala | 26 +--
pom.xml| 88 +-
repl/pom.xml | 3 +
repl/scala-2.11/pom.xml| 1 +
.../org/apache/livy/repl/SparkInterpreter.scala| 5 +-
repl/{scala-2.11 => scala-2.12}/pom.xml| 11 +--
.../org/apache/livy/repl/SparkInterpreter.scala| 17 ++---
.../apache/livy/repl/SparkInterpreterSpec.scala| 68 +
.../main/scala/org/apache/livy/repl/Session.scala | 4 +-
.../org/apache/livy/repl/SQLInterpreterSpec.scala | 4 +-
rsc/pom.xml| 6 +-
.../org/apache/livy/rsc/driver/SparkEntries.java | 7 +-
.../org/apache/livy/rsc/rpc/KryoMessageCodec.java | 7 --
{repl/scala-2.11 => scala-api/scala-2.12}/pom.xml | 17 ++---
scala-api/src/main/resources/build.marker | 0
.../org/apache/livy/scalaapi/ScalaJobHandle.scala | 8 ++
server/pom.xml | 9 ++-
.../org/apache/livy/server/SessionServlet.scala| 2 +-
.../server/interactive/InteractiveSession.scala| 6 +-
.../org/apache/livy/utils/LivySparkUtils.scala | 4 +-
.../apache/livy/server/BaseJsonServletSpec.scala | 3 +-
.../apache/livy/server/SessionServletSpec.scala| 2 +-
.../livy/server/batch/BatchServletSpec.scala | 2 +-
.../livy/server/batch/BatchSessionSpec.scala | 6 +-
.../InteractiveSessionServletSpec.scala| 3 +-
.../interactive/InteractiveSessionSpec.scala | 2 +-
.../livy/server/interactive/JobApiSpec.scala | 2 +-
.../server/interactive/SessionHeartbeatSpec.scala | 2 +-
.../server/recovery/FileSystemStateStoreSpec.scala | 2 +-
.../livy/server/recovery/SessionStoreSpec.scala| 2 +-
.../livy/server/recovery/StateStoreSpec.scala | 2 -
.../server/recovery/ZooKeeperStateStoreSpec.scala | 2 +-
.../apache/livy/sessions/SessionManagerSpec.scala | 2 +-
.../apache/livy/utils/LivySparkUtilsSuite.scala| 5 ++
.../org/apache/livy/utils/SparkYarnAppSpec.scala | 2 +-
.../org/apache/livy/test/jobs/SQLGetTweets.java| 2 +-
.../livy/thriftserver/types/DataTypeUtils.scala| 5 +-
.../livy/thriftserver/ThriftServerSuites.scala | 3 +-
thriftserver/session/pom.xml | 13
.../thriftserver/session/ColumnBufferTest.java | 16 ++--
55 files changed, 362 insertions(+), 189 deletions(-)
diff --git a/.gitignore b/.gitignore
index d46d49f..b1045ea 100644
--- a/.gitignore
+++ b/.gitignore
@@ -24,6 +24,7 @@ metastore_db/
derby.log
dependency-reduced-pom.xml
release-staging/
+venv/
# For python setup.py, which pollutes the source dirs.
python-api/dist
diff --git a/.rat-excludes b/.rat-excludes
index ac29fe6..1df6e9e 1006
