GitHub user vanzin opened a pull request:
https://github.com/apache/incubator-livy/pull/118
[LIVY-503] Separate thrift server session code in separate module.
This change creates a new module ("livy-thriftserver-session") with
the code related to the Thrift server that is expected to run inside
a Spark session. The module contains a few Job implementations that
are used by the Thrift server to run SQL statements on the session
side, plus some internal bookkeeping code.
The new module is a re-implementation of the current Scala code in
Java; Java is chosen because it becomes simpler to support different
Scala versions, for a couple of reasons:
- no need to compile the module separately for different Scala versions,
for when that becomes an issue.
- better control of serialization, instead of relying on Scala types that
may not be compatible across versions (imagine the Livy server and the
Spark session using different Scala versions).
The change also removes the "jarLocation" configuration; in a release
package, the session jars would be available with the rest of the RSC
jars; for development, some code was added to the RSC to add the
thrift server jars to the session when a local build is detected.
Also included is a small change in the RSC pom so that the shaded jar
is generated in the correct target directory and later copied into the
staging directory; this solves an issue with using the RSC in test code
in other modules (such as the one being added).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/incubator-livy LIVY-503
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-livy/pull/118.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #118
----
commit 5a033b7c89544d2e3a9b2536b095bae10212e91f
Author: Marcelo Vanzin <vanzin@...>
Date: 2018-09-10T18:32:29Z
[LIVY-503] Separate thrift server session code in separate module.
This change creates a new module ("livy-thriftserver-session") with
the code related to the Thrift server that is expected to run inside
a Spark session. The module contains a few Job implementations that
are used by the Thrift server to run SQL statements on the session
side, plus some internal bookkeeping code.
The new module is a re-implementation of the current Scala code in
Java; Java is chosen because it becomes simpler to support different
Scala versions, for a couple of reasons:
- no need to compile the module separately for different Scala versions,
for when that becomes an issue.
- better control of serialization, instead of relying on Scala types that
may not be compatible across versions (imagine the Livy server and the
Spark session using different Scala versions).
The change also removes the "jarLocation" configuration; in a release
package, the session jars would be available with the rest of the RSC
jars; for development, some code was added to the RSC to add the
thrift server jars to the session when a local build is detected.
Also included is a small change in the RSC pom so that the shaded jar
is generated in the correct target directory and later copied into the
staging directory; this solves an issue with using the RSC in test code
in other modules (such as the one being added).
----
---
