[GitHub] incubator-livy pull request #107: [LIVY-494] Add thriftserver to Livy server

2018-09-26 Thread vanzin 
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/107#discussion_r219612825
  
--- Diff: server/src/main/scala/org/apache/livy/server/LivyServer.scala ---
@@ -115,6 +117,16 @@ class LivyServer extends Logging {
 error("Failed to run kinit, stopping the server.")
 sys.exit(1)
   }
+  // This is and should be the only place where a login() on the UGI 
is performed.
--- End diff --

This should be pretty easy to check with a scalastyle rule that disallows 
calls to `UserGroupInformation.login*` (and disabling the check around this 
call). Not foolproof, but should catch inadvertent use at least.


---

[GitHub] incubator-livy issue #112: [LIVY-511][LIVY-512] Remove support for old Spark...

2018-09-26 Thread codecov-io 
Github user codecov-io commented on the issue:

https://github.com/apache/incubator-livy/pull/112
  
# 
[Codecov](https://codecov.io/gh/apache/incubator-livy/pull/112?src=pr=h1) 
Report
> Merging 
[#112](https://codecov.io/gh/apache/incubator-livy/pull/112?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-livy/commit/d39ab356b4e36d8780bd44b9d5020bfeec90cb2f?src=pr=desc)
 will **increase** coverage by `<.01%`.
> The diff coverage is `57.14%`.

[![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-livy/pull/112/graphs/tree.svg?width=650=0MkVbiUFwE=150=pr)](https://codecov.io/gh/apache/incubator-livy/pull/112?src=pr=tree)

```diff
@@ Coverage Diff  @@
## master #112  +/-   ##

+ Coverage 71.12%   71.12%   +<.01% 
- Complexity  796  928 +132 

  Files98   99   +1 
  Lines  5500 5483  -17 
  Branches829  828   -1 

- Hits   3912 3900  -12 
- Misses 1056 1059   +3 
+ Partials532  524   -8
```


| [Impacted 
Files](https://codecov.io/gh/apache/incubator-livy/pull/112?src=pr=tree) | 
Coverage Δ | Complexity Δ | |
|---|---|---|---|
| 
[.../scala/org/apache/livy/repl/SparkInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-cmVwbC9zY2FsYS0yLjExL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9yZXBsL1NwYXJrSW50ZXJwcmV0ZXIuc2NhbGE=)
 | `82.97% <ø> (ø)` | `10 <0> (?)` | |
| 
[...rver/src/main/scala/org/apache/livy/LivyConf.scala](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9MaXZ5Q29uZi5zY2FsYQ==)
 | `93.79% <ø> (ø)` | `21 <0> (ø)` | :arrow_down: |
| 
[...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh)
 | `85.22% <ø> (-0.17%)` | `13 <0> (ø)` | |
| 
[...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==)
 | `82.65% <100%> (-1.27%)` | `22 <0> (+4)` | |
| 
[...in/scala/org/apache/livy/repl/SQLInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TUUxJbnRlcnByZXRlci5zY2FsYQ==)
 | `70.37% <100%> (-2.61%)` | `7 <1> (+6)` | |
| 
[...n/scala/org/apache/livy/utils/LivySparkUtils.scala](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9MaXZ5U3BhcmtVdGlscy5zY2FsYQ==)
 | `67.6% <100%> (-4.02%)` | `0 <0> (ø)` | |
| 
[...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh)
 | `69.2% <25%> (+2.02%)` | `43 <0> (ø)` | :arrow_down: |
| 
[.../java/org/apache/livy/rsc/driver/SparkEntries.java](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvU3BhcmtFbnRyaWVzLmphdmE=)
 | `64.06% <33.33%> (-0.33%)` | `10 <0> (-1)` | |
| 
[...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==)
 | `91.11% <66.66%> (ø)` | `17 <0> (ø)` | :arrow_down: |
| 
[...rg/apache/livy/repl/AbstractSparkInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9BYnN0cmFjdFNwYXJrSW50ZXJwcmV0ZXIuc2NhbGE=)
 | `52.5% <0%> (-4.3%)` | `30% <0%> (+29%)` | |
| ... and [29 
more](https://codecov.io/gh/apache/incubator-livy/pull/112/diff?src=pr=tree-more)
 | |

--

[Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-livy/pull/112?src=pr=continue).
> **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute  (impact)`, `ø = not affected`, `? = missing 
data`
> Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-livy/pull/112?src=pr=footer).
 Last update 
[d39ab35...6459626](https://codecov.io/gh/apache/incubator-livy/pull/112?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).



---

[GitHub] incubator-livy pull request #112: [LIVY-511][LIVY-512] Remove support for ol...

2018-09-26 Thread vanzin 
GitHub user vanzin opened a pull request:

https://github.com/apache/incubator-livy/pull/112

[LIVY-511][LIVY-512] Remove support for old Spark, Scala versions.

This change restricts Livy support to Spark 2.2+ and Scala 2.11.  Both 
changes
are made together because by supporting Spark 2.2+ only, it becomes 
impossible
to test Scala 2.10.

As part of the change, a lot of code that used reflection to support 
different
versions of Spark could be cleaned up and directly call Spark APIs.

The Scala 2.10 parts of the builds also have been removed, but the actual
support for building and running with different Scala versions (and related
tests) have been left as is. This will allow us to support 2.12 in the 
future.

This change intentionally does not touch the public API (the "api/" module).
There are things that could be cleaned up now that Spark 1.x is not 
supported,
but that would mean an API breakage so I chose to leave those alone for now.

The test matrix and build profiles have also been simplified a lot.  There 
are
now two profiles to choose from (for Spark 2.2 and 2.3); integration tests 
can
be run against a different version of Spark by running just the integration
test module with the desired profile.

Tested with Spark 2.2 and 2.3, and also by building against 2.2 and running
integration tests against 2.3.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vanzin/incubator-livy LIVY-511

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-livy/pull/112.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #112


commit 6459626488a050935e25cceafe4e8b22e91e3012
Author: Marcelo Vanzin 
Date:   2018-09-20T18:39:23Z

[LIVY-511][LIVY-512] Remove support for old Spark, Scala versions.

This change restricts Livy support to Spark 2.2+ and Scala 2.11.  Both 
changes
are made together because by supporting Spark 2.2+ only, it becomes 
impossible
to test Scala 2.10.

As part of the change, a lot of code that used reflection to support 
different
versions of Spark could be cleaned up and directly call Spark APIs.

The Scala 2.10 parts of the builds also have been removed, but the actual
support for building and running with different Scala versions (and related
tests) have been left as is. This will allow us to support 2.12 in the 
future.

This change intentionally does not touch the public API (the "api/" module).
There are things that could be cleaned up now that Spark 1.x is not 
supported,
but that would mean an API breakage so I chose to leave those alone for now.

The test matrix and build profiles have also been simplified a lot.  There 
are
now two profiles to choose from (for Spark 2.2 and 2.3); integration tests 
can
be run against a different version of Spark by running just the integration
test module with the desired profile.

Tested with Spark 2.2 and 2.3, and also by building against 2.2 and running
integration tests against 2.3.




---