[GitHub] Jassy1994 commented on issue #138: [LIVY-547]do not check expire when session in state busy
Jassy1994 commented on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-455100356 > Please make this a configurable option. Hi meisam I don't think an interactiveSession in state busy should be expired.so I think its no need to set this into a configuration. or may you share me some counter-cases, Thanks a lot. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam commented on issue #138: [LIVY-547]do not check expire when session in state busy
meisam commented on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-455352091 > or may you share me some counter-cases If the number of jobs running are small and can be monitored manually, it makes sense to keep busy jobs. But in a system with hundreds of users and thousands of jobs running each day, it is impractical to check the status of each job manually. In many cases users submit jobs on a daily/hourly basis though an enterprise scheduler. In this case, if a job takes too long to run, it is better to fail it and figure out why it took so long. Also, users have the option to reset the last activity time for a job though `sessions/:id/connect` endpoint. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam opened a new pull request #48: [LIVY-41] Let users access sessions by session name
meisam opened a new pull request #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48 This commit enables Livy users to access sessions either by names or by auto-generated sessiond id's. It also prevents users from creating sessions that have the same name. This commit keeps API change minimal. Thse are palces that API change is needed: - `Session` and its subclasses adds a new field, `name`. - `RecoveryMetadata` and its subclasses adds a new field, `name`. - `SessionManager` adds a new method `getSession(name: String)` which lookups sessions by name. A more clean implementation would change the signature of `SessionManager.register` so it returns a proper container around the session value to determine if it failed to register the given session. For example, ```Scala def register(session S): Either[S, Throwable] ``` Task-url: https://issues.apache.org/jira/browse/LIVY-41 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam closed pull request #48: [LIVY-41] Let users access sessions by session name
meisam closed pull request #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name
codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48#issuecomment-330704048 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=h1) Report > Merging [#48](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.34%`. > The diff coverage is `78.16%`. [](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #48 +/- ## + Coverage 68.34% 68.69% +0.34% - Complexity 895 900 +5 Files 100 100 Lines 5604 5647 +43 Branches840 846 +6 + Hits 3830 3879 +49 + Misses 1225 1215 -10 - Partials549 553 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `68.96% <100%> (+1.89%)` | `17 <4> (+1)` | :arrow_up: | | [...apache/livy/server/batch/BatchSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `90.32% <100%> (+1.86%)` | `3 <0> (ø)` | :arrow_down: | | [...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh) | `86.31% <100%> (+0.92%)` | `14 <6> (+1)` | :arrow_up: | | [...va/org/apache/livy/client/common/HttpMessages.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL0h0dHBNZXNzYWdlcy5qYXZh) | `95.23% <100%> (+0.11%)` | `0 <0> (ø)` | :arrow_down: | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `66.93% <100%> (+0.54%)` | `6 <0> (ø)` | :arrow_down: | | [.../main/scala/org/apache/livy/sessions/Session.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uLnNjYWxh) | `70.53% <14.28%> (-3.05%)` | `18 <3> (+2)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `69.04% <69.44%> (+0.68%)` | `44 <4> (+1)` | :arrow_up: | | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `82.6% <81.81%> (-0.11%)` | `24 <1> (+1)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=footer). Last update
[GitHub] risdenk commented on issue #141: [LIVY-551] Add "doAs" impersonation support
risdenk commented on issue #141: [LIVY-551] Add "doAs" impersonation support URL: https://github.com/apache/incubator-livy/pull/141#issuecomment-455586202 Most of the changes are to pass the Session and HttpServletRequest around to allow for checking different parts of the request with the access manager. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] risdenk opened a new pull request #141: [LIVY-551] Add "doAs" impersonation support
risdenk opened a new pull request #141: [LIVY-551] Add "doAs" impersonation support URL: https://github.com/apache/incubator-livy/pull/141 ## What changes were proposed in this pull request? Currently `proxyuser` is used in batches and sessions for impersonation. This should be extended to support impersonation for all endpoints. Adding the query parameter `doAs` matches others in the Hadoop ecosystem and will work across endpoints. https://issues.apache.org/jira/browse/LIVY-551 ## How was this patch tested? Added unit and integration tests for this feature. Manually tested to ensure that `proxyuser` still works. Also checked that if both `proxyuser` and the new `doAs` parameter are specified that `doAs` takes precedence. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Jassy1994 opened a new pull request #140: quick failure when error happens in spark-submit process
Jassy1994 opened a new pull request #140: quick failure when error happens in spark-submit process URL: https://github.com/apache/incubator-livy/pull/140 ## What changes were proposed in this pull request? support batch session a quick failure when error happens in spark-submit process ## How was this patch tested? submit a batch session with wrong username or queuename，observe session state in livy ui This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #140: quick failure when error happens in spark-submit process
codecov-io commented on issue #140: quick failure when error happens in spark-submit process URL: https://github.com/apache/incubator-livy/pull/140#issuecomment-455508416 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/140?src=pr=h1) Report > Merging [#140](https://codecov.io/gh/apache/incubator-livy/pull/140?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.18%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-livy/pull/140?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #140 +/- ## + Coverage 68.34% 68.52% +0.18% + Complexity 895 893 -2 Files 100 100 Lines 5604 5605 +1 Branches840 840 + Hits 3830 3841 +11 + Misses 1225 1213 -12 - Partials549 551 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/140?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/140/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh) | `85.55% <100%> (+0.16%)` | `13 <0> (ø)` | :arrow_down: | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/140/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `77.96% <0%> (-1.28%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/140/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/140/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `60.69% <0%> (+2.31%)` | `14% <0%> (ø)` | :arrow_down: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/140/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+2.48%)` | `20% <0%> (+1%)` | :arrow_up: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/140/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `52.94% <0%> (+2.94%)` | `7% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/140?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/140?src=pr=footer). Last update [02550f7...135b0db](https://codecov.io/gh/apache/incubator-livy/pull/140?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] risdenk commented on issue #141: [LIVY-551] Add "doAs" impersonation support
risdenk commented on issue #141: [LIVY-551] Add "doAs" impersonation support URL: https://github.com/apache/incubator-livy/pull/141#issuecomment-455687407 The latest build succeeded except for "Spark 2.3 Unit Tests" ``` --- T E S T S --- Running org.apache.livy.rsc.TestJobHandle Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.634 sec - in org.apache.livy.rsc.TestJobHandle Running org.apache.livy.rsc.TestSparkClient Tests run: 19, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 57.489 sec <<< FAILURE! - in org.apache.livy.rsc.TestSparkClient testRemoteClient(org.apache.livy.rsc.TestSparkClient) Time elapsed: 4.546 sec <<< ERROR! java.util.concurrent.ExecutionException: java.io.IOException: RSCClient instance stopped. at org.apache.livy.rsc.TestSparkClient.runTest(TestSparkClient.java:588) at org.apache.livy.rsc.TestSparkClient.runTest(TestSparkClient.java:575) at org.apache.livy.rsc.TestSparkClient.testRemoteClient(TestSparkClient.java:213) Caused by: java.io.IOException: RSCClient instance stopped. Running org.apache.livy.rsc.rpc.TestKryoMessageCodec Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.042 sec - in org.apache.livy.rsc.rpc.TestKryoMessageCodec Running org.apache.livy.rsc.rpc.TestRpc Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.407 sec - in org.apache.livy.rsc.rpc.TestRpc Results : Tests in error: TestSparkClient.testRemoteClient:213->runTest:575->runTest:588 » Execution jav... ``` None of the other Spark versions failed. I can't rekick off the single job since it looks to be a transient failure. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #119: [LIVY-527] added autocompletion api and implementation for ipython
codecov-io edited a comment on issue #119: [LIVY-527] added autocompletion api and implementation for ipython URL: https://github.com/apache/incubator-livy/pull/119#issuecomment-427762007 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=h1) Report > Merging [#119](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/d87a34872069107329b67940d193c2962cf2026d?src=pr=desc) will **decrease** coverage by `2.34%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#119 +/- ## === - Coverage 70.94% 68.6% -2.35% + Complexity 922 896 -26 === Files 100 100 Lines 55005606 +106 Branches828 840 +12 === - Hits 39023846 -56 - Misses 10621211 +149 - Partials536 549 +13 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...scala/org/apache/livy/repl/PythonInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9QeXRob25JbnRlcnByZXRlci5zY2FsYQ==) | `47.53% <0%> (-0.6%)` | `9 <0> (ø)` | | | [.../main/java/org/apache/livy/rsc/ReplJobResults.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SZXBsSm9iUmVzdWx0cy5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...ain/java/org/apache/livy/rsc/driver/Statement.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvU3RhdGVtZW50LmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-6%)` | | | [...in/java/org/apache/livy/rsc/driver/AddFileJob.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvQWRkRmlsZUpvYi5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...ava/org/apache/livy/rsc/driver/StatementState.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvU3RhdGVtZW50U3RhdGUuamF2YQ==) | `0% <0%> (-93.34%)` | `0% <0%> (-9%)` | | | [...rc/main/java/org/apache/livy/rsc/BaseProtocol.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9CYXNlUHJvdG9jb2wuamF2YQ==) | `52.12% <0%> (-22.35%)` | `1% <0%> (ø)` | | | [.../java/org/apache/livy/rsc/driver/SparkEntries.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvU3BhcmtFbnRyaWVzLmphdmE=) | `46.87% <0%> (-17.19%)` | `7% <0%> (-3%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (-15.62%)` | `13% <0%> (-5%)` | | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (-8.7%)` | `20% <0%> (-6%)` | | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `73.23% <0%> (-6.34%)` | `33% <0%> (ø)` | | | ... and [20 more](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=footer). Last update [d87a348...44570f5](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache
[GitHub] wangqiaoshi commented on a change in pull request #119: [LIVY-527] added autocompletion api and implementation for ipython
wangqiaoshi commented on a change in pull request #119: [LIVY-527] added autocompletion api and implementation for ipython URL: https://github.com/apache/incubator-livy/pull/119#discussion_r249282542 ## File path: repl/src/main/resources/fake_shell.py ## @@ -49,6 +49,22 @@ TOP_FRAME_REGEX = re.compile(r'\s*File "".*in ') +def get_completer(): +try: +__IPYTHON__ +from IPython.core.completer import IPCompleter +ip = get_ipython() +ip_completer = IPCompleter(ip,global_namespace=global_dict) +global_dict['ip_completer'] = ip_completer Review comment: Completer will be different when python interpreter is different. The number of completer global variable will be two,I don't know if that's appropriate? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #119: [LIVY-527] added autocompletion api and implementation for ipython
codecov-io edited a comment on issue #119: [LIVY-527] added autocompletion api and implementation for ipython URL: https://github.com/apache/incubator-livy/pull/119#issuecomment-427762007 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=h1) Report > Merging [#119](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/d87a34872069107329b67940d193c2962cf2026d?src=pr=desc) will **decrease** coverage by `2.34%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#119 +/- ## === - Coverage 70.94% 68.6% -2.35% + Complexity 922 896 -26 === Files 100 100 Lines 55005606 +106 Branches828 840 +12 === - Hits 39023846 -56 - Misses 10621211 +149 - Partials536 549 +13 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...scala/org/apache/livy/repl/PythonInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9QeXRob25JbnRlcnByZXRlci5zY2FsYQ==) | `47.53% <0%> (-0.6%)` | `9 <0> (ø)` | | | [.../main/java/org/apache/livy/rsc/ReplJobResults.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SZXBsSm9iUmVzdWx0cy5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...ain/java/org/apache/livy/rsc/driver/Statement.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvU3RhdGVtZW50LmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-6%)` | | | [...in/java/org/apache/livy/rsc/driver/AddFileJob.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvQWRkRmlsZUpvYi5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...ava/org/apache/livy/rsc/driver/StatementState.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvU3RhdGVtZW50U3RhdGUuamF2YQ==) | `0% <0%> (-93.34%)` | `0% <0%> (-9%)` | | | [...rc/main/java/org/apache/livy/rsc/BaseProtocol.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9CYXNlUHJvdG9jb2wuamF2YQ==) | `52.12% <0%> (-22.35%)` | `1% <0%> (ø)` | | | [.../java/org/apache/livy/rsc/driver/SparkEntries.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvU3BhcmtFbnRyaWVzLmphdmE=) | `46.87% <0%> (-17.19%)` | `7% <0%> (-3%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (-15.62%)` | `13% <0%> (-5%)` | | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (-8.7%)` | `20% <0%> (-6%)` | | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `73.23% <0%> (-6.34%)` | `33% <0%> (ø)` | | | ... and [20 more](https://codecov.io/gh/apache/incubator-livy/pull/119/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=footer). Last update [d87a348...44570f5](https://codecov.io/gh/apache/incubator-livy/pull/119?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache
[GitHub] Tagar commented on issue #119: [LIVY-527] added autocompletion api and implementation for ipython
Tagar commented on issue #119: [LIVY-527] added autocompletion api and implementation for ipython URL: https://github.com/apache/incubator-livy/pull/119#issuecomment-454596543 ping @wangqiaoshi - that's a great improvement! do you mind fixing those minor issues after the last review thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam opened a new pull request #48: [LIVY-41] Let users access sessions by session name
meisam opened a new pull request #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48 This commit enables Livy users to access sessions either by names or by auto-generated sessiond id's. It also prevents users from creating sessions that have the same name. This commit keeps API change minimal. Thse are palces that API change is needed: - `Session` and its subclasses adds a new field, `name`. - `RecoveryMetadata` and its subclasses adds a new field, `name`. - `SessionManager` adds a new method `getSession(name: String)` which lookups sessions by name. A more clean implementation would change the signature of `SessionManager.register` so it returns a proper container around the session value to determine if it failed to register the given session. For example, ```Scala def register(session S): Either[S, Throwable] ``` Task-url: https://issues.apache.org/jira/browse/LIVY-41 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam commented on issue #48: [LIVY-41] Let users access sessions by session name
meisam commented on issue #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48#issuecomment-454596319 close/reopen to trigger the Travis CI... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam closed pull request #48: [LIVY-41] Let users access sessions by session name
meisam closed pull request #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48 As this is a foreign pull request (from a fork), the diff has been sent to your commit mailing list, comm...@livy.apache.org This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name
meisam commented on a change in pull request #48: [LIVY-41] Let users access
sessions by session name
URL: https://github.com/apache/incubator-livy/pull/48#discussion_r140117359
##
File path: server/src/main/scala/org/apache/livy/sessions/SessionManager.scala
##
@@ -92,13 +94,29 @@ class SessionManager[S <: Session, R <: RecoveryMetadata :
ClassTag](
def register(session: S): S = {
info(s"Registering new session ${session.id}")
synchronized {
- sessions.put(session.id, session)
+ // in InteractiveSessionServlet.createSession() and
BatchSessionServlet.createSession(),
+ // Livy checks to make sure another session with the same name does not
exist already.
+ // This case should not happen rarely.
+ // already exists. But looking up a session name and adding it to the
set of existing sessions
+ // should happen atomically.
+ if (sessionsByName.contains(session.name)) {
+val msg = s"Session ${session.name} already exists!"
+error(msg)
+delete(session)
Review comment:
This is an example use case we have here. I'll ask Prabhu to provide more.
Team AAA has an ETL pipeline everyday at 1pm. The pipeline is a chain of
jobs that work on the same dataset. They want to create a session on Livy and
use it to run the pipeline. They also want to push the metrics and stats of
their pipeline to alerting and monitoring services.
This PR allows them to access the session, manage it, and monitor it with a
human readable name like team_aaa_daily_etl.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name
meisam commented on a change in pull request #48: [LIVY-41] Let users access
sessions by session name
URL: https://github.com/apache/incubator-livy/pull/48#discussion_r248096967
##
File path: server/src/main/scala/org/apache/livy/server/SessionServlet.scala
##
@@ -214,16 +214,23 @@ abstract class SessionServlet[S <: Session, R <:
RecoveryMetadata](
private def doWithSession(fn: (S => Any),
allowAll: Boolean,
checkFn: Option[(String, HttpServletRequest) => Boolean]): Any = {
-val sessionId = params("id").toInt
-sessionManager.get(sessionId) match {
+val idOrNameParam: String = params("id")
+val session = if (idOrNameParam.forall(_.isDigit)) {
Review comment:
I changed this to force session names to have at least one non-digit
character.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name
meisam commented on a change in pull request #48: [LIVY-41] Let users access
sessions by session name
URL: https://github.com/apache/incubator-livy/pull/48#discussion_r248097005
##
File path: server/src/main/scala/org/apache/livy/sessions/SessionManager.scala
##
@@ -92,13 +94,27 @@ class SessionManager[S <: Session, R <: RecoveryMetadata :
ClassTag](
def register(session: S): S = {
info(s"Registering new session ${session.id}")
synchronized {
+ session.name.map { sessionName =>
Review comment:
Addressed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name
meisam commented on a change in pull request #48: [LIVY-41] Let users access
sessions by session name
URL: https://github.com/apache/incubator-livy/pull/48#discussion_r248097902
##
File path:
server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
##
@@ -451,6 +403,64 @@ class InteractiveSession(
}
}
+ override def start(): Unit = {
+app = mockApp.orElse {
+ val driverProcess = client.flatMap { c => Option(c.getDriverProcess) }
+.map(new LineBufferedProcess(_,
livyConf.getInt(LivyConf.SPARK_LOGS_SIZE)))
+ driverProcess.map { _ => SparkApp.create(appTag, appId, driverProcess,
livyConf, Some(this))
+ }
+}
+
+if (client.isEmpty) {
+ transition(Dead())
+ val msg = s"Cannot recover interactive session $id because its RSCDriver
URI is unknown."
+ info(msg)
+ sessionLog = IndexedSeq(msg)
+} else {
+ val uriFuture = Future { client.get.getServerUri.get() }
+
+ uriFuture onSuccess { case url =>
Review comment:
Addressed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name
meisam commented on a change in pull request #48: [LIVY-41] Let users access
sessions by session name
URL: https://github.com/apache/incubator-livy/pull/48#discussion_r248100093
##
File path:
server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala
##
@@ -53,8 +53,23 @@ class InteractiveSessionServlet(
override protected def createSession(req: HttpServletRequest):
InteractiveSession = {
val createRequest = bodyAs[CreateInteractiveRequest](req)
val proxyUser = checkImpersonation(createRequest.proxyUser, req)
+val sessionId: Int = sessionManager.nextId()
+val sessionName: String = createRequest.name match {
+ case Some(name) if sessionManager.get(name).isEmpty =>
Review comment:
It is refactored into one place now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name
meisam commented on a change in pull request #48: [LIVY-41] Let users access
sessions by session name
URL: https://github.com/apache/incubator-livy/pull/48#discussion_r248099616
##
File path:
integration-test/src/test/scala/org/apache/livy/test/InteractiveIT.scala
##
@@ -191,7 +191,7 @@ class InteractiveIT extends BaseIntegrationTestSuite {
waitForIdle: Boolean = true,
heartbeatTimeoutInSecond: Int = 0)
(f: (LivyRestClient#InteractiveSession) => R): R = {
-withSession(livyClient.startSession(kind, sparkConf,
heartbeatTimeoutInSecond)) { s =>
+withSession(livyClient.startSession(null, kind, sparkConf,
heartbeatTimeoutInSecond)) { s =>
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name
meisam commented on a change in pull request #48: [LIVY-41] Let users access
sessions by session name
URL: https://github.com/apache/incubator-livy/pull/48#discussion_r248099484
##
File path:
server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
##
@@ -451,6 +403,64 @@ class InteractiveSession(
}
}
+ override def start(): Unit = {
+app = mockApp.orElse {
+ val driverProcess = client.flatMap { c => Option(c.getDriverProcess) }
+.map(new LineBufferedProcess(_,
livyConf.getInt(LivyConf.SPARK_LOGS_SIZE)))
+ driverProcess.map { _ => SparkApp.create(appTag, appId, driverProcess,
livyConf, Some(this))
+ }
Review comment:
In these lines, I only changed the indentation. I think formatting issues
should be handled in a separate PR if needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] meisam commented on issue #138: [LIVY-547]do not check expire when session in state busy
meisam commented on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-454547943 Please make this a configurable option. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name
meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48#discussion_r248099838 ## File path: server/src/main/scala/org/apache/livy/sessions/SessionManager.scala ## @@ -73,6 +73,8 @@ class SessionManager[S <: Session, R <: RecoveryMetadata : ClassTag]( protected[this] final val idCounter = new AtomicInteger(0) protected[this] final val sessions = mutable.LinkedHashMap[Int, S]() + private[this] final val sessionsByName = mutable.Map[String, S]() Review comment: Addressed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam commented on a change in pull request #48: [LIVY-41] Let users access sessions by session name
meisam commented on a change in pull request #48: [LIVY-41] Let users access
sessions by session name
URL: https://github.com/apache/incubator-livy/pull/48#discussion_r24814
##
File path:
server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala
##
@@ -169,5 +174,5 @@ class BatchSession(
override def infoChanged(appInfo: AppInfo): Unit = { this.appInfo = appInfo }
override def recoveryMetadata: RecoveryMetadata =
-BatchRecoveryMetadata(id, appId, appTag, owner, proxyUser)
+BatchRecoveryMetadata(id, name, appId, appTag, owner, proxyUser)
Review comment:
I'll add test cases to confirm that Livy can recover from old data.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] meisam opened a new pull request #139: [MINOR] Fix instructions on how to build Livy.
meisam opened a new pull request #139: [MINOR] Fix instructions on how to build Livy. URL: https://github.com/apache/incubator-livy/pull/139 ## What changes were proposed in this pull request? Update the README section with instructions on building Livy ## How was this patch tested? - Checked the rendered markdown. - Executed the instructions. ## What remains to be done? This instruction should be updated in the documentation each time Livy's repository name changes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #139: [MINOR] Fix instructions on how to build Livy.
codecov-io commented on issue #139: [MINOR] Fix instructions on how to build Livy. URL: https://github.com/apache/incubator-livy/pull/139#issuecomment-454611468 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=h1) Report > Merging [#139](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.23%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #139 +/- ## + Coverage 68.34% 68.57% +0.23% + Complexity 895 893 -2 Files 100 100 Lines 5604 5604 Branches840 840 + Hits 3830 3843 +13 + Misses 1225 1211 -14 - Partials549 550 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `60.69% <0%> (+2.31%)` | `14% <0%> (ø)` | :arrow_down: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+2.48%)` | `20% <0%> (+1%)` | :arrow_up: | | [...c/main/scala/org/apache/livy/repl/ReplDriver.scala](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9SZXBsRHJpdmVyLnNjYWxh) | `33.33% <0%> (+2.56%)` | `7% <0%> (ø)` | :arrow_down: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `52.94% <0%> (+2.94%)` | `7% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=footer). Last update [02550f7...0ccc821](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #139: [MINOR] Fix instructions on how to build Livy.
codecov-io edited a comment on issue #139: [MINOR] Fix instructions on how to build Livy. URL: https://github.com/apache/incubator-livy/pull/139#issuecomment-454611468 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=h1) Report > Merging [#139](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.23%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #139 +/- ## + Coverage 68.34% 68.57% +0.23% + Complexity 895 893 -2 Files 100 100 Lines 5604 5604 Branches840 840 + Hits 3830 3843 +13 + Misses 1225 1211 -14 - Partials549 550 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `60.69% <0%> (+2.31%)` | `14% <0%> (ø)` | :arrow_down: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+2.48%)` | `20% <0%> (+1%)` | :arrow_up: | | [...c/main/scala/org/apache/livy/repl/ReplDriver.scala](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9SZXBsRHJpdmVyLnNjYWxh) | `33.33% <0%> (+2.56%)` | `7% <0%> (ø)` | :arrow_down: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/139/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `52.94% <0%> (+2.94%)` | `7% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=footer). Last update [02550f7...0ccc821](https://codecov.io/gh/apache/incubator-livy/pull/139?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgaido91 commented on issue #139: [MINOR] Fix instructions on how to build Livy.
mgaido91 commented on issue #139: [MINOR] Fix instructions on how to build Livy. URL: https://github.com/apache/incubator-livy/pull/139#issuecomment-456209741 The change seems fine to me, have you checked that all the other part of the README are accurate? I think we should update if there are other imprecisions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #142: LIVY-552 Add JWTFilter to validate incoming JWT tokens
codecov-io commented on issue #142: LIVY-552 Add JWTFilter to validate incoming JWT tokens URL: https://github.com/apache/incubator-livy/pull/142#issuecomment-456572678 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/142?src=pr=h1) Report > Merging [#142](https://codecov.io/gh/apache/incubator-livy/pull/142?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.28%`. > The diff coverage is `71.87%`. [](https://codecov.io/gh/apache/incubator-livy/pull/142?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #142 +/- ## + Coverage 68.34% 68.62% +0.28% - Complexity 895 913 +18 Files 100 103 +3 Lines 5604 5664 +60 Branches840 848 +8 + Hits 3830 3887 +57 + Misses 1225 1220 -5 - Partials549 557 +8 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/142?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/142/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `37.11% <0%> (-0.59%)` | `11 <0> (ø)` | | | [...rver/src/main/scala/org/apache/livy/LivyConf.scala](https://codecov.io/gh/apache/incubator-livy/pull/142/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9MaXZ5Q29uZi5zY2FsYQ==) | `95.91% <100%> (+0.06%)` | `21 <0> (ø)` | :arrow_down: | | [...n/scala/org/apache/livy/server/jwt/JWTFilter.scala](https://codecov.io/gh/apache/incubator-livy/pull/142/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvand0L0pXVEZpbHRlci5zY2FsYQ==) | `66.66% <66.66%> (ø)` | `6 <6> (?)` | | | [...cala/org/apache/livy/server/jwt/JWTValidator.scala](https://codecov.io/gh/apache/incubator-livy/pull/142/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvand0L0pXVFZhbGlkYXRvci5zY2FsYQ==) | `85.71% <85.71%> (ø)` | `9 <9> (?)` | | | [...g/apache/livy/server/jwt/JWSVerifierProvider.scala](https://codecov.io/gh/apache/incubator-livy/pull/142/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvand0L0pXU1ZlcmlmaWVyUHJvdmlkZXIuc2NhbGE=) | `93.75% <93.75%> (ø)` | `5 <5> (?)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/142/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `77.96% <0%> (-1.28%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/142/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/incubator-livy/pull/142/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/142?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/142?src=pr=footer). Last update [02550f7...101a6e2](https://codecov.io/gh/apache/incubator-livy/pull/142?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam opened a new pull request #48: [LIVY-41] Let users access sessions by session name
meisam opened a new pull request #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48 This commit enables Livy users to access sessions either by names or by auto-generated sessiond id's. It also prevents users from creating sessions that have the same name. This commit keeps API change minimal. These are places that API change is needed: - `Session` and its sub-classes adds a new field, `name`. - `RecoveryMetadata` and its sub-classes adds a new field, `name`. - `SessionManager` adds a new method `getSession(name: String)` which looks sessions up by name. A more clean implementation would change the signature of `SessionManager.register` so it returns a proper container around the session value to determine if it failed to register the given session. For example, ```Scala def register(session S): Either[S, Throwable] ``` Task-url: https://issues.apache.org/jira/browse/LIVY-41 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vanzin commented on a change in pull request #119: [LIVY-527] added autocompletion api and implementation for ipython
vanzin commented on a change in pull request #119: [LIVY-527] added autocompletion api and implementation for ipython URL: https://github.com/apache/incubator-livy/pull/119#discussion_r249967903 ## File path: repl/src/main/resources/fake_shell.py ## @@ -49,6 +49,22 @@ TOP_FRAME_REGEX = re.compile(r'\s*File "".*in ') +def get_completer(): +try: +__IPYTHON__ +from IPython.core.completer import IPCompleter +ip = get_ipython() +ip_completer = IPCompleter(ip,global_namespace=global_dict) +global_dict['ip_completer'] = ip_completer Review comment: I see what you mean (`ip_completer` vs. `p_completer`). I think it would be better to keep a single variable for the completer, and a boolean that says whether to use the IPython API. But my main question was: do they need to be stored in `global_dict` or can they just be a global variable in this module instead? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name
codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48#issuecomment-330704048 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=h1) Report > Merging [#48](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.38%`. > The diff coverage is `78.16%`. [](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #48 +/- ## + Coverage 68.34% 68.72% +0.38% - Complexity 895 900 +5 Files 100 100 Lines 5604 5647 +43 Branches840 846 +6 + Hits 3830 3881 +51 + Misses 1225 1213 -12 - Partials549 553 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `68.96% <100%> (+1.89%)` | `17 <4> (+1)` | :arrow_up: | | [...apache/livy/server/batch/BatchSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `90.32% <100%> (+1.86%)` | `3 <0> (ø)` | :arrow_down: | | [...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh) | `86.31% <100%> (+0.92%)` | `14 <6> (+1)` | :arrow_up: | | [...va/org/apache/livy/client/common/HttpMessages.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL0h0dHBNZXNzYWdlcy5qYXZh) | `95.23% <100%> (+0.11%)` | `0 <0> (ø)` | :arrow_down: | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `66.93% <100%> (+0.54%)` | `6 <0> (ø)` | :arrow_down: | | [.../main/scala/org/apache/livy/sessions/Session.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uLnNjYWxh) | `70.53% <14.28%> (-3.05%)` | `18 <3> (+2)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `69.04% <69.44%> (+0.68%)` | `44 <4> (+1)` | :arrow_up: | | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `82.6% <81.81%> (-0.11%)` | `24 <1> (+1)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `38.21% <0%> (+0.52%)` | `11% <0%> (ø)` | :arrow_down: | | ... and [5 more](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=footer). Last update
[GitHub] codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name
codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48#issuecomment-330704048 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=h1) Report > Merging [#48](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.38%`. > The diff coverage is `78.16%`. [](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #48 +/- ## + Coverage 68.34% 68.72% +0.38% - Complexity 895 900 +5 Files 100 100 Lines 5604 5647 +43 Branches840 846 +6 + Hits 3830 3881 +51 + Misses 1225 1213 -12 - Partials549 553 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `68.96% <100%> (+1.89%)` | `17 <4> (+1)` | :arrow_up: | | [...apache/livy/server/batch/BatchSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `90.32% <100%> (+1.86%)` | `3 <0> (ø)` | :arrow_down: | | [...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh) | `86.31% <100%> (+0.92%)` | `14 <6> (+1)` | :arrow_up: | | [...va/org/apache/livy/client/common/HttpMessages.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL0h0dHBNZXNzYWdlcy5qYXZh) | `95.23% <100%> (+0.11%)` | `0 <0> (ø)` | :arrow_down: | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `66.93% <100%> (+0.54%)` | `6 <0> (ø)` | :arrow_down: | | [.../main/scala/org/apache/livy/sessions/Session.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uLnNjYWxh) | `70.53% <14.28%> (-3.05%)` | `18 <3> (+2)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `69.04% <69.44%> (+0.68%)` | `44 <4> (+1)` | :arrow_up: | | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `82.6% <81.81%> (-0.11%)` | `24 <1> (+1)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `38.21% <0%> (+0.52%)` | `11% <0%> (ø)` | :arrow_down: | | ... and [5 more](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=footer). Last update
[GitHub] z-york opened a new pull request #142: LIVY-552 Add JWTFilter to validate incoming JWT tokens
z-york opened a new pull request #142: LIVY-552 Add JWTFilter to validate incoming JWT tokens URL: https://github.com/apache/incubator-livy/pull/142 ## What changes were proposed in this pull request? This PR adds JWT Authentication to Livy via a new filter, JWTFilter that validates the JWT and accepts or rejects the request. https://issues.apache.org/jira/browse/LIVY-552 (Please fill in changes proposed in this fix) (Include a link to the associated JIRA and make sure to add a link to this pr on the JIRA as well) ## How was this patch tested? Unit Tests passed. Tested manually as well. (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review https://livy.incubator.apache.org/community/ before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam closed pull request #48: [LIVY-41] Let users access sessions by session name
meisam closed pull request #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam opened a new pull request #48: [LIVY-41] Let users access sessions by session name
meisam opened a new pull request #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48 This commit enables Livy users to access sessions either by names or by auto-generated sessiond id's. It also prevents users from creating sessions that have the same name. This commit keeps API change minimal. These are places that API change is needed: - `Session` and its sub-classes adds a new field, `name`. - `RecoveryMetadata` and its sub-classes adds a new field, `name`. - `SessionManager` adds a new method `getSession(name: String)` which looks sessions up by name. A more clean implementation would change the signature of `SessionManager.register` so it returns a proper container around the session value to determine if it failed to register the given session. For example, ```Scala def register(session S): Either[S, Throwable] ``` Task-url: https://issues.apache.org/jira/browse/LIVY-41 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vanzin commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support
vanzin commented on a change in pull request #141: [LIVY-551] Add "doAs"
impersonation support
URL: https://github.com/apache/incubator-livy/pull/141#discussion_r249962561
##
File path: server/src/main/scala/org/apache/livy/server/AccessManager.scala
##
@@ -97,47 +99,71 @@ private[livy] class AccessManager(conf: LivyConf) extends
Logging {
*/
def isAccessControlOn: Boolean = aclsOn
+ def getRequestUser(request: HttpServletRequest): String = {
+request.getRemoteUser
+ }
+
+ def getImpersonatedUser(request: HttpServletRequest): Option[String] = {
+val impersonatedUser = Option(request.getParameter("doAs"))
+impersonatedUser.filter(checkImpersonation(request, _))
+ }
+
+ def getEffectiveUser(request: HttpServletRequest): String = {
+val requestUser = getRequestUser(request)
+val impersonatedUser = getImpersonatedUser(request)
+impersonatedUser.getOrElse(requestUser)
+ }
+
/**
* Checks that the requesting user can impersonate the target user.
* If the user does not have permission to impersonate, then throws an
`AccessControlException`.
- *
- * @return The user that should be impersonated. That can be the target user
if defined, the
- * request's user - which may not be defined - otherwise, or `None`
if impersonation is
- * disabled.
*/
- def checkImpersonation(
- target: Option[String],
- requestUser: String,
- livyConf: LivyConf): Option[String] = {
-if (livyConf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
- if (!target.forall(hasSuperAccess(_, requestUser))) {
-throw new AccessControlException(
- s"User '$requestUser' not allowed to impersonate '$target'.")
+ def checkImpersonation(request: HttpServletRequest, impersonatedUser:
String): Boolean = {
+if (conf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
+ if (hasSuperAccess(request, impersonatedUser) || checkProxyUser(request,
impersonatedUser)) {
+return true
}
- target.orElse(Option(requestUser))
-} else {
- None
+ val requestUser = getRequestUser(request)
+ throw new AccessControlException(
+s"User '$requestUser' not allowed to impersonate '$impersonatedUser'.")
}
+false
+ }
+
+ def checkProxyUser(request: HttpServletRequest, impersonatedUser: String):
Boolean = {
+val proxyUser = getRequestUser(request)
+val remoteHost = request.getRemoteHost
+val allowedHosts = conf.hadoopConf.get("hadoop.proxyuser." + proxyUser +
".hosts")
Review comment:
It doesn't feel right to look at Hadoop's configuration when deciding
whether impersonation is allowed in Livy. They'll probably end up being
different.
e.g. Hadoop's configuration could allow `livy` to impersonate anybody and
that's it. But Livy could allow `admin` to impersonate other people when
performing Livy operations.
IIRC the current Livy behavior is "superusers can impersonate". What are you
trying to achieve here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vanzin commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support
vanzin commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support URL: https://github.com/apache/incubator-livy/pull/141#discussion_r249964784 ## File path: server/src/main/scala/org/apache/livy/server/SessionServlet.scala ## @@ -158,7 +158,21 @@ abstract class SessionServlet[S <: Session, R <: RecoveryMetadata]( /** * Returns the remote user for the given request. Separate method so that tests can override it. */ - protected def remoteUser(req: HttpServletRequest): String = req.getRemoteUser() + protected def getOwner(req: HttpServletRequest): String = accessManager.getRequestUser(req) + + protected def checkBodyProxyUser(owner: String, bodyProxyUser: Option[String], Review comment: One parameter per line in multi-line declarations. Also it's not clear from the name what this method is doing. What is "body proxy user"? Or "proxyUser body" mentioned below? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgaido91 commented on issue #139: [MINOR] Fix instructions on how to build Livy.
mgaido91 commented on issue #139: [MINOR] Fix instructions on how to build Livy. URL: https://github.com/apache/incubator-livy/pull/139#issuecomment-456716191 LGTM, if I won't see other comments in couple of days, I will merge this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wangqiaoshi commented on issue #119: [LIVY-527] added autocompletion api and implementation for ipython
wangqiaoshi commented on issue #119: [LIVY-527] added autocompletion api and implementation for ipython URL: https://github.com/apache/incubator-livy/pull/119#issuecomment-455196868 > ping @wangqiaoshi - that's a great improvement! > do you mind fixing those minor issues after the last review > thank you! ok This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy
meisam edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-455352091 > or may you share me some counter-cases If the number of jobs running are small and can be monitored manually, it makes sense to keep busy jobs. But in a system with hundreds of users and thousands of jobs running each day, it is impractical to check the status of each job manually. In many cases, users submit jobs on a daily/hourly basis though an enterprise scheduler. In this case, if a job takes too long to run, it is better to fail it and figure out why it took so long. Also, users have the option to keep the jobs alive by reset the last activity time for a job though the `sessions/:id/connect` endpoint. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name
codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48#issuecomment-330704048 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=h1) Report > Merging [#48](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.34%`. > The diff coverage is `78.16%`. [](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #48 +/- ## + Coverage 68.34% 68.69% +0.34% - Complexity 895 900 +5 Files 100 100 Lines 5604 5647 +43 Branches840 846 +6 + Hits 3830 3879 +49 + Misses 1225 1215 -10 - Partials549 553 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `68.96% <100%> (+1.89%)` | `17 <4> (+1)` | :arrow_up: | | [...apache/livy/server/batch/BatchSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `90.32% <100%> (+1.86%)` | `3 <0> (ø)` | :arrow_down: | | [...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh) | `86.31% <100%> (+0.92%)` | `14 <6> (+1)` | :arrow_up: | | [...va/org/apache/livy/client/common/HttpMessages.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL0h0dHBNZXNzYWdlcy5qYXZh) | `95.23% <100%> (+0.11%)` | `0 <0> (ø)` | :arrow_down: | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `66.93% <100%> (+0.54%)` | `6 <0> (ø)` | :arrow_down: | | [.../main/scala/org/apache/livy/sessions/Session.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uLnNjYWxh) | `70.53% <14.28%> (-3.05%)` | `18 <3> (+2)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `69.04% <69.44%> (+0.68%)` | `44 <4> (+1)` | :arrow_up: | | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `82.6% <81.81%> (-0.11%)` | `24 <1> (+1)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=footer). Last update
[GitHub] codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name
codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48#issuecomment-330704048 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=h1) Report > Merging [#48](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.34%`. > The diff coverage is `78.16%`. [](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #48 +/- ## + Coverage 68.34% 68.69% +0.34% - Complexity 895 900 +5 Files 100 100 Lines 5604 5647 +43 Branches840 846 +6 + Hits 3830 3879 +49 + Misses 1225 1215 -10 - Partials549 553 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `68.96% <100%> (+1.89%)` | `17 <4> (+1)` | :arrow_up: | | [...apache/livy/server/batch/BatchSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `90.32% <100%> (+1.86%)` | `3 <0> (ø)` | :arrow_down: | | [...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh) | `86.31% <100%> (+0.92%)` | `14 <6> (+1)` | :arrow_up: | | [...va/org/apache/livy/client/common/HttpMessages.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL0h0dHBNZXNzYWdlcy5qYXZh) | `95.23% <100%> (+0.11%)` | `0 <0> (ø)` | :arrow_down: | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `66.93% <100%> (+0.54%)` | `6 <0> (ø)` | :arrow_down: | | [.../main/scala/org/apache/livy/sessions/Session.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uLnNjYWxh) | `70.53% <14.28%> (-3.05%)` | `18 <3> (+2)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `69.04% <69.44%> (+0.68%)` | `44 <4> (+1)` | :arrow_up: | | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `82.6% <81.81%> (-0.11%)` | `24 <1> (+1)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=footer). Last update
[GitHub] risdenk commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support
risdenk commented on a change in pull request #141: [LIVY-551] Add "doAs"
impersonation support
URL: https://github.com/apache/incubator-livy/pull/141#discussion_r250221878
##
File path: server/src/main/scala/org/apache/livy/server/AccessManager.scala
##
@@ -97,47 +99,71 @@ private[livy] class AccessManager(conf: LivyConf) extends
Logging {
*/
def isAccessControlOn: Boolean = aclsOn
+ def getRequestUser(request: HttpServletRequest): String = {
+request.getRemoteUser
+ }
+
+ def getImpersonatedUser(request: HttpServletRequest): Option[String] = {
+val impersonatedUser = Option(request.getParameter("doAs"))
+impersonatedUser.filter(checkImpersonation(request, _))
+ }
+
+ def getEffectiveUser(request: HttpServletRequest): String = {
+val requestUser = getRequestUser(request)
+val impersonatedUser = getImpersonatedUser(request)
+impersonatedUser.getOrElse(requestUser)
+ }
+
/**
* Checks that the requesting user can impersonate the target user.
* If the user does not have permission to impersonate, then throws an
`AccessControlException`.
- *
- * @return The user that should be impersonated. That can be the target user
if defined, the
- * request's user - which may not be defined - otherwise, or `None`
if impersonation is
- * disabled.
*/
- def checkImpersonation(
- target: Option[String],
- requestUser: String,
- livyConf: LivyConf): Option[String] = {
-if (livyConf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
- if (!target.forall(hasSuperAccess(_, requestUser))) {
-throw new AccessControlException(
- s"User '$requestUser' not allowed to impersonate '$target'.")
+ def checkImpersonation(request: HttpServletRequest, impersonatedUser:
String): Boolean = {
+if (conf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
+ if (hasSuperAccess(request, impersonatedUser) || checkProxyUser(request,
impersonatedUser)) {
+return true
}
- target.orElse(Option(requestUser))
-} else {
- None
+ val requestUser = getRequestUser(request)
+ throw new AccessControlException(
+s"User '$requestUser' not allowed to impersonate '$impersonatedUser'.")
}
+false
+ }
+
+ def checkProxyUser(request: HttpServletRequest, impersonatedUser: String):
Boolean = {
+val proxyUser = getRequestUser(request)
+val remoteHost = request.getRemoteHost
+val allowedHosts = conf.hadoopConf.get("hadoop.proxyuser." + proxyUser +
".hosts")
Review comment:
The idea was to model after
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html#Configurations
and how a bunch of other Hadoop ecosystem components work.
Superusers can impersonate anyone yes. There should be a limit as to who can
impersonate who. There should also be a limit from where superuser requests can
come from.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] risdenk commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support
risdenk commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support URL: https://github.com/apache/incubator-livy/pull/141#discussion_r250222080 ## File path: server/src/main/scala/org/apache/livy/server/SessionServlet.scala ## @@ -158,7 +158,21 @@ abstract class SessionServlet[S <: Session, R <: RecoveryMetadata]( /** * Returns the remote user for the given request. Separate method so that tests can override it. */ - protected def remoteUser(req: HttpServletRequest): String = req.getRemoteUser() + protected def getOwner(req: HttpServletRequest): String = accessManager.getRequestUser(req) + + protected def checkBodyProxyUser(owner: String, bodyProxyUser: Option[String], Review comment: I'll add more comments here to explain what this means. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vanzin opened a new pull request #137: Update Livy repo URLs.
vanzin opened a new pull request #137: Update Livy repo URLs. URL: https://github.com/apache/incubator-livy/pull/137 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit closed pull request #137: Update Livy repo URLs.
asfgit closed pull request #137: Update Livy repo URLs. URL: https://github.com/apache/incubator-livy/pull/137 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/dev/release-build.sh b/dev/release-build.sh index 6b882184..9a721641 100755 --- a/dev/release-build.sh +++ b/dev/release-build.sh @@ -96,7 +96,7 @@ rm -rf release-staging mkdir release-staging cd release-staging -git clone https://git-wip-us.apache.org/repos/asf/incubator-livy.git +git clone https://gitbox.apache.org/repos/asf/incubator-livy.git cd incubator-livy git checkout $GIT_REF git_hash=`git rev-parse --short HEAD` diff --git a/pom.xml b/pom.xml index 050bd3d0..2177134f 100644 --- a/pom.xml +++ b/pom.xml @@ -44,7 +44,7 @@ scm:git:g...@github.com:apache/incubator-livy.git - scm:git:https://git-wip-us.apache.org/repos/asf/incubator-livy.git + scm:git:https://gitbox.apache.org/repos/asf/incubator-livy.git scm:git:g...@github.com:apache/incubator-livy.git HEAD This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vanzin commented on issue #137: Update Livy repo URLs.
vanzin commented on issue #137: Update Livy repo URLs. URL: https://github.com/apache/incubator-livy/pull/137#issuecomment-452117135 Merging (failure is unrelated). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy
codecov-io edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-453984602 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=h1) Report > Merging [#138](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.24%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #138 +/- ## + Coverage 68.34% 68.58% +0.24% Complexity 895 895 Files 100 100 Lines 5604 5606 +2 Branches840 841 +1 + Hits 3830 3845 +15 + Misses 1225 1211 -14 - Partials549 550 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `80.72% <0%> (-2%)` | `23 <0> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `60.69% <0%> (+2.31%)` | `14% <0%> (ø)` | :arrow_down: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+2.48%)` | `20% <0%> (+1%)` | :arrow_up: | | [...in/java/org/apache/livy/rsc/driver/JobWrapper.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iV3JhcHBlci5qYXZh) | `88.57% <0%> (+5.71%)` | `9% <0%> (+1%)` | :arrow_up: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `55.88% <0%> (+5.88%)` | `7% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=footer). Last update [02550f7...768dfba](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy
codecov-io edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-453984602 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=h1) Report > Merging [#138](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.24%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #138 +/- ## + Coverage 68.34% 68.58% +0.24% Complexity 895 895 Files 100 100 Lines 5604 5606 +2 Branches840 841 +1 + Hits 3830 3845 +15 + Misses 1225 1211 -14 - Partials549 550 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `80.72% <0%> (-2%)` | `23 <0> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `60.69% <0%> (+2.31%)` | `14% <0%> (ø)` | :arrow_down: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+2.48%)` | `20% <0%> (+1%)` | :arrow_up: | | [...in/java/org/apache/livy/rsc/driver/JobWrapper.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iV3JhcHBlci5qYXZh) | `88.57% <0%> (+5.71%)` | `9% <0%> (+1%)` | :arrow_up: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `55.88% <0%> (+5.88%)` | `7% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=footer). Last update [02550f7...768dfba](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy
codecov-io edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-453984602 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=h1) Report > Merging [#138](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.24%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #138 +/- ## + Coverage 68.34% 68.58% +0.24% Complexity 895 895 Files 100 100 Lines 5604 5606 +2 Branches840 841 +1 + Hits 3830 3845 +15 + Misses 1225 1211 -14 - Partials549 550 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `80.72% <0%> (-2%)` | `23 <0> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `60.69% <0%> (+2.31%)` | `14% <0%> (ø)` | :arrow_down: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+2.48%)` | `20% <0%> (+1%)` | :arrow_up: | | [...in/java/org/apache/livy/rsc/driver/JobWrapper.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iV3JhcHBlci5qYXZh) | `88.57% <0%> (+5.71%)` | `9% <0%> (+1%)` | :arrow_up: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `55.88% <0%> (+5.88%)` | `7% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=footer). Last update [02550f7...768dfba](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Jassy1994 opened a new pull request #138: [LIVY-547]do not check expire when session in state busy
Jassy1994 opened a new pull request #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138 ## What changes were proposed in this pull request? add a logic in check expired function to ensure interactiveSession will not expired in state busy。 ## How was this patch tested? set livy.server.session.timeout to a short interval and submit a job by interactiveSession this session will not be recycled in state busy。 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #138: [LIVY-547]do not check expire when session in state busy
codecov-io commented on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-453984602 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=h1) Report > Merging [#138](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.26%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#138 +/- ## === + Coverage 68.34% 68.6% +0.26% - Complexity 895 896 +1 === Files 100 100 Lines 56045606 +2 Branches840 841 +1 === + Hits 38303846 +16 + Misses 12251209 -16 - Partials549 551 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `83.13% <100%> (+0.41%)` | `23 <0> (ø)` | :arrow_down: | | [.../scala/org/apache/livy/sessions/SessionState.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvU2Vzc2lvblN0YXRlLnNjYWxh) | `58.33% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `79.23% <0%> (ø)` | `42% <0%> (ø)` | :arrow_down: | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `60.69% <0%> (+2.31%)` | `14% <0%> (ø)` | :arrow_down: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+2.48%)` | `20% <0%> (+1%)` | :arrow_up: | | [...c/main/scala/org/apache/livy/repl/ReplDriver.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9SZXBsRHJpdmVyLnNjYWxh) | `33.33% <0%> (+2.56%)` | `7% <0%> (ø)` | :arrow_down: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `55.88% <0%> (+5.88%)` | `7% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=footer). Last update [02550f7...768dfba](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy
codecov-io edited a comment on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-453984602 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=h1) Report > Merging [#138](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.26%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#138 +/- ## === + Coverage 68.34% 68.6% +0.26% - Complexity 895 896 +1 === Files 100 100 Lines 56045606 +2 Branches840 841 +1 === + Hits 38303846 +16 + Misses 12251209 -16 - Partials549 551 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `83.13% <100%> (+0.41%)` | `23 <0> (ø)` | :arrow_down: | | [.../scala/org/apache/livy/sessions/SessionState.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvU2Vzc2lvblN0YXRlLnNjYWxh) | `58.33% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `79.23% <0%> (ø)` | `42% <0%> (ø)` | :arrow_down: | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `60.69% <0%> (+2.31%)` | `14% <0%> (ø)` | :arrow_down: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+2.48%)` | `20% <0%> (+1%)` | :arrow_up: | | [...c/main/scala/org/apache/livy/repl/ReplDriver.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9SZXBsRHJpdmVyLnNjYWxh) | `33.33% <0%> (+2.56%)` | `7% <0%> (ø)` | :arrow_down: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/138/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `55.88% <0%> (+5.88%)` | `7% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=footer). Last update [02550f7...768dfba](https://codecov.io/gh/apache/incubator-livy/pull/138?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgaido91 commented on issue #138: [LIVY-547]do not check expire when session in state busy
mgaido91 commented on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-457342220 Yes, polling refreshes the expiry date. The point is that maybe someone runs a job which he knows runs for hours and then he doesn't poll for it before the expiry date... And the config name is ambiguous, so this confusion can arise. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vanzin commented on issue #138: [LIVY-547]do not check expire when session in state busy
vanzin commented on issue #138: [LIVY-547]do not check expire when session in state busy URL: https://github.com/apache/incubator-livy/pull/138#issuecomment-457338491 I don't remember exactly how the clients monitor interactive sessions, but doesn't a client poll for the result of a statement? Doesn't that count as updating the session for activity? Although I guess that would still not account for the "fire a job and go out for a few days" use case. It sounds like a separate config is the only way to cover all user cases (e.g. a "busy timeout", which by default maintains the current behavior, and can be set to stop the session if it's been busy for a certain period). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name
codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48#issuecomment-330704048 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=h1) Report > Merging [#48](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.36%`. > The diff coverage is `80%`. [](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #48 +/- ## + Coverage 68.34% 68.71% +0.36% - Complexity 895 902 +7 Files 100 100 Lines 5604 5644 +40 Branches840 847 +7 + Hits 3830 3878 +48 + Misses 1225 1211 -14 - Partials549 555 +6 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `68.96% <100%> (+1.89%)` | `17 <4> (+1)` | :arrow_up: | | [...apache/livy/server/batch/BatchSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `90.32% <100%> (+1.86%)` | `3 <0> (ø)` | :arrow_down: | | [...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh) | `86.17% <100%> (+0.77%)` | `14 <5> (+1)` | :arrow_up: | | [...va/org/apache/livy/client/common/HttpMessages.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL0h0dHBNZXNzYWdlcy5qYXZh) | `95.23% <100%> (+0.11%)` | `0 <0> (ø)` | :arrow_down: | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `66.93% <100%> (+0.54%)` | `6 <0> (ø)` | :arrow_down: | | [.../main/scala/org/apache/livy/sessions/Session.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uLnNjYWxh) | `72.56% <50%> (-1.02%)` | `20 <5> (+4)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `69.04% <71.05%> (+0.68%)` | `44 <4> (+1)` | :arrow_up: | | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `82.02% <75%> (-0.7%)` | `24 <1> (+1)` | | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `72.53% <0%> (-0.71%)` | `33% <0%> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | ... and [5 more](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=footer). Last update
[GitHub] codecov-io edited a comment on issue #141: [LIVY-551] Add "doAs" impersonation support
codecov-io edited a comment on issue #141: [LIVY-551] Add "doAs" impersonation support URL: https://github.com/apache/incubator-livy/pull/141#issuecomment-457216469 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/141?src=pr=h1) Report > Merging [#141](https://codecov.io/gh/apache/incubator-livy/pull/141?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.17%`. > The diff coverage is `65.71%`. [](https://codecov.io/gh/apache/incubator-livy/pull/141?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #141 +/- ## + Coverage 68.34% 68.52% +0.17% - Complexity 895 896 +1 Files 100 100 Lines 5604 5620 +16 Branches840 847 +7 + Hits 3830 3851 +21 + Misses 1225 1213 -12 - Partials549 556 +7 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/141?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh) | `85.22% <ø> (-0.17%)` | `13 <0> (ø)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `68.25% <ø> (-0.11%)` | `43 <0> (ø)` | | | [.../main/scala/org/apache/livy/sessions/Session.scala](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uLnNjYWxh) | `73.58% <ø> (ø)` | `16 <0> (ø)` | :arrow_down: | | [...n/scala/org/apache/livy/server/AccessManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvQWNjZXNzTWFuYWdlci5zY2FsYQ==) | `76.47% <61.53%> (-4.39%)` | `43 <14> (-1)` | | | [...apache/livy/server/batch/BatchSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `86.2% <66.66%> (-2.26%)` | `3 <0> (ø)` | | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `66.4% <66.66%> (ø)` | `6 <0> (ø)` | :arrow_down: | | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `67.77% <70%> (+0.7%)` | `20 <5> (+4)` | :arrow_up: | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `37.17% <0%> (-0.53%)` | `11% <0%> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (+1.94%)` | `13% <0%> (ø)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/incubator-livy/pull/141/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/141?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/141?src=pr=footer). Last update [02550f7...b48bf12](https://codecov.io/gh/apache/incubator-livy/pull/141?src=pr=lastupdated). Read the [comment
[GitHub] codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name
codecov-io edited a comment on issue #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48#issuecomment-330704048 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=h1) Report > Merging [#48](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/02550f7919b7348b6a7270cf806e031670037b2f?src=pr=desc) will **increase** coverage by `0.36%`. > The diff coverage is `80%`. [](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #48 +/- ## + Coverage 68.34% 68.71% +0.36% - Complexity 895 902 +7 Files 100 100 Lines 5604 5644 +40 Branches840 847 +7 + Hits 3830 3878 +48 + Misses 1225 1211 -14 - Partials549 555 +6 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `68.96% <100%> (+1.89%)` | `17 <4> (+1)` | :arrow_up: | | [...apache/livy/server/batch/BatchSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `90.32% <100%> (+1.86%)` | `3 <0> (ø)` | :arrow_down: | | [...la/org/apache/livy/server/batch/BatchSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uLnNjYWxh) | `86.17% <100%> (+0.77%)` | `14 <5> (+1)` | :arrow_up: | | [...va/org/apache/livy/client/common/HttpMessages.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL0h0dHBNZXNzYWdlcy5qYXZh) | `95.23% <100%> (+0.11%)` | `0 <0> (ø)` | :arrow_down: | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `66.93% <100%> (+0.54%)` | `6 <0> (ø)` | :arrow_down: | | [.../main/scala/org/apache/livy/sessions/Session.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uLnNjYWxh) | `72.56% <50%> (-1.02%)` | `20 <5> (+4)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `69.04% <71.05%> (+0.68%)` | `44 <4> (+1)` | :arrow_up: | | [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `82.02% <75%> (-0.7%)` | `24 <1> (+1)` | | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `72.53% <0%> (-0.71%)` | `33% <0%> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (-0.43%)` | `41% <0%> (-1%)` | | | ... and [5 more](https://codecov.io/gh/apache/incubator-livy/pull/48/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/48?src=pr=footer). Last update
[GitHub] meisam commented on issue #139: [MINOR] Fix instructions on how to build Livy.
meisam commented on issue #139: [MINOR] Fix instructions on how to build Livy. URL: https://github.com/apache/incubator-livy/pull/139#issuecomment-456505631 > have you checked that all the other part of the README are accurate? I checked rest of the README file as well. I didn't see any other issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] meisam closed pull request #48: [LIVY-41] Let users access sessions by session name
meisam closed pull request #48: [LIVY-41] Let users access sessions by session name URL: https://github.com/apache/incubator-livy/pull/48 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgaido91 commented on issue #25: [MINOR] Updating contributors
mgaido91 commented on issue #25: [MINOR] Updating contributors URL: https://github.com/apache/incubator-livy-website/pull/25#issuecomment-458167328 cc @alex-the-man @ajbozarth This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgaido91 edited a comment on issue #25: [MINOR] Updating contributors
mgaido91 edited a comment on issue #25: [MINOR] Updating contributors URL: https://github.com/apache/incubator-livy-website/pull/25#issuecomment-458167328 cc @ajbozarth @alex-the-man This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgaido91 opened a new pull request #25: [MINOR] Updating contributors
mgaido91 opened a new pull request #25: [MINOR] Updating contributors URL: https://github.com/apache/incubator-livy-website/pull/25 Updating contributors list, adding new PPMC members and fixing a typo. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support
mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs"
impersonation support
URL: https://github.com/apache/incubator-livy/pull/141#discussion_r251421028
##
File path: server/src/main/scala/org/apache/livy/server/AccessManager.scala
##
@@ -98,46 +99,51 @@ private[livy] class AccessManager(conf: LivyConf) extends
Logging {
def isAccessControlOn: Boolean = aclsOn
/**
- * Checks that the requesting user can impersonate the target user.
- * If the user does not have permission to impersonate, then throws an
`AccessControlException`.
- *
- * @return The user that should be impersonated. That can be the target user
if defined, the
- * request's user - which may not be defined - otherwise, or `None`
if impersonation is
- * disabled.
+ * Checks that the request user can impersonate the target user.
+ * If impersonation is enabled and the user does not have permission to
impersonate
+ * then throws an `AccessControlException`. If impersonation is disabled
returns false
*/
def checkImpersonation(
- target: Option[String],
requestUser: String,
- livyConf: LivyConf): Option[String] = {
Review comment:
mmmh...now you are using the `conf` passed when building the
`AccessManager`, which may be different from the one which was used before this
patch. Despite this one seems a cleaner approach, I wonder if there may be some
behavioral changes due to this. cc @vanzin for his opinion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support
mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs"
impersonation support
URL: https://github.com/apache/incubator-livy/pull/141#discussion_r251422378
##
File path: server/src/main/scala/org/apache/livy/server/AccessManager.scala
##
@@ -98,46 +99,51 @@ private[livy] class AccessManager(conf: LivyConf) extends
Logging {
def isAccessControlOn: Boolean = aclsOn
/**
- * Checks that the requesting user can impersonate the target user.
- * If the user does not have permission to impersonate, then throws an
`AccessControlException`.
- *
- * @return The user that should be impersonated. That can be the target user
if defined, the
- * request's user - which may not be defined - otherwise, or `None`
if impersonation is
- * disabled.
+ * Checks that the request user can impersonate the target user.
+ * If impersonation is enabled and the user does not have permission to
impersonate
+ * then throws an `AccessControlException`. If impersonation is disabled
returns false
*/
def checkImpersonation(
- target: Option[String],
requestUser: String,
- livyConf: LivyConf): Option[String] = {
-if (livyConf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
- if (!target.forall(hasSuperAccess(_, requestUser))) {
-throw new AccessControlException(
- s"User '$requestUser' not allowed to impersonate '$target'.")
+ impersonatedUser: String): Boolean = {
+if (conf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
+ if (hasSuperAccess(requestUser, impersonatedUser)) {
+return true
}
- target.orElse(Option(requestUser))
-} else {
- None
+ throw new AccessControlException(
+s"User '$requestUser' not allowed to impersonate '$impersonatedUser'.")
}
+false
}
/**
- * Check that the requesting user has admin access to resources owned by the
given target user.
+ * Check that the request user has is able to impersonate the given user.
*/
- def hasSuperAccess(target: String, requestUser: String): Boolean = {
-requestUser == target || checkSuperUser(requestUser)
+ def hasSuperAccess(
+ requestUser: String,
+ impersonatedUser: String): Boolean = {
+requestUser == impersonatedUser || checkSuperUser(requestUser)
}
/**
- * Check that the request's user has modify access to resources owned by the
given target user.
+ * Check that the request user has modify access to the given session.
*/
- def hasModifyAccess(target: String, requestUser: String): Boolean = {
-requestUser == target || checkModifyPermissions(requestUser)
+ def hasModifyAccess(
+ session: Session,
+ effectiveUser: String): Boolean = {
+session.owner == effectiveUser ||
Review comment:
this is different from the previous implementation, why do we need this
change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support
mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs"
impersonation support
URL: https://github.com/apache/incubator-livy/pull/141#discussion_r251422686
##
File path: server/src/main/scala/org/apache/livy/server/SessionServlet.scala
##
@@ -156,9 +156,39 @@ abstract class SessionServlet[S <: Session, R <:
RecoveryMetadata](
}
/**
- * Returns the remote user for the given request. Separate method so that
tests can override it.
- */
- protected def remoteUser(req: HttpServletRequest): String =
req.getRemoteUser()
+* Gets the remote user from the given request
+*/
+ protected def getRemoteUser(request: HttpServletRequest): String = {
+request.getRemoteUser
+ }
+
+ protected def getImpersonatedUser(request: HttpServletRequest):
Option[String] = {
+val impersonatedUser = Option(request.getParameter("doAs"))
+
impersonatedUser.filter(accessManager.checkImpersonation(getRemoteUser(request),
_))
+ }
+
+
+ /**
+* Returns the proxy user as determined by the json request body or owner
if available.
+* This is necessary to preserve backwards compatibility prior to "doAs"
impersonation.
+*/
+ protected def getLegacyProxyUser(owner: String,
+ bodyProxyUser: Option[String]):
Option[String] = {
Review comment:
nit: indentation
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support
mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs"
impersonation support
URL: https://github.com/apache/incubator-livy/pull/141#discussion_r251421630
##
File path: server/src/main/scala/org/apache/livy/server/AccessManager.scala
##
@@ -98,46 +99,51 @@ private[livy] class AccessManager(conf: LivyConf) extends
Logging {
def isAccessControlOn: Boolean = aclsOn
/**
- * Checks that the requesting user can impersonate the target user.
- * If the user does not have permission to impersonate, then throws an
`AccessControlException`.
- *
- * @return The user that should be impersonated. That can be the target user
if defined, the
- * request's user - which may not be defined - otherwise, or `None`
if impersonation is
- * disabled.
+ * Checks that the request user can impersonate the target user.
+ * If impersonation is enabled and the user does not have permission to
impersonate
+ * then throws an `AccessControlException`. If impersonation is disabled
returns false
*/
def checkImpersonation(
- target: Option[String],
requestUser: String,
- livyConf: LivyConf): Option[String] = {
-if (livyConf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
- if (!target.forall(hasSuperAccess(_, requestUser))) {
-throw new AccessControlException(
- s"User '$requestUser' not allowed to impersonate '$target'.")
+ impersonatedUser: String): Boolean = {
+if (conf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
+ if (hasSuperAccess(requestUser, impersonatedUser)) {
+return true
}
- target.orElse(Option(requestUser))
-} else {
- None
+ throw new AccessControlException(
+s"User '$requestUser' not allowed to impersonate '$impersonatedUser'.")
}
+false
}
/**
- * Check that the requesting user has admin access to resources owned by the
given target user.
+ * Check that the request user has is able to impersonate the given user.
*/
- def hasSuperAccess(target: String, requestUser: String): Boolean = {
-requestUser == target || checkSuperUser(requestUser)
+ def hasSuperAccess(
Review comment:
nit: one line?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs" impersonation support
mgaido91 commented on a change in pull request #141: [LIVY-551] Add "doAs"
impersonation support
URL: https://github.com/apache/incubator-livy/pull/141#discussion_r251421684
##
File path: server/src/main/scala/org/apache/livy/server/AccessManager.scala
##
@@ -98,46 +99,51 @@ private[livy] class AccessManager(conf: LivyConf) extends
Logging {
def isAccessControlOn: Boolean = aclsOn
/**
- * Checks that the requesting user can impersonate the target user.
- * If the user does not have permission to impersonate, then throws an
`AccessControlException`.
- *
- * @return The user that should be impersonated. That can be the target user
if defined, the
- * request's user - which may not be defined - otherwise, or `None`
if impersonation is
- * disabled.
+ * Checks that the request user can impersonate the target user.
+ * If impersonation is enabled and the user does not have permission to
impersonate
+ * then throws an `AccessControlException`. If impersonation is disabled
returns false
*/
def checkImpersonation(
- target: Option[String],
requestUser: String,
- livyConf: LivyConf): Option[String] = {
-if (livyConf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
- if (!target.forall(hasSuperAccess(_, requestUser))) {
-throw new AccessControlException(
- s"User '$requestUser' not allowed to impersonate '$target'.")
+ impersonatedUser: String): Boolean = {
+if (conf.getBoolean(LivyConf.IMPERSONATION_ENABLED)) {
+ if (hasSuperAccess(requestUser, impersonatedUser)) {
+return true
}
- target.orElse(Option(requestUser))
-} else {
- None
+ throw new AccessControlException(
+s"User '$requestUser' not allowed to impersonate '$impersonatedUser'.")
}
+false
}
/**
- * Check that the requesting user has admin access to resources owned by the
given target user.
+ * Check that the request user has is able to impersonate the given user.
*/
- def hasSuperAccess(target: String, requestUser: String): Boolean = {
-requestUser == target || checkSuperUser(requestUser)
+ def hasSuperAccess(
+ requestUser: String,
+ impersonatedUser: String): Boolean = {
+requestUser == impersonatedUser || checkSuperUser(requestUser)
}
/**
- * Check that the request's user has modify access to resources owned by the
given target user.
+ * Check that the request user has modify access to the given session.
*/
- def hasModifyAccess(target: String, requestUser: String): Boolean = {
-requestUser == target || checkModifyPermissions(requestUser)
+ def hasModifyAccess(
Review comment:
nit: one line
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy-website] vanzin merged pull request #26: Update documentation for 0.6.0 release.
vanzin merged pull request #26: Update documentation for 0.6.0 release. URL: https://github.com/apache/incubator-livy-website/pull/26 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy-website] vanzin commented on issue #26: Update documentation for 0.6.0 release.
vanzin commented on issue #26: Update documentation for 0.6.0 release. URL: https://github.com/apache/incubator-livy-website/pull/26#issuecomment-479127468 I re-checked the updated docs, they look ok. Will merge then publish. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] vanzin commented on a change in pull request #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
vanzin commented on a change in pull request #162: [LIVY-572] Avoid usage of
spark classes in ColumnBuffer
URL: https://github.com/apache/incubator-livy/pull/162#discussion_r268902862
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -43,7 +63,13 @@ public void addRow(Object[] fields) {
}
for (int i = 0; i < fields.length; i++) {
- columns[i].add(fields[i]);
+ Object value;
+ if (columnIsString[i]) {
Review comment:
Checking `columns[i].getType().equals(DataType.STRING)` here avoids the
extra boolean array.
(It's also an enum so you could use `==`.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer URL: https://github.com/apache/incubator-livy/pull/162#issuecomment-476429537 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=h1) Report > Merging [#162](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/07d216d33a5420004262c9b8d2acc2751547cc96?src=pr=desc) will **increase** coverage by `0.21%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#162 +/- ## === + Coverage 68.59% 68.8% +0.21% Complexity 905 905 === Files 100 100 Lines 56625662 Branches848 848 === + Hits 38843896 +12 + Misses 12251213 -12 Partials553 553 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...c/main/scala/org/apache/livy/repl/ReplDriver.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9SZXBsRHJpdmVyLnNjYWxh) | `33.33% <0%> (+2.56%)` | `7% <0%> (ø)` | :arrow_down: | | [.../scala/org/apache/livy/sessions/SessionState.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvU2Vzc2lvblN0YXRlLnNjYWxh) | `61.11% <0%> (+2.77%)` | `2% <0%> (ø)` | :arrow_down: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `55.88% <0%> (+2.94%)` | `7% <0%> (ø)` | :arrow_down: | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `79.57% <0%> (+6.33%)` | `33% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=footer). Last update [07d216d...05d4863](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io commented on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
codecov-io commented on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer URL: https://github.com/apache/incubator-livy/pull/162#issuecomment-476429537 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=h1) Report > Merging [#162](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/07d216d33a5420004262c9b8d2acc2751547cc96?src=pr=desc) will **increase** coverage by `0.21%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#162 +/- ## === + Coverage 68.59% 68.8% +0.21% Complexity 905 905 === Files 100 100 Lines 56625662 Branches848 848 === + Hits 38843896 +12 + Misses 12251213 -12 Partials553 553 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...c/main/scala/org/apache/livy/repl/ReplDriver.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9SZXBsRHJpdmVyLnNjYWxh) | `33.33% <0%> (+2.56%)` | `7% <0%> (ø)` | :arrow_down: | | [.../scala/org/apache/livy/sessions/SessionState.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvU2Vzc2lvblN0YXRlLnNjYWxh) | `61.11% <0%> (+2.77%)` | `2% <0%> (ø)` | :arrow_down: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `55.88% <0%> (+2.94%)` | `7% <0%> (ø)` | :arrow_down: | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `79.57% <0%> (+6.33%)` | `33% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=footer). Last update [07d216d...05d4863](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] vanzin commented on a change in pull request #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
vanzin commented on a change in pull request #162: [LIVY-572] Avoid usage of
spark classes in ColumnBuffer
URL: https://github.com/apache/incubator-livy/pull/162#discussion_r269165870
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -43,7 +63,13 @@ public void addRow(Object[] fields) {
}
for (int i = 0; i < fields.length; i++) {
- columns[i].add(fields[i]);
+ Object value;
+ if (columnIsString[i]) {
Review comment:
Using `==` there's actually no more overhead at all. In fact, if you look at
CPU caches and things like that it's probably faster since there's less risk of
a cache miss...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] gyogal opened a new pull request #163: [LIVY-513][LIVY-517][TEST] Fix SessionHeartbeatSpec flakiness, occasional NPE
gyogal opened a new pull request #163: [LIVY-513][LIVY-517][TEST] Fix SessionHeartbeatSpec flakiness, occasional NPE URL: https://github.com/apache/incubator-livy/pull/163 ## What changes were proposed in this pull request? This PR aims to fix the following issues in the test suite `SessionHeartbeatSpec`: * Since there is no implementation provided for `stop()` in `mock[TestSession]`, Mockito returns `null` by default and this causes `SessionManager#delete` to throw `NullPointerException` when executing `map()` on the `Future[Unit]` object it expects. This can be observed in the unit test output (`server/target/unit-tests.log`): ``` 19/03/25 18:20:30.123 ScalaTest-main-running-SessionHeartbeatSpec INFO SessionHeartbeatSpec$$anonfun$2$TestWatchdog$1: Registering new session 0 19/03/25 18:20:30.123 ScalaTest-main-running-SessionHeartbeatSpec INFO SessionHeartbeatSpec$$anonfun$2$TestWatchdog$1: Registering new session 1 19/03/25 18:20:30.124 ScalaTest-main-running-SessionHeartbeatSpec INFO SessionHeartbeatSpec$$anonfun$2$TestWatchdog$1: Session 0 expired. Last heartbeat is at null. 19/03/25 18:20:30.124 ScalaTest-main-running-SessionHeartbeatSpec WARN SessionHeartbeatSpec$$anonfun$2$TestWatchdog$1: Exception was thrown when deleting expired session 0 java.lang.NullPointerException at org.apache.livy.sessions.SessionManager.delete(SessionManager.scala:123) at org.apache.livy.server.interactive.SessionHeartbeatWatchdog$$anonfun$deleteExpiredSessions$1.apply(SessionHeartbeat.scala:113) at org.apache.livy.server.interactive.SessionHeartbeatWatchdog$$anonfun$deleteExpiredSessions$1.apply(SessionHeartbeat.scala:110) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206) at org.apache.livy.server.interactive.SessionHeartbeatWatchdog$class.deleteExpiredSessions(SessionHeartbeat.scala:110) at org.apache.livy.server.interactive.SessionHeartbeatSpec$$anonfun$2$TestWatchdog$1.deleteExpiredSessions(SessionHeartbeatSpec.scala:60) ``` * In some rare cases, the SessionManager GC thread may run `collectGarbage()` after the test sessions are registered but before the test is finished. The GC thread will also attempt to delete the `mock[TestSession]` objects because the mocked session's `lastActivity()` function returns 0 by default which means`currentTime - session.lastActivity > sessionTimeout` will always be true. This causes a race condition that can have the following outcomes: 1. The `stop()` method may be called twice (both by `SessionHeartbeatWatchdog#deleteExpiredSessions` and `SessionManager#collectGarbage`), resulting in a `TooManyActualInvocations` exception (LIVY-513): ``` org.mockito.exceptions.verification.TooManyActualInvocations: testSession$1.stop(); Wanted 1 time: -> at org.apache.livy.server.interactive.SessionHeartbeatSpec$$anonfun$2$$anonfun$apply$mcV$sp$5.apply$mcV$sp(SessionHeartbeatSpec.scala:93) But was 2 times. Undesired invocation: -> at org.apache.livy.sessions.SessionManager.delete(SessionManager.scala:124) at org.apache.livy.server.interactive.SessionHeartbeatSpec$$anonfun$2$$anonfun$apply$mcV$sp$5.apply$mcV$sp(SessionHeartbeatSpec.scala:93) at org.apache.livy.server.interactive.SessionHeartbeatSpec$$anonfun$2$$anonfun$apply$mcV$sp$5.apply(SessionHeartbeatSpec.scala:70) at org.apache.livy.server.interactive.SessionHeartbeatSpec$$anonfun$2$$anonfun$apply$mcV$sp$5.apply(SessionHeartbeatSpec.scala:70) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSpecLike$$anon$1.apply(FunSpecLike.scala:422) at org.scalatest.Suite$class.withFixture(Suite.scala:1122) ... ``` 2. The `stop()` method may be called by `collectGarbage()` after `deleteExpiredSessions()`, but before the test is finished, which causes the following error message to appear on the console (LIVY-517). ``` Deleting Mock for TestSession$1, hashCode: 138753 because it was inactive for more than 360.0 ms. Exception in thread "session gc thread" java.lang.NullPointerException at
[GitHub] [incubator-livy] mgaido91 opened a new pull request #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
mgaido91 opened a new pull request #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer URL: https://github.com/apache/incubator-livy/pull/162 ## What changes were proposed in this pull request? The `ColumnBuffers` can be created both inside spark jobs and in the Livy server. The latter case happens when operation logs are returned and fails before this patch because we are using Spark classes in this code after the refactor in LIVY-503. Unfortunately, we do not have test coverage for operation logs retrieval and this is the reason why this wasn't spot out earlier. Since operation logs are retrieved by beeline for each query, this means that every query run through beeline fails, unless `livy.server.thrift.logging.operation.enabled` is set to `false`. ## How was this patch tested? manual tests using beeline This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] Ngone51 opened a new pull request #166: Avoid duplicate stopping log when stop/interrupt a session
Ngone51 opened a new pull request #166: Avoid duplicate stopping log when stop/interrupt a session URL: https://github.com/apache/incubator-livy/pull/166 ## What changes were proposed in this pull request? There's duplicate stopping log when stop/interrupt a session. This happens because server logs stopping info when we stopping/interrupting a session at the first time. And when the session expired, it would be removed from session manager and call stop() and logs stopping info for the second time(Though, it does not really stop the session again at this time).  The fix is quite simple by adding a `_stopped` flag in Session's `stop()` method. ## How was this patch tested? N.A.(?) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] Limmen opened a new pull request #165: [LIVY-581] Fix edge-case where livy overrides user-provided spark properties instead of appending
Limmen opened a new pull request #165: [LIVY-581] Fix edge-case where livy overrides user-provided spark properties instead of appending URL: https://github.com/apache/incubator-livy/pull/165 This commit fixes a bug in livy that yarn properties are overridden when hive is enabled in the spark sessions. ## What changes were proposed in this pull request? Bugfix, append livy spark properties to the correct spark property (spark.yarn.X or spark.X. If the user have supplied spark config in spark.yarn.X and livy sets spark.X, spark will ignore spark.yarn.X which causes undesired behaviour since the user properties are then ignored. https://issues.apache.org/jira/browse/LIVY-581 ## How was this patch tested? Unit tests pass and this patch have been in production in our cluster since November last year This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer URL: https://github.com/apache/incubator-livy/pull/162#issuecomment-476429537 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=h1) Report > Merging [#162](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/07d216d33a5420004262c9b8d2acc2751547cc96?src=pr=desc) will **increase** coverage by `0.14%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #162 +/- ## + Coverage 68.59% 68.73% +0.14% + Complexity 905 904 -1 Files 100 100 Lines 5662 5662 Branches848 848 + Hits 3884 3892 +8 + Misses 1225 1216 -9 - Partials553 554 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `77.96% <0%> (-0.85%)` | `41% <0%> (-1%)` | | | [.../scala/org/apache/livy/sessions/SessionState.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvU2Vzc2lvblN0YXRlLnNjYWxh) | `61.11% <0%> (+2.77%)` | `2% <0%> (ø)` | :arrow_down: | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `79.57% <0%> (+6.33%)` | `33% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=footer). Last update [07d216d...0c8400f](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer URL: https://github.com/apache/incubator-livy/pull/162#issuecomment-476429537 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=h1) Report > Merging [#162](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/07d216d33a5420004262c9b8d2acc2751547cc96?src=pr=desc) will **increase** coverage by `0.14%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #162 +/- ## + Coverage 68.59% 68.73% +0.14% + Complexity 905 904 -1 Files 100 100 Lines 5662 5662 Branches848 848 + Hits 3884 3892 +8 + Misses 1225 1216 -9 - Partials553 554 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `77.96% <0%> (-0.85%)` | `41% <0%> (-1%)` | | | [.../scala/org/apache/livy/sessions/SessionState.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvU2Vzc2lvblN0YXRlLnNjYWxh) | `61.11% <0%> (+2.77%)` | `2% <0%> (ø)` | :arrow_down: | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `79.57% <0%> (+6.33%)` | `33% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=footer). Last update [07d216d...0c8400f](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer URL: https://github.com/apache/incubator-livy/pull/162#issuecomment-476429537 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=h1) Report > Merging [#162](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/07d216d33a5420004262c9b8d2acc2751547cc96?src=pr=desc) will **increase** coverage by `0.14%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #162 +/- ## + Coverage 68.59% 68.73% +0.14% + Complexity 905 904 -1 Files 100 100 Lines 5662 5662 Branches848 848 + Hits 3884 3892 +8 + Misses 1225 1216 -9 - Partials553 554 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `77.96% <0%> (-0.85%)` | `41% <0%> (-1%)` | | | [.../scala/org/apache/livy/sessions/SessionState.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvU2Vzc2lvblN0YXRlLnNjYWxh) | `61.11% <0%> (+2.77%)` | `2% <0%> (ø)` | :arrow_down: | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `79.57% <0%> (+6.33%)` | `33% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=footer). Last update [07d216d...0c8400f](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
codecov-io edited a comment on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer URL: https://github.com/apache/incubator-livy/pull/162#issuecomment-476429537 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=h1) Report > Merging [#162](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/07d216d33a5420004262c9b8d2acc2751547cc96?src=pr=desc) will **increase** coverage by `0.14%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #162 +/- ## + Coverage 68.59% 68.73% +0.14% + Complexity 905 904 -1 Files 100 100 Lines 5662 5662 Branches848 848 + Hits 3884 3892 +8 + Misses 1225 1216 -9 - Partials553 554 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `77.96% <0%> (-0.85%)` | `41% <0%> (-1%)` | | | [.../scala/org/apache/livy/sessions/SessionState.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvU2Vzc2lvblN0YXRlLnNjYWxh) | `61.11% <0%> (+2.77%)` | `2% <0%> (ø)` | :arrow_down: | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/162/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `79.57% <0%> (+6.33%)` | `33% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=footer). Last update [07d216d...0c8400f](https://codecov.io/gh/apache/incubator-livy/pull/162?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy-website] vanzin opened a new pull request #26: Update documentation for 0.6.0 release.
vanzin opened a new pull request #26: Update documentation for 0.6.0 release. URL: https://github.com/apache/incubator-livy-website/pull/26 Updated the download links (since the archive names changed), added the auto-generated documentation, and updated the project contributors. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] vanzin opened a new pull request #164: [DOCS] Fix Javadoc 8 errors.
vanzin opened a new pull request #164: [DOCS] Fix Javadoc 8 errors. URL: https://github.com/apache/incubator-livy/pull/164 Seems like javadoc 8 doesn't like when you reference nested classes from parent types. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io commented on issue #164: [DOCS] Fix Javadoc 8 errors.
codecov-io commented on issue #164: [DOCS] Fix Javadoc 8 errors. URL: https://github.com/apache/incubator-livy/pull/164#issuecomment-478726478 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=h1) Report > Merging [#164](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/8c7f03b4a8c5fae73c14bac7d97c62cc0fd6241d?src=pr=desc) will **decrease** coverage by `3.79%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #164 +/- ## === - Coverage 68.75% 64.95% -3.8% + Complexity 904 877 -27 === Files 100 100 Lines 5662 5662 Branches848 848 === - Hits 3893 3678-215 - Misses 1216 1463+247 + Partials553 521 -32 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/livy/client/common/AbstractJobHandle.java](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL0Fic3RyYWN0Sm9iSGFuZGxlLmphdmE=) | `88.23% <ø> (ø)` | `13 <0> (ø)` | :arrow_down: | | [...a/org/apache/livy/server/ThriftServerFactory.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvVGhyaWZ0U2VydmVyRmFjdG9yeS5zY2FsYQ==) | `0% <0%> (-100%)` | `0% <0%> (ø)` | | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `1.97% <0%> (-33.5%)` | `2% <0%> (-9%)` | | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `55.63% <0%> (-23.95%)` | `29% <0%> (-4%)` | | | [...rc/main/scala/org/apache/livy/utils/SparkApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya0FwcC5zY2FsYQ==) | `56% <0%> (-20%)` | `1% <0%> (ø)` | | | [core/src/main/scala/org/apache/livy/Logging.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvTG9nZ2luZy5zY2FsYQ==) | `66.66% <0%> (-16.67%)` | `0% <0%> (ø)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `55.72% <0%> (-13.32%)` | `41% <0%> (-3%)` | | | [...src/main/scala/org/apache/livy/sessions/Kind.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvS2luZC5zY2FsYQ==) | `65% <0%> (-10%)` | `2% <0%> (ø)` | | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `60.8% <0%> (-6.41%)` | `6% <0%> (ø)` | | | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `64.83% <0%> (-5.5%)` | `19% <0%> (-1%)` | | | ... and [19 more](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=footer). Last update [8c7f03b...d8445cd](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[GitHub] [incubator-livy] codecov-io edited a comment on issue #164: [DOCS] Fix Javadoc 8 errors.
codecov-io edited a comment on issue #164: [DOCS] Fix Javadoc 8 errors. URL: https://github.com/apache/incubator-livy/pull/164#issuecomment-478726478 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=h1) Report > Merging [#164](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/8c7f03b4a8c5fae73c14bac7d97c62cc0fd6241d?src=pr=desc) will **decrease** coverage by `3.49%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #164 +/- ## === - Coverage 68.75% 65.25% -3.5% + Complexity 904 879 -25 === Files 100 100 Lines 5662 5662 Branches848 848 === - Hits 3893 3695-198 - Misses 1216 1448+232 + Partials553 519 -34 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/livy/client/common/AbstractJobHandle.java](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL0Fic3RyYWN0Sm9iSGFuZGxlLmphdmE=) | `88.23% <ø> (ø)` | `13 <0> (ø)` | :arrow_down: | | [...a/org/apache/livy/server/ThriftServerFactory.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvVGhyaWZ0U2VydmVyRmFjdG9yeS5zY2FsYQ==) | `0% <0%> (-100%)` | `0% <0%> (ø)` | | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `1.97% <0%> (-33.5%)` | `2% <0%> (-9%)` | | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `57.74% <0%> (-21.84%)` | `29% <0%> (-4%)` | | | [...rc/main/scala/org/apache/livy/utils/SparkApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya0FwcC5zY2FsYQ==) | `56% <0%> (-20%)` | `1% <0%> (ø)` | | | [core/src/main/scala/org/apache/livy/Logging.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvTG9nZ2luZy5zY2FsYQ==) | `66.66% <0%> (-16.67%)` | `0% <0%> (ø)` | | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `55.72% <0%> (-13.32%)` | `41% <0%> (-3%)` | | | [...src/main/scala/org/apache/livy/sessions/Kind.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-Y29yZS9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvc2Vzc2lvbnMvS2luZC5zY2FsYQ==) | `65% <0%> (-10%)` | `2% <0%> (ø)` | | | [...server/interactive/InteractiveSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `60.8% <0%> (-6.41%)` | `6% <0%> (ø)` | | | [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `64.83% <0%> (-5.5%)` | `19% <0%> (-1%)` | | | ... and [17 more](https://codecov.io/gh/apache/incubator-livy/pull/164/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=footer). Last update [8c7f03b...d8445cd](https://codecov.io/gh/apache/incubator-livy/pull/164?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[GitHub] [incubator-livy] asfgit closed pull request #164: [DOCS] Fix Javadoc 8 errors.
asfgit closed pull request #164: [DOCS] Fix Javadoc 8 errors. URL: https://github.com/apache/incubator-livy/pull/164 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] vanzin commented on issue #164: [DOCS] Fix Javadoc 8 errors.
vanzin commented on issue #164: [DOCS] Fix Javadoc 8 errors. URL: https://github.com/apache/incubator-livy/pull/164#issuecomment-478732470 Don't see the Travis update but the build passed on Travis: https://travis-ci.org/apache/incubator-livy/builds/514334116 Merging to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy-website] vanzin commented on issue #26: Update documentation for 0.6.0 release.
vanzin commented on issue #26: Update documentation for 0.6.0 release. URL: https://github.com/apache/incubator-livy-website/pull/26#issuecomment-478732122 Don't see the Travis update here but it passed. Merging to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] asfgit closed pull request #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
asfgit closed pull request #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer URL: https://github.com/apache/incubator-livy/pull/162 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] vanzin commented on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer
vanzin commented on issue #162: [LIVY-572] Avoid usage of spark classes in ColumnBuffer URL: https://github.com/apache/incubator-livy/pull/162#issuecomment-476913608 Merging to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] vanzin commented on issue #163: [LIVY-513][LIVY-517][TEST] Fix SessionHeartbeatSpec flakiness, occasional NPE
vanzin commented on issue #163: [LIVY-513][LIVY-517][TEST] Fix SessionHeartbeatSpec flakiness, occasional NPE URL: https://github.com/apache/incubator-livy/pull/163#issuecomment-476915265 Merging to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] asfgit closed pull request #163: [LIVY-513][LIVY-517][TEST] Fix SessionHeartbeatSpec flakiness, occasional NPE
asfgit closed pull request #163: [LIVY-513][LIVY-517][TEST] Fix SessionHeartbeatSpec flakiness, occasional NPE URL: https://github.com/apache/incubator-livy/pull/163 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] martinhartig edited a comment on issue #91: [LIVY-124] Allow jar/pyfile/file to be uploaded when creating a batch session
martinhartig edited a comment on issue #91: [LIVY-124] Allow jar/pyfile/file to be uploaded when creating a batch session URL: https://github.com/apache/incubator-livy/pull/91#issuecomment-467472464 Any progress in this functionality? For me, supporting uploads of jars via the REST API is very desirable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] martinhartig commented on issue #91: [LIVY-124] Allow jar/pyfile/file to be uploaded when creating a batch session
martinhartig commented on issue #91: [LIVY-124] Allow jar/pyfile/file to be uploaded when creating a batch session URL: https://github.com/apache/incubator-livy/pull/91#issuecomment-470175947 What is the "another solution"? Can you point me to a PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] Tagar commented on issue #135: [LIVY-544] Allow interpreterExecutor run in ThreadPool
Tagar commented on issue #135: [LIVY-544] Allow interpreterExecutor run in ThreadPool URL: https://github.com/apache/incubator-livy/pull/135#issuecomment-472971878 @WeiWenda I am not a committer. Thanks for this great improvement. It would be awesome to have concurrentSQL option in Livy, like we have in bare Zeppelin. cc @mgaido91 @jerryshao @vanzin @ajbozarth @zjffdu - please help to review. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
