[GitHub] [incubator-livy] codecov-io edited a comment on issue #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on projec
codecov-io edited a comment on issue #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report URL: https://github.com/apache/incubator-livy/pull/229#issuecomment-530197154 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=h1) Report > Merging [#229](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/4de6258aed7cbaff4a5909fe0d9db79fe869c46f?src=pr=desc) will **increase** coverage by `0.38%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#229 +/- ## === + Coverage 68.32% 68.7% +0.38% - Complexity 919 932 +13 === Files 100 100 Lines 57175822 +105 Branches867 881 +14 === + Hits 39064000 +94 - Misses 12471260 +13 + Partials564 562 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (ø)` | `13 <0> (ø)` | :arrow_down: | | [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `85.98% <100%> (+0.13%)` | `7 <0> (ø)` | :arrow_down: | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `69.36% <0%> (+0.25%)` | `46% <0%> (+2%)` | :arrow_up: | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `35.96% <0%> (+0.49%)` | `11% <0%> (ø)` | :arrow_down: | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `80.08% <0%> (+2.11%)` | `43% <0%> (+2%)` | :arrow_up: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `76.06% <0%> (+2.14%)` | `27% <0%> (+7%)` | :arrow_up: | | [...c/main/scala/org/apache/livy/repl/ReplDriver.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9SZXBsRHJpdmVyLnNjYWxh) | `33.33% <0%> (+2.56%)` | `7% <0%> (ø)` | :arrow_down: | | [...ala/org/apache/livy/scalaapi/LivyScalaClient.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9MaXZ5U2NhbGFDbGllbnQuc2NhbGE=) | `86.66% <0%> (+3.33%)` | `7% <0%> (ø)` | :arrow_down: | | [...in/java/org/apache/livy/rsc/driver/JobWrapper.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iV3JhcHBlci5qYXZh) | `88.57% <0%> (+5.71%)` | `9% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=footer). Last update [4de6258...3e98c2c](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,
[GitHub] [incubator-livy] codecov-io edited a comment on issue #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on projec
codecov-io edited a comment on issue #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report URL: https://github.com/apache/incubator-livy/pull/229#issuecomment-530197154 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=h1) Report > Merging [#229](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/4de6258aed7cbaff4a5909fe0d9db79fe869c46f?src=pr=desc) will **increase** coverage by `0.38%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#229 +/- ## === + Coverage 68.32% 68.7% +0.38% - Complexity 919 932 +13 === Files 100 100 Lines 57175822 +105 Branches867 881 +14 === + Hits 39064000 +94 - Misses 12471260 +13 + Partials564 562 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (ø)` | `13 <0> (ø)` | :arrow_down: | | [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `85.98% <100%> (+0.13%)` | `7 <0> (ø)` | :arrow_down: | | [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `69.36% <0%> (+0.25%)` | `46% <0%> (+2%)` | :arrow_up: | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `35.96% <0%> (+0.49%)` | `11% <0%> (ø)` | :arrow_down: | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `80.08% <0%> (+2.11%)` | `43% <0%> (+2%)` | :arrow_up: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `76.06% <0%> (+2.14%)` | `27% <0%> (+7%)` | :arrow_up: | | [...c/main/scala/org/apache/livy/repl/ReplDriver.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9SZXBsRHJpdmVyLnNjYWxh) | `33.33% <0%> (+2.56%)` | `7% <0%> (ø)` | :arrow_down: | | [...ala/org/apache/livy/scalaapi/LivyScalaClient.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9MaXZ5U2NhbGFDbGllbnQuc2NhbGE=) | `86.66% <0%> (+3.33%)` | `7% <0%> (ø)` | :arrow_down: | | [...in/java/org/apache/livy/rsc/driver/JobWrapper.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iV3JhcHBlci5qYXZh) | `88.57% <0%> (+5.71%)` | `9% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=footer). Last update [4de6258...3e98c2c](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,
[GitHub] [incubator-livy] jerryshao closed pull request #225: [LIVY-657][TEST]Fix travis failed on should not create sessions with duplicate names
jerryshao closed pull request #225: [LIVY-657][TEST]Fix travis failed on should not create sessions with duplicate names URL: https://github.com/apache/incubator-livy/pull/225 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] jerryshao commented on issue #228: [LIVY-663][API] exposing job id and session id in livy client
jerryshao commented on issue #228: [LIVY-663][API] exposing job id and session id in livy client URL: https://github.com/apache/incubator-livy/pull/228#issuecomment-532498159 I think such APIs should also be added in Scala and Python Client. Also would you please add some unit tests, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] yiheng opened a new pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project li
yiheng opened a new pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report URL: https://github.com/apache/incubator-livy/pull/229 ## What changes were proposed in this pull request? This patch fixes the flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report. When JVM shutdown no gracefully in a test, the code coverage data file generated by jacoco may be corrupt. Jacoco will throw an exception when generate code coverage report. In Livy integration test, two test cases shut down no gracefully(one of them uses System.exit). We can find random failure when jacoco process code coverage data file generated by that test case. In this patch, we turn off the code coverage analysis on these two test cases. ## How was this patch tested? Compare the jacoco data file generated in the integration test. Before the fix, there're 18 files, and after the fix there're 16 files, which means the fix works. Run 10 builds on Travis each before and after the fix: 1. Before the fix: 3 builds failed due to the jacoco code coverage exception 2. After the fix: No build failed Existing UTs and ITs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA support
ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA
support
URL: https://github.com/apache/incubator-livy/pull/212#discussion_r325415457
##
File path:
server/src/main/scala/org/apache/livy/server/CuratorElectorService.scala
##
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.server
+
+import java.io.Closeable
+import java.io.IOException
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.AtomicInteger
+
+import org.apache.curator.framework.CuratorFramework
+import org.apache.curator.framework.CuratorFrameworkFactory
+import org.apache.curator.framework.recipes.leader.LeaderLatch
+import org.apache.curator.framework.recipes.leader.LeaderLatchListener
+import org.apache.curator.retry.RetryNTimes
+
+import org.apache.livy.{LivyConf, Logging}
+import org.apache.livy.LivyConf.Entry
+
+object CuratorElectorService {
+ val HA_KEY_PREFIX_CONF = Entry("livy.server.ha.key-prefix", "livy_ha")
+ val HA_RETRY_CONF = Entry("livy.server.ha.retry-policy", "5,100")
+}
+
+class CuratorElectorService(livyConf : LivyConf, livyServer : LivyServer)
+ extends LeaderLatchListener
+ with Logging
+{
+
+ import CuratorElectorService._
+
+ val haAddress = livyConf.get(LivyConf.HA_ZOOKEEPER_URL)
+ require(!haAddress.isEmpty, s"Please configure
${LivyConf.HA_ZOOKEEPER_URL.key}.")
+ val haKeyPrefix = livyConf.get(HA_KEY_PREFIX_CONF)
+ val retryValue = livyConf.get(HA_RETRY_CONF)
+ // a regex to match patterns like "m, n" where m and n both are integer
values
+ val retryPattern = """\s*(\d+)\s*,\s*(\d+)\s*""".r
+ val retryPolicy = retryValue match {
+case retryPattern(n, sleepMs) => new RetryNTimes(n.toInt, sleepMs.toInt)
+case _ => throw new IllegalArgumentException(
+ s"$HA_KEY_PREFIX_CONF contains bad value: $retryValue. " +
+"Correct format is ,. e.g.
5,100")
+ }
+
+ val client: CuratorFramework = CuratorFrameworkFactory.newClient(haAddress,
retryPolicy)
+ val leaderKey = s"/$haKeyPrefix/leader"
+
+ var server : LivyServer = livyServer
+
+ var leaderLatch = new LeaderLatch(client, leaderKey)
+ leaderLatch.addListener(this)
+
+ object HAState extends Enumeration{
+type HAState = Value
+val Active, Standby = Value
+ }
+ var currentState = HAState.Standby
+
+ def isLeader() {
+transitionToActive();
+ }
+
+ def notLeader(){
Review comment:
space before `{`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA support
ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA
support
URL: https://github.com/apache/incubator-livy/pull/212#discussion_r325415549
##
File path:
server/src/main/scala/org/apache/livy/server/CuratorElectorService.scala
##
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.server
+
+import java.io.Closeable
+import java.io.IOException
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.AtomicInteger
+
+import org.apache.curator.framework.CuratorFramework
+import org.apache.curator.framework.CuratorFrameworkFactory
+import org.apache.curator.framework.recipes.leader.LeaderLatch
+import org.apache.curator.framework.recipes.leader.LeaderLatchListener
+import org.apache.curator.retry.RetryNTimes
+
+import org.apache.livy.{LivyConf, Logging}
+import org.apache.livy.LivyConf.Entry
+
+object CuratorElectorService {
+ val HA_KEY_PREFIX_CONF = Entry("livy.server.ha.key-prefix", "livy_ha")
+ val HA_RETRY_CONF = Entry("livy.server.ha.retry-policy", "5,100")
+}
+
+class CuratorElectorService(livyConf : LivyConf, livyServer : LivyServer)
+ extends LeaderLatchListener
+ with Logging
+{
+
+ import CuratorElectorService._
+
+ val haAddress = livyConf.get(LivyConf.HA_ZOOKEEPER_URL)
+ require(!haAddress.isEmpty, s"Please configure
${LivyConf.HA_ZOOKEEPER_URL.key}.")
+ val haKeyPrefix = livyConf.get(HA_KEY_PREFIX_CONF)
+ val retryValue = livyConf.get(HA_RETRY_CONF)
+ // a regex to match patterns like "m, n" where m and n both are integer
values
+ val retryPattern = """\s*(\d+)\s*,\s*(\d+)\s*""".r
+ val retryPolicy = retryValue match {
+case retryPattern(n, sleepMs) => new RetryNTimes(n.toInt, sleepMs.toInt)
+case _ => throw new IllegalArgumentException(
+ s"$HA_KEY_PREFIX_CONF contains bad value: $retryValue. " +
+"Correct format is ,. e.g.
5,100")
+ }
+
+ val client: CuratorFramework = CuratorFrameworkFactory.newClient(haAddress,
retryPolicy)
+ val leaderKey = s"/$haKeyPrefix/leader"
+
+ var server : LivyServer = livyServer
+
+ var leaderLatch = new LeaderLatch(client, leaderKey)
+ leaderLatch.addListener(this)
+
+ object HAState extends Enumeration{
+type HAState = Value
+val Active, Standby = Value
+ }
+ var currentState = HAState.Standby
+
+ def isLeader() {
+transitionToActive();
+ }
+
+ def notLeader(){
+transitionToStandby();
+ }
+
+ def start() : Unit = {
+transitionToStandby()
+
+client.start()
+leaderLatch.start()
+
+try {
+ Thread.currentThread.join()
+} finally {
+ transitionToStandby()
+}
+ }
+
+ def close() : Unit = {
Review comment:
no space before `:`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA support
ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA
support
URL: https://github.com/apache/incubator-livy/pull/212#discussion_r325415827
##
File path: server/src/main/scala/org/apache/livy/server/LivyServer.scala
##
@@ -394,16 +395,29 @@ class LivyServer extends Logging {
}
}
+object HighAvailabilitySettings {
+ val HA_ON = "on"
+ val HA_OFF = "off"
+}
+
object LivyServer {
def main(args: Array[String]): Unit = {
val server = new LivyServer()
-try {
- server.start()
- server.join()
-} finally {
- server.stop()
+val livyConf = new LivyConf().loadFromFile("livy.conf")
+
+if(livyConf.get(LivyConf.HA_MODE) == HighAvailabilitySettings.HA_ON){
Review comment:
space before `{`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA support
ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA
support
URL: https://github.com/apache/incubator-livy/pull/212#discussion_r325415571
##
File path:
server/src/main/scala/org/apache/livy/server/CuratorElectorService.scala
##
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.server
+
+import java.io.Closeable
+import java.io.IOException
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.AtomicInteger
+
+import org.apache.curator.framework.CuratorFramework
+import org.apache.curator.framework.CuratorFrameworkFactory
+import org.apache.curator.framework.recipes.leader.LeaderLatch
+import org.apache.curator.framework.recipes.leader.LeaderLatchListener
+import org.apache.curator.retry.RetryNTimes
+
+import org.apache.livy.{LivyConf, Logging}
+import org.apache.livy.LivyConf.Entry
+
+object CuratorElectorService {
+ val HA_KEY_PREFIX_CONF = Entry("livy.server.ha.key-prefix", "livy_ha")
+ val HA_RETRY_CONF = Entry("livy.server.ha.retry-policy", "5,100")
+}
+
+class CuratorElectorService(livyConf : LivyConf, livyServer : LivyServer)
+ extends LeaderLatchListener
+ with Logging
+{
+
+ import CuratorElectorService._
+
+ val haAddress = livyConf.get(LivyConf.HA_ZOOKEEPER_URL)
+ require(!haAddress.isEmpty, s"Please configure
${LivyConf.HA_ZOOKEEPER_URL.key}.")
+ val haKeyPrefix = livyConf.get(HA_KEY_PREFIX_CONF)
+ val retryValue = livyConf.get(HA_RETRY_CONF)
+ // a regex to match patterns like "m, n" where m and n both are integer
values
+ val retryPattern = """\s*(\d+)\s*,\s*(\d+)\s*""".r
+ val retryPolicy = retryValue match {
+case retryPattern(n, sleepMs) => new RetryNTimes(n.toInt, sleepMs.toInt)
+case _ => throw new IllegalArgumentException(
+ s"$HA_KEY_PREFIX_CONF contains bad value: $retryValue. " +
+"Correct format is ,. e.g.
5,100")
+ }
+
+ val client: CuratorFramework = CuratorFrameworkFactory.newClient(haAddress,
retryPolicy)
+ val leaderKey = s"/$haKeyPrefix/leader"
+
+ var server : LivyServer = livyServer
+
+ var leaderLatch = new LeaderLatch(client, leaderKey)
+ leaderLatch.addListener(this)
+
+ object HAState extends Enumeration{
+type HAState = Value
+val Active, Standby = Value
+ }
+ var currentState = HAState.Standby
+
+ def isLeader() {
+transitionToActive();
+ }
+
+ def notLeader(){
+transitionToStandby();
+ }
+
+ def start() : Unit = {
+transitionToStandby()
+
+client.start()
+leaderLatch.start()
+
+try {
+ Thread.currentThread.join()
+} finally {
+ transitionToStandby()
+}
+ }
+
+ def close() : Unit = {
+transitionToStandby();
+leaderLatch.close();
+ }
+
+ def transitionToActive() : Unit = {
Review comment:
no space before `:`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA support
ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA
support
URL: https://github.com/apache/incubator-livy/pull/212#discussion_r325415370
##
File path:
server/src/main/scala/org/apache/livy/server/CuratorElectorService.scala
##
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.server
+
+import java.io.Closeable
+import java.io.IOException
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.AtomicInteger
+
+import org.apache.curator.framework.CuratorFramework
+import org.apache.curator.framework.CuratorFrameworkFactory
+import org.apache.curator.framework.recipes.leader.LeaderLatch
+import org.apache.curator.framework.recipes.leader.LeaderLatchListener
+import org.apache.curator.retry.RetryNTimes
+
+import org.apache.livy.{LivyConf, Logging}
+import org.apache.livy.LivyConf.Entry
+
+object CuratorElectorService {
+ val HA_KEY_PREFIX_CONF = Entry("livy.server.ha.key-prefix", "livy_ha")
+ val HA_RETRY_CONF = Entry("livy.server.ha.retry-policy", "5,100")
+}
+
+class CuratorElectorService(livyConf : LivyConf, livyServer : LivyServer)
+ extends LeaderLatchListener
+ with Logging
+{
+
+ import CuratorElectorService._
+
+ val haAddress = livyConf.get(LivyConf.HA_ZOOKEEPER_URL)
+ require(!haAddress.isEmpty, s"Please configure
${LivyConf.HA_ZOOKEEPER_URL.key}.")
+ val haKeyPrefix = livyConf.get(HA_KEY_PREFIX_CONF)
+ val retryValue = livyConf.get(HA_RETRY_CONF)
+ // a regex to match patterns like "m, n" where m and n both are integer
values
+ val retryPattern = """\s*(\d+)\s*,\s*(\d+)\s*""".r
+ val retryPolicy = retryValue match {
+case retryPattern(n, sleepMs) => new RetryNTimes(n.toInt, sleepMs.toInt)
+case _ => throw new IllegalArgumentException(
+ s"$HA_KEY_PREFIX_CONF contains bad value: $retryValue. " +
+"Correct format is ,. e.g.
5,100")
+ }
+
+ val client: CuratorFramework = CuratorFrameworkFactory.newClient(haAddress,
retryPolicy)
+ val leaderKey = s"/$haKeyPrefix/leader"
+
+ var server : LivyServer = livyServer
Review comment:
no space before `:`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA support
ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA
support
URL: https://github.com/apache/incubator-livy/pull/212#discussion_r325415504
##
File path:
server/src/main/scala/org/apache/livy/server/CuratorElectorService.scala
##
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.server
+
+import java.io.Closeable
+import java.io.IOException
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.AtomicInteger
+
+import org.apache.curator.framework.CuratorFramework
+import org.apache.curator.framework.CuratorFrameworkFactory
+import org.apache.curator.framework.recipes.leader.LeaderLatch
+import org.apache.curator.framework.recipes.leader.LeaderLatchListener
+import org.apache.curator.retry.RetryNTimes
+
+import org.apache.livy.{LivyConf, Logging}
+import org.apache.livy.LivyConf.Entry
+
+object CuratorElectorService {
+ val HA_KEY_PREFIX_CONF = Entry("livy.server.ha.key-prefix", "livy_ha")
+ val HA_RETRY_CONF = Entry("livy.server.ha.retry-policy", "5,100")
+}
+
+class CuratorElectorService(livyConf : LivyConf, livyServer : LivyServer)
+ extends LeaderLatchListener
+ with Logging
+{
+
+ import CuratorElectorService._
+
+ val haAddress = livyConf.get(LivyConf.HA_ZOOKEEPER_URL)
+ require(!haAddress.isEmpty, s"Please configure
${LivyConf.HA_ZOOKEEPER_URL.key}.")
+ val haKeyPrefix = livyConf.get(HA_KEY_PREFIX_CONF)
+ val retryValue = livyConf.get(HA_RETRY_CONF)
+ // a regex to match patterns like "m, n" where m and n both are integer
values
+ val retryPattern = """\s*(\d+)\s*,\s*(\d+)\s*""".r
+ val retryPolicy = retryValue match {
+case retryPattern(n, sleepMs) => new RetryNTimes(n.toInt, sleepMs.toInt)
+case _ => throw new IllegalArgumentException(
+ s"$HA_KEY_PREFIX_CONF contains bad value: $retryValue. " +
+"Correct format is ,. e.g.
5,100")
+ }
+
+ val client: CuratorFramework = CuratorFrameworkFactory.newClient(haAddress,
retryPolicy)
+ val leaderKey = s"/$haKeyPrefix/leader"
+
+ var server : LivyServer = livyServer
+
+ var leaderLatch = new LeaderLatch(client, leaderKey)
+ leaderLatch.addListener(this)
+
+ object HAState extends Enumeration{
+type HAState = Value
+val Active, Standby = Value
+ }
+ var currentState = HAState.Standby
+
+ def isLeader() {
+transitionToActive();
+ }
+
+ def notLeader(){
+transitionToStandby();
+ }
+
+ def start() : Unit = {
Review comment:
no space before `:`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA support
ajbozarth commented on a change in pull request #212: [LIVY-11] Enable HA
support
URL: https://github.com/apache/incubator-livy/pull/212#discussion_r325415712
##
File path:
server/src/main/scala/org/apache/livy/server/CuratorElectorService.scala
##
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.server
+
+import java.io.Closeable
+import java.io.IOException
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.AtomicInteger
+
+import org.apache.curator.framework.CuratorFramework
+import org.apache.curator.framework.CuratorFrameworkFactory
+import org.apache.curator.framework.recipes.leader.LeaderLatch
+import org.apache.curator.framework.recipes.leader.LeaderLatchListener
+import org.apache.curator.retry.RetryNTimes
+
+import org.apache.livy.{LivyConf, Logging}
+import org.apache.livy.LivyConf.Entry
+
+object CuratorElectorService {
+ val HA_KEY_PREFIX_CONF = Entry("livy.server.ha.key-prefix", "livy_ha")
+ val HA_RETRY_CONF = Entry("livy.server.ha.retry-policy", "5,100")
+}
+
+class CuratorElectorService(livyConf : LivyConf, livyServer : LivyServer)
+ extends LeaderLatchListener
+ with Logging
+{
+
+ import CuratorElectorService._
+
+ val haAddress = livyConf.get(LivyConf.HA_ZOOKEEPER_URL)
+ require(!haAddress.isEmpty, s"Please configure
${LivyConf.HA_ZOOKEEPER_URL.key}.")
+ val haKeyPrefix = livyConf.get(HA_KEY_PREFIX_CONF)
+ val retryValue = livyConf.get(HA_RETRY_CONF)
+ // a regex to match patterns like "m, n" where m and n both are integer
values
+ val retryPattern = """\s*(\d+)\s*,\s*(\d+)\s*""".r
+ val retryPolicy = retryValue match {
+case retryPattern(n, sleepMs) => new RetryNTimes(n.toInt, sleepMs.toInt)
+case _ => throw new IllegalArgumentException(
+ s"$HA_KEY_PREFIX_CONF contains bad value: $retryValue. " +
+"Correct format is ,. e.g.
5,100")
+ }
+
+ val client: CuratorFramework = CuratorFrameworkFactory.newClient(haAddress,
retryPolicy)
+ val leaderKey = s"/$haKeyPrefix/leader"
+
+ var server : LivyServer = livyServer
+
+ var leaderLatch = new LeaderLatch(client, leaderKey)
+ leaderLatch.addListener(this)
+
+ object HAState extends Enumeration{
+type HAState = Value
+val Active, Standby = Value
+ }
+ var currentState = HAState.Standby
+
+ def isLeader() {
+transitionToActive();
+ }
+
+ def notLeader(){
+transitionToStandby();
+ }
+
+ def start() : Unit = {
+transitionToStandby()
+
+client.start()
+leaderLatch.start()
+
+try {
+ Thread.currentThread.join()
+} finally {
+ transitionToStandby()
+}
+ }
+
+ def close() : Unit = {
+transitionToStandby();
+leaderLatch.close();
+ }
+
+ def transitionToActive() : Unit = {
+info("Transitioning to Active state")
+if(currentState == HAState.Active){
+ info("Already in Active State")
+}
+else {
+ server.start()
+ currentState = HAState.Active
+ info("Transition complete")
+}
+ }
+
+ def transitionToStandby() : Unit = {
+info("Transitioning to Standby state")
+if(currentState == HAState.Standby){
Review comment:
space before `{`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] mgaido91 commented on a change in pull request #231: [LIVY-356][SERVER]Add LDAP authentication for livy-server.
mgaido91 commented on a change in pull request #231: [LIVY-356][SERVER]Add LDAP authentication for livy-server. URL: https://github.com/apache/incubator-livy/pull/231#discussion_r325150420 ## File path: conf/livy.conf.template ## @@ -160,6 +160,13 @@ # livy.server.auth.kerberos.keytab = # livy.server.auth.kerberos.name-rules = DEFAULT # +# livy.server.auth.type = ldap Review comment: sounds great. I'd remove it from here anyway. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] mgaido91 commented on a change in pull request #231: [LIVY-356][SERVER]Add LDAP authentication for livy-server.
mgaido91 commented on a change in pull request #231: [LIVY-356][SERVER]Add LDAP
authentication for livy-server.
URL: https://github.com/apache/incubator-livy/pull/231#discussion_r325149564
##
File path: server/src/main/scala/org/apache/livy/server/LivyServer.scala
##
@@ -260,6 +261,26 @@ class LivyServer extends Logging {
server.context.addFilter(holder, "/*",
EnumSet.allOf(classOf[DispatcherType]))
info(s"SPNEGO auth enabled (principal = $principal)")
+ case authType @ LdapAuthenticationHandlerImpl.TYPE =>
Review comment:
ah sorry, I rushed checking it, thanks for the explanation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] captainzmc commented on a change in pull request #231: [LIVY-356][SERVER]Add LDAP authentication for livy-server.
captainzmc commented on a change in pull request #231: [LIVY-356][SERVER]Add LDAP authentication for livy-server. URL: https://github.com/apache/incubator-livy/pull/231#discussion_r325149382 ## File path: conf/livy.conf.template ## @@ -160,6 +160,13 @@ # livy.server.auth.kerberos.keytab = # livy.server.auth.kerberos.name-rules = DEFAULT # +# livy.server.auth.type = ldap Review comment: I couldn't agree with you more. The doc about livy is not complete at present, and our team has some plans to improve it. Next we will make a plan to improve livy's doc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] captainzmc commented on a change in pull request #231: [LIVY-356][SERVER]Add LDAP authentication for livy-server.
captainzmc commented on a change in pull request #231: [LIVY-356][SERVER]Add LDAP authentication for livy-server. URL: https://github.com/apache/incubator-livy/pull/231#discussion_r325146353 ## File path: pom.xml ## @@ -121,6 +121,10 @@ false + Review comment: This section is for testing only, and needs to simulate a Mini ldap environment, with reference to [hadoop](https://github.com/apache/hadoop/blob/release-2.8.0-RC0/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestLdapAuthenticationHandler.java) and [hive](https://github.com/apache/hive/blob/master/service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java) practices This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] captainzmc commented on a change in pull request #231: [LIVY-356][SERVER]Add LDAP authentication for livy-server.
captainzmc commented on a change in pull request #231: [LIVY-356][SERVER]Add
LDAP authentication for livy-server.
URL: https://github.com/apache/incubator-livy/pull/231#discussion_r325143141
##
File path: server/src/main/scala/org/apache/livy/server/LivyServer.scala
##
@@ -260,6 +261,26 @@ class LivyServer extends Logging {
server.context.addFilter(holder, "/*",
EnumSet.allOf(classOf[DispatcherType]))
info(s"SPNEGO auth enabled (principal = $principal)")
+ case authType @ LdapAuthenticationHandlerImpl.TYPE =>
Review comment:
Sorry for not adding JIRA in title. This part is to add ldap authentication
in livy-server (rest). The authentication unification in rest before referred
to hadoop's filters, so this part is consistent.
Now I am currently working on ldap authentication in thrift-server (not yet
complete) and refer to this PR #183
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] mgaido91 commented on a change in pull request #231: Add LDAP authentication for livy-server.
mgaido91 commented on a change in pull request #231: Add LDAP authentication
for livy-server.
URL: https://github.com/apache/incubator-livy/pull/231#discussion_r325133826
##
File path: server/src/main/scala/org/apache/livy/server/LivyServer.scala
##
@@ -260,6 +261,26 @@ class LivyServer extends Logging {
server.context.addFilter(holder, "/*",
EnumSet.allOf(classOf[DispatcherType]))
info(s"SPNEGO auth enabled (principal = $principal)")
+ case authType @ LdapAuthenticationHandlerImpl.TYPE =>
Review comment:
I think the approach in https://github.com/apache/incubator-livy/pull/183 is
to be preferred
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] mgaido91 commented on a change in pull request #231: Add LDAP authentication for livy-server.
mgaido91 commented on a change in pull request #231: Add LDAP authentication for livy-server. URL: https://github.com/apache/incubator-livy/pull/231#discussion_r325132718 ## File path: pom.xml ## @@ -121,6 +121,10 @@ false + Review comment: not only for testing, right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] mgaido91 commented on a change in pull request #231: Add LDAP authentication for livy-server.
mgaido91 commented on a change in pull request #231: Add LDAP authentication for livy-server. URL: https://github.com/apache/incubator-livy/pull/231#discussion_r325132488 ## File path: conf/livy.conf.template ## @@ -160,6 +160,13 @@ # livy.server.auth.kerberos.keytab = # livy.server.auth.kerberos.name-rules = DEFAULT # +# livy.server.auth.type = ldap Review comment: I'd argue probably `ldap` is not the most common choice so I am not sure adding it here. In general, one thing livy lacks really is a good doc. I think it would be great creating a proper doc for the project and these should be put there IMHO. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] mgaido91 commented on issue #231: Add LDAP authentication for livy-server.
mgaido91 commented on issue #231: Add LDAP authentication for livy-server. URL: https://github.com/apache/incubator-livy/pull/231#issuecomment-532193892 @captainzmc please add the JIRA to the PR title This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] mgaido91 commented on issue #183: [LIVY-609] LDAP auth for Livy thriftserver
mgaido91 commented on issue #183: [LIVY-609] LDAP auth for Livy thriftserver URL: https://github.com/apache/incubator-livy/pull/183#issuecomment-532193530 @captainzmc thanks, I'd just enforce that the tasks are complementary and adding a feature doesn't require rework. Apart from that I am fine splitting the implementation in subtasks. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] yiheng commented on a change in pull request #230: [WIP][LIVY-664][Server] Spark application still running when Livy session creating was rejected
yiheng commented on a change in pull request #230: [WIP][LIVY-664][Server]
Spark application still running when Livy session creating was rejected
URL: https://github.com/apache/incubator-livy/pull/230#discussion_r325087100
##
File path: server/src/main/scala/org/apache/livy/server/SessionServlet.scala
##
@@ -131,6 +131,11 @@ abstract class SessionServlet[S <: Session, R <:
RecoveryMetadata](
if (tooManySessions) {
BadRequest(ResponseMessage("Rejected, too many sessions are being
created!"))
} else {
+// LIVY-664: Check if name is duplicated before create a session
+val newSessionName = sessionName(request)
+if (newSessionName.isDefined &&
sessionManager.get(newSessionName.get).isDefined) {
Review comment:
sure
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] yiheng commented on a change in pull request #230: [WIP][LIVY-664][Server] Spark application still running when Livy session creating was rejected
yiheng commented on a change in pull request #230: [WIP][LIVY-664][Server] Spark application still running when Livy session creating was rejected URL: https://github.com/apache/incubator-livy/pull/230#discussion_r325087158 ## File path: server/src/main/scala/org/apache/livy/server/SessionServlet.scala ## @@ -160,6 +165,11 @@ abstract class SessionServlet[S <: Session, R <: RecoveryMetadata]( */ protected def remoteUser(req: HttpServletRequest): String = req.getRemoteUser() + /** + * Return the session name Review comment: will fix it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] jerryshao commented on issue #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json
jerryshao commented on issue #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json URL: https://github.com/apache/incubator-livy/pull/234#issuecomment-532146892 This should have been fixed in #186 , what is difference here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] Jassy1994 commented on a change in pull request #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json
Jassy1994 commented on a change in pull request #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json URL: https://github.com/apache/incubator-livy/pull/234#discussion_r325074450 ## File path: repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala ## @@ -91,6 +91,9 @@ class SQLInterpreter( // Convert java BigDecimal type to Scala BigDecimal, because current version of // Json4s doesn't support java BigDecimal as a primitive type (LIVY-455). case i: java.math.BigDecimal => BigDecimal(i) +// Convert java.sql.Date type data to String, because java.sql.Date cannot extracted +// correctly with Json4s (LIVY-683). +case d: java.sql.Date => d.toString Review comment: OK，I will make it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #225: [LIVY-657][TEST]Fix travis failed on should not create sessions with duplicate names
codecov-io edited a comment on issue #225: [LIVY-657][TEST]Fix travis failed on should not create sessions with duplicate names URL: https://github.com/apache/incubator-livy/pull/225#issuecomment-528246905 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/225?src=pr=h1) Report > Merging [#225](https://codecov.io/gh/apache/incubator-livy/pull/225?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/e6b6ec060d18d33a300ffa7d766ad46d6c4c70cf?src=pr=desc) will **increase** coverage by `0.3%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-livy/pull/225?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #225 +/- ## === + Coverage 68.25% 68.56% +0.3% - Complexity 918 922 +4 === Files 100 100 Lines 5715 5717 +2 Branches867 867 === + Hits 3901 3920 +19 + Misses 1246 1236 -10 + Partials568 561 -7 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/225?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...in/java/org/apache/livy/rsc/driver/JobWrapper.java](https://codecov.io/gh/apache/incubator-livy/pull/225/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iV3JhcHBlci5qYXZh) | `82.85% <0%> (-5.72%)` | `8% <0%> (-1%)` | | | [...cala/org/apache/livy/repl/ProcessInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/225/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9Qcm9jZXNzSW50ZXJwcmV0ZXIuc2NhbGE=) | `57.14% <0%> (-2.39%)` | `6% <0%> (ø)` | | | [...scala/org/apache/livy/repl/PythonInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/225/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9QeXRob25JbnRlcnByZXRlci5zY2FsYQ==) | `47.5% <0%> (-0.63%)` | `9% <0%> (ø)` | | | [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/225/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `60.11% <0%> (-0.58%)` | `14% <0%> (ø)` | | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/225/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `35.46% <0%> (-0.5%)` | `11% <0%> (ø)` | | | [...apache/livy/server/batch/BatchSessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/225/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvYmF0Y2gvQmF0Y2hTZXNzaW9uU2VydmxldC5zY2FsYQ==) | `91.66% <0%> (+0.49%)` | `3% <0%> (ø)` | :arrow_down: | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/225/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `67.32% <0%> (+3.26%)` | `40% <0%> (ø)` | :arrow_down: | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/225/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `80.08% <0%> (+3.38%)` | `42% <0%> (+3%)` | :arrow_up: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/225/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `78.88% <0%> (+6.21%)` | `20% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/225?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/225?src=pr=footer). Last update [e6b6ec0...6ae5392](https://codecov.io/gh/apache/incubator-livy/pull/225?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] yiheng commented on a change in pull request #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json
yiheng commented on a change in pull request #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json URL: https://github.com/apache/incubator-livy/pull/234#discussion_r325049178 ## File path: repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala ## @@ -91,6 +91,9 @@ class SQLInterpreter( // Convert java BigDecimal type to Scala BigDecimal, because current version of // Json4s doesn't support java BigDecimal as a primitive type (LIVY-455). case i: java.math.BigDecimal => BigDecimal(i) +// Convert java.sql.Date type data to String, because java.sql.Date cannot extracted +// correctly with Json4s (LIVY-683). +case d: java.sql.Date => d.toString Review comment: Can we have a unit test to cover this case? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] jerryshao closed pull request #225: [LIVY-657][TEST]Fix travis failed on should not create sessions with duplicate names
jerryshao closed pull request #225: [LIVY-657][TEST]Fix travis failed on should not create sessions with duplicate names URL: https://github.com/apache/incubator-livy/pull/225 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang opened a new pull request #225: [LIVY-657][TEST]Fix travis failed on should not create sessions with duplicate names
runzhiwang opened a new pull request #225: [LIVY-657][TEST]Fix travis failed on should not create sessions with duplicate names URL: https://github.com/apache/incubator-livy/pull/225 ## What changes were proposed in this pull request? Fix travis failed on "should not create sessions with duplicate names" The cause of failed is as follows: 1. When session2 was stopped, it will call stop() method in Session.scala asynchronously by Scala Future. So when assert(session2.stopped), if the stop() method has not been executed asynchronously, the test will fail. ## How was this patch tested? Existed UT and IT. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] yiheng opened a new pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project li
yiheng opened a new pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report URL: https://github.com/apache/incubator-livy/pull/229 ## What changes were proposed in this pull request? This patch fixes the flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report. When JVM shutdown no gracefully in a test, the code coverage data file generated by jacoco may be corrupt. Jacoco will throw an exception when generate code coverage report. In Livy integration test, two test cases shut down no gracefully(one of them uses System.exit). We can find random failure when jacoco process code coverage data file generated by that test case. In this patch, we turn off the code coverage analysis on these two test cases. ## How was this patch tested? Compare the jacoco data file generated in the integration test. Before the fix, there're 18 files, and after the fix there're 16 files, which means the fix works. Run 10 builds on Travis each before and after the fix: 1. Before the fix: 3 builds failed due to the jacoco code coverage exception 2. After the fix: No build failed Existing UTs and ITs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] jerryshao closed pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-
jerryshao closed pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report URL: https://github.com/apache/incubator-livy/pull/229 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on projec
codecov-io edited a comment on issue #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report URL: https://github.com/apache/incubator-livy/pull/229#issuecomment-530197154 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=h1) Report > Merging [#229](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/4de6258aed7cbaff4a5909fe0d9db79fe869c46f?src=pr=desc) will **increase** coverage by `0.11%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #229 +/- ## + Coverage 68.32% 68.43% +0.11% - Complexity 919 922 +3 Files 100 100 Lines 5717 5718 +1 Branches867 867 + Hits 3906 3913 +7 + Misses 1247 1243 -4 + Partials564 562 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (ø)` | `13 <0> (ø)` | :arrow_down: | | [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `85.98% <100%> (+0.13%)` | `7 <0> (ø)` | :arrow_down: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `52.94% <0%> (-2.95%)` | `7% <0%> (ø)` | | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `35.96% <0%> (+0.49%)` | `11% <0%> (ø)` | :arrow_down: | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (+0.84%)` | `42% <0%> (+1%)` | :arrow_up: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+1.24%)` | `20% <0%> (ø)` | :arrow_down: | | [...in/java/org/apache/livy/rsc/driver/JobWrapper.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iV3JhcHBlci5qYXZh) | `88.57% <0%> (+5.71%)` | `9% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=footer). Last update [4de6258...3e98c2c](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on projec
codecov-io edited a comment on issue #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report URL: https://github.com/apache/incubator-livy/pull/229#issuecomment-530197154 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=h1) Report > Merging [#229](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/4de6258aed7cbaff4a5909fe0d9db79fe869c46f?src=pr=desc) will **increase** coverage by `0.11%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #229 +/- ## + Coverage 68.32% 68.43% +0.11% - Complexity 919 922 +3 Files 100 100 Lines 5717 5718 +1 Branches867 867 + Hits 3906 3913 +7 + Misses 1247 1243 -4 + Partials564 562 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `66.01% <0%> (ø)` | `13 <0> (ø)` | :arrow_down: | | [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `85.98% <100%> (+0.13%)` | `7 <0> (ø)` | :arrow_down: | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `52.94% <0%> (-2.95%)` | `7% <0%> (ø)` | | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `35.96% <0%> (+0.49%)` | `11% <0%> (ø)` | :arrow_down: | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `78.81% <0%> (+0.84%)` | `42% <0%> (+1%)` | :arrow_up: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (+1.24%)` | `20% <0%> (ø)` | :arrow_down: | | [...in/java/org/apache/livy/rsc/driver/JobWrapper.java](https://codecov.io/gh/apache/incubator-livy/pull/229/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iV3JhcHBlci5qYXZh) | `88.57% <0%> (+5.71%)` | `9% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=footer). Last update [4de6258...3e98c2c](https://codecov.io/gh/apache/incubator-livy/pull/229?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] captainzmc commented on issue #231: Add LDAP authentication for livy-server.
captainzmc commented on issue #231: Add LDAP authentication for livy-server. URL: https://github.com/apache/incubator-livy/pull/231#issuecomment-532103646 Hi @jerryshao This PR is based on the previous #20 , could you please help to review it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] codecov-io commented on issue #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json
codecov-io commented on issue #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json URL: https://github.com/apache/incubator-livy/pull/234#issuecomment-532103511 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/234?src=pr=h1) Report > Merging [#234](https://codecov.io/gh/apache/incubator-livy/pull/234?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/a9b2ddb951a043a960fc9feb69f0bf30f7e18fdd?src=pr=desc) will **decrease** coverage by `0.3%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-livy/pull/234?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #234 +/- ## - Coverage 68.63% 68.32% -0.31% Complexity 920 920 Files 100 100 Lines 5717 5718 +1 Branches867 868 +1 - Hits 3924 3907 -17 - Misses 1231 1246 +15 - Partials562 565 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/234?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...in/scala/org/apache/livy/repl/SQLInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/234/diff?src=pr=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TUUxJbnRlcnByZXRlci5zY2FsYQ==) | `63.63% <100%> (+1.13%)` | `9 <0> (ø)` | :arrow_down: | | [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/234/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `75.15% <0%> (-3.73%)` | `20% <0%> (ø)` | | | [...in/java/org/apache/livy/rsc/rpc/RpcDispatcher.java](https://codecov.io/gh/apache/incubator-livy/pull/234/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvUnBjRGlzcGF0Y2hlci5qYXZh) | `59.18% <0%> (-3.07%)` | `19% <0%> (-1%)` | | | [rsc/src/main/java/org/apache/livy/rsc/rpc/Rpc.java](https://codecov.io/gh/apache/incubator-livy/pull/234/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvUnBjLmphdmE=) | `79.28% <0%> (-2.96%)` | `12% <0%> (ø)` | | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/234/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `50% <0%> (-2.95%)` | `7% <0%> (ø)` | | | [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/234/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `66.01% <0%> (-1.31%)` | `40% <0%> (ø)` | | | [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/234/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `35.46% <0%> (-0.5%)` | `11% <0%> (ø)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/234/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `77.96% <0%> (ø)` | `40% <0%> (-1%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/234?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/234?src=pr=footer). Last update [a9b2ddb...0fbf548](https://codecov.io/gh/apache/incubator-livy/pull/234?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] jerryshao commented on issue #208: [LIVY-643][Server] Support for a Livy Server running on Windows Operating System
jerryshao commented on issue #208: [LIVY-643][Server] Support for a Livy Server running on Windows Operating System URL: https://github.com/apache/incubator-livy/pull/208#issuecomment-532098684 @davidOSUL sorry for late response, can you please tell what are the left issues for Livy running on Windows? >This PR also partially resolves some of the test issues when building in a Windows environment as described in https://issues.apache.org/jira/browse/LIVY-611. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] jerryshao closed pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-
jerryshao closed pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report URL: https://github.com/apache/incubator-livy/pull/229 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] yiheng opened a new pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project li
yiheng opened a new pull request #229: [LIVY-644][TEST] Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report URL: https://github.com/apache/incubator-livy/pull/229 ## What changes were proposed in this pull request? This patch fixes the flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report. When JVM shutdown no gracefully in a test, the code coverage data file generated by jacoco may be corrupt. Jacoco will throw an exception when generate code coverage report. In Livy integration test, two test cases shut down no gracefully(one of them uses System.exit). We can find random failure when jacoco process code coverage data file generated by that test case. In this patch, we turn off the code coverage analysis on these two test cases. ## How was this patch tested? Compare the jacoco data file generated in the integration test. Before the fix, there're 18 files, and after the fix there're 16 files, which means the fix works. Run 10 builds on Travis each before and after the fix: 1. Before the fix: 3 builds failed due to the jacoco code coverage exception 2. After the fix: No build failed Existing UTs and ITs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] Jassy1994 opened a new pull request #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json
Jassy1994 opened a new pull request #234: [LIVY-683] Livy SQLInterpreter get empty array when extract date format rows to json URL: https://github.com/apache/incubator-livy/pull/234 ## What changes were proposed in this pull request? The issue link: [LIVY-683](https://issues.apache.org/jira/browse/LIVY-683) to avoid this situation, we can transform date data to string data in rows, and this time rows can be extracted correctly. ## How was this patch tested? manual test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] captainzmc closed pull request #231: Add LDAP authentication for livy-server.
captainzmc closed pull request #231: Add LDAP authentication for livy-server. URL: https://github.com/apache/incubator-livy/pull/231 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] captainzmc opened a new pull request #231: Add LDAP authentication for livy-server.
captainzmc opened a new pull request #231: Add LDAP authentication for livy-server. URL: https://github.com/apache/incubator-livy/pull/231 ## What changes were proposed in this pull request? Currently, livy-server doesn't support LDAP Authentication from client to server(livy). We need to add LDAP authentication as that's preferable method due to security reasons. Here we reimplement LdapAuthenticationHandle, which is new in hadoop2.8+. ## How was this patch tested? UTs tests for this part have been added. We can test in UTs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
