[GitHub] spark issue #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark dispatche...

2017-08-07 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18528 @srowen could I get a merge pls? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-07 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @srowen or @vanzin can I get a merge pls? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18837: [Spark-20812][Mesos] Add secrets support to the dispatch...

2017-08-04 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18837 Btw a lot of secrets are defined in the code base: spark.secret.mesos,spark.mesos.driver.secret.name, spark.authenticate.secret (https://spark.apache.org/docs/latest/security.html) do you

[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...

2017-08-04 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131354746 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -529,18 +560,54 @@ private

[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...

2017-08-04 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131351705 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -510,12 +510,20 @@ trait

[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...

2017-08-04 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131350894 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config

[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...

2017-08-04 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131348687 --- Diff: resource-managers/mesos/pom.xml --- @@ -29,7 +29,7 @@ Spark Project Mesos mesos -1.0.0 +1.3.0-rc1

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-03 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @vanzin fixed the issues. Please give it another try or merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark dispatche...

2017-08-03 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18528 @srowen fixed, ready for merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark di...

2017-08-02 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18528#discussion_r130912865 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala --- @@ -76,6 +77,17 @@ private[mesos] class

[GitHub] spark pull request #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark di...

2017-08-02 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18528#discussion_r130912963 --- Diff: docs/running-on-mesos.md --- @@ -153,6 +153,8 @@ can find the results of the driver from the Mesos Web UI. To use cluster mode, you must

[GitHub] spark pull request #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark di...

2017-08-02 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18528#discussion_r130912751 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala --- @@ -76,6 +77,17 @@ private[mesos] class

[GitHub] spark issue #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark dispatche...

2017-08-02 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18528 thnx @srowen will fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-02 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r130911557 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -66,4 +70,16 @@ object DriverWrapper { System.exit(-1

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-02 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r130909789 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -1315,6 +1294,80 @@ private[spark] object SparkSubmitUtils

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-02 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r130908799 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -1315,6 +1294,80 @@ private[spark] object SparkSubmitUtils

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-02 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r130908694 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -1315,6 +1294,80 @@ private[spark] object SparkSubmitUtils

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-02 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r130908550 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -1315,6 +1294,80 @@ private[spark] object SparkSubmitUtils

[GitHub] spark issue #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark dispatche...

2017-08-01 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18528 @srowen Could you pls have a look and merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode

2017-07-31 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18784 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode

2017-07-31 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18784 @srowen this test https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80079/ seems flaky, I run the suite locally and works fine (13 tests finished successfully). --- If your

[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode

2017-07-31 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18784 @srowen I made a new commit but didn't get a new build... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode

2017-07-31 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18784 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #18784: [SPARK-21559][Mesos] remove mesos fine-grained mo...

2017-07-31 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18784#discussion_r130383697 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -784,42 +784,6 @@ class TaskSchedulerImplSuite extends

[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode

2017-07-31 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18784 Fixed the test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode

2017-07-31 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18784 @srowen ok, yes. I have discussed this with Art and Suzan from Mesosphere and we made the decision to remove it as it is deprecated for so long. I am waiting for their comments here. In the

[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode

2017-07-31 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18784 @ArtRand @susanxhuynh pls review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #18784: [SPARK-21559][Mesos] remove mesos fine-grained mo...

2017-07-31 Thread skonto
GitHub user skonto opened a pull request: https://github.com/apache/spark/pull/18784 [SPARK-21559][Mesos] remove mesos fine-grained mode ## What changes were proposed in this pull request? Removes mesos fine-grained mode. Specifically: - Updates docs. - Renames

[GitHub] spark issue #13143: [SPARK-15359] [Mesos] Mesos dispatcher should handle DRI...

2017-07-27 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/13143 @ArtRand @susanxhuynh could please review this before we call for a merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark dispatche...

2017-07-26 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18528 @vanzin Could we get a merge please and/or a review? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark dispatche...

2017-07-25 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18528 @ArtRand I tested the PR on 1.4 have a look pls. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #18098: [SPARK-16944][Mesos] Improve data locality when l...

2017-07-24 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18098#discussion_r129176526 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -72,6 +72,10 @@ package object config { private[spark] val

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-07-24 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @vanzin I did the required refactoring, addressing the issues you mentioned and added to the description a better message so its available in git log when this PR is merged. --- If your project is

[GitHub] spark issue #18705: [SPARK-21502][Mesos] fix --supervise for mesos in cluste...

2017-07-23 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18705 @vanzin thnx for the review. I updated the PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-23 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r128920834 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -18,8 +18,17 @@ package org.apache.spark.deploy.worker

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-21 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r128835800 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -43,7 +52,7 @@ object DriverWrapper

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-21 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r128835610 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -66,4 +77,68 @@ object DriverWrapper { System.exit(-1

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-21 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r128835523 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -66,4 +77,68 @@ object DriverWrapper { System.exit(-1

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-21 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r128834993 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -66,4 +77,68 @@ object DriverWrapper { System.exit(-1

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-21 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r128834674 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -66,4 +77,68 @@ object DriverWrapper { System.exit(-1

[GitHub] spark issue #18705: [SPARK-21502][Mesos] fix --supervise for mesos in cluste...

2017-07-21 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18705 @susanxhuynh @ArtRand pls review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #18705: [SPARK-21502][Mesos] fix --supervise for mesos in...

2017-07-21 Thread skonto
GitHub user skonto opened a pull request: https://github.com/apache/spark/pull/18705 [SPARK-21502][Mesos] fix --supervise for mesos in cluster mode ## What changes were proposed in this pull request? With supervise enabled for a driver so far re-launching it was failing because

[GitHub] spark issue #18674: [SPARK-21456][MESOS] Make the driver failover_timeout co...

2017-07-19 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18674 @susanxhuynh no its fine. @vanzin @srowen could we have a merge pls or if you want to go through it first? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #18674: [SPARK-21456][MESOS] Make the driver failover_timeout co...

2017-07-19 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18674 I saw this issue: https://issues.apache.org/jira/browse/SPARK-21458 refering to supervision. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #18674: [SPARK-21456][MESOS] Make the driver failover_timeout co...

2017-07-19 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18674 So the task updates will not have any effect cool, and the framework tear down will come later on if the timer expires. while checking the docs since this is obviously useful for the

[GitHub] spark issue #18674: [SPARK-21456][MESOS] Make the driver failover_timeout co...

2017-07-19 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18674 @susanxhuynh what are the implications for the dispatcher? The driver runs as a task so the dispatcher will get an task status update earlier if the task has failed correct? --- If your project is

[GitHub] spark pull request #18674: [SPARK-21456][MESOS] Make the driver failover_tim...

2017-07-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18674#discussion_r128257884 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala --- @@ -58,9 +58,16 @@ package object config

[GitHub] spark pull request #18098: [SPARK-16944][Mesos] Improve data locality when l...

2017-07-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18098#discussion_r128243453 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -72,6 +72,10 @@ package object config { private[spark] val

[GitHub] spark pull request #18098: [SPARK-16944][Mesos] Improve data locality when l...

2017-07-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18098#discussion_r128235076 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -502,6 +526,25

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-07-18 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @vanzin thnx done, I changed the title. I am willing to work on the distributed cache but needs time for sure. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-07-17 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 I agree that's why I didnt try to do the refactoring... I thought it would take a long time... As for the cluster mode some people use it due to the supervise feature, guess not too many, but

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-07-17 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @jerryshao I agree I had shared this thought with @vanzin for utilizing the distributed cache or whatever is available there in the other PR for mesos. For mesos I am also going to discuss this with

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-07-17 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @jerryshao @vanzin I updated the PR, pls review. In standalone cluster mode with this new update I can resolve jars paths of the form: /tmp/tmp7255787727862122372/jarname.jar, file:///home

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-07-17 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 One thing I noticed is that if you use https you need to set : --conf spark.ssl.enabled=true --conf spark.ssl.protocol=TLS but this does nto apply to the user jar. I think that is an issue. --- If

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-07-16 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @jerryshao > Are you trying to support --packages in standalone cluster? Yes that is my intention as stated in the description. I will proceed with the changes thnx

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r127614213 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -66,4 +75,50 @@ object DriverWrapper { System.exit(-1

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r127614182 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -66,4 +75,50 @@ object DriverWrapper { System.exit(-1

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r127613972 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -473,6 +474,12 @@ object SparkSubmit extends CommandLineUtils

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for standalon...

2017-07-13 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @vanzin pls review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-07-13 Thread skonto
GitHub user skonto opened a pull request: https://github.com/apache/spark/pull/18630 [SPARK-12559][SPARK SUBMIT] fix --packages for standalone ## What changes were proposed in this pull request? Fixes --packages flag for the standalone mode. ## How was this patch

[GitHub] spark issue #18587: [SPARK-21403][Mesos] fix --packages for mesos

2017-07-13 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18587 thnx @vanzin I am working on some code for the standalone case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18587: [SPARK-12559][Mesos] fix --packages for mesos

2017-07-12 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18587 @vanzin > args.jars = mergeFileLists(args.jars, resolvedMavenCoordinates) My understanding: Files intended for the appmaster are handled by copying to hdfs: https://github.

[GitHub] spark pull request #18587: [SPARK-12559][Mesos] fix --packages for mesos

2017-07-10 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18587#discussion_r126382374 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -269,6 +269,25 @@ object SparkSubmit extends CommandLineUtils

[GitHub] spark pull request #18587: [SPARK-12559][mesos] fix --packages for mesos

2017-07-10 Thread skonto
GitHub user skonto opened a pull request: https://github.com/apache/spark/pull/18587 [SPARK-12559][mesos] fix --packages for mesos ## What changes were proposed in this pull request? Fixes --packages flag for mesos in cluster mode. Probably I will handle standalone and Yarn in

[GitHub] spark issue #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark dispatche...

2017-07-04 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18528 @mgummelt @HyukjinKwon pls review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #18528: [SPARK-13041][Mesos] Adds sandbox uri to spark di...

2017-07-04 Thread skonto
GitHub user skonto opened a pull request: https://github.com/apache/spark/pull/18528 [SPARK-13041][Mesos] Adds sandbox uri to spark dispatcher ui ## What changes were proposed in this pull request? Adds a sandbox link per driver in the dispatcher ui with minimal changes after a

[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...

2017-07-04 Thread skonto
Github user skonto closed the pull request at: https://github.com/apache/spark/pull/11887 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...

2017-07-04 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/11887 @HyukjinKwon I will close this one and start fresh. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...

2017-06-20 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/11887 Great thnx @haosdent :)I will proceed with updating the PR. Thnx a lot! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...

2017-06-07 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/11887 @HyukjinKwon I will have a look and let you know, please don't close it for now. There was finally progress at the mesos side: https://reviews.apache.org/r/58872/ --- If your proje

[GitHub] spark pull request #17109: [SPARK-19740][MESOS]Add support in Spark to pass ...

2017-04-11 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17109#discussion_r111045370 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -99,6 +99,26 @@ private

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Increase default refuse_seconds tim...

2017-03-04 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 @mgummelt do we wan to keep the suppress/revive technique? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Increase default refuse_seconds tim...

2017-03-03 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 @srown There are parts for refactoring only purposes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Increase default refuse_seconds tim...

2017-03-03 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 @mgummelt LGTM. Thanks fo rthe clarifications. @srowen can we get a merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Increase default refuse_seconds tim...

2017-03-01 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 But this time is the refuse time correct? As stated here: https://issues.apache.org/jira/browse/MESOS-3202 I have 30 seconds for osme other framework to accept resources in the list otherwise the

[GitHub] spark issue #17045: [SPARK-19373][MESOS] Base spark.scheduler.minRegisteredR...

2017-02-28 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17045 LGTM run it locally works fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #17045: [SPARK-19373][MESOS] Base spark.scheduler.minRegi...

2017-02-28 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17045#discussion_r103411216 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -256,7 +259,7

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 Ok I see. LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 Ok like the Cassandra case you mean right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17031: [SPARK-19702][MESOS] Add suppress/revive support ...

2017-02-27 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17031#discussion_r103287098 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala --- @@ -24,6 +24,7

[GitHub] spark pull request #17031: [SPARK-19702][MESOS] Add suppress/revive support ...

2017-02-27 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17031#discussion_r103281854 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -582,141 +688,33 @@ private

[GitHub] spark pull request #17031: [SPARK-19702][MESOS] Add suppress/revive support ...

2017-02-27 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17031#discussion_r103281366 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -737,13 +735,75 @@ private

[GitHub] spark pull request #17031: [SPARK-19702][MESOS] Add suppress/revive support ...

2017-02-27 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17031#discussion_r103280797 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -737,13 +735,75 @@ private

[GitHub] spark pull request #17031: [SPARK-19702][MESOS] Add suppress/revive support ...

2017-02-27 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17031#discussion_r103280133 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala --- @@ -32,7 +32,7 @@ private[mesos] class

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 "The only way to fix this generally is to implement some periodic timer that calls reviveOffers() if there are queued/pending drivers to be scheduled. This can be chatty and complicates the cod

[GitHub] spark pull request #17031: [SPARK-19702][MESOS] Add suppress/revive support ...

2017-02-27 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17031#discussion_r103261951 --- Diff: resource-managers/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterSchedulerSuite.scala --- @@ -48,45 +48,50 @@ class

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 @mgummelt Yes they should look at the logs but how do they know this is something that requires action from their side and not a cluster issue or anything else. It should be documented since it is

[GitHub] spark issue #17031: [SPARK-19702] Add suppress/revive support to the Mesos S...

2017-02-23 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 How the operator should know about starvation? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17031: [SPARK-19702] Add suppress/revive support to the ...

2017-02-23 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17031#discussion_r102707406 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala --- @@ -24,6 +24,7

[GitHub] spark issue #13072: [SPARK-15288] [Mesos] Mesos dispatcher should handle gra...

2017-02-15 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/13072 Ideally all processes we have should handle thread termination correctly, same applies to MesosClusterDispatcher. Btw I think the call in Executor.scala should be done in static code, for example in

[GitHub] spark issue #13072: [SPARK-15288] [Mesos] Mesos dispatcher should handle gra...

2017-02-15 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/13072 @devaraj-kavali Which threads/Executor you are referring to in MesosClusterDispatcher? Could you provide some pointers? --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #16755: [MESOS] Support constraints in spark-dispatcher

2017-02-15 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/16755 Please fix the title for the PR and include the related jira issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...

2017-02-10 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/11887 @HyukjinKwon probably. Let me ask Michael. @mgummelt what do you think? what options do we have? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #13072: [SPARK-15288] [Mesos] Mesos dispatcher should handle gra...

2017-01-21 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/13072 @mgummelt @srowen should we get a merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14650: [SPARK-17062][MESOS] add conf option to mesos dispatcher

2016-11-20 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/14650 @vanzin if there is anything to fix I can do it if you want I can do another parse to check it against the style guide. For all the style nit stuff I guess sometimes is a matter of taste... --- If

[GitHub] spark issue #14650: [SPARK-17062][MESOS] add conf option to mesos dispatcher

2016-11-18 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/14650 @rxin @srowen could I get merge pls if there are no other issues? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14650: [SPARK-17062][MESOS] add conf option to mesos dispatcher

2016-11-17 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/14650 @vanzin I updated the PR. Thank you for the review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #14650: [SPARK-17062][MESOS] add conf option to mesos dis...

2016-11-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/14650#discussion_r88572655 --- Diff: core/src/main/scala/org/apache/spark/util/CommandLineUtils.scala --- @@ -0,0 +1,57 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #14650: [SPARK-17062][MESOS] add conf option to mesos dis...

2016-11-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/14650#discussion_r88570905 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1962,6 +1962,21 @@ private[spark] object Utils extends Logging { path

[GitHub] spark pull request #14650: [SPARK-17062][MESOS] add conf option to mesos dis...

2016-11-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/14650#discussion_r88570567 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -412,10 +412,8 @@ private[deploy] class SparkSubmitArguments(args

<    1   2   3   4   5   6   7   8   9   10   >