[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/16061 The diff showing our progress so far has moved to https://github.com/apache-spark-on-k8s/spark/pull/200 for anyone following along. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/16061 +1 I think we should close this PR to avoid confusion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/16061 Work on this feature has moved to https://github.com/apache-spark-on-k8s/spark. The exact diff we are working with is this: https://github.com/apache-spark-on-k8s/spark/pull/1. Feel free to provide feedback on the fork's PR. @erikerlandson can you close this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16061 We should acknowledge people that have done this in the merged commit, but I don't think it'd make a lot of sense to do a merge rather than squash. The initial Spark SQL commit was a single commit even though it was a much larger project https://github.com/apache/spark/pull/146 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/16061 @iyanuobidele the current head of this branch should be equivalent, but it can't hurt to rebase just in case. @foxish yesterday occurred to me that it would've been sensible set up a GH organization for this little consortium, and run the PR from there. Presumably we still could. The drawback to making any change to it now is losing the dialog that's happened here already. The problem I'm interested in solving even more is ensuring that everybody who contributes to this PR gets logged somehow in the upstream commit history when this PR eventually gets merged. I don't think that would happen in the case of a typical squash-merge, although I assume it _would_ using a rebase merge. @rxin do you have any thoughts on that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/16061 I would think that it would be easier if we continue to keep PRs and issues in foxish/spark since we have the previous PRs and issues over there. It should be easy enough to pick those commits up and update this branch once we review and merge them. It doesn't matter where it lives really, except that it seems like most of it is already on one place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user iyanuobidele commented on the issue: https://github.com/apache/spark/pull/16061 @erikerlandson are there any new commits to this branch ? Or it's at the same level as @foxish's k8s-support. In case I need to rebase --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/16061 At the risk of redundant announcements, I am going to present this spark-on-kube topic at the [Kubernetes SIG-Apps](https://github.com/kubernetes/community/tree/master/sig-apps) next Monday (Dec 5). It will be similar to my earlier [OpenShift briefing](https://blog.openshift.com/openshift-commons-big-data-sig-2-running-apache-spark-natively-on-kubernetes-with-openshift/) but the focus will be on kube specifically, and I'll update it to cover all the latest developments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/16061 Since we have at least four people with interest in working on the code, I have a workflow proposal, which is that we all submit candidate updates to #16061 in the form of PRs against [erikerlandson:spark-18278-k8s-native](https://github.com/erikerlandson/spark/tree/spark-18278-k8s-native) (recursive PRs against a PR!) I did this with Anirudh against his `k8s-support` branch, and it seemed to work fine. It should help keep asynchronous contributions sane, and provides the usual durable public forum for discussing the updates as they evolve. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/16061 Cross posting link to [the proposal](https://issues.apache.org/jira/secure/attachment/12840988/SPARK-18278%20-%20Spark%20on%20Kubernetes%20Design%20Proposal.pdf) here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16061 Another external scheduler backend I'm aware of is Two Sigma's scheduler backend for the system they've created called [Cook](https://github.com/twosigma/Cook). See [CoarseCookSchedulerBackend.scala](https://github.com/twosigma/spark/blob/2.0.2-cook/core/src/main/scala/org/apache/spark/scheduler/CoarseCookSchedulerBackend.scala) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/16061 @rxin, when you say "move all resource managers" does that mean "move scheduler back-ends for mesos, yarn, etc, into some `resource-managers` sub-project" ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16061 @erikerlandson For the RAT failure, you may either add Apache license header to newly added files or add the file to `dev/.rat-excludes`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user tnachen commented on the issue: https://github.com/apache/spark/pull/16061 @rxin Makes sense, @srowen also talked about starting the discussion of having a better support for external cluster managers as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16061 One thing - can we submit a separate pr to move all resource managers into resource-managers/yarn resource-managers/mesos ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16061 This is pretty cool. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16061 **[Test build #69335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69335/consoleFull)** for PR 16061 at commit [`8584913`](https://github.com/apache/spark/commit/8584913d51c1f68c8f9d3cdf142947cd35cdfcc2). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16061 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16061 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69335/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16061 **[Test build #69335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69335/consoleFull)** for PR 16061 at commit [`8584913`](https://github.com/apache/spark/commit/8584913d51c1f68c8f9d3cdf142947cd35cdfcc2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
