[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13713 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13713#discussion_r83297642
--- Diff:
mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
---
@@ -369,9 +369,11 @@ trait MesosSchedulerUtils extends Logging {
sc.executorMemory
}
- def setupUris(uris: String, builder: CommandInfo.Builder): Unit = {
+ def setupUris(uris: String,
+builder: CommandInfo.Builder,
+useFetchCache: Boolean = false): Unit = {
uris.split(",").foreach { uri =>
- builder.addUris(CommandInfo.URI.newBuilder().setValue(uri.trim()))
+
builder.addUris(CommandInfo.URI.newBuilder().setValue(uri.trim()).setCache(useFetchCache))
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/13713#discussion_r83271593
--- Diff:
mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
---
@@ -369,9 +369,11 @@ trait MesosSchedulerUtils extends Logging {
sc.executorMemory
}
- def setupUris(uris: String, builder: CommandInfo.Builder): Unit = {
+ def setupUris(uris: String,
+builder: CommandInfo.Builder,
+useFetchCache: Boolean = false): Unit = {
uris.split(",").foreach { uri =>
- builder.addUris(CommandInfo.URI.newBuilder().setValue(uri.trim()))
+
builder.addUris(CommandInfo.URI.newBuilder().setValue(uri.trim()).setCache(useFetchCache))
--- End diff --
s/fetch/fetcher
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13713#discussion_r83113285
--- Diff:
mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala
---
@@ -59,6 +59,8 @@ private[spark] class MesosFineGrainedSchedulerBackend(
private[mesos] val mesosExecutorCores =
sc.conf.getDouble("spark.mesos.mesosExecutor.cores", 1)
+ private val useFetchCache =
sc.conf.getBoolean("spark.mesos.fetchCache.enable", false)
--- End diff --
Removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13713#discussion_r83113266
--- Diff:
mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -129,6 +129,7 @@ private[spark] class MesosClusterScheduler(
private val queuedCapacity = conf.getInt("spark.mesos.maxDrivers", 200)
private val retainedDrivers = conf.getInt("spark.mesos.retainedDrivers",
200)
private val maxRetryWaitTime =
conf.getInt("spark.mesos.cluster.retry.wait.max", 60) // 1 minute
+ private val useFetchCache =
conf.getBoolean("spark.mesos.fetchCache.enable", false)
--- End diff --
Good idea but outside scope of this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user drcrallen commented on a diff in the pull request: https://github.com/apache/spark/pull/13713#discussion_r83113237 --- Diff: docs/running-on-mesos.md --- @@ -506,8 +506,13 @@ See the [configuration page](configuration.html) for information on Spark config since this configuration is just a upper limit and not a guaranteed amount. - - + + spark.mesos.fetchCache.enable + false + +If set to `true`, all URIs (example: `spark.executor.uri`, `spark.mesos.uris`) will be eligible for caching by the [Mesos fetch cache](http://mesos.apache.org/documentation/latest/fetcher/) --- End diff -- fixed both --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13713#discussion_r83112712
--- Diff:
mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala
---
@@ -463,6 +463,21 @@ class MesosCoarseGrainedSchedulerBackendSuite extends
SparkFunSuite
assert(launchedTasks.head.getCommand.getUrisList.asScala(0).getValue
== url)
}
+ test("mesos supports setting fetcher cache") {
+val url = "spark.spark.spark.com"
+setBackend(Map(
+ "spark.mesos.fetchCache.enable" -> "true",
+ "spark.executor.uri" -> url
+), false)
+val offers = List(Resources(backend.executorMemory(sc), 1))
+offerResources(offers)
+// Don't crash!
--- End diff --
it originally didn't do much except check that it didn't fail. I have since
added proper checking of the task info for if the fetcher cache was enabled.
Will remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user mgummelt commented on a diff in the pull request: https://github.com/apache/spark/pull/13713#discussion_r82846312 --- Diff: docs/running-on-mesos.md --- @@ -506,8 +506,13 @@ See the [configuration page](configuration.html) for information on Spark config since this configuration is just a upper limit and not a guaranteed amount. - - + + spark.mesos.fetchCache.enable --- End diff -- s/fetchCache/fetcherCache (et al) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user mgummelt commented on a diff in the pull request: https://github.com/apache/spark/pull/13713#discussion_r82846146 --- Diff: docs/running-on-mesos.md --- @@ -506,8 +506,13 @@ See the [configuration page](configuration.html) for information on Spark config since this configuration is just a upper limit and not a guaranteed amount. - - + + spark.mesos.fetchCache.enable + false + +If set to `true`, all URIs (example: `spark.executor.uri`, `spark.mesos.uris`) will be eligible for caching by the [Mesos fetch cache](http://mesos.apache.org/documentation/latest/fetcher/) --- End diff -- s/fetch/fetcher s/eligible for caching/cached (they're not just eligible, they are in fact cached) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/13713#discussion_r82846761
--- Diff:
mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala
---
@@ -463,6 +463,21 @@ class MesosCoarseGrainedSchedulerBackendSuite extends
SparkFunSuite
assert(launchedTasks.head.getCommand.getUrisList.asScala(0).getValue
== url)
}
+ test("mesos supports setting fetcher cache") {
+val url = "spark.spark.spark.com"
+setBackend(Map(
+ "spark.mesos.fetchCache.enable" -> "true",
+ "spark.executor.uri" -> url
+), false)
+val offers = List(Resources(backend.executorMemory(sc), 1))
+offerResources(offers)
+// Don't crash!
--- End diff --
??
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/13713#discussion_r82846430
--- Diff:
mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -129,6 +129,7 @@ private[spark] class MesosClusterScheduler(
private val queuedCapacity = conf.getInt("spark.mesos.maxDrivers", 200)
private val retainedDrivers = conf.getInt("spark.mesos.retainedDrivers",
200)
private val maxRetryWaitTime =
conf.getInt("spark.mesos.cluster.retry.wait.max", 60) // 1 minute
+ private val useFetchCache =
conf.getBoolean("spark.mesos.fetchCache.enable", false)
--- End diff --
We really need to factor out all these conf vars like YARN does at some
point
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/13713#discussion_r82846894
--- Diff:
mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala
---
@@ -59,6 +59,8 @@ private[spark] class MesosFineGrainedSchedulerBackend(
private[mesos] val mesosExecutorCores =
sc.conf.getDouble("spark.mesos.mesosExecutor.cores", 1)
+ private val useFetchCache =
sc.conf.getBoolean("spark.mesos.fetchCache.enable", false)
--- End diff --
Please remove changes to the fine grained scheduler. It's deprecated, and
we shouldn't be adding any new features to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user drcrallen commented on a diff in the pull request: https://github.com/apache/spark/pull/13713#discussion_r82714540 --- Diff: docs/running-on-mesos.md --- @@ -506,8 +506,13 @@ See the [configuration page](configuration.html) for information on Spark config since this configuration is just a upper limit and not a guaranteed amount. - - + + spark.mesos.fetchCache.enable + false + +If set to `true`, all URIs in `spark.mesos.uris` will be eligible for caching by the [Mesos fetch cache](http://mesos.apache.org/documentation/latest/fetcher/) --- End diff -- Added more info. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/13713#discussion_r82695978 --- Diff: docs/running-on-mesos.md --- @@ -506,8 +506,13 @@ See the [configuration page](configuration.html) for information on Spark config since this configuration is just a upper limit and not a guaranteed amount. - - + + spark.mesos.fetchCache.enable + false + +If set to `true`, all URIs in `spark.mesos.uris` will be eligible for caching by the [Mesos fetch cache](http://mesos.apache.org/documentation/latest/fetcher/) --- End diff -- From the implementation you actually set all downloadable URIs (like spark.executor.uri, jarUrl, etc) to be fetcher cachable. I think we need to be more explicit here that it's more than just spark.mesos.uris --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13713#discussion_r82695810
--- Diff:
mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala
---
@@ -463,6 +463,21 @@ class MesosCoarseGrainedSchedulerBackendSuite extends
SparkFunSuite
assert(launchedTasks.head.getCommand.getUrisList.asScala(0).getValue
== url)
}
+ test("mesos supports setting fetcher") {
--- End diff --
s/supports setting fetcher/supports setting fetcher cache/g
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch ...
GitHub user drcrallen opened a pull request: https://github.com/apache/spark/pull/13713 [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache in coarse executor backend Mesos 0.23.0 introduces a Fetch Cache feature http://mesos.apache.org/documentation/latest/fetcher/ which allows caching of resources specified in command URIs. This patch: * Updates the Mesos shaded protobuf dependency to 0.23.0 * Allows setting `spark.mesos.fetchCache.enable` to enable the fetch cache for all specified URIs. (URIs must be specified for the setting to have any affect) * Updates documentation for Mesos configuration with the new setting. This patch does NOT: * Allow for per-URI caching configuration. The cache setting is global to ALL URIs for the command. You can merge this pull request into a Git repository by running: $ git pull https://github.com/metamx/spark SPARK15994 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13713.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13713 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
