[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18837 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r135852456
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -529,18 +566,113 @@ private[spark] class MesosClusterScheduler(
val appName = desc.conf.get("spark.app.name")
+val driverLabels =
MesosProtoUtils.mesosLabels(desc.conf.get(config.DRIVER_LABELS)
+ .getOrElse(""))
+
TaskInfo.newBuilder()
.setTaskId(taskId)
.setName(s"Driver for ${appName}")
.setSlaveId(offer.offer.getSlaveId)
.setCommand(buildDriverCommand(desc))
+ .setContainer(getContainerInfo(desc))
.addAllResources(cpuResourcesToUse.asJava)
.addAllResources(memResourcesToUse.asJava)
-
.setLabels(MesosProtoUtils.mesosLabels(desc.conf.get(config.DRIVER_LABELS).getOrElse("")))
- .setContainer(MesosSchedulerBackendUtil.containerInfo(desc.conf))
+ .setLabels(driverLabels)
.build
}
+ private def getContainerInfo(desc: MesosDriverDescription):
ContainerInfo.Builder = {
+val containerInfo = MesosSchedulerBackendUtil.containerInfo(desc.conf)
+
+getSecretVolume(desc).foreach { volume =>
+ if (volume.getSource.getSecret.getReference.isInitialized) {
+logInfo(s"Setting reference secret
${volume.getSource.getSecret.getReference.getName}" +
+ s"on file ${volume.getContainerPath}")
+ } else {
+logInfo(s"Setting secret on file name=${volume.getContainerPath}")
+ }
+ containerInfo.addVolumes(volume)
+}
+
+containerInfo
+ }
+
+
+ private def getSecrets(desc: MesosDriverDescription): Seq[Secret] = {
+def createValueSecret(data: String): Secret = {
+ Secret.newBuilder()
+.setType(Secret.Type.VALUE)
+
.setValue(Secret.Value.newBuilder().setData(ByteString.copyFrom(data.getBytes)))
+.build()
+}
+
+def createReferenceSecret(name: String): Secret = {
+ Secret.newBuilder()
+.setReference(Secret.Reference.newBuilder().setName(name))
+.setType(Secret.Type.REFERENCE)
+.build()
+}
+
+val referenceSecrets: Seq[Secret] = {
+ if (desc.conf.get(config.SECRET_NAME).isDefined) {
--- End diff --
Are you going to fix this kind of code according to my previous suggestion?
```
desc.conf.get(config.SECRET_NAME).getOrElse(Nil).map(...)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r135851961 --- Diff: docs/running-on-mesos.md --- @@ -481,6 +483,43 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey --- End diff -- The documentation now doesn't match the code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user ArtRand commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r135787616
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala
---
@@ -58,12 +58,43 @@ package object config {
private[spark] val DRIVER_LABELS =
ConfigBuilder("spark.mesos.driver.labels")
- .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
+ .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
"pairs should be separated by a colon, and commas used to list
more than one." +
"Ex. key:value,key2:value2")
.stringConf
.createOptional
+ private[spark] val SECRET_NAME =
+ConfigBuilder("spark.mesos.driver.secret.name")
--- End diff --
I suppose this naming came from the Mesos protos where
[secret](https://github.com/apache/mesos/blob/1.3.0/include/mesos/mesos.proto#L2255)
and
[name](https://github.com/apache/mesos/blob/1.3.0/include/mesos/mesos.proto#L2041)
are singular and
[volumes](https://github.com/apache/mesos/blob/1.3.0/include/mesos/mesos.proto#L2532)
is plural. However the last of which is hidden in this case. I agree `names`
is perhaps more intuitive?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r135640958
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -383,15 +385,57 @@ private[spark] class MesosClusterScheduler(
v => s"$v
-Dspark.mesos.driver.frameworkId=${getDriverFrameworkID(desc)}"
)
-val env = desc.conf.getAllWithPrefix("spark.mesos.driverEnv.") ++
commandEnv
+val driverEnv = desc.conf.getAllWithPrefix("spark.mesos.driverEnv.")
+val env = driverEnv ++ commandEnv
val envBuilder = Environment.newBuilder()
+
+// add normal environment variables
env.foreach { case (k, v) =>
envBuilder.addVariables(Variable.newBuilder().setName(k).setValue(v))
}
+
+// add secret environment variables
+getSecretEnvVar(desc).foreach { variable =>
+ if (variable.getSecret.getReference.isInitialized) {
+logInfo(s"Setting reference secret
${variable.getSecret.getReference.getName}" +
+ s"on file ${variable.getName}")
+ } else {
+logInfo(s"Setting secret on environment variable
name=${variable.getName}")
+ }
+ envBuilder.addVariables(variable)
+}
+
envBuilder.build()
}
+ private def getSecretEnvVar(desc: MesosDriverDescription):
List[Variable] = {
+val secrets = getSecrets(desc)
+val secretEnvKeys = {
+ if (desc.conf.get(config.SECRET_ENVKEY).isDefined) {
--- End diff --
This kind of pattern is not really necessary.
You can either use `conf.get(BLAH).getOrElse(Nil)`, or declare `Nil` as the
default when creating the config constant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r135641205
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala
---
@@ -58,12 +58,43 @@ package object config {
private[spark] val DRIVER_LABELS =
ConfigBuilder("spark.mesos.driver.labels")
- .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
+ .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
"pairs should be separated by a colon, and commas used to list
more than one." +
"Ex. key:value,key2:value2")
.stringConf
.createOptional
+ private[spark] val SECRET_NAME =
+ConfigBuilder("spark.mesos.driver.secret.name")
--- End diff --
Are you going to keep the config names in the singular? That's both odd
since they're lists, and inconsistent with the config above
("spark.mesos.driver.labels").
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r135353557
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala
---
@@ -58,13 +58,40 @@ package object config {
private[spark] val DRIVER_LABELS =
ConfigBuilder("spark.mesos.driver.labels")
- .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
+ .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
"pairs should be separated by a colon, and commas used to list
more than one." +
"Ex. key:value,key2:value2")
.stringConf
.createOptional
- private[spark] val DRIVER_FAILOVER_TIMEOUT =
+ private[spark] val SECRET_NAME =
+ConfigBuilder("spark.mesos.driver.secret.name")
+ .doc("A comma-separated list of secret references. Consult the Mesos
Secret protobuf for " +
+"more information.")
+ .stringConf
+ .createOptional
+
+ private[spark] val SECRET_VALUE =
+ConfigBuilder("spark.mesos.driver.secret.value")
+ .doc("A comma-separated list of secret values.")
--- End diff --
Ditto.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r135353586
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala
---
@@ -58,13 +58,40 @@ package object config {
private[spark] val DRIVER_LABELS =
ConfigBuilder("spark.mesos.driver.labels")
- .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
+ .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
"pairs should be separated by a colon, and commas used to list
more than one." +
"Ex. key:value,key2:value2")
.stringConf
.createOptional
- private[spark] val DRIVER_FAILOVER_TIMEOUT =
+ private[spark] val SECRET_NAME =
+ConfigBuilder("spark.mesos.driver.secret.name")
+ .doc("A comma-separated list of secret references. Consult the Mesos
Secret protobuf for " +
+"more information.")
+ .stringConf
+ .createOptional
+
+ private[spark] val SECRET_VALUE =
+ConfigBuilder("spark.mesos.driver.secret.value")
+ .doc("A comma-separated list of secret values.")
+ .stringConf
+ .createOptional
+
+ private[spark] val SECRET_ENVKEY =
+ConfigBuilder("spark.mesos.driver.secret.envkey")
+ .doc("A comma-separated list of the environment variables to contain
the secrets." +
--- End diff --
Ditto.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r135353454
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala
---
@@ -58,13 +58,40 @@ package object config {
private[spark] val DRIVER_LABELS =
ConfigBuilder("spark.mesos.driver.labels")
- .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
+ .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
"pairs should be separated by a colon, and commas used to list
more than one." +
"Ex. key:value,key2:value2")
.stringConf
.createOptional
- private[spark] val DRIVER_FAILOVER_TIMEOUT =
+ private[spark] val SECRET_NAME =
+ConfigBuilder("spark.mesos.driver.secret.name")
+ .doc("A comma-separated list of secret references. Consult the Mesos
Secret protobuf for " +
--- End diff --
If this is a list it should be created with `.toSequence`; it returns the
value properly parsed as a list to you. And it should probably be called
`.names`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r135353607
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala
---
@@ -58,13 +58,40 @@ package object config {
private[spark] val DRIVER_LABELS =
ConfigBuilder("spark.mesos.driver.labels")
- .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
+ .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
"pairs should be separated by a colon, and commas used to list
more than one." +
"Ex. key:value,key2:value2")
.stringConf
.createOptional
- private[spark] val DRIVER_FAILOVER_TIMEOUT =
+ private[spark] val SECRET_NAME =
+ConfigBuilder("spark.mesos.driver.secret.name")
+ .doc("A comma-separated list of secret references. Consult the Mesos
Secret protobuf for " +
+"more information.")
+ .stringConf
+ .createOptional
+
+ private[spark] val SECRET_VALUE =
+ConfigBuilder("spark.mesos.driver.secret.value")
+ .doc("A comma-separated list of secret values.")
+ .stringConf
+ .createOptional
+
+ private[spark] val SECRET_ENVKEY =
+ConfigBuilder("spark.mesos.driver.secret.envkey")
+ .doc("A comma-separated list of the environment variables to contain
the secrets." +
+"The environment variable will be set on the driver.")
+ .stringConf
+ .createOptional
+
+ private[spark] val SECRET_FILENAME =
+ConfigBuilder("spark.mesos.driver.secret.filename")
+ .doc("A comma-seperated list of file paths secret will be written
to. Consult the Mesos " +
--- End diff --
Ditto.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r135030520 --- Diff: resource-managers/mesos/pom.xml --- @@ -29,7 +29,7 @@ Spark Project Mesos mesos -1.0.0 +1.3.0 shaded-protobuf --- End diff -- So we don't think this will require users to use Mesos 1.3+? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r135031225
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
---
@@ -510,12 +510,20 @@ trait MesosSchedulerUtils extends Logging {
}
def mesosToTaskState(state: MesosTaskState): TaskState.TaskState = state
match {
-case MesosTaskState.TASK_STAGING | MesosTaskState.TASK_STARTING =>
TaskState.LAUNCHING
-case MesosTaskState.TASK_RUNNING | MesosTaskState.TASK_KILLING =>
TaskState.RUNNING
+case MesosTaskState.TASK_STAGING |
+ MesosTaskState.TASK_STARTING => TaskState.LAUNCHING
+case MesosTaskState.TASK_RUNNING |
+ MesosTaskState.TASK_KILLING => TaskState.RUNNING
case MesosTaskState.TASK_FINISHED => TaskState.FINISHED
-case MesosTaskState.TASK_FAILED => TaskState.FAILED
+case MesosTaskState.TASK_FAILED |
+ MesosTaskState.TASK_GONE |
+ MesosTaskState.TASK_GONE_BY_OPERATOR => TaskState.FAILED
--- End diff --
Just so my thinking's clear, this still doesn't require the Mesos cluster
to be 1.2+, because the wire protocol is still the same? there won't be older
Mesos client libs lying around that would cause this to fail at runtime.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user ArtRand commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r132597784
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -529,18 +570,120 @@ private[spark] class MesosClusterScheduler(
val appName = desc.conf.get("spark.app.name")
+val driverLabels =
MesosProtoUtils.mesosLabels(desc.conf.get(config.DRIVER_LABELS)
+ .getOrElse(""))
+
TaskInfo.newBuilder()
.setTaskId(taskId)
.setName(s"Driver for ${appName}")
.setSlaveId(offer.offer.getSlaveId)
.setCommand(buildDriverCommand(desc))
+ .setContainer(getContainerInfo(desc))
.addAllResources(cpuResourcesToUse.asJava)
.addAllResources(memResourcesToUse.asJava)
-
.setLabels(MesosProtoUtils.mesosLabels(desc.conf.get(config.DRIVER_LABELS).getOrElse("")))
- .setContainer(MesosSchedulerBackendUtil.containerInfo(desc.conf))
+ .setLabels(driverLabels)
.build
}
+ private def getContainerInfo(desc: MesosDriverDescription):
ContainerInfo.Builder = {
+val containerInfo = MesosSchedulerBackendUtil.containerInfo(desc.conf)
+
+getSecretVolume(desc).foreach { volume =>
+ logInfo(s"Setting secret
name=${volume.getSource.getSecret.getReference.getName} " +
--- End diff --
Good catch, I changed this to be conditioned on the secret being a
`REFERENCE` type. (You probably don't want the content of your secret printed
to the logs with `VALUE` type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r132593586 --- Diff: resource-managers/mesos/pom.xml --- @@ -29,7 +29,7 @@ Spark Project Mesos mesos -1.0.0 +1.3.0 shaded-protobuf --- End diff -- I don't think anything _requires_ 1.4, only that DC/OS with secrets runs 1.4. On a <1.3 Mesos cluster all of the protobuf messages will still be valid. I added to the documentation w.r.t secrets requiring 1.3+. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r132008642 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name + (none) + +Set the secret's reference name. Consult the Mesos Secret +protobuf for more information. + + + --- End diff -- Document `spark.mesos.driver.secret.value` here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r132009351 --- Diff: resource-managers/mesos/pom.xml --- @@ -29,7 +29,7 @@ Spark Project Mesos mesos -1.0.0 +1.3.0 shaded-protobuf --- End diff -- https://spark.apache.org/docs/latest/running-on-mesos.html#installing-mesos mentions that Spark is designed for use with Mesos 1.0.0+. Is that still true? Do these changes affect users who have clusters that are < 1.3.0? Also, how much of this change requires 1.4? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r132040995
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -529,18 +570,120 @@ private[spark] class MesosClusterScheduler(
val appName = desc.conf.get("spark.app.name")
+val driverLabels =
MesosProtoUtils.mesosLabels(desc.conf.get(config.DRIVER_LABELS)
+ .getOrElse(""))
+
TaskInfo.newBuilder()
.setTaskId(taskId)
.setName(s"Driver for ${appName}")
.setSlaveId(offer.offer.getSlaveId)
.setCommand(buildDriverCommand(desc))
+ .setContainer(getContainerInfo(desc))
.addAllResources(cpuResourcesToUse.asJava)
.addAllResources(memResourcesToUse.asJava)
-
.setLabels(MesosProtoUtils.mesosLabels(desc.conf.get(config.DRIVER_LABELS).getOrElse("")))
- .setContainer(MesosSchedulerBackendUtil.containerInfo(desc.conf))
+ .setLabels(driverLabels)
.build
}
+ private def getContainerInfo(desc: MesosDriverDescription):
ContainerInfo.Builder = {
+val containerInfo = MesosSchedulerBackendUtil.containerInfo(desc.conf)
+
+getSecretVolume(desc).foreach { volume =>
+ logInfo(s"Setting secret
name=${volume.getSource.getSecret.getReference.getName} " +
--- End diff --
Is this assuming the type is `Reference` when it could be either
`Reference` or `Value`? Same question for the logInfo() for env vars above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r132009978
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala
---
@@ -58,12 +58,39 @@ package object config {
private [spark] val DRIVER_LABELS =
ConfigBuilder("spark.mesos.driver.labels")
- .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
+ .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value" +
--- End diff --
Missing a space after "Key-value"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r132010482
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala
---
@@ -58,12 +58,39 @@ package object config {
private [spark] val DRIVER_LABELS =
ConfigBuilder("spark.mesos.driver.labels")
- .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value " +
+ .doc("Mesos labels to add to the driver. Labels are free-form
key-value pairs. Key-value" +
"pairs should be separated by a colon, and commas used to list
more than one." +
"Ex. key:value,key2:value2")
.stringConf
.createOptional
+ private[spark] val SECRET_NAME =
+ConfigBuilder("spark.mesos.driver.secret.name")
+ .doc("A comma-seperated list of secret references. Consult the Mesos
Secret protobuf for " +
--- End diff --
Sp: "sep**a**rated". Check for same typo below also.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r132012345 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + --- End diff -- Maybe update to mention this can be a comma-separated list of env vars. Same for filename below. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131986331 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name --- End diff -- Hey @susanxhuynh, so the way it works now is you can specify a secret as a `REFERENCE` or as a `VALUE` type by using the `spark.mesos.driver.secret.name` or `spark.mesos.driver.secret.value` configs, respectively. These secrets are then made file-based and/or env-based depending on the contents of `spark.mesos.driver.secret.filename` and `spark.mesos.driver.secret.envkey`. I allow for multiple secrets (but not multiple types) as comma-seperated lists (like Mesos URIs). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131806416 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name --- End diff -- Ah you're right, we'll need a separate system for value secrets. I would imagine that `VALUE` secrets are mainly used as environment-based secrets? Let me re-do this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131792566 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name --- End diff -- @ArtRand Looking at the protobuf definition, https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L2335, I see that unlike a `Reference`, a `message Value` does not have a name: it has a field called `data`. So, I was wondering how the user would pass in that data. @ArtRand Looking at the protobuf definition, https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L2335, I see that unlike a `Reference`, a `message Value` does not have a name: it has a field called `data`. So, I was wondering how the user would pass in that data. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131784043 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name --- End diff -- @susanxhuynh the type would only be set on the secret `name` for example ``` ... --conf spark.mesos.driver.secret.name=/mysecret:REFERENCE \ --conf spark.mesos.driver.secret.filename=asecret ``` Also, while we're on the topic, what about multiple secrets? Is comma-separated entries the typical way? E.g: ``` --conf spark.mesos.driver.secret.name=/mysecret:REFERENCE,/super/secret:REFERENCE \ --conf spark.mesos.driver.secret.filename=asecret,topsecret ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131780454 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name --- End diff -- @ArtRand What would the `path` mean for TYPE=VALUE. Would it be a path to a local file containing the value of the secret? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131723907 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name --- End diff -- @susanxhuynh @skonto This is an oversight on my part. We should allow the user to specify which secret `Type` they are going to use. Would the format `--conf spark.mesos.driver.secret.name=/path:TYPE` be acceptable? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user ArtRand commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r131719283
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
---
@@ -510,12 +510,20 @@ trait MesosSchedulerUtils extends Logging {
}
def mesosToTaskState(state: MesosTaskState): TaskState.TaskState = state
match {
-case MesosTaskState.TASK_STAGING | MesosTaskState.TASK_STARTING =>
TaskState.LAUNCHING
-case MesosTaskState.TASK_RUNNING | MesosTaskState.TASK_KILLING =>
TaskState.RUNNING
+case MesosTaskState.TASK_STAGING |
+ MesosTaskState.TASK_STARTING => TaskState.LAUNCHING
+case MesosTaskState.TASK_RUNNING |
+ MesosTaskState.TASK_KILLING => TaskState.RUNNING
case MesosTaskState.TASK_FINISHED => TaskState.FINISHED
-case MesosTaskState.TASK_FAILED => TaskState.FAILED
+case MesosTaskState.TASK_FAILED |
+ MesosTaskState.TASK_GONE |
+ MesosTaskState.TASK_GONE_BY_OPERATOR => TaskState.FAILED
--- End diff --
Actually they were introduced in 1.2
https://github.com/apache/mesos/blob/1.2.0/include/mesos/mesos.proto#L1672
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131431005 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name --- End diff -- Actually, the `Secret` protobuf message is new in Mesos 1.3. What are the implications for compatibility with older (<1.3) Mesos clusters? Also, is there anything in Mesos 1.4 that this relies on? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user susanxhuynh commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131429404 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name --- End diff -- I think this refers to the `message Secret` here https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L2335 It says a secret can have one of two types: reference or value. Presumably the type here is `REFERENCE` and this is the name of the reference. And, in practice this could be the path in a secret store. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r131354746
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -529,18 +560,54 @@ private[spark] class MesosClusterScheduler(
val appName = desc.conf.get("spark.app.name")
+
TaskInfo.newBuilder()
.setTaskId(taskId)
.setName(s"Driver for ${appName}")
.setSlaveId(offer.offer.getSlaveId)
.setCommand(buildDriverCommand(desc))
+ .setContainer(getContainerInfo(desc))
.addAllResources(cpuResourcesToUse.asJava)
.addAllResources(memResourcesToUse.asJava)
.setLabels(MesosProtoUtils.mesosLabels(desc.conf.get(config.DRIVER_LABELS).getOrElse("")))
--- End diff --
More clean I think to do here:
val labels =
MesosProtoUtils.mesosLabels(desc.conf.get(config.DRIVER_LABELS).getOrElse(""))
.setLebels(labels)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/18837#discussion_r131351705
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
---
@@ -510,12 +510,20 @@ trait MesosSchedulerUtils extends Logging {
}
def mesosToTaskState(state: MesosTaskState): TaskState.TaskState = state
match {
-case MesosTaskState.TASK_STAGING | MesosTaskState.TASK_STARTING =>
TaskState.LAUNCHING
-case MesosTaskState.TASK_RUNNING | MesosTaskState.TASK_KILLING =>
TaskState.RUNNING
+case MesosTaskState.TASK_STAGING |
+ MesosTaskState.TASK_STARTING => TaskState.LAUNCHING
+case MesosTaskState.TASK_RUNNING |
+ MesosTaskState.TASK_KILLING => TaskState.RUNNING
case MesosTaskState.TASK_FINISHED => TaskState.FINISHED
-case MesosTaskState.TASK_FAILED => TaskState.FAILED
+case MesosTaskState.TASK_FAILED |
+ MesosTaskState.TASK_GONE |
+ MesosTaskState.TASK_GONE_BY_OPERATOR => TaskState.FAILED
--- End diff --
These are new to 1.3?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131350894 --- Diff: docs/running-on-mesos.md --- @@ -479,6 +479,35 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.driver.secret.envkey + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +environment variable in the driver's process. + + + +spark.mesos.driver.secret.filename + (none) + +If set, the contents of the secret referenced by +spark.mesos.driver.secret.name will be written to the provided +file. Relative paths are relative to the container's work +directory. Absolute paths must already exist. Consult the Mesos Secret +protobuf for more information. + + + + spark.mesos.driver.secret.name --- End diff -- Is this the fully qualified path to the secret in the store? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18837: [Spark-20812][Mesos] Add secrets support to the d...
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18837#discussion_r131348687 --- Diff: resource-managers/mesos/pom.xml --- @@ -29,7 +29,7 @@ Spark Project Mesos mesos -1.0.0 +1.3.0-rc1 --- End diff -- 1.3.0 is out I think right? Should we depend on a stable release? Does this change anything? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
