[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-17 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18519


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-17 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r133841268
  
--- Diff: resource-managers/mesos/pom.xml ---
@@ -74,6 +74,17 @@
   test
 
 
+
--- End diff --

Hmm, ok... the credential manager code should be safe when Hive classes 
aren't present, but if there's a problem in that area it's not your fault.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-15 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r133294684
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -95,6 +99,12 @@ class CoarseGrainedSchedulerBackend(scheduler: 
TaskSchedulerImpl, val rpcEnv: Rp
   // The num of current max ExecutorId used to re-register appMaster
   @volatile protected var currentExecutorIdCounter = 0
 
+  // hadoop token manager used by some sub-classes (e.g. Mesos)
+  protected var hadoopDelegationTokenManager: 
Option[HadoopDelegationTokenManager] = None
+
+  // Hadoop delegation tokens to be sent to the executors.
+  protected var hadoopDelegationCreds: Option[Array[Byte]] = None
--- End diff --

So, another option here is have this be a `val` and have 
`hadoopDelegationTokenManager` be a `def`. The latter means there's no 
initialization order issue (so no need for the `lazy val` hack). Since this is 
really only done once, that should be fine.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-15 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r133290641
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -95,6 +99,12 @@ class CoarseGrainedSchedulerBackend(scheduler: 
TaskSchedulerImpl, val rpcEnv: Rp
   // The num of current max ExecutorId used to re-register appMaster
   @volatile protected var currentExecutorIdCounter = 0
 
+  // hadoop token manager used by some sub-classes (e.g. Mesos)
+  protected var hadoopDelegationTokenManager: 
Option[HadoopDelegationTokenManager] = None
+
+  // Hadoop delegation tokens to be sent to the executors.
+  protected var hadoopDelegationCreds: Option[Array[Byte]] = None
--- End diff --

This should be a `val`; you don't need to set in in the subclass. Because 
there might be some initialization order issue, it might need to be a `lazy 
val`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-15 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r133290499
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -95,6 +99,12 @@ class CoarseGrainedSchedulerBackend(scheduler: 
TaskSchedulerImpl, val rpcEnv: Rp
   // The num of current max ExecutorId used to re-register appMaster
   @volatile protected var currentExecutorIdCounter = 0
 
+  // hadoop token manager used by some sub-classes (e.g. Mesos)
+  protected var hadoopDelegationTokenManager: 
Option[HadoopDelegationTokenManager] = None
--- End diff --

This should be a `val`. Just override it in the subclass.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-11 Thread ArtRand
Github user ArtRand commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r132774771
  
--- Diff: resource-managers/mesos/pom.xml ---
@@ -74,6 +74,17 @@
   test
 
 
+
--- End diff --

Yes, `MesosClusterManagerSuite` creates a 
`MesosCoarseGrainedSchedulerBackend` which contains a 
`HadoopDelegationTokenManager`... etc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r132741326
  
--- Diff: 
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -53,9 +53,16 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
 sc: SparkContext,
 master: String,
 securityManager: SecurityManager)
-  extends CoarseGrainedSchedulerBackend(scheduler, sc.env.rpcEnv)
-  with org.apache.mesos.Scheduler
-  with MesosSchedulerUtils {
+  extends CoarseGrainedSchedulerBackend(
+scheduler,
+sc.env.rpcEnv)
+with org.apache.mesos.Scheduler
+with MesosSchedulerUtils {
+
+  override val hadoopDelegationTokenManager: 
Option[HadoopDelegationTokenManager] =
--- End diff --

`protected`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r132741250
  
--- Diff: 
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -53,9 +53,16 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
 sc: SparkContext,
 master: String,
 securityManager: SecurityManager)
-  extends CoarseGrainedSchedulerBackend(scheduler, sc.env.rpcEnv)
-  with org.apache.mesos.Scheduler
-  with MesosSchedulerUtils {
+  extends CoarseGrainedSchedulerBackend(
+scheduler,
+sc.env.rpcEnv)
+with org.apache.mesos.Scheduler
+with MesosSchedulerUtils {
+
+  override val hadoopDelegationTokenManager: 
Option[HadoopDelegationTokenManager] =
+Some(new HadoopDelegationTokenManager(sc.conf, sc.hadoopConfiguration))
+
+  override val hadoopDelegationCreds: Option[Array[Byte]] = 
getHadoopDelegationCreds()
--- End diff --

No need to override this guy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r132740552
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -41,9 +45,11 @@ import org.apache.spark.util.{RpcUtils, 
SerializableBuffer, ThreadUtils, Utils}
  * (spark.deploy.*).
  */
 private[spark]
-class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val 
rpcEnv: RpcEnv)
-  extends ExecutorAllocationClient with SchedulerBackend with Logging
-{
+class CoarseGrainedSchedulerBackend(
--- End diff --

You're not changing anything here now, are you?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r132741194
  
--- Diff: resource-managers/mesos/pom.xml ---
@@ -74,6 +74,17 @@
   test
 
 
+
--- End diff --

Is this really needed?

I don't see you adding specific tests for this, so wonder why you need the 
explicit dependency when other modules that depend on `spark-core` don't.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-10 Thread ArtRand
Github user ArtRand commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r132514413
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -680,6 +686,18 @@ object SparkSubmit extends CommandLineUtils {
 (childArgs, childClasspath, sysProps, childMainClass)
   }
 
+  // [SPARK-20328]. HadoopRDD calls into a Hadoop library that fetches 
delegation tokens with
+  // renewer set to the YARN ResourceManager.  Since YARN isn't configured 
in Mesos mode, we
+  // must trick it into thinking we're YARN.
+  private def setRMPrincipal(sysProps: HashMap[String, String]): Unit = {
+val shortUserName = 
UserGroupInformation.getCurrentUser.getShortUserName
+val key = s"spark.hadoop.${YarnConfiguration.RM_PRINCIPAL}"
+// scalastyle:off println
+printStream.println(s"Setting ${key} to ${shortUserName}")
--- End diff --

It only prints when `UserGroupInformation.isSecurityEnabled` and I think 
it's useful information whenever a job is run. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-08 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r132033130
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -41,8 +45,11 @@ import org.apache.spark.util.{RpcUtils, 
SerializableBuffer, ThreadUtils, Utils}
  * (spark.deploy.*).
  */
 private[spark]
-class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val 
rpcEnv: RpcEnv)
-  extends ExecutorAllocationClient with SchedulerBackend with Logging
+class CoarseGrainedSchedulerBackend(
+scheduler: TaskSchedulerImpl,
+val rpcEnv: RpcEnv,
+hadoopDelegationTokenManager: Option[HadoopDelegationTokenManager])
+extends ExecutorAllocationClient with SchedulerBackend with Logging
--- End diff --

nit: unindent this line one level.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-08 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r132032720
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -680,6 +686,18 @@ object SparkSubmit extends CommandLineUtils {
 (childArgs, childClasspath, sysProps, childMainClass)
   }
 
+  // [SPARK-20328]. HadoopRDD calls into a Hadoop library that fetches 
delegation tokens with
+  // renewer set to the YARN ResourceManager.  Since YARN isn't configured 
in Mesos mode, we
+  // must trick it into thinking we're YARN.
+  private def setRMPrincipal(sysProps: HashMap[String, String]): Unit = {
+val shortUserName = 
UserGroupInformation.getCurrentUser.getShortUserName
+val key = s"spark.hadoop.${YarnConfiguration.RM_PRINCIPAL}"
+// scalastyle:off println
+printStream.println(s"Setting ${key} to ${shortUserName}")
--- End diff --

Do you want this to be printed out every time someone runs spark-submit? 
Sounds a bit noisy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-08-08 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r132034176
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -41,8 +45,11 @@ import org.apache.spark.util.{RpcUtils, 
SerializableBuffer, ThreadUtils, Utils}
  * (spark.deploy.*).
  */
 private[spark]
-class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val 
rpcEnv: RpcEnv)
-  extends ExecutorAllocationClient with SchedulerBackend with Logging
+class CoarseGrainedSchedulerBackend(
+scheduler: TaskSchedulerImpl,
+val rpcEnv: RpcEnv,
+hadoopDelegationTokenManager: Option[HadoopDelegationTokenManager])
--- End diff --

I'm a little torn on having this as a constructor argument. It seems 
cleaner at first, but it kinda make the constructor of sub-classes (like the 
mesos one) kinda ugly.

How about having a `protected val hadoopDelegationTokenManager = None` and 
overriding it where needed? That makes initialization in the sub-class more 
readable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-07-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r126792121
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -42,7 +45,10 @@ import org.apache.spark.util.{RpcUtils, 
SerializableBuffer, ThreadUtils, Utils}
  * (spark.deploy.*).
  */
 private[spark]
-class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val 
rpcEnv: RpcEnv)
+class CoarseGrainedSchedulerBackend(
+  scheduler: TaskSchedulerImpl,
--- End diff --

nit: one more indent level.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-07-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r126790869
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -65,7 +67,7 @@ private[deploy] object SparkSubmitAction extends 
Enumeration {
  * This program handles setting up the classpath with relevant Spark 
dependencies and provides
  * a layer over the different cluster managers and deploy modes that Spark 
supports.
  */
-object SparkSubmit extends CommandLineUtils {
+object SparkSubmit extends CommandLineUtils with Logging {
--- End diff --

You can't do this. This breaks the logging configuration of `spark-shell` 
and other shells (which is WARN by default instead of INFO).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-07-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r126791240
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/security/CredentialsSerializer.scala
 ---
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, DataOutputStream}
+
+import org.apache.hadoop.security.Credentials
+
+class CredentialsSerializer {
--- End diff --

`private[spark]`? Also feels like this could be just a couple of methods in 
`SparkHadoopUtil` instead of a new separate class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-07-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r126791991
  
--- Diff: 
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
 ---
@@ -232,6 +240,14 @@ private[spark] object CoarseGrainedExecutorBackend 
extends Logging {
 }
   }
 
+  /** Add Credentials to the currently logged in user. */
+  private def addCredentials(creds: Credentials): Unit = {
+logInfo(s"Adding ${creds.numberOfTokens()} tokens and 
${creds.numberOfSecretKeys()} secret" +
+  s"keys to the current user's credentials.")
+
+UserGroupInformation.getCurrentUser().addCredentials(creds)
--- End diff --

This looks like `SparkHadoopUtil.addCurrentUserCredentials`. That's only 
implemented on `YarnSparkHadoopUtil` for historical reasons, but since we 
dropped Hadoop 1.x support, the implementation can move to `core/` now, and 
you'd avoid this copy of that code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-07-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r126792287
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -577,7 +589,6 @@ class CoarseGrainedSchedulerBackend(scheduler: 
TaskSchedulerImpl, val rpcEnv: Rp
* of requesting a delta of executors risks double counting new 
executors when there are
* insufficient resources to satisfy the first request. We make the 
assumption here that the
* cluster manager will eventually fulfill all requests when resources 
free up.
-   *
--- End diff --

nit: leave these as they were.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-07-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r126792996
  
--- Diff: 
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -52,9 +55,14 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
 sc: SparkContext,
 master: String,
 securityManager: SecurityManager)
-  extends CoarseGrainedSchedulerBackend(scheduler, sc.env.rpcEnv)
-  with org.apache.mesos.Scheduler
-  with MesosSchedulerUtils {
+  extends CoarseGrainedSchedulerBackend(
+scheduler,
+sc.env.rpcEnv,
+Some(new HadoopDelegationTokenManager(
+  sc.conf,
+  SparkHadoopUtil.get.newConfiguration(sc.conf
--- End diff --

`sc.hadoopConfiguration`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-07-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r126792217
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -254,6 +265,7 @@ class CoarseGrainedSchedulerBackend(scheduler: 
TaskSchedulerImpl, val rpcEnv: Rp
   "messages.")))
 }
 
+
--- End diff --

nit: not needed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #18519: [SPARK-16742] Mesos Kerberos Support

2017-07-11 Thread vanzin
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18519#discussion_r126791349
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/security/CredentialsSerializer.scala
 ---
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, DataOutputStream}
+
+import org.apache.hadoop.security.Credentials
+
+class CredentialsSerializer {
+  def serialize(creds: Credentials): Array[Byte] = {
+val byteStream = new ByteArrayOutputStream
+val dataStream = new DataOutputStream(byteStream)
+creds.writeTokenStorageToStream(dataStream)
+byteStream.toByteArray
+  }
+
+  def deserialize(tokenBytes: Array[Byte]): Credentials = {
+val tokensBuf = new java.io.ByteArrayInputStream(tokenBytes)
--- End diff --

Why not import it like other classes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org