[
https://issues.apache.org/jira/browse/SPARK-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14253059#comment-14253059
]
Jongyoul Lee commented on SPARK-4465:
-------------------------------------
I've dug into mesos code and jni and java.security which is used by
UserGroupInformation in runAsSparkUser. Finally I've found an exact wrong point
of MesosExecutorBackend, these are about java.security scope and how mesos c++
executor works. Unlike to standalone and yarn, in a task under mesos cluster,
jni runs each task in a native method via callback. There are several problems
about running java code under jni environment, especially it's not covered by
java.security. JDK spec warn the following
{quote}
Some important points about being privileged: Firstly, this concept only exists
within a single thread. As soon as the privileged code completes, the privilege
is guaranteed to be erased or revoked.
--
http://docs.oracle.com/javase/7/docs/technotes/guides/security/spec/security-spec.doc4.html#20573
{quote}
{code:title=MesosExecutorBackend.scala}
SparkHadoopUtil.get.runAsSparkUser { () =>
MesosNativeLibrary.load()
// Create a new Executor and start it running
val runner = new MesosExecutorBackend()
new MesosExecutorDriver(runner).run()
}
{code}
Show that code. Only run() method is only covered by runAsSparkUser, however,
the actual code running tasks is not covered by runAsSparkUser. As a result,
MesosExecutorBackend.launchTask doesn't affect by runAsSparkUser.
I'll fix that bug by adding runAsSparkUser into MesosExecutorBackend.launchTask
rather and adding them into Executor.launchTask, because Executor.launchTask is
already covered in case of standalone and yarn. Thus I judged that code is
exact point of fix that bug.
> runAsSparkUser doesn't affect TaskRunner in Mesos environment at all.
> ---------------------------------------------------------------------
>
> Key: SPARK-4465
> URL: https://issues.apache.org/jira/browse/SPARK-4465
> Project: Spark
> Issue Type: Bug
> Components: Input/Output, Mesos
> Reporter: Jongyoul Lee
> Fix For: 1.2.0, 1.3.0
>
>
> runAsSparkUser enable classes using hadoop library to change an active user
> to spark User, however in Mesos environment, because the function calls
> before running within JNI, runAsSparkUser doesn't affect anything, and
> meaningless code. fix the Appropriate scope of function and remove
> meaningless code. That's a bug because of running program incorrectly. That's
> related to SPARK-3223 but setting framework user is not perfect solution in
> my tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
