[
https://issues.apache.org/jira/browse/SPARK-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616085#comment-14616085
]
Ankur Chauhan edited comment on SPARK-8009 at 7/7/15 2:42 AM:
--------------------------------------------------------------
I think the correct way to handle this sort of problem is to allow the user to
specify extra Uris that can be downloaded into the sandbox before the actual
executor is started. Something via a property like
{code}spark.executor.additionalUris =>
http://www.example.com/qa/file1.ext,http://www.example.com/qa/file2.ext{code}
The specific problem of being able to specify the log4j.properties would be the
combination of specifying this property along with
{code}spark.executor.extraJavaOptions =>
-Dlog4j.configuration=${MESOS_SANDBOX}/log4j.properties{code}
[~tnachen] Does this sound reasonable? I have a simple change to support this
use case but would like more input, this can be used to download any additional
data etc. Patch:
https://github.com/apache/spark/compare/master...ankurcha:fetch_extra_uris
Also, Does this solution deserve a separate issue or right here is fine (
because it solves a broader problem ).
was (Author: ankurcha):
I think the correct way to handle this sort of problem is to allow the user to
specify extra Uris that can be downloaded into the sandbox before the actual
executor is started. Something via a property like
{code}spark.executor.additionalUris =>
http://www.example.com/qa/file1.ext,http://www.example.com/qa/file2.ext{code}.
The specific problem of being able to specify the log4j.properties would be the
combination of specifying this property along with
{code}spark.executor.extraJavaOptions =>
-Dlog4j.configuration=${MESOS_SANDBOX}/log4j.properties{code}
[~tnachen] Does this sound reasonable? I have a simple change to support this
use case but would like more input, this can be used to download any additional
data etc. Patch:
https://github.com/apache/spark/compare/master...ankurcha:fetch_extra_uris
Also, Does this solution deserve a separate issue or right here is fine (
because it solves a broader problem ).
> [Mesos] Allow provisioning of executor logging configuration
> -------------------------------------------------------------
>
> Key: SPARK-8009
> URL: https://issues.apache.org/jira/browse/SPARK-8009
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
> Affects Versions: 1.3.1
> Environment: Mesos executor
> Reporter: Gerard Maas
> Labels: logging, mesos
>
> It's currently not possible to provide a custom logging configuration for the
> Mesos executors.
> Upon startup of the executor JVM, it loads a default config file from the
> Spark assembly, visible by this line in stderr:
> > Using Spark's default log4j profile:
> > org/apache/spark/log4j-defaults.properties
> That line comes from Logging.scala [1] where a default config is loaded if
> none is found in the classpath upon the startup of the Spark Mesos executor
> in the Mesos sandbox. At that point in time, none of the application-specific
> resources have been shipped yet, as the executor JVM is just starting up.
> To load a custom configuration file we should have it already on the sandbox
> before the executor JVM starts and add it to the classpath on the startup
> command.
> For the classpath customization, It looks like it should be possible to pass
> a -Dlog4j.configuration property by using the
> 'spark.executor.extraClassPath' that will be picked up at [2] and that should
> be added to the command that starts the executor JVM, but the resource must
> be already on the host before we can do that. Therefore we need some means of
> 'shipping' the log4j.configuration file to the allocated executor.
> This all boils down to the need of shipping extra files to the sandbox.
> There's a workaround: open up the Spark assembly, replace the
> log4j-default.properties and pack it up again. That would work, although
> kind of rudimentary as people may use the same assembly for many jobs.
> Probably, accessing the log4j API programmatically should also work (we
> didn't try that yet)
> [1]
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/Logging.scala#L128
> [2]
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala#L77
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
