So sounds like some generic downloadable uris support can solve this problem, that Mesos automatically places in your sandbox and you can refer to it.
If so please file a jira and this is a pretty simple fix on the Spark side. Tim On Sat, May 30, 2015 at 7:34 AM, andy petrella <andy.petre...@gmail.com> wrote: > Hello, > > I'm currently exploring DCOS for the spark notebook, and while looking at > the spark configuration I found something interesting which is actually > converging to what we've discovered: > > https://github.com/mesosphere/universe/blob/master/repo/packages/S/spark/0/marathon.json > > So the logging is working fine here because the spark package is using the > spark-class which is able to configure the log4j file. But the interesting > part comes with the fact that the `uris` parameter is filled in with a > downloadable path to the log4j file! > > However, it's not possible when creating the spark context ourselfves and > relying on the mesos sheduler backend only. Unles the spark.executor.uri > (or a another one) can take more than one downloadable path. > > my.2ยข > > andy > > > On Fri, May 29, 2015 at 5:09 PM Gerard Maas <gerard.m...@gmail.com> wrote: > >> Hi Tim, >> >> Thanks for the info. We (Andy Petrella and myself) have been diving a >> bit deeper into this log config: >> >> The log line I was referring to is this one (sorry, I provided the others >> just for context) >> >> *Using Spark's default log4j profile: >> org/apache/spark/log4j-defaults.properties* >> >> That line comes from Logging.scala [1] where a default config is loaded >> is none is found in the classpath upon the startup of the Spark Mesos >> executor in the Mesos sandbox. At that point in time, none of the >> application-specific resources have been shipped yet as the executor JVM is >> just starting up. To load a custom configuration file we should have it >> already on the sandbox before the executor JVM starts and add it to the >> classpath on the startup command. Is that correct? >> >> For the classpath customization, It looks like it should be possible to >> pass a -Dlog4j.configuration property by using the >> 'spark.executor.extraClassPath' that will be picked up at [2] and that >> should be added to the command that starts the executor JVM, but the >> resource must be already on the host before we can do that. Therefore we >> also need some means of 'shipping' the log4j.configuration file to the >> allocated executor. >> >> This all boils down to your statement on the need of shipping extra files >> to the sandbox. Bottom line: It's currently not possible to specify a >> config file for your mesos executor. (ours grows several GB/day). >> >> The only workaround I found so far is to open up the Spark assembly, >> replace the log4j-default.properties and pack it up again. That would >> work, although kind of rudimentary as we use the same assembly for many >> jobs. Probably, accessing the log4j API programmatically should also work >> (I didn't try that yet) >> >> Should we open a JIRA for this functionality? >> >> -kr, Gerard. >> >> >> >> >> [1] >> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/Logging.scala#L128 >> [2] >> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala#L77 >> >> On Thu, May 28, 2015 at 7:50 PM, Tim Chen <t...@mesosphere.io> wrote: >> >>> >>> ---------- Forwarded message ---------- >>> From: Tim Chen <t...@mesosphere.io> >>> Date: Thu, May 28, 2015 at 10:49 AM >>> Subject: Re: [Streaming] Configure executor logging on Mesos >>> To: Gerard Maas <gerard.m...@gmail.com> >>> >>> >>> Hi Gerard, >>> >>> The log line you referred to is not Spark logging but Mesos own logging, >>> which is using glog. >>> >>> Our own executor logs should only contain very few lines though. >>> >>> Most of the log lines you'll see is from Spark, and it can be controled >>> by specifiying a log4j.properties to be downloaded with your Mesos task. >>> Alternatively if you are downloading Spark executor via spark.executor.uri, >>> you can include log4j.properties in that tar ball. >>> >>> I think we probably need some more configurations for Spark scheduler to >>> pick up extra files to be downloaded into the sandbox. >>> >>> Tim >>> >>> >>> >>> >>> >>> On Thu, May 28, 2015 at 6:46 AM, Gerard Maas <gerard.m...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I'm trying to control the verbosity of the logs on the Mesos executors >>>> with no luck so far. The default behaviour is INFO on stderr dump with an >>>> unbounded growth that gets too big at some point. >>>> >>>> I noticed that when the executor is instantiated, it locates a default >>>> log configuration in the spark assembly: >>>> >>>> I0528 13:36:22.958067 26890 exec.cpp:206] Executor registered on slave >>>> 20150528-063307-780930314-5050-8152-S5 >>>> Spark assembly has been built with Hive, including Datanucleus jars on >>>> classpath >>>> Using Spark's default log4j profile: >>>> org/apache/spark/log4j-defaults.properties >>>> >>>> So, no matter what I provide in my job jar files (or also tried with >>>> (spark.executor.extraClassPath=log4j.properties) takes effect in the >>>> executor's configuration. >>>> >>>> How should I configure the log on the executors? >>>> >>>> thanks, Gerard. >>>> >>> >>> >>> >>
