I've done a little reconnoitering, and the terrain looks to me as follows: 1. Docker maintains container log files at /var/lib/docker/containers/<CONTAINER_ID>/<CONTAINER_ID>-json.log 2. Mesos maintains container STDOUT files at a slave/framework/application specific location, e.g., /tmp/mesos/slaves/20151102-082316-370041927-5050-32381-S1/frameworks/20151102-082316-370041927-5050-32381-0000/executors/ecxprimary1.80750071-81a0-11e5-8596-82d195a34239/runs/5c767378-9599-40af-8010-a31f4c55f9dc 3. The latter is mapped to the container's /mnt/mesos/sandbox 4. These two files (-json.log and the STDOUT file) are different, *each* consumes disk space.
I think that the answer to (1) is Docker's logrotate. As to (2), I am considering a cron job at host (not container) level that drives truncate cmd (GNU coreutils) to prune these files at a certain size. Obviously requires knowing the fully-qualified path under /tmp/mesos/slaves, but this is readily available via "docker inspect". -Paul On Fri, Nov 6, 2015 at 7:17 AM, Paul Bell <arach...@gmail.com> wrote: > Hi Mauricio, > > Yeah....I see your point; thank you. > > My approach would be akin to closing the barn door after the horse got > out. Both Mesos & Docker are doing their own writing of STDOUT. Docker's > rotation won't address Mesos's behavior. > > I need to find a solution here. > > -Paul > > > On Thu, Nov 5, 2015 at 10:46 PM, Mauricio Garavaglia < > mauriciogaravag...@gmail.com> wrote: > >> Hi Paul, >> >> I don't think that's going to help :( >> Even if you configure a different docker log driver, Docker still send >> things to stdout, which is catched by mesos and dumped in the .logs >> directory in the job sandbox. For example, by default docker logs into a >> json file in /var/lib/docker but mesos still writes to the sandbox. >> Hi Mauricio, >> >> I'm grappling with the same issue. >> >> I'm not yet sure if it represents a viable solution, but I plan to look >> at Docker's log rotation facility. It was introduced in Docker 1.8. >> >> If you beat me to it & it looks like a solution, please let us know! >> >> Thanks. >> >> Cordially, >> >> Paul >> >> > On Nov 5, 2015, at 9:40 PM, Mauricio Garavaglia < >> mauriciogaravag...@gmail.com> wrote: >> > >> > Hi guys, >> > >> > How can I manage the stdout/err log files generated by jobs in mesos? >> for long running docker apps launched using marathon the log files can >> deplete the disk of an agent, and using quotas makes the jobs to be killed >> which is also not ideal. I'd like to have a way to rotate them. >> > >> > Is it correct to just go to the mesos agent workdir and go through each >> sandbox stdout/err and rotate them? I know that could break the log UI but >> it doesn't scale very well having logs of several of GB. >> > >> > Thanks! >> > >
