see https://github.com/eirslett/mesos/commit/7eac3da95fdd39a0fb96b8d8a883e72b6b9f9943
2015-11-06 15:34 GMT+01:00 Kamil Chmielewski <[email protected]>: > Try to cantact with https://github.com/eirslett > He can have a PoC from MesosCon Europe hackathon with Mesos writting > containers logs to Logstash. > > Cheers, > Kamil > > > 2015-11-06 15:26 GMT+01:00 Paul Bell <[email protected]>: > >> I've done a little reconnoitering, and the terrain looks to me as follows: >> >> 1. Docker maintains container log files at >> /var/lib/docker/containers/<CONTAINER_ID>/<CONTAINER_ID>-json.log >> 2. Mesos maintains container STDOUT files at a >> slave/framework/application specific location, e.g., >> >> /tmp/mesos/slaves/20151102-082316-370041927-5050-32381-S1/frameworks/20151102-082316-370041927-5050-32381-0000/executors/ecxprimary1.80750071-81a0-11e5-8596-82d195a34239/runs/5c767378-9599-40af-8010-a31f4c55f9dc >> 3. The latter is mapped to the container's /mnt/mesos/sandbox >> 4. These two files (-json.log and the STDOUT file) are different, >> *each* consumes disk space. >> >> I think that the answer to (1) is Docker's logrotate. >> >> As to (2), I am considering a cron job at host (not container) level that >> drives truncate cmd (GNU coreutils) to prune these files at a certain size. >> Obviously requires knowing the fully-qualified path under >> /tmp/mesos/slaves, but this is readily available via "docker inspect". >> >> -Paul >> >> >> On Fri, Nov 6, 2015 at 7:17 AM, Paul Bell <[email protected]> wrote: >> >>> Hi Mauricio, >>> >>> Yeah....I see your point; thank you. >>> >>> My approach would be akin to closing the barn door after the horse got >>> out. Both Mesos & Docker are doing their own writing of STDOUT. Docker's >>> rotation won't address Mesos's behavior. >>> >>> I need to find a solution here. >>> >>> -Paul >>> >>> >>> On Thu, Nov 5, 2015 at 10:46 PM, Mauricio Garavaglia < >>> [email protected]> wrote: >>> >>>> Hi Paul, >>>> >>>> I don't think that's going to help :( >>>> Even if you configure a different docker log driver, Docker still send >>>> things to stdout, which is catched by mesos and dumped in the .logs >>>> directory in the job sandbox. For example, by default docker logs into a >>>> json file in /var/lib/docker but mesos still writes to the sandbox. >>>> Hi Mauricio, >>>> >>>> I'm grappling with the same issue. >>>> >>>> I'm not yet sure if it represents a viable solution, but I plan to look >>>> at Docker's log rotation facility. It was introduced in Docker 1.8. >>>> >>>> If you beat me to it & it looks like a solution, please let us know! >>>> >>>> Thanks. >>>> >>>> Cordially, >>>> >>>> Paul >>>> >>>> > On Nov 5, 2015, at 9:40 PM, Mauricio Garavaglia < >>>> [email protected]> wrote: >>>> > >>>> > Hi guys, >>>> > >>>> > How can I manage the stdout/err log files generated by jobs in mesos? >>>> for long running docker apps launched using marathon the log files can >>>> deplete the disk of an agent, and using quotas makes the jobs to be killed >>>> which is also not ideal. I'd like to have a way to rotate them. >>>> > >>>> > Is it correct to just go to the mesos agent workdir and go through >>>> each sandbox stdout/err and rotate them? I know that could break the log UI >>>> but it doesn't scale very well having logs of several of GB. >>>> > >>>> > Thanks! >>>> >>> >>> >> >
