Try to cantact with https://github.com/eirslett He can have a PoC from MesosCon Europe hackathon with Mesos writting containers logs to Logstash.
Cheers, Kamil 2015-11-06 15:26 GMT+01:00 Paul Bell <[email protected]>: > I've done a little reconnoitering, and the terrain looks to me as follows: > > 1. Docker maintains container log files at > /var/lib/docker/containers/<CONTAINER_ID>/<CONTAINER_ID>-json.log > 2. Mesos maintains container STDOUT files at a > slave/framework/application specific location, e.g., > > /tmp/mesos/slaves/20151102-082316-370041927-5050-32381-S1/frameworks/20151102-082316-370041927-5050-32381-0000/executors/ecxprimary1.80750071-81a0-11e5-8596-82d195a34239/runs/5c767378-9599-40af-8010-a31f4c55f9dc > 3. The latter is mapped to the container's /mnt/mesos/sandbox > 4. These two files (-json.log and the STDOUT file) are different, > *each* consumes disk space. > > I think that the answer to (1) is Docker's logrotate. > > As to (2), I am considering a cron job at host (not container) level that > drives truncate cmd (GNU coreutils) to prune these files at a certain size. > Obviously requires knowing the fully-qualified path under > /tmp/mesos/slaves, but this is readily available via "docker inspect". > > -Paul > > > On Fri, Nov 6, 2015 at 7:17 AM, Paul Bell <[email protected]> wrote: > >> Hi Mauricio, >> >> Yeah....I see your point; thank you. >> >> My approach would be akin to closing the barn door after the horse got >> out. Both Mesos & Docker are doing their own writing of STDOUT. Docker's >> rotation won't address Mesos's behavior. >> >> I need to find a solution here. >> >> -Paul >> >> >> On Thu, Nov 5, 2015 at 10:46 PM, Mauricio Garavaglia < >> [email protected]> wrote: >> >>> Hi Paul, >>> >>> I don't think that's going to help :( >>> Even if you configure a different docker log driver, Docker still send >>> things to stdout, which is catched by mesos and dumped in the .logs >>> directory in the job sandbox. For example, by default docker logs into a >>> json file in /var/lib/docker but mesos still writes to the sandbox. >>> Hi Mauricio, >>> >>> I'm grappling with the same issue. >>> >>> I'm not yet sure if it represents a viable solution, but I plan to look >>> at Docker's log rotation facility. It was introduced in Docker 1.8. >>> >>> If you beat me to it & it looks like a solution, please let us know! >>> >>> Thanks. >>> >>> Cordially, >>> >>> Paul >>> >>> > On Nov 5, 2015, at 9:40 PM, Mauricio Garavaglia < >>> [email protected]> wrote: >>> > >>> > Hi guys, >>> > >>> > How can I manage the stdout/err log files generated by jobs in mesos? >>> for long running docker apps launched using marathon the log files can >>> deplete the disk of an agent, and using quotas makes the jobs to be killed >>> which is also not ideal. I'd like to have a way to rotate them. >>> > >>> > Is it correct to just go to the mesos agent workdir and go through >>> each sandbox stdout/err and rotate them? I know that could break the log UI >>> but it doesn't scale very well having logs of several of GB. >>> > >>> > Thanks! >>> >> >> >
