[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951779#comment-16951779 ] Gregoire Seux commented on MESOS-6003: -- We would be interested in having such logger in mesos code base Our use case is that we forward all task logs to local rsyslog (which forward them eventually to our centralized logging). To do this, we ask all our tasks to start a wrapper script which setup stdout/stderr redirection. This solution is not great because it forces our users to update their tasks definition to benefit from this feature. Our intent is to make this fully transparent. Is anyone still working on this or should we try to use https://github.com/wjoel/mesos-external-container-logger directly > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272711#comment-16272711 ] Marc LeBourdais commented on MESOS-6003: [~wrouesnel] I second [~wjoel]'s request. We're using Mesos at DigitalOcean and using the external container logger module to configure container logging to the local syslog socket on each Mesos host to be aggregated and centralized. In order to keep up to date with Mesos 1.3 and 1.4, we've made a couple new releases in our fork: https://github.com/digitalocean/mesos-external-container-logger/releases I'd rather not take over "maintaining" this module, but as long as we're still following the upgrades to Mesos, we'll likely keep updating the module unless it ends up getting merged (the ideal scenario). > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268280#comment-16268280 ] Joel Wilsson commented on MESOS-6003: - [~wrouesnel] you might also find the changes at https://github.com/wjoel/mesos-external-container-logger useful, Marc LeBourdais from DigitalOcean has helped to keep it up to date with Mesos 1.3 and now 1.4. If this doesn't get merged they will probably take over maintenance of the module, but this is really a very useful feature that should be included by default. Can we get this back on your radar, [~kaysoky]? If not, maybe someone else can take over as shepherd? > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264839#comment-16264839 ] Will Rouesnel commented on MESOS-6003: -- Is there any further changes needed to try and advance this to being merged? > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049820#comment-16049820 ] Will Rouesnel commented on MESOS-6003: -- [~wjoel] From a prettiness perspective I agree, but I wanted to preserve the ability to find MESOS_LOG injected env vars, and just specifying "executorinfo" didn't seem descriptive enough. Since you almost always need to post-process that field in some way, the user script/program is pretty likely to want to do something with it. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968267#comment-15968267 ] Pranay Kanwar commented on MESOS-6003: -- Might run into MESOS-7386 if using docker and logger process dies > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947914#comment-15947914 ] Joel Wilsson commented on MESOS-6003: - I guess this needs another look from [~kaysoky]? It would be nice to have this feature merged. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934680#comment-15934680 ] Pranay Kanwar commented on MESOS-6003: -- 0.28.2 port in case anyone else needs it https://github.com/r4um/mesos-external-container-logger/tree/mesos-0.28.2-ubuntu-14.04.5 > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884917#comment-15884917 ] Joel Wilsson commented on MESOS-6003: - Thanks, [~wrouesnel]. I've updated my backport. I think I found one issue: executor_info_json_field should have a default value of "EXECUTORINFO_JSON" instead of "MESOS_EXECUTORINFO_JSON", since (as the help message explains) it's concatenated with the value of --mesos_field_prefix. Otherwise it gets turned into MESOS_LOG_MESOS_EXECUTORINFO_JSON. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874111#comment-15874111 ] Will Rouesnel commented on MESOS-6003: -- Huh I thought I had notified this thread - anyway - I've rebased the patchset for the changes that have been made to ContainerLogger interface, so this should be good to go again. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865618#comment-15865618 ] Pranay Kanwar commented on MESOS-6003: -- Yes that would do, thanks a lot [~wjoel]. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865377#comment-15865377 ] Joel Wilsson commented on MESOS-6003: - [~r4um] I have back port of this as a standalone module here, if that's what you mean: https://github.com/wjoel/mesos-external-container-logger I'll update it with the latest changes from Will later this week. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865362#comment-15865362 ] Pranay Kanwar commented on MESOS-6003: -- Can this also be available for other/older versions in a separate repo like https://github.com/dcos/dcos-mesos-modules/tree/master/journald ? > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864615#comment-15864615 ] Will Rouesnel commented on MESOS-6003: -- Rebased and building on master now, with added support for the new USER variable if someone could review. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857543#comment-15857543 ] Joel Wilsson commented on MESOS-6003: - [~kaysoky], I use it to do [log forwarding with Filebeat|https://wjoel.com/posts/mesos-container-log-forwarding-with-filebeat.html] and we're about to implement the same setup at work. It would be great if I didn't need to maintain and update a custom compiled module with this code for each new release. [~wrouesnel], that would be fantastic, thanks! This is a very useful feature, especially because it enables the combination of log forwarding and log rotation. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857149#comment-15857149 ] Joseph Wu commented on MESOS-6003: -- Sorry, this dropped off my radar a while ago. The ContainerLogger code is subtly different now. If you want to revive the reviews, you can post new versions of them, rebased to master. (Just curious: what do you intend to use a module like this for?) > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854405#comment-15854405 ] Joel Wilsson commented on MESOS-6003: - [~wrouesnel], [~kaysoky], is there anything I can do to help this move along? > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805126#comment-15805126 ] Joel Wilsson commented on MESOS-6003: - Looks good to me as an outsider, but most of the code changes ended up in the documentation review request. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608482#comment-15608482 ] Will Rouesnel commented on MESOS-6003: -- Latest round of fixups is waiting for review again. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531689#comment-15531689 ] Will Rouesnel commented on MESOS-6003: -- Rebased and awaiting code review again against the latest master. > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6003) Add logging module for logging to an external program
[ https://issues.apache.org/jira/browse/MESOS-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508298#comment-15508298 ] Joseph Wu commented on MESOS-6003: -- Module: https://reviews.apache.org/r/51257/ Docs: https://reviews.apache.org/r/51258/ > Add logging module for logging to an external program > - > > Key: MESOS-6003 > URL: https://issues.apache.org/jira/browse/MESOS-6003 > Project: Mesos > Issue Type: Improvement > Components: modules >Reporter: Will Rouesnel >Assignee: Will Rouesnel >Priority: Minor > > In the vein of the logrotate module for logging, there should be a similar > module which provides support for logging to an arbitrary log handling > program, with suitable task metadata provided by environment variables or > command line arguments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
