Re: Structured logging for Mesos (or c++ glog)

[Otis Gospodnetić]

Hi Zhitao,

Is there a JIRA for this?  I looked at
http://search-hadoop.com/?project=Mesos=issue=log but didn't see
anything matching 1).

I'd love for Logagent to ship with log parser/pattern for Mesos OOTB.

Thanks,
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Tue, Dec 20, 2016 at 12:06 PM, Zhitao Li  wrote:

> Hi Otis,
>
> Thanks for the good summary. The conversation is mostly about 1) in this
> thread, because right now Mesos logs are not really structured, or at least
> most of it.
>
> On Tue, Dec 20, 2016 at 6:57 AM, Otis Gospodnetić <
> otis.gospodne...@gmail.com> wrote:
>
>> Hi Zhitao,
>>
>> When people talk about structure and logging it typically means two
>> things:
>>
>> 1) make the log format a known/standard format where all its elements are
>> known, and thus it's easy to parse them; a log event can still be a single
>> line, but it can also be multi-line or JSON or some other (even binary)
>> format.  As long as the format/structure is known, the log event *is*
>> structured.
>>
>> 2) I want tools/configs/patterns that will let me easily parse this log
>> event structure and send it somewhere (e.g. Elasticsearch or Logsene
>>  or ...) where this structure will be
>> handled in the way that lets me easy filtering/slicing and dicing by one or
>> more attributes/fields extracted from the log event structure.
>>
>> *For 1*):
>> I'm assuming Mesos logs already are structured.  I assume their format is
>> either widely known (like Apache common log format, for example), or
>> well-documented (again like Apache common log format).  If that is not
>> true, then yes, Mesos devs will want to do document the structure.  I've
>> looked at https://mesos.apache.org/documentation/latest/logging/ but saw
>> nothing mentioning the structure.  Maybe this info is somewhere else?
>>
>> *For 2)*
>> This is where modern log shippers come in. We open-sourced our Logagent
>>  (more info here
>> ), which has log parsing (and thus
>> structuring) built-in.  It ships with a bunch of log patterns/parsers, and
>> one can add new ones (e.g. for Mesos).  Elasticsearch, mentioned in this
>> thread, is one of the outputs.  It's sort of like Filebeat+Logstash in one,
>> and it's often used in Dockerized deployments, as part of this Docker
>> agent .  One could also use Logstash for
>> parsing/structuring, but Logstash is a bit heavy.
>>
>> I hope this helps.
>>
>> Otis
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>> On Mon, Dec 19, 2016 at 6:03 PM, Zhitao Li  wrote:
>>
>>> Charles,
>>>
>>> Thanks for sharing the pattern. If my reading is right, this will
>>> extract the entire message line as one string. What I'm looking for is: on
>>> top of extracting the entire message line, also break it into structured
>>> fields automatically.
>>>
>>>
>>>
>>> On Mon, Dec 19, 2016 at 1:59 PM, Charles Allen <
>>> charles.al...@metamarkets.com> wrote:
>>>
 For what its worth we use SumoLogic and the magic parsing search looks
 like
 this:

 parse regex field=message "^(?[IWE])(?>>> og_date>[0-9]{4}
 [0-9:.]*) [0-9]*
 (?[0-9a-zA-Z.]*):(?[0-9]*)]
 (?.*)$"



 On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere <
 jo...@mesosphere.io>
 wrote:

 > @Zhitao are you looking specifically for structure or just for
 tagging?
 > glog does already have support for custom tags in the header. I don't
 know
 > if this is enough for your use case though.
 >
 > —
 > *Joris Van Remoortere*
 > Mesosphere
 >
 > On Mon, Dec 19, 2016 at 9:58 AM, James Peach 
 wrote:
 >
 >
 > > On Dec 19, 2016, at 9:43 AM, Zhitao Li 
 wrote:
 > >
 > > Hi,
 > >
 > > I'm looking at how to better utilize ElasticSearch to perform log
 > analysis for logs from Mesos. It seems like ElasticSearch would
 generally
 > work better for structured logging, but Mesos still uses glog thus
 all logs
 > produced are old-school unstructured lines.
 > >
 > > I wonder whether anyone has brought the conversation of making Mesos
 > logs easier to process, or if anyone has experience to share.
 >
 > Are you trying to stitch together sequences of events? I that case,
 would
 > direct event logging be more useful?
 >
 > J
 >
 >
 >

>>>
>>>
>>>
>>> --
>>> Cheers,
>>>
>>> Zhitao Li
>>>
>>
>>
>
>
> --
> Cheers,
>
> Zhitao Li
>

Re: Structured logging for Mesos (or c++ glog)

[Ilya Pronin]

Hi,

>From my experience both structured and less structured logs are useful and
they aren't 100% interchangeable but most of the time having a more or less
structured human readable log would is sufficient​.

A good example of a more or less structured general purpose log is Postfix
maillog. It contains a "session ID", a name of subsystem that makes an
entry, a bunch of key=value parameters (with standardized keys and values
formats) and human readable explanations where needed. That makes those
logs both readable and grepable.

On Tue, Dec 20, 2016 at 5:43 PM, Ilya Pronin 
wrote:

> Hi,
>
> From my experience both structured and less structured logs are useful and
> they aren't 100% interchangeable but most of the time having a more or less
> structured human readable log would is sufficient​.
>
> A good example of a more or less structured general purpose log is Postfix
> maillog. It contains a "session ID", a name of subsystem that makes an
> entry, a bunch of key=value parameters (with standardized keys and values
> formats) and human readable explanations where needed. That makes those
> logs both readable and grepable.
>
>
> On Tue, Dec 20, 2016 at 5:06 PM, Zhitao Li  wrote:
>
>> Hi Otis,
>>
>> Thanks for the good summary. The conversation is mostly about 1) in this
>> thread, because right now Mesos logs are not really structured, or at least
>> most of it.
>>
>> On Tue, Dec 20, 2016 at 6:57 AM, Otis Gospodnetić <
>> otis.gospodne...@gmail.com> wrote:
>>
>>> Hi Zhitao,
>>>
>>> When people talk about structure and logging it typically means two
>>> things:
>>>
>>> 1) make the log format a known/standard format where all its elements
>>> are known, and thus it's easy to parse them; a log event can still be a
>>> single line, but it can also be multi-line or JSON or some other (even
>>> binary) format.  As long as the format/structure is known, the log event
>>> *is* structured.
>>>
>>> 2) I want tools/configs/patterns that will let me easily parse this log
>>> event structure and send it somewhere (e.g. Elasticsearch or Logsene
>>>  or ...) where this structure will be
>>> handled in the way that lets me easy filtering/slicing and dicing by one or
>>> more attributes/fields extracted from the log event structure.
>>>
>>> *For 1*):
>>> I'm assuming Mesos logs already are structured.  I assume their format
>>> is either widely known (like Apache common log format, for example), or
>>> well-documented (again like Apache common log format).  If that is not
>>> true, then yes, Mesos devs will want to do document the structure.  I've
>>> looked at https://mesos.apache.org/documentation/latest/logging/ but
>>> saw nothing mentioning the structure.  Maybe this info is somewhere else?
>>>
>>> *For 2)*
>>> This is where modern log shippers come in. We open-sourced our Logagent
>>>  (more info here
>>> ), which has log parsing (and thus
>>> structuring) built-in.  It ships with a bunch of log patterns/parsers, and
>>> one can add new ones (e.g. for Mesos).  Elasticsearch, mentioned in this
>>> thread, is one of the outputs.  It's sort of like Filebeat+Logstash in one,
>>> and it's often used in Dockerized deployments, as part of this Docker
>>> agent .  One could also use Logstash for
>>> parsing/structuring, but Logstash is a bit heavy.
>>>
>>> I hope this helps.
>>>
>>> Otis
>>> --
>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>>
>>>
>>> On Mon, Dec 19, 2016 at 6:03 PM, Zhitao Li 
>>> wrote:
>>>
 Charles,

 Thanks for sharing the pattern. If my reading is right, this will
 extract the entire message line as one string. What I'm looking for is: on
 top of extracting the entire message line, also break it into structured
 fields automatically.



 On Mon, Dec 19, 2016 at 1:59 PM, Charles Allen <
 charles.al...@metamarkets.com> wrote:

> For what its worth we use SumoLogic and the magic parsing search looks
> like
> this:
>
> parse regex field=message "^(?[IWE])(? og_date>[0-9]{4}
> [0-9:.]*) [0-9]*
> (?[0-9a-zA-Z.]*):(?[0-9]*)]
> (?.*)$"
>
>
>
> On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere <
> jo...@mesosphere.io>
> wrote:
>
> > @Zhitao are you looking specifically for structure or just for
> tagging?
> > glog does already have support for custom tags in the header. I
> don't know
> > if this is enough for your use case though.
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Mon, Dec 19, 2016 at 9:58 AM, James Peach 
> wrote:
> >
> >
> > > On Dec 19, 2016, at 9:43 AM, Zhitao Li 
> 

Re: Structured logging for Mesos (or c++ glog)

[Zhitao Li]

Hi Otis,

Thanks for the good summary. The conversation is mostly about 1) in this
thread, because right now Mesos logs are not really structured, or at least
most of it.

On Tue, Dec 20, 2016 at 6:57 AM, Otis Gospodnetić <
otis.gospodne...@gmail.com> wrote:

> Hi Zhitao,
>
> When people talk about structure and logging it typically means two things:
>
> 1) make the log format a known/standard format where all its elements are
> known, and thus it's easy to parse them; a log event can still be a single
> line, but it can also be multi-line or JSON or some other (even binary)
> format.  As long as the format/structure is known, the log event *is*
> structured.
>
> 2) I want tools/configs/patterns that will let me easily parse this log
> event structure and send it somewhere (e.g. Elasticsearch or Logsene
>  or ...) where this structure will be
> handled in the way that lets me easy filtering/slicing and dicing by one or
> more attributes/fields extracted from the log event structure.
>
> *For 1*):
> I'm assuming Mesos logs already are structured.  I assume their format is
> either widely known (like Apache common log format, for example), or
> well-documented (again like Apache common log format).  If that is not
> true, then yes, Mesos devs will want to do document the structure.  I've
> looked at https://mesos.apache.org/documentation/latest/logging/ but saw
> nothing mentioning the structure.  Maybe this info is somewhere else?
>
> *For 2)*
> This is where modern log shippers come in. We open-sourced our Logagent
>  (more info here
> ), which has log parsing (and thus
> structuring) built-in.  It ships with a bunch of log patterns/parsers, and
> one can add new ones (e.g. for Mesos).  Elasticsearch, mentioned in this
> thread, is one of the outputs.  It's sort of like Filebeat+Logstash in one,
> and it's often used in Dockerized deployments, as part of this Docker
> agent .  One could also use Logstash for
> parsing/structuring, but Logstash is a bit heavy.
>
> I hope this helps.
>
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
> On Mon, Dec 19, 2016 at 6:03 PM, Zhitao Li  wrote:
>
>> Charles,
>>
>> Thanks for sharing the pattern. If my reading is right, this will extract
>> the entire message line as one string. What I'm looking for is: on top of
>> extracting the entire message line, also break it into structured fields
>> automatically.
>>
>>
>>
>> On Mon, Dec 19, 2016 at 1:59 PM, Charles Allen <
>> charles.al...@metamarkets.com> wrote:
>>
>>> For what its worth we use SumoLogic and the magic parsing search looks
>>> like
>>> this:
>>>
>>> parse regex field=message "^(?[IWE])(?[0-9]{4}
>>> [0-9:.]*) [0-9]*
>>> (?[0-9a-zA-Z.]*):(?[0-9]*)]
>>> (?.*)$"
>>>
>>>
>>>
>>> On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere <
>>> jo...@mesosphere.io>
>>> wrote:
>>>
>>> > @Zhitao are you looking specifically for structure or just for tagging?
>>> > glog does already have support for custom tags in the header. I don't
>>> know
>>> > if this is enough for your use case though.
>>> >
>>> > —
>>> > *Joris Van Remoortere*
>>> > Mesosphere
>>> >
>>> > On Mon, Dec 19, 2016 at 9:58 AM, James Peach  wrote:
>>> >
>>> >
>>> > > On Dec 19, 2016, at 9:43 AM, Zhitao Li 
>>> wrote:
>>> > >
>>> > > Hi,
>>> > >
>>> > > I'm looking at how to better utilize ElasticSearch to perform log
>>> > analysis for logs from Mesos. It seems like ElasticSearch would
>>> generally
>>> > work better for structured logging, but Mesos still uses glog thus all
>>> logs
>>> > produced are old-school unstructured lines.
>>> > >
>>> > > I wonder whether anyone has brought the conversation of making Mesos
>>> > logs easier to process, or if anyone has experience to share.
>>> >
>>> > Are you trying to stitch together sequences of events? I that case,
>>> would
>>> > direct event logging be more useful?
>>> >
>>> > J
>>> >
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Cheers,
>>
>> Zhitao Li
>>
>
>


-- 
Cheers,

Zhitao Li

Re: Structured logging for Mesos (or c++ glog)

[Otis Gospodnetić]

Hi Zhitao,

When people talk about structure and logging it typically means two things:

1) make the log format a known/standard format where all its elements are
known, and thus it's easy to parse them; a log event can still be a single
line, but it can also be multi-line or JSON or some other (even binary)
format.  As long as the format/structure is known, the log event *is*
structured.

2) I want tools/configs/patterns that will let me easily parse this log
event structure and send it somewhere (e.g. Elasticsearch or Logsene
 or ...) where this structure will be handled
in the way that lets me easy filtering/slicing and dicing by one or more
attributes/fields extracted from the log event structure.

*For 1*):
I'm assuming Mesos logs already are structured.  I assume their format is
either widely known (like Apache common log format, for example), or
well-documented (again like Apache common log format).  If that is not
true, then yes, Mesos devs will want to do document the structure.  I've
looked at https://mesos.apache.org/documentation/latest/logging/ but saw
nothing mentioning the structure.  Maybe this info is somewhere else?

*For 2)*
This is where modern log shippers come in. We open-sourced our Logagent
 (more info here
), which has log parsing (and thus
structuring) built-in.  It ships with a bunch of log patterns/parsers, and
one can add new ones (e.g. for Mesos).  Elasticsearch, mentioned in this
thread, is one of the outputs.  It's sort of like Filebeat+Logstash in one,
and it's often used in Dockerized deployments, as part of this Docker agent
.  One could also use Logstash for
parsing/structuring, but Logstash is a bit heavy.

I hope this helps.

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Mon, Dec 19, 2016 at 6:03 PM, Zhitao Li  wrote:

> Charles,
>
> Thanks for sharing the pattern. If my reading is right, this will extract
> the entire message line as one string. What I'm looking for is: on top of
> extracting the entire message line, also break it into structured fields
> automatically.
>
>
>
> On Mon, Dec 19, 2016 at 1:59 PM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
>> For what its worth we use SumoLogic and the magic parsing search looks
>> like
>> this:
>>
>> parse regex field=message "^(?[IWE])(?[0-9]{4}
>> [0-9:.]*) [0-9]*
>> (?[0-9a-zA-Z.]*):(?[0-9]*)]
>> (?.*)$"
>>
>>
>>
>> On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere <
>> jo...@mesosphere.io>
>> wrote:
>>
>> > @Zhitao are you looking specifically for structure or just for tagging?
>> > glog does already have support for custom tags in the header. I don't
>> know
>> > if this is enough for your use case though.
>> >
>> > —
>> > *Joris Van Remoortere*
>> > Mesosphere
>> >
>> > On Mon, Dec 19, 2016 at 9:58 AM, James Peach  wrote:
>> >
>> >
>> > > On Dec 19, 2016, at 9:43 AM, Zhitao Li  wrote:
>> > >
>> > > Hi,
>> > >
>> > > I'm looking at how to better utilize ElasticSearch to perform log
>> > analysis for logs from Mesos. It seems like ElasticSearch would
>> generally
>> > work better for structured logging, but Mesos still uses glog thus all
>> logs
>> > produced are old-school unstructured lines.
>> > >
>> > > I wonder whether anyone has brought the conversation of making Mesos
>> > logs easier to process, or if anyone has experience to share.
>> >
>> > Are you trying to stitch together sequences of events? I that case,
>> would
>> > direct event logging be more useful?
>> >
>> > J
>> >
>> >
>> >
>>
>
>
>
> --
> Cheers,
>
> Zhitao Li
>

Re: Structured logging for Mesos (or c++ glog)

[Zhitao Li]

Charles,

Thanks for sharing the pattern. If my reading is right, this will extract
the entire message line as one string. What I'm looking for is: on top of
extracting the entire message line, also break it into structured fields
automatically.



On Mon, Dec 19, 2016 at 1:59 PM, Charles Allen <
charles.al...@metamarkets.com> wrote:

> For what its worth we use SumoLogic and the magic parsing search looks like
> this:
>
> parse regex field=message "^(?[IWE])(?[0-9]{4}
> [0-9:.]*) [0-9]*
> (?[0-9a-zA-Z.]*):(?[0-9]*)]
> (?.*)$"
>
>
>
> On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere  >
> wrote:
>
> > @Zhitao are you looking specifically for structure or just for tagging?
> > glog does already have support for custom tags in the header. I don't
> know
> > if this is enough for your use case though.
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Mon, Dec 19, 2016 at 9:58 AM, James Peach  wrote:
> >
> >
> > > On Dec 19, 2016, at 9:43 AM, Zhitao Li  wrote:
> > >
> > > Hi,
> > >
> > > I'm looking at how to better utilize ElasticSearch to perform log
> > analysis for logs from Mesos. It seems like ElasticSearch would generally
> > work better for structured logging, but Mesos still uses glog thus all
> logs
> > produced are old-school unstructured lines.
> > >
> > > I wonder whether anyone has brought the conversation of making Mesos
> > logs easier to process, or if anyone has experience to share.
> >
> > Are you trying to stitch together sequences of events? I that case, would
> > direct event logging be more useful?
> >
> > J
> >
> >
> >
>



-- 
Cheers,

Zhitao Li

Re: Structured logging for Mesos (or c++ glog)

[Zhitao Li]

Joris,

I am particular looking for structure. We have mechanism to add static tags
easily to log collected into ELK.

If there is a way to dynamically inject tags like "framework_id" at actual
logging call, it might be a starting point for me.

I cannot find a good reference on how to add tagging for glog though. Do
you have any reference?

On Mon, Dec 19, 2016 at 11:15 AM, Joris Van Remoortere 
wrote:

> @Zhitao are you looking specifically for structure or just for tagging?
> glog does already have support for custom tags in the header. I don't know
> if this is enough for your use case though.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Mon, Dec 19, 2016 at 9:58 AM, James Peach  wrote:
>
> >
> > > On Dec 19, 2016, at 9:43 AM, Zhitao Li  wrote:
> > >
> > > Hi,
> > >
> > > I'm looking at how to better utilize ElasticSearch to perform log
> > analysis for logs from Mesos. It seems like ElasticSearch would generally
> > work better for structured logging, but Mesos still uses glog thus all
> logs
> > produced are old-school unstructured lines.
> > >
> > > I wonder whether anyone has brought the conversation of making Mesos
> > logs easier to process, or if anyone has experience to share.
> >
> > Are you trying to stitch together sequences of events? I that case, would
> > direct event logging be more useful?
> >
> > J
>



-- 
Cheers,

Zhitao Li

Re: Structured logging for Mesos (or c++ glog)

[Charles Allen]

For what its worth we use SumoLogic and the magic parsing search looks like
this:

parse regex field=message "^(?[IWE])(?[0-9]{4}
[0-9:.]*) [0-9]*
(?[0-9a-zA-Z.]*):(?[0-9]*)]
(?.*)$"



On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere 
wrote:

> @Zhitao are you looking specifically for structure or just for tagging?
> glog does already have support for custom tags in the header. I don't know
> if this is enough for your use case though.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Mon, Dec 19, 2016 at 9:58 AM, James Peach  wrote:
>
>
> > On Dec 19, 2016, at 9:43 AM, Zhitao Li  wrote:
> >
> > Hi,
> >
> > I'm looking at how to better utilize ElasticSearch to perform log
> analysis for logs from Mesos. It seems like ElasticSearch would generally
> work better for structured logging, but Mesos still uses glog thus all logs
> produced are old-school unstructured lines.
> >
> > I wonder whether anyone has brought the conversation of making Mesos
> logs easier to process, or if anyone has experience to share.
>
> Are you trying to stitch together sequences of events? I that case, would
> direct event logging be more useful?
>
> J
>
>
>

Re: Structured logging for Mesos (or c++ glog)

[Joris Van Remoortere]

@Zhitao are you looking specifically for structure or just for tagging?
glog does already have support for custom tags in the header. I don't know
if this is enough for your use case though.

—
*Joris Van Remoortere*
Mesosphere

On Mon, Dec 19, 2016 at 9:58 AM, James Peach  wrote:

>
> > On Dec 19, 2016, at 9:43 AM, Zhitao Li  wrote:
> >
> > Hi,
> >
> > I'm looking at how to better utilize ElasticSearch to perform log
> analysis for logs from Mesos. It seems like ElasticSearch would generally
> work better for structured logging, but Mesos still uses glog thus all logs
> produced are old-school unstructured lines.
> >
> > I wonder whether anyone has brought the conversation of making Mesos
> logs easier to process, or if anyone has experience to share.
>
> Are you trying to stitch together sequences of events? I that case, would
> direct event logging be more useful?
>
> J

Re: Structured logging for Mesos (or c++ glog)

[James Peach]

> On Dec 19, 2016, at 9:43 AM, Zhitao Li  wrote:
> 
> Hi,
> 
> I'm looking at how to better utilize ElasticSearch to perform log analysis 
> for logs from Mesos. It seems like ElasticSearch would generally work better 
> for structured logging, but Mesos still uses glog thus all logs produced are 
> old-school unstructured lines.
> 
> I wonder whether anyone has brought the conversation of making Mesos logs 
> easier to process, or if anyone has experience to share.

Are you trying to stitch together sequences of events? I that case, would 
direct event logging be more useful?

J