Re: [openstack-dev] [kolla] Can Heka solve all the deficiencies in the current rsyslog implementation: was Re: [kolla] Introduction of Heka in Kolla

[Lei Zhang]

Just FYI.

Docker support multi log driver now[0].

Version New Logging Drivers
1.6.0 json-file, syslog, none
1.7.0 journald
1.8.0 fluentd, gelf
1.9.0 awslogs

For the stdout from container, we can use gelf/syslog driver to forward.

[0] https://docs.docker.com/engine/reference/logging/overview/

On Fri, Jan 15, 2016 at 8:28 AM, Steven Dake (stdake) 
wrote:

> Eric,
>
> Comments inline.
>
> On 1/14/16, 3:31 AM, "Eric LEMOINE"  wrote:
>
> >On Wed, Jan 13, 2016 at 1:27 PM, Steven Dake (stdake) 
> >wrote:
> >> Eric,
> >
> >
> >Hi Steven
>
>
> Feel free to call me Steve
>
> >
> >
> >>
> >> Apologies for top post, not really sure where in this thread to post
> >>this
> >> list of questions as its sort of a change in topic so I changed the
> >> subject line :)
> >>
> >> 1.
> >> Somewhere I read when researching this Heka topic, that Heka cannot log
> >> all details from /dev/log.  Some services like mariadb for example don't
> >> log to stdout as I think Heka requires to operate correctly.  Would you
> >> mind responding on the question "Would Heka be able to effectively log
> >> every piece of information coming off the system related to OpenStack
> >>(our
> >> infrastructure services like ceph/mariadb/etc as well as the OpenStack
> >> services)?
> >
> >
> >My first reaction to this is: if we have services, such as mariadb,
> >that can only send their logs to syslog then let's continue using
> >Rsyslog.  And with Rsyslog we can easily store logs on the local
> >filesystem as well (your requirement #3 below).
>
> Rsyslog may be the best tool for this job.  I'll be looking to your POC to
> get a feel for the validity of that claim :)
>
> >
> >That being said, it appears that Heka supports reading logs from
> >/dev/log.  This can be done using the UdpInput plugin with "net" set
> >to "unixgram".  See
> > for the original
> >issue.  Heka also supports writing logs to files on the local
> >filesystem, through the FileOutput plugin.  We do not currently use
> >the UdpInput plugin, so we need to test it and see if it can work for
> >Kolla.  We will work on these tests, and report back to the list.
> >
> >
> >
> >> 2.
> >> Also, I want to make sure we can fix up the backtrace defeciency.
> >> Currently rsyslog doesn't log backtraces in python code.  Perhaps Sam or
> >> inc0 know the reason behind it, but I want to make sure we can fix up
> >>this
> >> annoyance, because backtraces are mightily important.
> >
> >
> >I've had a look on my AIO Kolla.  And I do see Python tracebacks in
> >OpenStack log files created by Rsyslog (in
> >/var/lib/docker/volumes/rsyslog/_data/nova/nova-api.log for example).
> >They're just on a single line, with "#012" used as the separator [*].
> >So they are hard to read, but they are there.  I think that is
> >consistent with what SamYaple and inc0 said yesterday on IRC.
> >
> >[*] This is related to Rsyslog's $EscapeControlCharactersOnReceive
> >setting. See
> ><
> http://www.rsyslog.com/doc/master/configuration/input_directives/rsconf1_
> >escapecontrolcharactersonreceive.html>.
> >
>
> Apparently the one-lining of the tracebacks is a new feature because of a
> bug fix in oslo.log that dims fixed for us.  So my original comments about
> missing backtraces was dated information from December.
>
> >
> >> 3.
> >> Also I want to make sure each node ends up with log files in a data
> >> container (or data volume or whatever we just recently replaced the data
> >> containers with) for all the services for individual node diagnostics.
> >> This helps fill the gap of the Kibana visualization and Elasticsearch
> >> where we may not have a perfect diagnostic solution at the conclusion of
> >> Mitaka and in need of individual node inspection of the logs.  Can Heka
> >>be
> >> made to do this?  Our rsyslog implementation does today, and its a hard
> >> requirement for the moment.  If we need some special software to run in
> >> addition to Heka, I could live with that.
> >
> >
> >That "special software" could be Rsyslog :)  Seriously, Rsyslog
> >provides a solution for getting logs from services that only log to
> >syslog.  We can also easily configure Rsyslog to write logs on the
> >local filesystem, as done in Kolla already today.  And using Heka we
> >can easily make Python tracebacks look good in Kibana.
> >
> >I would like to point out that our initial intent was not to remove
> >Rsyslog.  Our intent was to propose a scalable/decentralized log
> >processing architecture based on Heka running on each node, instead of
> >relying on a centralized Logstash instance.  Using Heka we eliminate
> >the need to deploy and manage a resilient Logstash/Redis cluster.  And
> >it is to be noted that Heka gives us a lot of flexibility.  In
> >particular, Heka makes it possible to collect logs from services that
> >don't speak syslog (RabbitMQ for example, whose logs are not currently
> >collected!).
>

Re: [openstack-dev] [kolla] Can Heka solve all the deficiencies in the current rsyslog implementation: was Re: [kolla] Introduction of Heka in Kolla

[Eric LEMOINE]

On Wed, Jan 13, 2016 at 1:27 PM, Steven Dake (stdake)  wrote:
> Eric,


Hi Steven


>
> Apologies for top post, not really sure where in this thread to post this
> list of questions as its sort of a change in topic so I changed the
> subject line :)
>
> 1.
> Somewhere I read when researching this Heka topic, that Heka cannot log
> all details from /dev/log.  Some services like mariadb for example don't
> log to stdout as I think Heka requires to operate correctly.  Would you
> mind responding on the question "Would Heka be able to effectively log
> every piece of information coming off the system related to OpenStack (our
> infrastructure services like ceph/mariadb/etc as well as the OpenStack
> services)?


My first reaction to this is: if we have services, such as mariadb,
that can only send their logs to syslog then let's continue using
Rsyslog.  And with Rsyslog we can easily store logs on the local
filesystem as well (your requirement #3 below).

That being said, it appears that Heka supports reading logs from
/dev/log.  This can be done using the UdpInput plugin with "net" set
to "unixgram".  See
 for the original
issue.  Heka also supports writing logs to files on the local
filesystem, through the FileOutput plugin.  We do not currently use
the UdpInput plugin, so we need to test it and see if it can work for
Kolla.  We will work on these tests, and report back to the list.



> 2.
> Also, I want to make sure we can fix up the backtrace defeciency.
> Currently rsyslog doesn't log backtraces in python code.  Perhaps Sam or
> inc0 know the reason behind it, but I want to make sure we can fix up this
> annoyance, because backtraces are mightily important.


I've had a look on my AIO Kolla.  And I do see Python tracebacks in
OpenStack log files created by Rsyslog (in
/var/lib/docker/volumes/rsyslog/_data/nova/nova-api.log for example).
They're just on a single line, with "#012" used as the separator [*].
So they are hard to read, but they are there.  I think that is
consistent with what SamYaple and inc0 said yesterday on IRC.

[*] This is related to Rsyslog's $EscapeControlCharactersOnReceive
setting. See 
.


> 3.
> Also I want to make sure each node ends up with log files in a data
> container (or data volume or whatever we just recently replaced the data
> containers with) for all the services for individual node diagnostics.
> This helps fill the gap of the Kibana visualization and Elasticsearch
> where we may not have a perfect diagnostic solution at the conclusion of
> Mitaka and in need of individual node inspection of the logs.  Can Heka be
> made to do this?  Our rsyslog implementation does today, and its a hard
> requirement for the moment.  If we need some special software to run in
> addition to Heka, I could live with that.


That "special software" could be Rsyslog :)  Seriously, Rsyslog
provides a solution for getting logs from services that only log to
syslog.  We can also easily configure Rsyslog to write logs on the
local filesystem, as done in Kolla already today.  And using Heka we
can easily make Python tracebacks look good in Kibana.

I would like to point out that our initial intent was not to remove
Rsyslog.  Our intent was to propose a scalable/decentralized log
processing architecture based on Heka running on each node, instead of
relying on a centralized Logstash instance.  Using Heka we eliminate
the need to deploy and manage a resilient Logstash/Redis cluster.  And
it is to be noted that Heka gives us a lot of flexibility.  In
particular, Heka makes it possible to collect logs from services that
don't speak syslog (RabbitMQ for example, whose logs are not currently
collected!).

As mentioned above Heka provides plugins that we could possibly
leverage to remove Rsyslog completely, but at this point we cannot
guarantee that they will do the job.  Our coming tests will tell.

Thanks.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Can Heka solve all the deficiencies in the current rsyslog implementation: was Re: [kolla] Introduction of Heka in Kolla

[Steven Dake (stdake)]

Eric,

Comments inline.

On 1/14/16, 3:31 AM, "Eric LEMOINE"  wrote:

>On Wed, Jan 13, 2016 at 1:27 PM, Steven Dake (stdake) 
>wrote:
>> Eric,
>
>
>Hi Steven


Feel free to call me Steve

>
>
>>
>> Apologies for top post, not really sure where in this thread to post
>>this
>> list of questions as its sort of a change in topic so I changed the
>> subject line :)
>>
>> 1.
>> Somewhere I read when researching this Heka topic, that Heka cannot log
>> all details from /dev/log.  Some services like mariadb for example don't
>> log to stdout as I think Heka requires to operate correctly.  Would you
>> mind responding on the question "Would Heka be able to effectively log
>> every piece of information coming off the system related to OpenStack
>>(our
>> infrastructure services like ceph/mariadb/etc as well as the OpenStack
>> services)?
>
>
>My first reaction to this is: if we have services, such as mariadb,
>that can only send their logs to syslog then let's continue using
>Rsyslog.  And with Rsyslog we can easily store logs on the local
>filesystem as well (your requirement #3 below).

Rsyslog may be the best tool for this job.  I'll be looking to your POC to
get a feel for the validity of that claim :)

>
>That being said, it appears that Heka supports reading logs from
>/dev/log.  This can be done using the UdpInput plugin with "net" set
>to "unixgram".  See
> for the original
>issue.  Heka also supports writing logs to files on the local
>filesystem, through the FileOutput plugin.  We do not currently use
>the UdpInput plugin, so we need to test it and see if it can work for
>Kolla.  We will work on these tests, and report back to the list.
>
>
>
>> 2.
>> Also, I want to make sure we can fix up the backtrace defeciency.
>> Currently rsyslog doesn't log backtraces in python code.  Perhaps Sam or
>> inc0 know the reason behind it, but I want to make sure we can fix up
>>this
>> annoyance, because backtraces are mightily important.
>
>
>I've had a look on my AIO Kolla.  And I do see Python tracebacks in
>OpenStack log files created by Rsyslog (in
>/var/lib/docker/volumes/rsyslog/_data/nova/nova-api.log for example).
>They're just on a single line, with "#012" used as the separator [*].
>So they are hard to read, but they are there.  I think that is
>consistent with what SamYaple and inc0 said yesterday on IRC.
>
>[*] This is related to Rsyslog's $EscapeControlCharactersOnReceive
>setting. See 
>escapecontrolcharactersonreceive.html>.
>

Apparently the one-lining of the tracebacks is a new feature because of a
bug fix in oslo.log that dims fixed for us.  So my original comments about
missing backtraces was dated information from December.

>
>> 3.
>> Also I want to make sure each node ends up with log files in a data
>> container (or data volume or whatever we just recently replaced the data
>> containers with) for all the services for individual node diagnostics.
>> This helps fill the gap of the Kibana visualization and Elasticsearch
>> where we may not have a perfect diagnostic solution at the conclusion of
>> Mitaka and in need of individual node inspection of the logs.  Can Heka
>>be
>> made to do this?  Our rsyslog implementation does today, and its a hard
>> requirement for the moment.  If we need some special software to run in
>> addition to Heka, I could live with that.
>
>
>That "special software" could be Rsyslog :)  Seriously, Rsyslog
>provides a solution for getting logs from services that only log to
>syslog.  We can also easily configure Rsyslog to write logs on the
>local filesystem, as done in Kolla already today.  And using Heka we
>can easily make Python tracebacks look good in Kibana.
>
>I would like to point out that our initial intent was not to remove
>Rsyslog.  Our intent was to propose a scalable/decentralized log
>processing architecture based on Heka running on each node, instead of
>relying on a centralized Logstash instance.  Using Heka we eliminate
>the need to deploy and manage a resilient Logstash/Redis cluster.  And
>it is to be noted that Heka gives us a lot of flexibility.  In
>particular, Heka makes it possible to collect logs from services that
>don't speak syslog (RabbitMQ for example, whose logs are not currently
>collected!).


FWIW the concern that logstash isn't scalable is unsubstantiated.  IMNSHO
the reason to use Heka is it does two things we need with one component
while removing JVMs from each compute node.  This is my main attraction to
the adoption of Heka.

>
>As mentioned above Heka provides plugins that we could possibly
>leverage to remove Rsyslog completely, but at this point we cannot
>guarantee that they will do the job.  Our coming tests will tell.

If the plugins work in a way in which we can remove rsyslog completely, I
would prefer that outcome.  Less dependencies in software systems 

[openstack-dev] [kolla] Can Heka solve all the deficiencies in the current rsyslog implementation: was Re: [kolla] Introduction of Heka in Kolla

[Steven Dake (stdake)]

Eric,

Apologies for top post, not really sure where in this thread to post this
list of questions as its sort of a change in topic so I changed the
subject line :)

1.
Somewhere I read when researching this Heka topic, that Heka cannot log
all details from /dev/log.  Some services like mariadb for example don't
log to stdout as I think Heka requires to operate correctly.  Would you
mind responding on the question "Would Heka be able to effectively log
every piece of information coming off the system related to OpenStack (our
infrastructure services like ceph/mariadb/etc as well as the OpenStack
services)?

2.
Also, I want to make sure we can fix up the backtrace defeciency.
Currently rsyslog doesn't log backtraces in python code.  Perhaps Sam or
inc0 know the reason behind it, but I want to make sure we can fix up this
annoyance, because backtraces are mightily important.

3.
Also I want to make sure each node ends up with log files in a data
container (or data volume or whatever we just recently replaced the data
containers with) for all the services for individual node diagnostics.
This helps fill the gap of the Kibana visualization and Elasticsearch
where we may not have a perfect diagnostic solution at the conclusion of
Mitaka and in need of individual node inspection of the logs.  Can Heka be
made to do this?  Our rsyslog implementation does today, and its a hard
requirement for the moment.  If we need some special software to run in
addition to Heka, I could live with that.

inc0,

Beyond the scheduling logistics which I split off into a different thread,
are there other hard requirements we need to make Eric aware of up front
where you struggled with rsyslog?

Regards,
-steve


On 1/13/16, 4:11 AM, "Kwasniewska, Alicja" 
wrote:

>Eric, Patrick, Simon, Clark thanks for your comments.
>
>I don't know Heka, so that's why I ask a lot of questions. I hope you are
>fine with that:) I am not against Heka, I was just curious how reliable
>it is  and how much experience you have with setting it up in Docker
>environment in order to know both advantages and disadvantages of this
>solution. 
>
>@Eric, great that you are going to create POC, it will explain a lot and
>it will show us possible problems.
>
>Kind regards,
>Alicja Kwaśniewska
>
>
>-Original Message-
>From: Eric LEMOINE [mailto:elemo...@mirantis.com]
>Sent: Wednesday, January 13, 2016 10:55 AM
>To: OpenStack Development Mailing List (not for usage questions)
>Subject: Re: [openstack-dev] [kolla] Introduction of Heka in Kolla
>
>Hi Alicja
>
>
>Thank you for your comments.  Answers and comments below.
>
>
>
>On Tue, Jan 12, 2016 at 1:19 PM, Kwasniewska, Alicja
> wrote:
>> Unfortunately I do not have any experience in working or testing Heka,
>> so it's hard for me to compare its performance vs Logstash
>> performance. However I've read that Heka possess a lot advantages over
>>Logstash in this scope.
>>
>>
>> But which version of Logstash did you test? One guy from the Logstash
>> community said that: "The next release of logstash (1.2.0 is in beta)
>> has a 3.5x improvement in event throughput. For numbers: on my
>> workstation at home
>> (6 vcpu on virtualbox, host OS windows, 8 GB ram, host cpu is FX-8150)
>> - with logstash 1.1.13, I can process roughly 31,000 events/sec
>> parsing apache logs. With logstash 1.2.0.beta1, I can process 102,000
>>events/sec."
>
>
>I've used the latest Docker image:
>.  It uses Logstash 2.1.1,
>which is the most recent stable version.
>
>
>
>> You also said that Heka is a unified data processing, but do we need
>>this?
>
>
>Heka, as a unified data processing, enables to derive metrics from logs,
>HTTP response times for example.  Alerts can also be triggered on
>specific log patterns.
>
>
>> Heka seems to address stream processing needs, while Logstash focuses
>> mainly on processing logs. We want to create a central logging
>> service, and Logstash was created especially for it and seems to work
>> well for this application.
>>
>>
>> One thing that is obvious is the fact that the Logstash is better
>> known, more popular and tested. Maybe it has some performance
>> disadvantages, but at least we know what we can expect from it. Also,
>> it has more pre-built plugins and has a lot examples of usage, while
>> Heka doesn't have many of them yet and is nowhere near the range of
>> plugins and integrations provided by Logstash.
>
>
>As Simon said Heka already includes quite a lot of plugins.  See the Heka
>documentation [*] for an exhaustive list.  It may indeed be the case that
>Logstash includes even more plugins, but Heka has taken us pretty far
>already.
>
>
>
>> In the case of adding plugins, I've read that in order to add Go
>> plugins, the binary has to be recompiled, what is a little bit
>> frustrating (static linking - to wire in new plugins, have to
>> recompile). On the other hand, the Lua plugins 

Re: [openstack-dev] [kolla] Can Heka solve all the deficiencies in the current rsyslog implementation: was Re: [kolla] Introduction of Heka in Kolla

[Fox, Kevin M]

Some random thoughts...

I've been looking into how to log our production docker containers better, and 
a couple of things have shown up that may be of interest to you...

1, docker now has a journald backend. so rather then log to files, you just set 
your daemons to log to stdout, and docker+journald takes care of log rolling, 
log deletion, and log shipping for you. Setup a remote journald to funnel the 
traffic off to and your good to go.

2. filebeat can tail log files and ship them to logstash. should be very easy 
to add a filebeat docker container as a sidecar to watch and upload logs rather 
then try and cram them through syslog.

and my 2 cents on ELK as an Op. I normally have a similar reaction to Java as 
Steven Dake does. ELK is becoming very common and EleasticSearch does work fine 
on the fully open source version though. We've been running it here for a long 
time now and its quite nice. We already have existing ELK stacks and being able 
to point Kolla at existing ELK's would be quite useful.

Thanks,
Kevin

From: Steven Dake (stdake) [std...@cisco.com]
Sent: Wednesday, January 13, 2016 4:27 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [kolla] Can Heka solve all the deficiencies in the 
current rsyslog implementation: was Re: [kolla] Introduction of Heka in Kolla

Eric,

Apologies for top post, not really sure where in this thread to post this
list of questions as its sort of a change in topic so I changed the
subject line :)

1.
Somewhere I read when researching this Heka topic, that Heka cannot log
all details from /dev/log.  Some services like mariadb for example don't
log to stdout as I think Heka requires to operate correctly.  Would you
mind responding on the question "Would Heka be able to effectively log
every piece of information coming off the system related to OpenStack (our
infrastructure services like ceph/mariadb/etc as well as the OpenStack
services)?

2.
Also, I want to make sure we can fix up the backtrace defeciency.
Currently rsyslog doesn't log backtraces in python code.  Perhaps Sam or
inc0 know the reason behind it, but I want to make sure we can fix up this
annoyance, because backtraces are mightily important.

3.
Also I want to make sure each node ends up with log files in a data
container (or data volume or whatever we just recently replaced the data
containers with) for all the services for individual node diagnostics.
This helps fill the gap of the Kibana visualization and Elasticsearch
where we may not have a perfect diagnostic solution at the conclusion of
Mitaka and in need of individual node inspection of the logs.  Can Heka be
made to do this?  Our rsyslog implementation does today, and its a hard
requirement for the moment.  If we need some special software to run in
addition to Heka, I could live with that.

inc0,

Beyond the scheduling logistics which I split off into a different thread,
are there other hard requirements we need to make Eric aware of up front
where you struggled with rsyslog?

Regards,
-steve


On 1/13/16, 4:11 AM, "Kwasniewska, Alicja" <alicja.kwasniew...@intel.com>
wrote:

>Eric, Patrick, Simon, Clark thanks for your comments.
>
>I don't know Heka, so that's why I ask a lot of questions. I hope you are
>fine with that:) I am not against Heka, I was just curious how reliable
>it is  and how much experience you have with setting it up in Docker
>environment in order to know both advantages and disadvantages of this
>solution.
>
>@Eric, great that you are going to create POC, it will explain a lot and
>it will show us possible problems.
>
>Kind regards,
>Alicja Kwaśniewska
>
>
>-Original Message-
>From: Eric LEMOINE [mailto:elemo...@mirantis.com]
>Sent: Wednesday, January 13, 2016 10:55 AM
>To: OpenStack Development Mailing List (not for usage questions)
>Subject: Re: [openstack-dev] [kolla] Introduction of Heka in Kolla
>
>Hi Alicja
>
>
>Thank you for your comments.  Answers and comments below.
>
>
>
>On Tue, Jan 12, 2016 at 1:19 PM, Kwasniewska, Alicja
><alicja.kwasniew...@intel.com> wrote:
>> Unfortunately I do not have any experience in working or testing Heka,
>> so it's hard for me to compare its performance vs Logstash
>> performance. However I've read that Heka possess a lot advantages over
>>Logstash in this scope.
>>
>>
>> But which version of Logstash did you test? One guy from the Logstash
>> community said that: "The next release of logstash (1.2.0 is in beta)
>> has a 3.5x improvement in event throughput. For numbers: on my
>> workstation at home
>> (6 vcpu on virtualbox, host OS windows, 8 GB ram, host cpu is FX-8150)
>> - with logstash 1.1.13, I can process roughly 31,000 events/sec
>> parsing apache logs. With logstash 1.