Re: [openstack-dev] [kolla] Vagrant environment for kolla-kubernetes

[Eric LEMOINE]

On Fri, May 13, 2016 at 10:36 AM, Martin André  wrote:
> On Fri, May 13, 2016 at 8:57 AM, Britt Houser (bhouser)
>  wrote:
>> Would we ever run AIO-k8s vagrant w/o Kolla repo?  If not, then it makes 
>> sense to me just to extend Vagrant kolla repo.
>
> Agreed. Vagrant is for developers and developers will want the kolla
> repo for building the container images.
> Having a config options to setup the Vagrant environment for the
> different deployment methods seems appropriate here.

Agree. Duplicating the Vagrantfile would cause maintenance issues.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][vote] Proposing Alicja Kwasniewska for core reviewer

[Eric LEMOINE]

+1

Looking forward to more collaboration on Diagnostics and other stuff :)

Le 4 mars 2016 17:58, "Steven Dake (stdake)"  a écrit :
>
> Core Reviewers,
>
> Alicja has been instrumental in our work around jinja2 docker file
creation, removing our symlink madness.  She has also been instrumental in
actually getting Diagnostics implemented in a sanitary fashion.  She has
also done a bunch of other work that folks in the community already know
about that I won't repeat here.
>
> I had always hoped she would start reviewing so we could invite her to
the core review team, and over the last several months she has reviewed
quite a bit!  Her 90 day stats[1] place her at #9 with a solid ratio of
72%.  Her 30 day stats[2] are even better and place her at #6 with an
improving ratio of 67%.  She also just doesn't rubber stamp reviews or jump
in reviews at the end; she sticks with them from beginning to end and finds
real problems, not trivial things.  Finally Alicja is full time on Kolla as
funded by her employer so she will be around for the long haul and always
available.
>
> Please consider my proposal to be a +1 vote.
>
> To be approved for the core reviewer team, Alicja requires a majority
vote of 6 total votes with no veto within the one week period beginning now
and ending Friday March 11th.  If your on the fence, you can always
abstain.  If the vote is unanimous before the voting ends, I will make
appropriate changes to gerrit's acls.  If their is a veto vote, voting will
close prior to March 11th.
>
> Regards,
> -steve
>
> [1] http://stackalytics.com/report/contribution/kolla-group/90
> [2] http://stackalytics.com/report/contribution/kolla-group/30
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kolla][logging] log rotation

[Eric LEMOINE]

Hi Kolla devs

So with the Heka work services write their logs into files (in the
"kolla_logs" volume). This means that we need log rotation. This was
mentioned in the "logging-with-heka" spec [*].

I've just created a change request [**] that adds a "cron" Dockerfile
and Ansible tasks/plays to deploy a "cron" container on every node.
I've opened this mainly as a base for discussion.

So the "cron" container runs the crond daemon, which runs logrotate
daily for rotating the log files of kolla services. In the future the
"cron" container could be used for other cron-type tasks.

Thoughts? Feedback?

Thanks.

[*] 

[**] 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] discussion about core reviewer limitations by company

[Eric LEMOINE]

Le 20 févr. 2016 18:12, "Steven Dake (stdake)"  a écrit :
>
> Hey folks,
>
> Mirantis has been developing a big footprint in the core review team, and
Red Hat already has a big footprint in the core review team.  These are all
good things, but I want to avoid in the future a situation in which one
company has a majority of core reviewers.  Since core reviewers set policy
for the project, the project could be harmed if one company has such a
majority.  This is one reason why project diversity is so important and has
its own special snowflake tag in the governance repository.
>
> I'd like your thoughts on how to best handle this situation, before I
trigger  a vote we can all agree on.
>
> I was thinking of something simple like:
> "1 company may not have more then 33% of core reviewers.  At the
conclusion of PTL elections, the current cycle's 6 months of reviews
completed will be used as a metric to select the core reviewers from that
particular company if the core review team has shrunk as a result of
removal of core reviewers during the cycle."
>
> Thoughts, comments, questions, concerns, etc?


I think that introducing this policy would not be fair and would even be
counter-productive. For example, my motivation would be low if I knew I
couldn't be accepted as a core reviewer because my company already has "too
many" core reviewers.  So this policy would close the doors to developers
who could potentially be great contributors to the success of the project.
If anything, a policy where "2 people from the same company should not
approve a third person from that same company's patch" (as stated by Sam
Yaple) would sound more acceptable to me.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Make "central logging" optional

[Eric LEMOINE]

On Fri, Feb 5, 2016 at 4:46 PM, Eric LEMOINE <elemo...@mirantis.com> wrote:
> Hi Kolla devs
>
> The other day inc0 said that we would like "central logging" to be
> optional in Mitaka, and still use Rsyslog and keep the current
> behavior if "central logging" is disabled.
>
> I would like to propose an alternative, where we do remove Rsyslog as
> planned in the spec.
>
> I like the idea of an enable_central_logging switch, because it makes
> sense to me that an operator have the possibility to NOT centralize
> his logs in Elasticsearch (or anywhere else).
>
> When enable_central_logging is false I suggest to just not deploy
> Heka, Elasticsearch and Kibana. (Rsyslog would not be deployed
> either.)
>
> So when enable_central_logging is false the OpenStack services will
> still write their logs in the "log" named volume
> (/var/lib/docker/volumes/log/_data), but no one will read/process
> these logs.  They will just sit there at the disposal of the operator.
>
> For services that log to Syslog (HAProxy and Keepalived) we won't
> collect their logs if enable_central_logging is false, but these
> services' logs are not collected today, so no regression there.
>
> I think this would make a simple alternative.



I implemented that option in
<https://review.openstack.org/#/c/275841/> and
<https://review.openstack.org/#/c/275842/>.  More specifically see
<https://review.openstack.org/#/c/275841/11/ansible/roles/common/tasks/start.yml>
and 
<https://review.openstack.org/#/c/275842/12/ansible/roles/haproxy/templates/haproxy.cfg.j2>
and the use of the enable_elk variable.

So when enable_elk is false the Heka container will not be started,
and syslog logging will be disabled in HAProxy, but the OpenStack
containers will still write their logs in the "log" volume.

PS: I'd like to rename enable_elk to enable_central_logging, but this
can be done later, when both the Heka and Elasticsearch patches are
merged.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Location of Heka Lua plugins

[Eric LEMOINE]

Le 6 févr. 2016 20:39, "Steven Dake (stdake)" <std...@cisco.com> a écrit :
>
>
>
> On 2/5/16, 1:14 AM, "Eric LEMOINE" <elemo...@mirantis.com> wrote:
>
> >On Thu, Feb 4, 2016 at 5:13 PM, Jeff Peeler <jpee...@redhat.com> wrote:
> >I totally agree with you Jeff.
> >
> >It is to be noted that we (my team at Mirantis) want to avoid
> >duplicating our Lua plugins, as we obviously don't want to maintain
> >two sets of identical plugins.  So there are mulitple reasons for
> >creating separate packages for these plugins: a) make it easy the
> >share the plugins across different projects, b) avoid maintaining
> >multiple sets of identical plugins, and c) avoid clobbering Kolla with
> >code not directly related to Kolla - for example, would you really
> >like to see Lua tests in Kolla and run Lua tests in the Kolla gates?
> >It would indeed be best to have these plugins in the OpenStack Git
> >namespace (as Steve Dake said), but we will have to see if that's
> >possible in practice.
> >
> >Thank you all for your responses.
>
> Eric,
>
> If I read that correctly, there is some implied resistance to placing
> these LUA plugins in the openstack git namespace.  Could you enumerate the
> issues now please?

We have no problem with placing these Lua plugins in the openstack git
namespace.  At this point I just don't know if others from the OpenStack
community would see this as appropriate.  That's all I'm saying.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kolla] Make "central logging" optional

[Eric LEMOINE]

Hi Kolla devs

The other day inc0 said that we would like "central logging" to be
optional in Mitaka, and still use Rsyslog and keep the current
behavior if "central logging" is disabled.

I would like to propose an alternative, where we do remove Rsyslog as
planned in the spec.

I like the idea of an enable_central_logging switch, because it makes
sense to me that an operator have the possibility to NOT centralize
his logs in Elasticsearch (or anywhere else).

When enable_central_logging is false I suggest to just not deploy
Heka, Elasticsearch and Kibana. (Rsyslog would not be deployed
either.)

So when enable_central_logging is false the OpenStack services will
still write their logs in the "log" named volume
(/var/lib/docker/volumes/log/_data), but no one will read/process
these logs.  They will just sit there at the disposal of the operator.

For services that log to Syslog (HAProxy and Keepalived) we won't
collect their logs if enable_central_logging is false, but these
services' logs are not collected today, so no regression there.

I think this would make a simple alternative.

Thoughts?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Location of Heka Lua plugins

[Eric LEMOINE]

On Thu, Feb 4, 2016 at 5:13 PM, Jeff Peeler  wrote:
> On Thu, Feb 4, 2016 at 9:22 AM, Michał Jastrzębski  wrote:
>> TLDR; +1 to have lua in tree of kolla, not sure if we want to switch later
>>
>> So I'm not so sure about switching. If these git repos are in
>> /openstack/ namespace, then sure, otherwise I'd be -1 to this, we
>> don't want to add dependency here. Also we're looking at pretty simple
>> set of files that won't change anytime soon probably. Also we might
>> introduce new service that fuel does not have, and while I'm sure we
>> can push new file to this repo, it's bigger issue than just coding it
>> in tree.
>
> I wouldn't think there'd be any opposition to having additional
> contributed Lua scripts for additional services that Fuel doesn't yet
> have. It's always easier to commit to one tree, but distinct
> components like this separated out encourages code sharing (done
> properly). I did assume that the new repo would be in the OpenStack
> namespace, but even if it weren't I'd still think a separate repo is
> best. Sure the files are small, but given the purpose of these files
> small changes could potentially have a huge impact on the logs.
>
> In summary, +1 to temporarily having the Lua scripts in tree until
> they can be properly migrated to a new repository.


I totally agree with you Jeff.

It is to be noted that we (my team at Mirantis) want to avoid
duplicating our Lua plugins, as we obviously don't want to maintain
two sets of identical plugins.  So there are mulitple reasons for
creating separate packages for these plugins: a) make it easy the
share the plugins across different projects, b) avoid maintaining
multiple sets of identical plugins, and c) avoid clobbering Kolla with
code not directly related to Kolla – for example, would you really
like to see Lua tests in Kolla and run Lua tests in the Kolla gates?
It would indeed be best to have these plugins in the OpenStack Git
namespace (as Steve Dake said), but we will have to see if that's
possible in practice.

Thank you all for your responses.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kolla] Location of Heka Lua plugins

[Eric LEMOINE]

Hi

As discussed yesterday in our IRC meeting we'll need specific Lua
plugins for parsing OpenStack, MariaDB and RabbitMQ logs.  We already
have these Lua plugins in one of our Fuel plugins [*].  So our plan is
to move these Lua plugins in their own Git repo, and distribute them
as deb and rpm packages in the future.  This will allow to easily
share the Lua plugins between projects, and having a separate Git repo
will facilitate testing, documentation, etc.

But we may not have time to do that by the 4th of March (for Mitaka
3), so my suggestion is to copy the Lua plugins that we need in Kolla.
This would be a temporary thing.  When our Lua plugins are
available/distributed as deb and rpm packages we will remove the Lua
plugins from the kolla repository and change the Heka Dockerfile to
install the Lua plugins from the package.

Please tell me if you agree with the approach.  Thank you!

[*] 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kolla] Heka POC

[Eric LEMOINE]

Hi Kolla devs

So I've identified the following tests/POCs:

* Verify Heka can read logs from containers that log to stdout
* Verify Heka can read logs from containers that log to Syslog (/dev/log)
* Verify Heka can write logs to local files (as currently done using Rsyslog)

If these three POCs "succeed" then it'll mean that we may be able to
remove Rsyslog entirely, and I'll write the specs in that sense.

When I am done with these tests I'll report back to the mailing list,
and then continue working on the specs.

Do we agree with that?

Thanks.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Eric LEMOINE]

On Fri, Jan 15, 2016 at 2:40 PM, Michał Jastrzębski  wrote:
> Yeah that's true. We did all of openstack systems but we didn't
> implement infra around yet. I'd guess most of services can log either
> to stdout or file, and both sources should be accessible by heka.
> Also, I'd be surprised if heka wouldn't have syslog driver?

Heka's UdpInput plugin supports Unix datagram sockets [*], so this
plugin can possibly be used for reading from /dev/log.  This is
something I'll be testing soon.

[*] 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Eric LEMOINE]

On Fri, Jan 15, 2016 at 11:57 AM, Michal Rostecki
 wrote:
> On 01/15/2016 11:14 AM, Simon Pasquier wrote:
>>
>> My 2 cents on RabbitMQ logging...
>>
>> On Fri, Jan 15, 2016 at 8:39 AM, Michal Rostecki > > wrote:
>>
>> I'd suggest to check the similar options in RabbitMQ and other
>> non-OpenStack components.
>>
>>
>> AFAICT RabbitMQ can't log to Syslog anyway. But you have option to make
>> RabbitMQ log to stdout [1].
>> BR,
>> Simon.
>> [1] http://www.superpumpup.com/docker-rabbitmq-stdout
>>
>
> That's OK for Heka/Mesos/k8s approach.
>
> Just for the curiosity,
> @inc0: so we don't receive any logs from RabbitMQ in the current rsyslog
> approach?


/var/lib/docker/volumes/rsyslog/_data is where logs are stored, and
you'll see that there is no file for RabbitMQ.  This is related to
RabbitMQ not logging to syslog.  So our impression is that Kolla
doesn't at all collect RabbitMQ logs today.  I guess this should be
fixed.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Eric LEMOINE]

>
> In case of MariaDB I see that you can
>
> - do --skip-syslog, then all things go to "error log"
> - set "error log" path by --log-error
>
> So maybe setting stdout/stderr may work here?
>
> I'd suggest to check the similar options in RabbitMQ and other non-OpenStack
> components.
>
> I may help with this, because in my opinion, logging to stdout is the best
> default option for Mesos and Kubernetes - i.e. Mesos UI shows a "sandbox" of
> application, which generally is stdout/stderr. So if someone will not want
> to use Heka/ELK, then having everything in Mesos/Kubernetes would be
> probably the lesser evil than trying rsyslog to work here.


That makes sense to me Michal. Thanks!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Eric LEMOINE]

On Wed, Jan 13, 2016 at 1:15 PM, Steven Dake (stdake)  wrote:
> Hey folks,
>
> I'd like to have a mailing list discussion about logistics of the ELKSTACK
> solution that Alicja has sorted out vs the Heka implementation that Eric is
> proposing.
>
> My take on that is Eric wants to replace rsyslog and logstash with Heka.


See my other email on this point.  At this point, given the
requirements we have (get logs from services that only speak syslog
and write logs to local files), we cannot guarantee that Heka will
replace Rsyslog.  We are going to test the use of Heka's UdpInput
(with "net" set to "unixgram") et FileOutput plugins for that.  Stay
tuned!


> That seems fine, but I want to make certain this doesn't happen in a way
> that leaves Kolla completely non-functional as we finish up Mitaka.  Liberty
> is the first version of Kolla people will deploy, and Mitaka is the first
> version of Kolla people will upgrade to, so making sure that we don't
> completely bust diagnostics (and I recognize diags as is are a little weak
> is critical).
>
> It sounds like from my reading of the previous thread on this topic, unless
> there is some intractable problem, our goal is to use Heka to replace
> resyslog and logstash.  I'd ask inc0 (who did the rsyslog work) and Alicja
> (who did the elkstack work) to understand that replacement often happens on
> work that has already been done, and its not a "waste of time" so to speak
> as an evolution of the system.
>
> Here are the deadlines:
> http://docs.openstack.org/releases/schedules/mitaka.html
>
> Let me help decode that for folks. March 4th is the final deadline to have a
> completely working solution based upon Heka if its to enter Mitaka.


Understood.


>
> Unlike previous releases of Kolla, I want to hand off release management of
> Kolla to the release management team, and to do that, we need to show a
> track record of hitting our deadlines and not adding features past feature
> freeze (the m3 milestone on March 4th).  In the past releases of Kolla we as
> a team were super loose on this requirement – going forward I prefer us
> being super strict.  Handing off to release management is a sign of maturity
> and would have an overall positive impact, assuming we can get the software
> written in time :)
>
> Eric,
>
> I'd like a plan and commitment to either hit Mitaka 3, or the N cycle.  It
> must work well first on Ansible, and second on Mesos.  If it doesn't work at
> all on Mesos, I could live with that -  I think the Mesos implementation
> will really not be ready for prime time until the middle or completion of
> the N cycle.  We lead with Ansible, and I don't see that changing any time
> soon – as a result, I want our Ansible deployment to be rock solid and
> usable out of the gate.  I don't expect to "Market" Mitaka Mesos (with the
> OpenStack foundation's help) as "production ready" but rather as "tech
> preview" and something for folks to evaluate.


It is our intent to meet the March 4th deadline.



>
> Alicja,
>
> I think a parallel development effort with the ELKSTACK that your working on
> makes sense.  In case the Heka development fails entirely, or misses Mitaka
> 3, I don't want us left lacking a diagnostics solution for Mitaka.
> Diagnostics is my priority #2 for Kolla (#1 is upgrades).  Unfortunately
> what this means is you may end up wasting your time doing development that
> is replaced at the last minute in Mitaka 3, or later in the N cycle.  This
> is very common in software development (all the code I wrote for Magnum has
> been sadly replaced).  I know you can be a good team player here and take
> one for the team so to speak, but I'm asking you if you would take offense
> to this approach.


I'd like to moderate this a bit.  We want to build on Alicja's work,
and we will reuse everything that Alicja has done/will do on
Elasticsearch and Kibana, as this part of the stack will be the same.



>
> I'd like comments/questions/concerns on the above logistics approach
> discussed, and a commitment from Eric as to when he thinks all the code
> would land as one patch stream unit.
>
> I'd also like to see the code come in as one super big patch stream (think
> 30 patches in the stream) so the work can be evaluated and merged as one
> unit.  I could also live with 2-3 different patch streams with 10-15 patches
> per stream, just so we can eval as a unit.  This means lots of rebasing on
> your part Eric ;-)  It also means a commitment from the core reviewer team
> to test and review this critical change.  If there isn't a core reviewer on
> board with this approach, please speak up now.


Makes total sense to me.


Thanks.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Can Heka solve all the deficiencies in the current rsyslog implementation: was Re: [kolla] Introduction of Heka in Kolla

[Eric LEMOINE]

On Wed, Jan 13, 2016 at 1:27 PM, Steven Dake (stdake)  wrote:
> Eric,


Hi Steven


>
> Apologies for top post, not really sure where in this thread to post this
> list of questions as its sort of a change in topic so I changed the
> subject line :)
>
> 1.
> Somewhere I read when researching this Heka topic, that Heka cannot log
> all details from /dev/log.  Some services like mariadb for example don't
> log to stdout as I think Heka requires to operate correctly.  Would you
> mind responding on the question "Would Heka be able to effectively log
> every piece of information coming off the system related to OpenStack (our
> infrastructure services like ceph/mariadb/etc as well as the OpenStack
> services)?


My first reaction to this is: if we have services, such as mariadb,
that can only send their logs to syslog then let's continue using
Rsyslog.  And with Rsyslog we can easily store logs on the local
filesystem as well (your requirement #3 below).

That being said, it appears that Heka supports reading logs from
/dev/log.  This can be done using the UdpInput plugin with "net" set
to "unixgram".  See
 for the original
issue.  Heka also supports writing logs to files on the local
filesystem, through the FileOutput plugin.  We do not currently use
the UdpInput plugin, so we need to test it and see if it can work for
Kolla.  We will work on these tests, and report back to the list.



> 2.
> Also, I want to make sure we can fix up the backtrace defeciency.
> Currently rsyslog doesn't log backtraces in python code.  Perhaps Sam or
> inc0 know the reason behind it, but I want to make sure we can fix up this
> annoyance, because backtraces are mightily important.


I've had a look on my AIO Kolla.  And I do see Python tracebacks in
OpenStack log files created by Rsyslog (in
/var/lib/docker/volumes/rsyslog/_data/nova/nova-api.log for example).
They're just on a single line, with "#012" used as the separator [*].
So they are hard to read, but they are there.  I think that is
consistent with what SamYaple and inc0 said yesterday on IRC.

[*] This is related to Rsyslog's $EscapeControlCharactersOnReceive
setting. See 
.


> 3.
> Also I want to make sure each node ends up with log files in a data
> container (or data volume or whatever we just recently replaced the data
> containers with) for all the services for individual node diagnostics.
> This helps fill the gap of the Kibana visualization and Elasticsearch
> where we may not have a perfect diagnostic solution at the conclusion of
> Mitaka and in need of individual node inspection of the logs.  Can Heka be
> made to do this?  Our rsyslog implementation does today, and its a hard
> requirement for the moment.  If we need some special software to run in
> addition to Heka, I could live with that.


That "special software" could be Rsyslog :)  Seriously, Rsyslog
provides a solution for getting logs from services that only log to
syslog.  We can also easily configure Rsyslog to write logs on the
local filesystem, as done in Kolla already today.  And using Heka we
can easily make Python tracebacks look good in Kibana.

I would like to point out that our initial intent was not to remove
Rsyslog.  Our intent was to propose a scalable/decentralized log
processing architecture based on Heka running on each node, instead of
relying on a centralized Logstash instance.  Using Heka we eliminate
the need to deploy and manage a resilient Logstash/Redis cluster.  And
it is to be noted that Heka gives us a lot of flexibility.  In
particular, Heka makes it possible to collect logs from services that
don't speak syslog (RabbitMQ for example, whose logs are not currently
collected!).

As mentioned above Heka provides plugins that we could possibly
leverage to remove Rsyslog completely, but at this point we cannot
guarantee that they will do the job.  Our coming tests will tell.

Thanks.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Introduction of Heka in Kolla

[Eric LEMOINE]

Hi Alicja


Thank you for your comments.  Answers and comments below.



On Tue, Jan 12, 2016 at 1:19 PM, Kwasniewska, Alicja
 wrote:
> Unfortunately I do not have any experience in working or testing Heka, so
> it’s hard for me to compare its performance vs Logstash performance. However
> I’ve read that Heka possess a lot advantages over Logstash in this scope.
>
>
> But which version of Logstash did you test? One guy from the Logstash
> community said that: “The next release of logstash (1.2.0 is in beta) has a
> 3.5x improvement in event throughput. For numbers: on my workstation at home
> (6 vcpu on virtualbox, host OS windows, 8 GB ram, host cpu is FX-8150) -
> with logstash 1.1.13, I can process roughly 31,000 events/sec parsing apache
> logs. With logstash 1.2.0.beta1, I can process 102,000 events/sec.”


I've used the latest Docker image:
.  It uses Logstash 2.1.1,
which is the most recent stable version.



> You also said that Heka is a unified data processing, but do we need this?


Heka, as a unified data processing, enables to derive metrics from
logs, HTTP response times for example.  Alerts can also be triggered
on specific log patterns.


> Heka seems to address stream processing needs, while Logstash focuses mainly
> on processing logs. We want to create a central logging service, and
> Logstash was created especially for it and seems to work well for this
> application.
>
>
> One thing that is obvious is the fact that the Logstash is better known,
> more popular and tested. Maybe it has some performance disadvantages, but at
> least we know what we can expect from it. Also, it has more pre-built
> plugins and has a lot examples of usage, while Heka doesn’t have many of
> them yet and is nowhere near the range of plugins and integrations provided
> by Logstash.


As Simon said Heka already includes quite a lot of plugins.  See the
Heka documentation [*] for an exhaustive list.  It may indeed be the
case that Logstash includes even more plugins, but Heka has taken us
pretty far already.



> In the case of adding plugins, I’ve read that in order to add Go plugins,
> the binary has to be recompiled, what is a little bit frustrating (static
> linking - to wire in new plugins, have to recompile). On the other hand, the
> Lua plugins do not require it, but the question is whether Lua plugins are
> sufficient? Or maybe adding Go plugins is not so bad?


See Simon's answer.



> You also said that you didn’t test the Heka with Docker, right?


I did test Heka with Docker.  In my performance tests both Heka and
Logstash ran in Docker containers.  What I haven't tested yet is the
Docker Log Input plugin.  We'll do more tests as part of the work on
specifications.



> But do you
> have any experience in setting up Heka in Docker container? I saw that with
> Heka 0.8.0 new Docker features were implemented (included Dockerfiles to
> generate Heka Docker containers for both development and deployment), but
> did you test it? If you didn’t, we could not be sure whether there are any
> issues with it.
>
>
> Moreover you will have to write your own Dockerfile for Heka that inherits
> from Kolla base image (as we discussed during last meeting, we would like to
> have our own images), you won’t be able to inherit from ianneub/heka:0.10 as
> specified in the link that you sent
> http://www.ianneubert.com/wp/2015/03/03/how-to-use-heka-docker-and-tutum/.



As I said in my first email Heka has no dependencies, so creating a
Dockerfile for Heka is quite easy.  See

for the super-simple Dockerfile I've used so far.


> There are also some issues with DockerInput Module which you want to use.
> For example splitters are not available in DockerInput
> (https://github.com/mozilla-services/heka/issues/1643). I can’t say that it
> will affect us, but we also don’t know which new issues may arise during
> first tests, as any of us has ever tried Heka in and with Dockers.



Yes, we're aware of that limitation.  But, we're not sure this is a
problem, as the decoder can be the component coalescing log lines.  We
already have a Lua decoder that does that, accumulating lines of
Python Tracebacks.  I am going to look at this in more detail when
working on the blueprint.



> I am not stick to any specific solution, however just not sure whether Heka
> won’t surprise us with something hard to solve, configure, etc.


We chose Heka because it's lightweight and fast, while providing us
with the flexibility we need for processing different types of data
streams.  The distributed architecture we think is necessary for large
environments requires running the logs processing component on each
cluster node, and we did not want to run a JVM on each node,
especially on compute nodes where user VMs run.



Thanks,

Re: [openstack-dev] [kolla] Introduction of Heka in Kolla

[Eric LEMOINE]

On Tue, Jan 12, 2016 at 5:36 PM, Michał Jastrzębski  wrote:
> So tracebacks sort of works, they're there just ugly. That's why I'm
> also happy if we change rsyslog to heka.
>
> Eric, I hope I wont ask too much, but could you please prepare PoC of
> kolla+heka, for what I care heka can log to local log file same as
> rsyslog for now. Would that be big problem?


I'll certainly do that POC while working on the blueprint.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Introduction of Heka in Kolla

[Eric LEMOINE]

On Tue, Jan 12, 2016 at 8:26 PM, Steven Dake (stdake) <std...@cisco.com> wrote:
> Eric,
>
> Thanks for using the mailing list for this discussion.  I like to see more
> mailing list conversations on big changes related to kolla, which this one
> is :)
> Responses inline.
>
> Please put document #3 (the design document) in gerrit rather than google
> docs in the main kolla repo as a spec using our special snowflake (read
> less cumbersome) spec template.


Sure.  The Google doc is a temporary thing.


> On 1/11/16, 8:16 AM, "Eric LEMOINE" <elemo...@mirantis.com> wrote:
>>In the proposed architecture each cluster node runs an instance of
>>Heka for collecting and processing logs.  And instead of sending the
>>processed logs to a centralized Logstash instance, logs are directly
>>sent to Elasticsearch, which itself can be distributed across multiple
>>nodes for high-availability and scaling.  The proposed architecture is
>>based on Heka, and it doesn't use Logstash.
>
> How are the elasticsearch network addresses discovered by Heka here?


Initially one Elasticsearch instance will be used, as in Alicja's work
(<https://review.openstack.org/#/c/252968>).  In the future, HAProxy,
which is already included in Kolla, could be used between Heka and
Elasticsearch.  Another option would be to extend Heka's Elasticsearch
Ouput plugin to work with a list a Elasticsearch hosts instead of just
one host.



>>That being said, it is important to note that the intent of this
>>proposal is not strictly directed at replacing Logstash by Heka.  The
>>intent is to propose a distributed architecture with Heka running on
>>each cluster node rather than having Logstash run as a centralized
>>logs processing component.  For such a distributed architecture we
>>think that Heka is more appropriate, with a smaller memory footprint
>>and better performances in general.  In addition, Heka is also more
>>than a logs processing tool, as it's designed to process streams of
>>any type of data, including events, logs and metrics.
>
> I think the followup was that the intent of this proposal was to replace
> both logstash and rsyslog.  Could you comment on that?



Yes, it may make sense to remove Rsyslog entirely and only rely on
Heka.  This is something we want/need to assess in the specifications.



> It may be that
> this work has to be punted to the N cycle if that¹s the case - we are
> super short on time, and need updates done.



Yes, sure, that makes sense.


>
> Will you be making it to the Kolla Midcycle Feb 9th and 10th to discuss
> this system face to face?


I won't be able to go myself unfortunately.  But there will probably
be someone from Mirantis (from the kolla-mesos team) whom you can talk
to.




>>* Heka is written in Go and has no dependencies.  Go programs are
>>compiled to native code.  This in contrast to Logstash which uses
>>JRuby and as such requires running a Java Virtual Machine.  Besides
>>this native versus interpreted code aspect, this also can raise the
>>question of which JVM to use (Oracle, OpenJDK?) and which version
>>(6,7,8?).
>
> This I did not know.  I was aware kibana (visualization) was implemented
> in Java.
>
> I would prefer to avoid any Java dependencies in the Kolla project.  The
> reason being there is basically a fork of the virtual machines, the Oracle
> version and the openjdk version.  This creates licensing problems for our
> downstreams.  If Kibana and Elasticsearch are developed in Java, I guess
> we will just have to live with that but the less Java dependencies the
> better.


Confirming that Elasticsearch and Logstash run in a JVM.  However,
Kibana is written in JavaScript, and Kibana version 4 includes a
server-side component that runs in NodeJS.



>>* There are six types of Heka plugins: Inputs, Splitters, Decoders,
>>Filters, Encoders, and Outputs.  Heka plugins are written in Go or
>>Lua.  When written in Lua their executions are sandbox'ed, where
>>misbehaving plugins may be shut down by Heka.  Lua plugins may also be
>>dynamically added to Heka with no config changes or Heka restart. This
>>is an important property on container environments such as Mesos,
>>where workloads are changed dynamically.
>
> For any update to the Kolla environment, I expect a full pull, stop, start
> of the service. This preserves immutability which is a magical property of
> a container.  For more details on my opinions on this matter, please take
> 10 minutes and read:
>
> http://sdake.io/2015/11/11/the-tldr-on-immutable-infrastructure/


Thanks for the link.


>>* Heka is faster than Logstash for processing logs, and its memory
>>footprint is smaller.  I ran tests, where 3,

Re: [openstack-dev] [kolla] Introduction of Heka in Kolla

[Eric LEMOINE]

Le 11 janv. 2016 18:45, "Michał Jastrzębski" <inc...@gmail.com> a écrit :
>
> On 11 January 2016 at 10:55, Eric LEMOINE <elemo...@mirantis.com> wrote:
> > Currently the services running in containers send their logs to
> > rsyslog. And rsyslog stores the logs in local files, located in the
> > host's /var/log directory.
>
> Yeah, however plan was to teach rsyslog to forward logs to central
> logging stack once this thing is implemented.

Yes. With the current ELK Change Request, Rsyslog sends logs to the central
Logstash instance. If you read my design doc you'll understand that it's
precisely what we're proposing changing.

> > I know. Our plan is to rely on Docker. Basically: containers write
> > their logs to stdout. The logs are collected by Docker Engine, which
> > makes them available through the unix:///var/run/docker.sock socket.
> > The socket is mounted into the Heka container, which uses the Docker
> > Log Input plugin [*] to reads the logs from that that socket.
> >
> > [*] <
http://hekad.readthedocs.org/en/latest/config/inputs/docker_log.html>
>
> So docker logs isn't best thing there is, however I'd suspect that's
> mostly console output fault. If you can tap into stdout efficiently,
> I'd say that's pretty good option.

I'm not following you. Could you please be more specific?

> >> Seems to me we need additional comparason of heka vs rsyslog;) Also
> >> this would have to be hands down better because rsyslog is already
> >> implemented, working and most of operators knows how to use it.
> >
> >
> > We don't need to remove Rsyslog. Services running in containers can
> > write their logs to both Rsyslog and stdout, which even is what they
> > do today (at least for the OpenStack services).
> >
>
> There is no point for that imho. I don't want to have several systems
> doing the same thing. Let's make decision of one, but optimal toolset.
> Could you please describe bottoms up what would your logging stack
> look like? Heka listening on stdout, transfering stuff to
> elasticsearch and kibana on top of it?

My plan is to provide details in the blueprint document, that I'll continue
working on if the core developers agree with the principles of the proposed
architecture and change.

But here's our plan—as already described in my previous email: the Kolla
services, which run in containers, write their logs to stdout. Logs are
collected by the Docker engine. Heka's Docker Log Input plugin is used to
read the container logs from the Docker endpoint (Unix socket). Since Heka
will run in a container a volume is necessary for accessing the Docker
endpoint. The Docker Log Input plugin inserts the logs into the Heka
pipeline, at the end of which an Elasticsearch Output plugin will send the
log messages to Elasticsearch. Here's a blog post reporting on that
approach: <
http://www.ianneubert.com/wp/2015/03/03/how-to-use-heka-docker-and-tutum/>.
We haven't tested that approach yet, but we plan to experiment with it as
we work on the specs.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Introduction of Heka in Kolla

[Eric LEMOINE]

On Mon, Jan 11, 2016 at 5:01 PM, Michał Jastrzębski <inc...@gmail.com> wrote:
> On 11 January 2016 at 09:16, Eric LEMOINE <elemo...@mirantis.com> wrote:

>> * Logstash was designed for logs processing.  Heka is a "unified data
>> processing" software, designed to process streams of any type of data.
>> So Heka is about running one service on each box instead of many.
>> Using a single service for processing different types of data also
>> makes it possible to do correlations, and derive metrics from logs and
>> events.  See Rob Miller's presentation [4] for more details.
>
> Right now we use rsyslog for that.


Currently the services running in containers send their logs to
rsyslog. And rsyslog stores the logs in local files, located in the
host's /var/log directory.


> As I understand Heka right now
> would be actually alternative to rsyslog, and that is already
> implemented. Also with Heka case, we might run into same problem we've
> seen with rsyslog - transporting logs from service to heka. Keep in
> mind we're in dockers and heka will be in different container than
> service it's supposed to listen to. We do that by sharing faked
> /dev/log across containers and rsyslog can handle that.


I know. Our plan is to rely on Docker. Basically: containers write
their logs to stdout. The logs are collected by Docker Engine, which
makes them available through the unix:///var/run/docker.sock socket.
The socket is mounted into the Heka container, which uses the Docker
Log Input plugin [*] to reads the logs from that that socket.

[*] <http://hekad.readthedocs.org/en/latest/config/inputs/docker_log.html>


> Also Heka seems to be just processing mechanism while logstash is
> well...stash, so it saves and persists logs, so it seems to me they're
> different layers of log processing.

No. Logstash typically stores the logs in Elasticsearch. And we'd do
the same with Heka.


>
> Seems to me we need additional comparason of heka vs rsyslog;) Also
> this would have to be hands down better because rsyslog is already
> implemented, working and most of operators knows how to use it.


We don't need to remove Rsyslog. Services running in containers can
write their logs to both Rsyslog and stdout, which even is what they
do today (at least for the OpenStack services).


Hope that makes sense!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kolla] Introduction of Heka in Kolla

[Eric LEMOINE]

Hi

As discussed on IRC the other day [1] we want to propose a distributed
logs processing architecture based on Heka [2], built on Alicja
Kwasniewska's ELK work with
.  Please take a look at the
design document I've started working on [3].  The document is still
work-in-progress, but the "Problem statement" and "Proposed change"
sections should provide you with a good overview of the architecture
we have in mind.

In the proposed architecture each cluster node runs an instance of
Heka for collecting and processing logs.  And instead of sending the
processed logs to a centralized Logstash instance, logs are directly
sent to Elasticsearch, which itself can be distributed across multiple
nodes for high-availability and scaling.  The proposed architecture is
based on Heka, and it doesn't use Logstash.

That being said, it is important to note that the intent of this
proposal is not strictly directed at replacing Logstash by Heka.  The
intent is to propose a distributed architecture with Heka running on
each cluster node rather than having Logstash run as a centralized
logs processing component.  For such a distributed architecture we
think that Heka is more appropriate, with a smaller memory footprint
and better performances in general.  In addition, Heka is also more
than a logs processing tool, as it's designed to process streams of
any type of data, including events, logs and metrics.

Some elements of comparison between Heka and Logstash:

* Logstash was designed for logs processing.  Heka is a "unified data
processing" software, designed to process streams of any type of data.
So Heka is about running one service on each box instead of many.
Using a single service for processing different types of data also
makes it possible to do correlations, and derive metrics from logs and
events.  See Rob Miller's presentation [4] for more details.

* The virtual size of the Logstash Docker image is 447 MB, while the
virtual size of an Heka image built from the same base image
(debian:jessie) is 177 MB.  For comparison the virtual size of the
Elasticsearch image is 345 MB.

* Heka is written in Go and has no dependencies.  Go programs are
compiled to native code.  This in contrast to Logstash which uses
JRuby and as such requires running a Java Virtual Machine.  Besides
this native versus interpreted code aspect, this also can raise the
question of which JVM to use (Oracle, OpenJDK?) and which version
(6,7,8?).

* There are six types of Heka plugins: Inputs, Splitters, Decoders,
Filters, Encoders, and Outputs.  Heka plugins are written in Go or
Lua.  When written in Lua their executions are sandbox'ed, where
misbehaving plugins may be shut down by Heka.  Lua plugins may also be
dynamically added to Heka with no config changes or Heka restart. This
is an important property on container environments such as Mesos,
where workloads are changed dynamically.

* To avoid losing logs under high load it is often recommend to use
Logstash together with Redis [5].  Redis plays the role of a buffer,
where logs are queued when Logstash or Elasticsearch cannot keep up
with the load.  Heka, as a "unified data processing" software,
includes its own resilient message queue, making it unnecessary to use
an external queue (Redis for example).

* Heka is faster than Logstash for processing logs, and its memory
footprint is smaller.  I ran tests, where 3,400,000 log messages were
read from 500 input files and then written to a single output file.
Heka processed the 3,400,000 log messages in 12 seconds, consuming
500M of RAM.  Logstash processed the 3,400,000 log messages in 1mn
35s, consuming 1.1G of RAM.  Adding a grok filter to parse and
structure logs, Logstash processed the 3,400,000 log messages in 2mn
15s, consuming 1.5G of RAM. Using an equivalent filtering plugin, Heka
processed the 3,400,000 log messages in 27s, consuming 730M of RAM.
See my GitHub repo [6] for more information about the test
environment.

Also, I want to say that our team has been using Heka in production
for about a year, in clusters of up to 200 nodes.  Heka has proven to
be very robust, efficient and flexible enough to address our logs
processing and monitoring use-cases.  We've also acquired a solid
experience with it.

Any comments are welcome!

Thanks.


[1] 

[2] 
[3] 

[4] 
[5] 
[6] 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe