Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Eric LEMOINE]

On Fri, Jan 15, 2016 at 2:40 PM, Michał Jastrzębski  wrote:
> Yeah that's true. We did all of openstack systems but we didn't
> implement infra around yet. I'd guess most of services can log either
> to stdout or file, and both sources should be accessible by heka.
> Also, I'd be surprised if heka wouldn't have syslog driver?

Heka's UdpInput plugin supports Unix datagram sockets [*], so this
plugin can possibly be used for reading from /dev/log.  This is
something I'll be testing soon.

[*] 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Simon Pasquier]

My 2 cents on RabbitMQ logging...

On Fri, Jan 15, 2016 at 8:39 AM, Michal Rostecki 
wrote:

I'd suggest to check the similar options in RabbitMQ and other
> non-OpenStack components.
>

AFAICT RabbitMQ can't log to Syslog anyway. But you have option to make
RabbitMQ log to stdout [1].
BR,
Simon.
[1] http://www.superpumpup.com/docker-rabbitmq-stdout
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Michal Rostecki]

On 01/15/2016 11:14 AM, Simon Pasquier wrote:

My 2 cents on RabbitMQ logging...

On Fri, Jan 15, 2016 at 8:39 AM, Michal Rostecki > wrote:

I'd suggest to check the similar options in RabbitMQ and other
non-OpenStack components.


AFAICT RabbitMQ can't log to Syslog anyway. But you have option to make
RabbitMQ log to stdout [1].
BR,
Simon.
[1] http://www.superpumpup.com/docker-rabbitmq-stdout



That's OK for Heka/Mesos/k8s approach.

Just for the curiosity,
@inc0: so we don't receive any logs from RabbitMQ in the current rsyslog 
approach?


Cheers,
Michal

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Eric LEMOINE]

On Fri, Jan 15, 2016 at 11:57 AM, Michal Rostecki
 wrote:
> On 01/15/2016 11:14 AM, Simon Pasquier wrote:
>>
>> My 2 cents on RabbitMQ logging...
>>
>> On Fri, Jan 15, 2016 at 8:39 AM, Michal Rostecki > > wrote:
>>
>> I'd suggest to check the similar options in RabbitMQ and other
>> non-OpenStack components.
>>
>>
>> AFAICT RabbitMQ can't log to Syslog anyway. But you have option to make
>> RabbitMQ log to stdout [1].
>> BR,
>> Simon.
>> [1] http://www.superpumpup.com/docker-rabbitmq-stdout
>>
>
> That's OK for Heka/Mesos/k8s approach.
>
> Just for the curiosity,
> @inc0: so we don't receive any logs from RabbitMQ in the current rsyslog
> approach?


/var/lib/docker/volumes/rsyslog/_data is where logs are stored, and
you'll see that there is no file for RabbitMQ.  This is related to
RabbitMQ not logging to syslog.  So our impression is that Kolla
doesn't at all collect RabbitMQ logs today.  I guess this should be
fixed.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Michał Jastrzębski]

Yeah that's true. We did all of openstack systems but we didn't
implement infra around yet. I'd guess most of services can log either
to stdout or file, and both sources should be accessible by heka.
Also, I'd be surprised if heka wouldn't have syslog driver? That
should be one of first:) Maybe worth writing one? I wanted an excuse
to write some golang;)

Regards,
Michal

On 15 January 2016 at 06:42, Eric LEMOINE  wrote:
> On Fri, Jan 15, 2016 at 11:57 AM, Michal Rostecki
>  wrote:
>> On 01/15/2016 11:14 AM, Simon Pasquier wrote:
>>>
>>> My 2 cents on RabbitMQ logging...
>>>
>>> On Fri, Jan 15, 2016 at 8:39 AM, Michal Rostecki >> > wrote:
>>>
>>> I'd suggest to check the similar options in RabbitMQ and other
>>> non-OpenStack components.
>>>
>>>
>>> AFAICT RabbitMQ can't log to Syslog anyway. But you have option to make
>>> RabbitMQ log to stdout [1].
>>> BR,
>>> Simon.
>>> [1] http://www.superpumpup.com/docker-rabbitmq-stdout
>>>
>>
>> That's OK for Heka/Mesos/k8s approach.
>>
>> Just for the curiosity,
>> @inc0: so we don't receive any logs from RabbitMQ in the current rsyslog
>> approach?
>
>
> /var/lib/docker/volumes/rsyslog/_data is where logs are stored, and
> you'll see that there is no file for RabbitMQ.  This is related to
> RabbitMQ not logging to syslog.  So our impression is that Kolla
> doesn't at all collect RabbitMQ logs today.  I guess this should be
> fixed.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Patrick Petit]

On 15 Jan 2016 at 14:43:39, Michał Jastrzębski (inc...@gmail.com) wrote:
Yeah that's true. We did all of openstack systems but we didn't 
implement infra around yet. I'd guess most of services can log either 
to stdout or file, and both sources should be accessible by heka. 
Also, I'd be surprised if heka wouldn't have syslog driver? That 
should be one of first:) Maybe worth writing one? I wanted an excuse 
to write some golang;) 
Well writing in golang plugin would require to rebuild Heka.

The beauty is that you don’t have to do that.

Just a decoder plugin in Lua like it’s already the case for rsyslog…

Cheers,

Patrick



Regards, 
Michal 

On 15 January 2016 at 06:42, Eric LEMOINE  wrote: 
> On Fri, Jan 15, 2016 at 11:57 AM, Michal Rostecki 
>  wrote: 
>> On 01/15/2016 11:14 AM, Simon Pasquier wrote: 
>>> 
>>> My 2 cents on RabbitMQ logging... 
>>> 
>>> On Fri, Jan 15, 2016 at 8:39 AM, Michal Rostecki >> > wrote: 
>>> 
>>> I'd suggest to check the similar options in RabbitMQ and other 
>>> non-OpenStack components. 
>>> 
>>> 
>>> AFAICT RabbitMQ can't log to Syslog anyway. But you have option to make 
>>> RabbitMQ log to stdout [1]. 
>>> BR, 
>>> Simon. 
>>> [1] http://www.superpumpup.com/docker-rabbitmq-stdout 
>>> 
>> 
>> That's OK for Heka/Mesos/k8s approach. 
>> 
>> Just for the curiosity, 
>> @inc0: so we don't receive any logs from RabbitMQ in the current rsyslog 
>> approach? 
> 
> 
> /var/lib/docker/volumes/rsyslog/_data is where logs are stored, and 
> you'll see that there is no file for RabbitMQ. This is related to 
> RabbitMQ not logging to syslog. So our impression is that Kolla 
> doesn't at all collect RabbitMQ logs today. I guess this should be 
> fixed. 
> 
> __ 
> OpenStack Development Mailing List (not for usage questions) 
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 

__ 
OpenStack Development Mailing List (not for usage questions) 
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Eric LEMOINE]

>
> In case of MariaDB I see that you can
>
> - do --skip-syslog, then all things go to "error log"
> - set "error log" path by --log-error
>
> So maybe setting stdout/stderr may work here?
>
> I'd suggest to check the similar options in RabbitMQ and other non-OpenStack
> components.
>
> I may help with this, because in my opinion, logging to stdout is the best
> default option for Mesos and Kubernetes - i.e. Mesos UI shows a "sandbox" of
> application, which generally is stdout/stderr. So if someone will not want
> to use Heka/ELK, then having everything in Mesos/Kubernetes would be
> probably the lesser evil than trying rsyslog to work here.


That makes sense to me Michal. Thanks!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Eric LEMOINE]

On Wed, Jan 13, 2016 at 1:15 PM, Steven Dake (stdake)  wrote:
> Hey folks,
>
> I'd like to have a mailing list discussion about logistics of the ELKSTACK
> solution that Alicja has sorted out vs the Heka implementation that Eric is
> proposing.
>
> My take on that is Eric wants to replace rsyslog and logstash with Heka.


See my other email on this point.  At this point, given the
requirements we have (get logs from services that only speak syslog
and write logs to local files), we cannot guarantee that Heka will
replace Rsyslog.  We are going to test the use of Heka's UdpInput
(with "net" set to "unixgram") et FileOutput plugins for that.  Stay
tuned!


> That seems fine, but I want to make certain this doesn't happen in a way
> that leaves Kolla completely non-functional as we finish up Mitaka.  Liberty
> is the first version of Kolla people will deploy, and Mitaka is the first
> version of Kolla people will upgrade to, so making sure that we don't
> completely bust diagnostics (and I recognize diags as is are a little weak
> is critical).
>
> It sounds like from my reading of the previous thread on this topic, unless
> there is some intractable problem, our goal is to use Heka to replace
> resyslog and logstash.  I'd ask inc0 (who did the rsyslog work) and Alicja
> (who did the elkstack work) to understand that replacement often happens on
> work that has already been done, and its not a "waste of time" so to speak
> as an evolution of the system.
>
> Here are the deadlines:
> http://docs.openstack.org/releases/schedules/mitaka.html
>
> Let me help decode that for folks. March 4th is the final deadline to have a
> completely working solution based upon Heka if its to enter Mitaka.


Understood.


>
> Unlike previous releases of Kolla, I want to hand off release management of
> Kolla to the release management team, and to do that, we need to show a
> track record of hitting our deadlines and not adding features past feature
> freeze (the m3 milestone on March 4th).  In the past releases of Kolla we as
> a team were super loose on this requirement – going forward I prefer us
> being super strict.  Handing off to release management is a sign of maturity
> and would have an overall positive impact, assuming we can get the software
> written in time :)
>
> Eric,
>
> I'd like a plan and commitment to either hit Mitaka 3, or the N cycle.  It
> must work well first on Ansible, and second on Mesos.  If it doesn't work at
> all on Mesos, I could live with that -  I think the Mesos implementation
> will really not be ready for prime time until the middle or completion of
> the N cycle.  We lead with Ansible, and I don't see that changing any time
> soon – as a result, I want our Ansible deployment to be rock solid and
> usable out of the gate.  I don't expect to "Market" Mitaka Mesos (with the
> OpenStack foundation's help) as "production ready" but rather as "tech
> preview" and something for folks to evaluate.


It is our intent to meet the March 4th deadline.



>
> Alicja,
>
> I think a parallel development effort with the ELKSTACK that your working on
> makes sense.  In case the Heka development fails entirely, or misses Mitaka
> 3, I don't want us left lacking a diagnostics solution for Mitaka.
> Diagnostics is my priority #2 for Kolla (#1 is upgrades).  Unfortunately
> what this means is you may end up wasting your time doing development that
> is replaced at the last minute in Mitaka 3, or later in the N cycle.  This
> is very common in software development (all the code I wrote for Magnum has
> been sadly replaced).  I know you can be a good team player here and take
> one for the team so to speak, but I'm asking you if you would take offense
> to this approach.


I'd like to moderate this a bit.  We want to build on Alicja's work,
and we will reuse everything that Alicja has done/will do on
Elasticsearch and Kibana, as this part of the stack will be the same.



>
> I'd like comments/questions/concerns on the above logistics approach
> discussed, and a commitment from Eric as to when he thinks all the code
> would land as one patch stream unit.
>
> I'd also like to see the code come in as one super big patch stream (think
> 30 patches in the stream) so the work can be evaluated and merged as one
> unit.  I could also live with 2-3 different patch streams with 10-15 patches
> per stream, just so we can eval as a unit.  This means lots of rebasing on
> your part Eric ;-)  It also means a commitment from the core reviewer team
> to test and review this critical change.  If there isn't a core reviewer on
> board with this approach, please speak up now.


Makes total sense to me.


Thanks.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Michal Rostecki]

On 01/14/2016 03:56 PM, Michał Jastrzębski wrote:

On 14 January 2016 at 04:46, Eric LEMOINE  wrote:

On Wed, Jan 13, 2016 at 1:15 PM, Steven Dake (stdake)  wrote:

Hey folks,

I'd like to have a mailing list discussion about logistics of the ELKSTACK
solution that Alicja has sorted out vs the Heka implementation that Eric is
proposing.

My take on that is Eric wants to replace rsyslog and logstash with Heka.



See my other email on this point.  At this point, given the
requirements we have (get logs from services that only speak syslog
and write logs to local files), we cannot guarantee that Heka will
replace Rsyslog.  We are going to test the use of Heka's UdpInput
(with "net" set to "unixgram") et FileOutput plugins for that.  Stay
tuned!


Yeah, also please try out different configs of only-rsyslog services.
Maybe Mariadb can be set up in a way compliant with heka?



In case of MariaDB I see that you can

- do --skip-syslog, then all things go to "error log"
- set "error log" path by --log-error

So maybe setting stdout/stderr may work here?

I'd suggest to check the similar options in RabbitMQ and other 
non-OpenStack components.


I may help with this, because in my opinion, logging to stdout is the 
best default option for Mesos and Kubernetes - i.e. Mesos UI shows a 
"sandbox" of application, which generally is stdout/stderr. So if 
someone will not want to use Heka/ELK, then having everything in 
Mesos/Kubernetes would be probably the lesser evil than trying rsyslog 
to work here.


Cheers,
Michal

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Michał Jastrzębski]

On 14 January 2016 at 04:46, Eric LEMOINE  wrote:
> On Wed, Jan 13, 2016 at 1:15 PM, Steven Dake (stdake)  
> wrote:
>> Hey folks,
>>
>> I'd like to have a mailing list discussion about logistics of the ELKSTACK
>> solution that Alicja has sorted out vs the Heka implementation that Eric is
>> proposing.
>>
>> My take on that is Eric wants to replace rsyslog and logstash with Heka.
>
>
> See my other email on this point.  At this point, given the
> requirements we have (get logs from services that only speak syslog
> and write logs to local files), we cannot guarantee that Heka will
> replace Rsyslog.  We are going to test the use of Heka's UdpInput
> (with "net" set to "unixgram") et FileOutput plugins for that.  Stay
> tuned!

Yeah, also please try out different configs of only-rsyslog services.
Maybe Mariadb can be set up in a way compliant with heka?

>> That seems fine, but I want to make certain this doesn't happen in a way
>> that leaves Kolla completely non-functional as we finish up Mitaka.  Liberty
>> is the first version of Kolla people will deploy, and Mitaka is the first
>> version of Kolla people will upgrade to, so making sure that we don't
>> completely bust diagnostics (and I recognize diags as is are a little weak
>> is critical).
>>
>> It sounds like from my reading of the previous thread on this topic, unless
>> there is some intractable problem, our goal is to use Heka to replace
>> resyslog and logstash.  I'd ask inc0 (who did the rsyslog work) and Alicja
>> (who did the elkstack work) to understand that replacement often happens on
>> work that has already been done, and its not a "waste of time" so to speak
>> as an evolution of the system.
>>
>> Here are the deadlines:
>> http://docs.openstack.org/releases/schedules/mitaka.html
>>
>> Let me help decode that for folks. March 4th is the final deadline to have a
>> completely working solution based upon Heka if its to enter Mitaka.
>
>
> Understood.
>
>
>>
>> Unlike previous releases of Kolla, I want to hand off release management of
>> Kolla to the release management team, and to do that, we need to show a
>> track record of hitting our deadlines and not adding features past feature
>> freeze (the m3 milestone on March 4th).  In the past releases of Kolla we as
>> a team were super loose on this requirement – going forward I prefer us
>> being super strict.  Handing off to release management is a sign of maturity
>> and would have an overall positive impact, assuming we can get the software
>> written in time :)
>>
>> Eric,
>>
>> I'd like a plan and commitment to either hit Mitaka 3, or the N cycle.  It
>> must work well first on Ansible, and second on Mesos.  If it doesn't work at
>> all on Mesos, I could live with that -  I think the Mesos implementation
>> will really not be ready for prime time until the middle or completion of
>> the N cycle.  We lead with Ansible, and I don't see that changing any time
>> soon – as a result, I want our Ansible deployment to be rock solid and
>> usable out of the gate.  I don't expect to "Market" Mitaka Mesos (with the
>> OpenStack foundation's help) as "production ready" but rather as "tech
>> preview" and something for folks to evaluate.
>
>
> It is our intent to meet the March 4th deadline.
>
>
>
>>
>> Alicja,
>>
>> I think a parallel development effort with the ELKSTACK that your working on
>> makes sense.  In case the Heka development fails entirely, or misses Mitaka
>> 3, I don't want us left lacking a diagnostics solution for Mitaka.
>> Diagnostics is my priority #2 for Kolla (#1 is upgrades).  Unfortunately
>> what this means is you may end up wasting your time doing development that
>> is replaced at the last minute in Mitaka 3, or later in the N cycle.  This
>> is very common in software development (all the code I wrote for Magnum has
>> been sadly replaced).  I know you can be a good team player here and take
>> one for the team so to speak, but I'm asking you if you would take offense
>> to this approach.
>
>
> I'd like to moderate this a bit.  We want to build on Alicja's work,
> and we will reuse everything that Alicja has done/will do on
> Elasticsearch and Kibana, as this part of the stack will be the same.
>
>
>
>>
>> I'd like comments/questions/concerns on the above logistics approach
>> discussed, and a commitment from Eric as to when he thinks all the code
>> would land as one patch stream unit.
>>
>> I'd also like to see the code come in as one super big patch stream (think
>> 30 patches in the stream) so the work can be evaluated and merged as one
>> unit.  I could also live with 2-3 different patch streams with 10-15 patches
>> per stream, just so we can eval as a unit.  This means lots of rebasing on
>> your part Eric ;-)  It also means a commitment from the core reviewer team
>> to test and review this critical change.  If there isn't a core reviewer on
>> board with this approach, please speak up now.

I'm on board:)

>
> Makes 

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[David Moreau Simard]

So is it decided that we want Heka instead of ELK in Kolla and that it is
just a matter of time, then ?

Clark Boyle put forward some very good points [1] which seem to have gone
sadly mostly ignored.

What are we trying to address by replacing ELK ? Performance ? Clark's
numbers are far from being bad and ELK effectively scales in any direction
you want.

I'll put on my operator hat and would like to give my +1 to keep ELK
instead of Heka.
ELK at this point is all but a golden standard. People know it, people use
it, people troubleshoot it. If something goes wrong, I can go on Google, on
IRC or mailing lists and expect someone to be able to help.
This is worth a lot to operators. OpenStack is already expensive enough,
even if you don't take the vendor route.

Python is slow but you don't see OpenStack being rewritten in Go (ok,
Swift, you're an exception). Python just has that massive community of
developers that OpenStack can tap into. This is worth a lot and in that
respect, I am happy that OpenStack is in Python, even if it is slow.

I'm not saying Heka is a bad decision or that it's an eccentric/exotic
choice. But please let the decision be mindful of the people that will be
deploying, configuring and supporting this. I don't believe a performance
increase is worth it unless ELK was a real and painful bottleneck, which it
is not.

My 0.02$CAD (definitely not worth a lot right now)

[1]:
http://lists.openstack.org/pipermail/openstack-dev/2016-January/083916.html

David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]
On Jan 13, 2016 7:20 AM, "Steven Dake (stdake)"  wrote:

> Hey folks,
>
> I'd like to have a mailing list discussion about logistics of the ELKSTACK
> solution that Alicja has sorted out vs the Heka implementation that Eric is
> proposing.
>
> My take on that is Eric wants to replace rsyslog and logstash with Heka.
> That seems fine, but I want to make certain this doesn't happen in a way
> that leaves Kolla completely non-functional as we finish up Mitaka.
> Liberty is the first version of Kolla people will deploy, and Mitaka is the
> first version of Kolla people will upgrade to, so making sure that we don't
> completely bust diagnostics (and I recognize diags as is are a little weak
> is critical).
>
> It sounds like from my reading of the previous thread on this topic,
> unless there is some intractable problem, our goal is to use Heka to
> replace resyslog and logstash.  I'd ask inc0 (who did the rsyslog work) and
> Alicja (who did the elkstack work) to understand that replacement often
> happens on work that has already been done, and its not a "waste of time"
> so to speak as an evolution of the system.
>
> Here are the deadlines:
> http://docs.openstack.org/releases/schedules/mitaka.html
>
> Let me help decode that for folks. March 4th is the final deadline to have
> a completely working solution based upon Heka if its to enter Mitaka.
>
> Unlike previous releases of Kolla, I want to hand off release management
> of Kolla to the release management team, and to do that, we need to show a
> track record of hitting our deadlines and not adding features past feature
> freeze (the m3 milestone on March 4th).  In the past releases of Kolla we
> as a team were super loose on this requirement – going forward I prefer us
> being super strict.  Handing off to release management is a sign of
> maturity and would have an overall positive impact, assuming we can get the
> software written in time :)
>
> Eric,
>
> I'd like a plan and commitment to either hit Mitaka 3, or the N cycle.  It
> must work well first on Ansible, and second on Mesos.  If it doesn't work
> at all on Mesos, I could live with that -  I think the Mesos implementation
> will really not be ready for prime time until the middle or completion of
> the N cycle.  We lead with Ansible, and I don't see that changing any time
> soon – as a result, I want our Ansible deployment to be rock solid and
> usable out of the gate.  I don't expect to "Market" Mitaka Mesos (with the
> OpenStack foundation's help) as "production ready" but rather as "tech
> preview" and something for folks to evaluate.
>
> Alicja,
>
> I think a parallel development effort with the ELKSTACK that your working
> on makes sense.  In case the Heka development fails entirely, or misses
> Mitaka 3, I don't want us left lacking a diagnostics solution for Mitaka.
> Diagnostics is my priority #2 for Kolla (#1 is upgrades).  Unfortunately
> what this means is you may end up wasting your time doing development that
> is replaced at the last minute in Mitaka 3, or later in the N cycle.  This
> is very common in software development (all the code I wrote for Magnum has
> been sadly replaced).  I know you can be a good team player here and take
> one for the team so to speak, but I'm asking you if you would take offense
> to this approach.
>
> I'd like comments/questions/concerns on the above logistics approach
> 

Re: [openstack-dev] [kolla] Heka v ELK stack logistics

[Steven Dake (stdake)]

From: David Moreau Simard <d...@redhat.com<mailto:d...@redhat.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Wednesday, January 13, 2016 at 5:55 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [kolla] Heka v ELK stack logistics


So is it decided that we want Heka instead of ELK in Kolla and that it is just 
a matter of time, then ?

Before "deciding" a vote is necessary of the core reviewers.  I just want to 
get the ball rolling because it seems like the core reviewers are leaning 
towards a Heka based solution and if we do use Heka, I'd like it done in Mitaka 
if at all possible so our operators don't have to learn a new diagnostics 
system.

My beef against rsyslog is it has a bunch of gaps that we will never be able to 
fix.  I don't recall the exact details, but inc0 or SamYaple could probably 
point them out in more detail.  In my opinion this isn't about replacing 
logstash, this is about replacing rsyslog.  This also isn't about performance 
or scalability – I think logstash can do the performance job needed for an 
OpenStack diagnostics system.

It may be that Heka turns out to be a waste of time – the only way to know for 
certain is to see a proof of concept implementation – and if its better then 
logstash, merge it.

I also have a beef against Java in general, because of the JVM fork between 
open source and Oracle.  To me this makes dependencies that use Java less 
viable – even though I recognize Elasticsearch is implemented in Java.

I recognize all your points about the well known mature nature of logstash.  To 
me this is a huge advantage.  Choosing dependencies wisely is one of the most 
important decisions made in software design and maturity goes a long way in the 
decision making process about dependency acceptance.  I don't know much about 
Heka's maturity or even suitability for our problems – I am just basing this 
discussion on the direction I see happening on the mailing list.

It would be nice if we could have both and select one or the other, but we are 
not going to be implementing in that manner as we need to choose one or the 
other.

A POC that fixes all the gaps in rsyslog will be necessary to obtain my +2 on 
the patch stream reviews.

Regards
-steve



Clark Boyle put forward some very good points [1] which seem to have gone sadly 
mostly ignored.

What are we trying to address by replacing ELK ? Performance ? Clark's numbers 
are far from being bad and ELK effectively scales in any direction you want.

I'll put on my operator hat and would like to give my +1 to keep ELK instead of 
Heka.
ELK at this point is all but a golden standard. People know it, people use it, 
people troubleshoot it. If something goes wrong, I can go on Google, on IRC or 
mailing lists and expect someone to be able to help.
This is worth a lot to operators. OpenStack is already expensive enough, even 
if you don't take the vendor route.

Python is slow but you don't see OpenStack being rewritten in Go (ok, Swift, 
you're an exception). Python just has that massive community of developers that 
OpenStack can tap into. This is worth a lot and in that respect, I am happy 
that OpenStack is in Python, even if it is slow.

I'm not saying Heka is a bad decision or that it's an eccentric/exotic choice. 
But please let the decision be mindful of the people that will be deploying, 
configuring and supporting this. I don't believe a performance increase is 
worth it unless ELK was a real and painful bottleneck, which it is not.

My 0.02$CAD (definitely not worth a lot right now)

[1]: http://lists.openstack.org/pipermail/openstack-dev/2016-January/083916.html

David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]

On Jan 13, 2016 7:20 AM, "Steven Dake (stdake)" 
<std...@cisco.com<mailto:std...@cisco.com>> wrote:
Hey folks,

I'd like to have a mailing list discussion about logistics of the ELKSTACK 
solution that Alicja has sorted out vs the Heka implementation that Eric is 
proposing.

My take on that is Eric wants to replace rsyslog and logstash with Heka.  That 
seems fine, but I want to make certain this doesn't happen in a way that leaves 
Kolla completely non-functional as we finish up Mitaka.  Liberty is the first 
version of Kolla people will deploy, and Mitaka is the first version of Kolla 
people will upgrade to, so making sure that we don't completely bust 
diagnostics (and I recognize diags as is are a little weak is critical).

It sounds like from my reading of the previous thread on this topic, unless 
there is some intractable problem, our goal is to use Heka to replace resyslog 
and logstash.  I'd ask inc0 (who did