Re: [Wikitech-l] [Ops] Help evaluate upgrade path for logstash.wikimedia.org

[Brad Jorsch (Anomie)]

On Fri, Jun 24, 2016 at 12:01 PM, Bryan Davis  wrote:

> * Date picker works in local browser TZ instead of UTC [2]. Honestly I
> think this bug or a variant of it exists in our Kibana3 install as
> well.
>

It does, I always have to convert UTC to local time when I'm trying to get
Kibana to tell me the log events around some error; then Kibana displays
the results in UTC again in the timeline and so on. :/


-- 
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Ops] Help evaluate upgrade path for logstash.wikimedia.org

[Bryan Davis]

On Thu, Jun 23, 2016 at 7:47 PM, Erik Bernhardson
 wrote:
> That url should have been: https://kibana4.wmflabs.org/
>
> On Thu, Jun 23, 2016 at 6:26 PM, Erik Bernhardson
>  wrote:
>>
>> The elasticsearch servers that run logstash.wikimedia.org will soon be
>> upgraded from 1.7 to 2.3. As part of this upgrade we also need to upgrade
>> kibana (the web interface that runs on logstash.wikimedia.org). The current
>> version uses elasticsearch features that have been long deprecated (since
>> 1.4) and as of 2.0 have been removed. I've setup a test environment in the
>> beta cluster running this new version of elasticsearch along with the new
>> version of kibana.
>>
>> Be warned, kibana 4 is a complete rewrite. It needs to be evaluated for
>> our purposes and we need to make sure it does what we need. I've recreated
>> most of the dashboards that were linked from the homepage of
>> logstash-beta.wmflabs.org to help with the evaluation. The first few took a
>> short while, but due to the way kibana re-uses visualizations it became a
>> good bit faster/easier after that.
>>
>> The main feature i've noticed missing from our old dashboards is the
>> trends panel. This is the part that told you how the volume of messages
>> compared to the previous hour/day/week/etc. There is a long standing github
>> issue about replacing this functionality but it has been stalled out. I
>> don't personally use that part of the visualization much, but perhaps you
>> do?
>>
>> Please try it out, and give some feedback. https://kibana.wmflabs.org/

Thanks for working on this Erik!

Kibana4 is a complete rewrite from Kibana3 (and I think the 5th or 6th
ground up rewrite of Kibana in its history). Elastic.co seems to have
high hopes for this iteration providing a platform for better
visualizations and performance. As a rewrite there are some feature
regressions that haven't been worked out upstream yet. We (well I)
didn't upgrade us to Kibana4 for over a year due to a lack of support
for pinning the dates shown in the UI to a particular timezone [0].
Thankfully that regression has been fixed in the version that Erik is
demoing in the beta cluster. I haven't found any other regressions
that are worthy of blocking yet, and that's good because we really do
have to upgrade in order to have a sane state of services in
production.

There are a few issues that I have noticed that others may find as well:

* No support for "markers" on a time histogram [1]. These were used in
Kibana3 to show when scap changed files on the servers. The markers
weren't highly reliable so its not the end of the world that this is
missing for now.

* Date picker works in local browser TZ instead of UTC [2]. Honestly I
think this bug or a variant of it exists in our Kibana3 install as
well. Working with dates is a pain in just about every language. :/

There are some neat new features too, like permalinks to individual
log events and quite a few new visualization types [3].

[0]: https://github.com/elastic/kibana/issues/1600
[1]: https://github.com/elastic/kibana/issues/2706
[2]: https://github.com/elastic/kibana/issues/5370
[3]: https://www.elastic.co/guide/en/kibana/4.0/whats-new.html

Bryan
-- 
Bryan Davis  Wikimedia Foundation
[[m:User:BDavis_(WMF)]]  Sr Software EngineerBoise, ID USA
irc: bd808v:415.839.6885 x6855

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l