Re: [Openstack-operators] OpenSource ELK configurations
On 27 May 2015 at 20:58, Joe Gordon joe.gord...@gmail.com wrote: On Wed, May 27, 2015 at 12:27 PM, Joseph Bajin josephba...@gmail.com wrote: I think those from the Infra Project are a great start, but I do think they are missing a lot. I am sure they are, and we would love contributions to make them better. The filters from the infra project are used every day by developers use to hunt down issues. And the more similar the developer and operator experience is here the easier it will be for the two groups to talk about the same problems. As part of development for Rackspace Private Cloud we've done a ton of work in this space and are happy to contribute the work upstream. The current work is exposed in https://github.com/rcbops/rpc-extras/pull/123 and is available for anyone to review/comment/consume. Instead of breaking down the message into parts, it just breaks down the type (INFO, DEBUG, etc) and then makes the rest of the message greedy. That doesn't help searching or graphing or anything like that (or at least makes it more difficult over time). At least that is what I have seen. One of the reasons for this is because most of the log messages are very unstructured. But I have a hunch we can do better then what we have today. We've done quite a bit of this. We extract transaction ID's for each project we deploy (including swift), response times to API queries, byte sizes, etc. We are using a little bit tweaked version of the godaddy's scripts. That seems to give us a good amount of detail that we can search and filter on. I'll see if I can post my versions of them to the repo. We derived some stuff from https://github.com/godaddy/openstack-logstash and other stuff from https://github.com/osops/tools-logging - I think it's time that we find a common place to consolidate and standardise. openstack-infra would appear to be a good place to do this. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] OpenSource ELK configurations
Sounds like a fine thing to point people to…thanks Joe. https://github.com/osops/tools-logging/pull/3 At Your Service, Mark T. Voelker On May 27, 2015, at 1:12 PM, Joe Gordon joe.gord...@gmail.com wrote: On Mon, May 18, 2015 at 3:11 PM, Tom Fifield t...@openstack.org wrote: There's some stuff in the osops repo: https://github.com/osops/tools-logging Please contribute if you can! Regards, Tom On 18/05/15 14:56, Anand Kumar Sankaran wrote: Hi all Is there a set of open source ELK configurations available? (log stash filters, templates, kibana dashboards). I see a github repository from Godaddy, wondering if there is a standard set that is used. OpenStack has an ELK stack running at logstash.openstack.org to help debug test jobs. The logstash filters can be found at: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/logstash/indexer.conf.erb Thanks. — anand ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] OpenSource ELK configurations
On Mon, May 18, 2015 at 3:11 PM, Tom Fifield t...@openstack.org wrote: There's some stuff in the osops repo: https://github.com/osops/tools-logging Please contribute if you can! Regards, Tom On 18/05/15 14:56, Anand Kumar Sankaran wrote: Hi all Is there a set of open source ELK configurations available? (log stash filters, templates, kibana dashboards). I see a github repository from Godaddy, wondering if there is a standard set that is used. OpenStack has an ELK stack running at logstash.openstack.org to help debug test jobs. The logstash filters can be found at: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/logstash/indexer.conf.erb Thanks. — anand ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] OpenSource ELK configurations
I think those from the Infra Project are a great start, but I do think they are missing a lot. Instead of breaking down the message into parts, it just breaks down the type (INFO, DEBUG, etc) and then makes the rest of the message greedy. That doesn't help searching or graphing or anything like that (or at least makes it more difficult over time). At least that is what I have seen. We are using a little bit tweaked version of the godaddy's scripts. That seems to give us a good amount of detail that we can search and filter on. I'll see if I can post my versions of them to the repo. On Wed, May 27, 2015 at 2:00 PM, Mark Voelker mvoel...@vmware.com wrote: Sounds like a fine thing to point people to…thanks Joe. https://github.com/osops/tools-logging/pull/3 At Your Service, Mark T. Voelker On May 27, 2015, at 1:12 PM, Joe Gordon joe.gord...@gmail.com wrote: On Mon, May 18, 2015 at 3:11 PM, Tom Fifield t...@openstack.org wrote: There's some stuff in the osops repo: https://github.com/osops/tools-logging Please contribute if you can! Regards, Tom On 18/05/15 14:56, Anand Kumar Sankaran wrote: Hi all Is there a set of open source ELK configurations available? (log stash filters, templates, kibana dashboards). I see a github repository from Godaddy, wondering if there is a standard set that is used. OpenStack has an ELK stack running at logstash.openstack.org to help debug test jobs. The logstash filters can be found at: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/logstash/indexer.conf.erb Thanks. — anand ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] OpenSource ELK configurations
On Wed, May 27, 2015 at 12:27 PM, Joseph Bajin josephba...@gmail.com wrote: I think those from the Infra Project are a great start, but I do think they are missing a lot. I am sure they are, and we would love contributions to make them better. The filters from the infra project are used every day by developers use to hunt down issues. And the more similar the developer and operator experience is here the easier it will be for the two groups to talk about the same problems. Instead of breaking down the message into parts, it just breaks down the type (INFO, DEBUG, etc) and then makes the rest of the message greedy. That doesn't help searching or graphing or anything like that (or at least makes it more difficult over time). At least that is what I have seen. One of the reasons for this is because most of the log messages are very unstructured. But I have a hunch we can do better then what we have today. We are using a little bit tweaked version of the godaddy's scripts. That seems to give us a good amount of detail that we can search and filter on. I'll see if I can post my versions of them to the repo. Are you talking about https://github.com/osops/tools-logging https://github.com/osops/tools-logging/pull/3 or the infra repo? It would be great to have both developer and operator communities collaborate on this and improve everyones ELK configuration. On Wed, May 27, 2015 at 2:00 PM, Mark Voelker mvoel...@vmware.com wrote: Sounds like a fine thing to point people to…thanks Joe. https://github.com/osops/tools-logging/pull/3 At Your Service, Mark T. Voelker On May 27, 2015, at 1:12 PM, Joe Gordon joe.gord...@gmail.com wrote: On Mon, May 18, 2015 at 3:11 PM, Tom Fifield t...@openstack.org wrote: There's some stuff in the osops repo: https://github.com/osops/tools-logging Please contribute if you can! Regards, Tom On 18/05/15 14:56, Anand Kumar Sankaran wrote: Hi all Is there a set of open source ELK configurations available? (log stash filters, templates, kibana dashboards). I see a github repository from Godaddy, wondering if there is a standard set that is used. OpenStack has an ELK stack running at logstash.openstack.org to help debug test jobs. The logstash filters can be found at: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/logstash/indexer.conf.erb Thanks. — anand ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
