Re: [Openstack-operators] ElasticSearch on OpenStack

[Jeremy Stanley]

On 2016-09-06 17:49:08 + (+), Randall, Nathan X wrote:
> For the storage backing Elasticsearch data nodes, we have been
> using one 500GB Cinder volume (backed by a Ceph cluster built from
> DL380s filled with 1.2TB 10k SAS drives) per data node. However,
> we've found that a VM with 8 vCPU and 64GB RAM can make use of
> more than 500GB disk capacity without bottlenecking on CPU or
> memory, so we are experimenting with 1TB or 1.5TB options per data
> node.
[...]

As a point of comparison, the 6 ES cluster members OpenStack Infra
is using are a 60GiB RAM/16 vCPU flavor in Rackspace's DFW region
and each of them has a 1TiB Cinder SATA volume formatted ext4 (~50%
full). You can see system utilization metrics for one of those at
http://cacti.openstack.org/cacti/graph_view.php?action=tree_id=1_id=123
(though we're apparently missing a graph for the
/var/lib/elasticsearch filesystem).
-- 
Jeremy Stanley

_______
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] ElasticSearch on OpenStack

[Randall, Nathan X]

Tim,

For the storage backing Elasticsearch data nodes, we have been using one 500GB 
Cinder volume (backed by a Ceph cluster built from DL380s filled with 1.2TB 10k 
SAS drives) per data node. However, we've found that a VM with 8 vCPU and 64GB 
RAM can make use of more than 500GB disk capacity without bottlenecking on CPU 
or memory, so we are experimenting with 1TB or 1.5TB options per data node. We 
are also moving to a different tier of storage that uses an array of SSDs 
instead of spinning rust, though this change has very little to do with 
performance and very much to do with the automatic deduplication, compression, 
and encryption offered by the hardware backend (Solidfire) for that storage 
tier.  # <-- Not a vendor promo; just letting you know what we're using for 
that tier.

We get a lot of duplicated messages in Elasticsearch since we're using if for 
log monitoring, and JSON documents compress very well... so it actually costs 
us significantly less to leverage a storage hardware platform that provides 
native deduplication and compression. Having SSDs in the mix probably helps 
reduce latency a bit (due to lower seek times), but honestly we didn't have 
enough of a latency problem to justify moving away from volumes backed by Ceph.

Guidance from Elastic is always going to advocate using local SSDs when 
possible, but I'm pretty sure that's not what Elastic uses for their own cloud 
offering...

Thanks,
Nathan Randall

From: Tim Bell <tim.b...@cern.ch<mailto:tim.b...@cern.ch>>
Date: Saturday, September 3, 2016 at 1:12 AM
To: David Medberry <openst...@medberry.net<mailto:openst...@medberry.net>>
Cc: openstack-operators 
<openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>>
Subject: Re: [Openstack-operators] ElasticSearch on OpenStack

Thanks. How's the storage handled ?

We're seeing slow I/O on local storage (which is also limited on space) and 
latencies with Ceph for block storage.

Tim

From: <medbe...@gmail.com<mailto:medbe...@gmail.com>> on behalf of David 
Medberry <openst...@medberry.net<mailto:openst...@medberry.net>>
Date: Friday 2 September 2016 at 22:18
To: Tim Bell <tim.b...@cern.ch<mailto:tim.b...@cern.ch>>
Cc: openstack-operators 
<openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>>
Subject: Re: [Openstack-operators] ElasticSearch on OpenStack

Nathan: The page at 
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html 
gives you good advice on a maximum size for the elasticsearch VM's memory.

Nathan: suggest you pick a flavor with 64GB RAM or less, then base other sizing 
things off of that (i.e. choose a flavor with 64GB of RAM and as many CPUs as 
possible for that RAM allocation, then base disk size on testing of your use 
case)

Nathan: give java heap 30GB, and leave the rest of the memory to the OS 
filesystem cache so that Lucene can make best use of it.

Nathan: that's mostly it for tuning. elasticsearch publishes many other docs 
for tuning recommendations, but there isn't anything specific to openstack 
besides the flavor choice. i personally chose CPU size (8CPU) such that all 
vCPUs for each VM would fit on a single NUMA node, which is a best practice for 
ESXi but not sure if it applies to KVM.

(resending for clarity)

On Fri, Sep 2, 2016 at 6:46 AM, David Medberry 
<openst...@medberry.net<mailto:openst...@medberry.net>> wrote:
Hey Tim,
We've just started this effort. I'll see if the guy running the service can 
comment today.

On Fri, Sep 2, 2016 at 6:36 AM, Tim Bell 
<tim.b...@cern.ch<mailto:tim.b...@cern.ch>> wrote:

Has anyone had experience running ElasticSearch on top of OpenStack VMs ?

Are there any tuning recommendations ?

Thanks
Tim

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] ElasticSearch on OpenStack

[Tim Bell]

Thanks. How’s the storage handled ?

We’re seeing slow I/O on local storage (which is also limited on space) and 
latencies with Ceph for block storage.

Tim

From: <medbe...@gmail.com> on behalf of David Medberry <openst...@medberry.net>
Date: Friday 2 September 2016 at 22:18
To: Tim Bell <tim.b...@cern.ch>
Cc: openstack-operators <openstack-operators@lists.openstack.org>
Subject: Re: [Openstack-operators] ElasticSearch on OpenStack

Nathan: The page at 
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html 
gives you good advice on a maximum size for the elasticsearch VM's memory.

Nathan: suggest you pick a flavor with 64GB RAM or less, then base other sizing 
things off of that (i.e. choose a flavor with 64GB of RAM and as many CPUs as 
possible for that RAM allocation, then base disk size on testing of your use 
case)

Nathan: give java heap 30GB, and leave the rest of the memory to the OS 
filesystem cache so that Lucene can make best use of it.

Nathan: that's mostly it for tuning. elasticsearch publishes many other docs 
for tuning recommendations, but there isn't anything specific to openstack 
besides the flavor choice. i personally chose CPU size (8CPU) such that all 
vCPUs for each VM would fit on a single NUMA node, which is a best practice for 
ESXi but not sure if it applies to KVM.

(resending for clarity)

On Fri, Sep 2, 2016 at 6:46 AM, David Medberry 
<openst...@medberry.net<mailto:openst...@medberry.net>> wrote:
Hey Tim,
We've just started this effort. I'll see if the guy running the service can 
comment today.

On Fri, Sep 2, 2016 at 6:36 AM, Tim Bell 
<tim.b...@cern.ch<mailto:tim.b...@cern.ch>> wrote:

Has anyone had experience running ElasticSearch on top of OpenStack VMs ?

Are there any tuning recommendations ?

Thanks
Tim

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] ElasticSearch on OpenStack

[Jeremy Stanley]

On 2016-09-02 12:36:27 + (+), Tim Bell wrote:
> Has anyone had experience running ElasticSearch on top of OpenStack VMs ?

The OpenStack community infrastructure includes a modest
ElasticSearch cluster indexing CI job results, behind the
http://logstash.openstack.org/ and
http://status.openstack.org/elastic-recheck/ services. We also use
the CirrusSearch extension for Mediawiki (which relies on a local
ElasticSearch instance) providing page searching for the
https://wiki.openstack.org/ service. All of these run on virtual
machine instances in Rackspace at the moment.

> Are there any tuning recommendations ?

Our use case isn't nearly as large as some, but you may be able to
find some ideas in our Puppet module for ES and the customization
class from which we instantiate it:

https://git.openstack.org/cgit/openstack-infra/puppet-elasticsearch/tree/

https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/elasticsearch_node.pp

-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] ElasticSearch on OpenStack

[David Medberry]

omitted on more line:

give java heap 30GB, and leave the rest of the memory to the OS filesystem
cache so that Lucene can make best use of it.

On Fri, Sep 2, 2016 at 2:15 PM, David Medberry 
wrote:

> From Nathan (on TWC Cloud team):
>
> Nathan: The page at https://www.elastic.co/guide/en/elasticsearch/guide/
> current/heap-sizing.html gives you good advice on a maximum size for the
> elasticsearch VM's memory.
>
> Nathan: suggest you pick a flavor with 64GB RAM or less, then base other
> sizing things off of that (i.e. choose a flavor with 64GB of RAM and as
> many CPUs as possible for that RAM allocation, then base disk size on
> testing of your use case)
>
> On Fri, Sep 2, 2016 at 6:46 AM, David Medberry 
> wrote:
>
>> Hey Tim,
>> We've just started this effort. I'll see if the guy running the service
>> can comment today.
>>
>> On Fri, Sep 2, 2016 at 6:36 AM, Tim Bell  wrote:
>>
>>>
>>>
>>> Has anyone had experience running ElasticSearch on top of OpenStack VMs ?
>>>
>>>
>>>
>>> Are there any tuning recommendations ?
>>>
>>>
>>>
>>> Thanks
>>>
>>> Tim
>>>
>>> ___
>>> OpenStack-operators mailing list
>>> OpenStack-operators@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>>
>>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] ElasticSearch on OpenStack

[David Medberry]

>From Nathan (on TWC Cloud team):

Nathan: The page at
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
gives you good advice on a maximum size for the elasticsearch VM's memory.

Nathan: suggest you pick a flavor with 64GB RAM or less, then base other
sizing things off of that (i.e. choose a flavor with 64GB of RAM and as
many CPUs as possible for that RAM allocation, then base disk size on
testing of your use case)

On Fri, Sep 2, 2016 at 6:46 AM, David Medberry 
wrote:

> Hey Tim,
> We've just started this effort. I'll see if the guy running the service
> can comment today.
>
> On Fri, Sep 2, 2016 at 6:36 AM, Tim Bell  wrote:
>
>>
>>
>> Has anyone had experience running ElasticSearch on top of OpenStack VMs ?
>>
>>
>>
>> Are there any tuning recommendations ?
>>
>>
>>
>> Thanks
>>
>> Tim
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] ElasticSearch on OpenStack

[Van Leeuwen, Robert]

Hi,

I had some “interesting” issues in the past with sparse files on xfs with 
elasticsearch:
http://engineering.spilgames.com/hypervisor-kernel-panics-hit-2014-sl6-5/

If you pre-allocate your files you should be good.

Cheers,
Robert van Leeuwen

From: Tim Bell <tim.b...@cern.ch>
Date: Friday, September 2, 2016 at 2:36 PM
To: openstack-operators <openstack-operators@lists.openstack.org>
Subject: [Openstack-operators] ElasticSearch on OpenStack


Has anyone had experience running ElasticSearch on top of OpenStack VMs ?

Are there any tuning recommendations ?

Thanks
Tim
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] ElasticSearch on OpenStack

[Matt Van Winkle]

We are running entire ELK clusters on our undercloud.  Let me see if I can 
point our resident metrics expert at this as well.

VW

Sent from my iPhone

On Sep 2, 2016, at 7:50 AM, David Medberry 
> wrote:

Hey Tim,
We've just started this effort. I'll see if the guy running the service can 
comment today.

On Fri, Sep 2, 2016 at 6:36 AM, Tim Bell 
> wrote:

Has anyone had experience running ElasticSearch on top of OpenStack VMs ?

Are there any tuning recommendations ?

Thanks
Tim

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] ElasticSearch on OpenStack

[Tim Bell]

Has anyone had experience running ElasticSearch on top of OpenStack VMs ?

Are there any tuning recommendations ?

Thanks
Tim
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators