Hi all,
i would like to start this conversation to discuss about the best
architecture of ELK based on our hardware and needed for a test envirorment.
What we have:
- 4+ ES nodes
- x2 with 24 gb of rams and 800 gb of HD SAS 4 x2 CPU
- x2 with 16 gb of rams and 500 gb of HD SAS 4 x2 CPU
- 10+ LS Collectors
- 2+ Kibana instances
we have 2 separate Datacentre infact, as i show, we have the specular
resources on the above list, so for example we have 2 ES nodes on the first
location and the other 2 in the second location that are linked with double
redundant fiber 10 gbit .
Our test is to understand how ELK stack performing with indexing of all
Application and Server Events, so we are talking about 200 Events for
seconds in the test lab. We would like to have a retention of 2 or 3
mounth, so seraching with kibana that logs, and then close and backup old
index that we test is a working well with curator plugin.
*What is the best configuration for Load balancing events across the two
locatio*n i mean every collectors should have 2 available choice for the
output in case of one node go down or is performing bad , what do you
suggests ?
we try Nginx with health check but i think that ES should do something
similar for load balancing indexing process with a node master false, data
false , even if we raed in the community that this type of node is reserved
for balancing search and not indexing that go every time across the master
of the cluster, am i right?
*What is the best configuration that you test ?* i mean how many shards how
many replicas for a full High availability and redundant solution ?
we try to play with 2 shard and one replica for 4 data node, because as we
see replcas are involved in search process so it can be a good solution to
reserve some nodes only for replicas but what we miss is if a node go down
or a datacentre died can we have all data automatically on the other side
(just with replicas) ? ( we know that for the golden rule we need to have 5
nodes and 3 minimum master node for a cluster so if we have only 2 DC could
be critical because one DC need to have more nodes and become the leader of
the all cluster... )
*What is your best configuration for a security prospective ? *
we test nginx also as reverse proxy with standard autentcation to prevent
unwanted DELETE and PUT but we are looking for a more strong solution with
more flexibility and roles/premissions configuration like a standard SQL
DB. Our network layer is really strong every ELK layer has his own DMZ, ACL
and firewall rule
iam worried about espacially on the ES configuration like shards replica
and load balancing i think that this conversation should be helpfull for a
very large community auditor that have some doubts about ES and ELK stack
in general.
Best Regards,
Stefano
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d3ca4aaf-a1ee-4732-9ae5-629dd8198e7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
