Hej to y'all:
I am looking for some guidance regarding elasticsearch together with logstash. I am new to all of these and I want to find if the setup we want is possible. We are setting up a couple of nodes in AWS EC2 which are located in different availability zones within AWS. I have resolved the issue of setting up a Cluster inside AWS without the plugin just using security groups and iptables. But the setup that we want requires the cluster mainly for searching. A bigger issue for us is about cost since AWS charges for the data transfering between availability zones, and that, we want to avoid. So back to what our scenario looks like, we would like to have the following: We want to have two separate servers that have all their indexes and shards located locally (meaning that when indexing information via logstash we need that all new and existing shards are not spread across the two nodes in the cluster). Node A will receive log information from availability zone 1 and needs to keep that info only in that node. And Node B will receive log information from availability zone 2 with the same behavior. But (and here is where I have an issue with) we would like to be able to go into one server (lets say Node A) and be able to query information from both Node A and Node B. Our setup is not critical and that is why we can skip the nice clustering and distributed functionality of elasticsearch. I have tried (after a lot of reading and "googling") with the parameters cluster.routing.allocation.awareness.attributes, cluster.routing.allocation.awareness.force.zone.values, node.zone, index.routing.allocation.total_shards_per_node, index.routing.allocation.require.zone and index.routing.allocation.exclude.zone. I have set these in the elasticsearch.yml file starting the elasticsearch cluster and I also put these values in elasticsearch.yml file for logstash. But with every try (I delete the data directory everytime to test) when I index some test logs from Node B some shards are written in Node A anyway. Do you think this setup is at all possible??? or maybe the elasticsearch distributed behavior cannot be changed like this and that is ok as long as I know then I can move on with another setup since, as I said, is nothing critical but I cannot take much more time investigating this (already been at it for a couple of weeks). Thank you to anyone for their time, attention and help, best regards, s.r./Fernando P.S.: I am using logstash 1.3.3 and elasticsearch 0.90.9 with Kibana 3 Milestone 4. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a770c8dd-6498-4722-9e6e-4fd034523434%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
