I have 2 ES nodes configured with 2 shards, 0 replica. I'm testing how fast logstash can push logs from a dummy "access_log" file to this clusters.
>From my test, with m3.xlarge on EC2, I can push around 4500 logs/sec. But I noticed that my 2 shards were on the same node. I still don't get how ES black magic works, why does it not split the shards ? Will that allows me to push 9000/sec ? I can't find if logstash or ES is the bottleneck there. /usr/bin/java -Xms7g -Xmx7g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-0.90.10.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.ElasticSearch /usr/bin/java -Xmx1G -Xms1G -cp /usr/local/bin/logstash/logstash.jar:/usr/local/bin/logstash/cloud-aws/* logstash.runner agent --config /etc/logstash/mylogstash.conf --log /var/log/logstash/logstash.log -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0abf0a40-94c4-4bae-adb4-5ecebc1ebb2a%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
