I have used Tsung to load test my clusters. It's very easy to install and configure. It will give more insight about memory usage, i/o, network i/o etc.
I suggest that you read this blog entry <https://engineering.helpshift.com/2014/tsung/> about it. You'll find here <http://tsung.erlang-projects.org/> the official page for Tsung. Le mardi 2 décembre 2014 12:38:37 UTC+1, Siddharth Trikha a écrit : > > I had tested my ELK setup on a small set of logs and got it working. > > Today I started it's testing on actual log files of a server. So one > server's one month data with around 10 type of logs. > So approximately 500 files to be monitored each approx 1k lines, comes > upto 5 lac log lines. > > **: I am using default setting for elasticsearch. > > So when I started logstash ES, first I saw logstash kept running for 2 > hours just reading logs and after that it gave following error on console: > > > > > *log4j, [2014-12-02T16:03:19.087] DEBUG: > org.elasticsearch.transport.netty: [logstash-XX-] disconnected from > [[#zen_unicast_2#][XX[inet[localhost/127.0.0.1:9301]]]log4j, > [2014-12-02T16:03:19.087] DEBUG: org.elasticsearch.discovery.zen: > [logstashXX] filtered ping responses: (filter_client[true], > filter_data[false]) {none}log4j, [2014-12-02T16:03:19.089] DEBUG: > org.elasticsearch.transport.netty: [logstashXX] connected to node > [[#zen_unicast_2#][inet[localhost/127.0.0.1:9301]]]* > ES console: > > > > > > > > > > *[2014-12-02 12:23:24,289][INFO ][cluster.metadata ] [Mantis] > [logstash-2014.12.02] creating index, cause [auto(bulk api)], shards > [5]/[1], mappings [_default_][2014-12-02 12:23:24,972][INFO > ][cluster.metadata ] [Mantis] [logstash-2014.12.02] update_mapping > [logs] (dynamic)[2014-12-02 12:23:39,675][INFO ][cluster.metadata ] > [Mantis] [logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02 > 12:51:13,975][INFO ][cluster.metadata ] [Mantis] > [logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02 > 12:51:26,389][INFO ][cluster.metadata ] [Mantis] > [logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02 > 12:55:18,142][INFO ][cluster.metadata ] [Mantis] > [logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02 > 12:55:31,564][INFO ][cluster.metadata ] [Mantis] > [logstash-2014.12.02] update_mapping [logs] (dynamic)Write failed: Broken > pipe* > I guess the connection is lost bw logstash and ES. > > I am using ELK first time with such amount data . > > Any guidance?? > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/25189c47-a9c4-4933-a29e-9599bec454d9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
