I had tested my ELK setup on a small set of logs and got it working.
Today I started it's testing on actual log files of a server. So one
server's one month data with around 10 type of logs.
So approximately 500 files to be monitored each approx 1k lines, comes upto
5 lac log lines.
**: I am using default setting for elasticsearch.
So when I started logstash ES, first I saw logstash kept running for 2
hours just reading logs and after that it gave following error on console:
*log4j, [2014-12-02T16:03:19.087] DEBUG: org.elasticsearch.transport.netty:
[logstash-XX-] disconnected from
[[#zen_unicast_2#][XX[inet[localhost/127.0.0.1:9301]]]log4j,
[2014-12-02T16:03:19.087] DEBUG: org.elasticsearch.discovery.zen:
[logstashXX] filtered ping responses: (filter_client[true],
filter_data[false]) {none}log4j, [2014-12-02T16:03:19.089] DEBUG:
org.elasticsearch.transport.netty: [logstashXX] connected to node
[[#zen_unicast_2#][inet[localhost/127.0.0.1:9301]]]*
ES console:
*[2014-12-02 12:23:24,289][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] creating index, cause [auto(bulk api)], shards
[5]/[1], mappings [_default_][2014-12-02 12:23:24,972][INFO
][cluster.metadata ] [Mantis] [logstash-2014.12.02] update_mapping
[logs] (dynamic)[2014-12-02 12:23:39,675][INFO ][cluster.metadata ]
[Mantis] [logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:51:13,975][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:51:26,389][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:55:18,142][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:55:31,564][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)Write failed: Broken
pipe*
I guess the connection is lost bw logstash and ES.
I am using ELK first time with such amount data .
Any guidance??
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0d820088-3604-426e-b50e-d5c3dccd7ee5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
