Hi Mark, Does data volume that you mean is my current log files that need to be processed ? If yes, it is 1.5-3 gb/hour. If index rate that you mean is Indexing request per second in bigdesk, it is 74839326 /seconds. If search rate means search request per second in bigdesk, it is 500-700 query/second. Now i have create a better implementation of a cluster. Master node index to 2 data nodes, and kibana talks to client node. But the query now takes about 18-21 seconds, query that i mean is by changing the time filter in kibana (from 4 hours to 1 hour or by changing absolute time filter) until the data/visualization finished to load.
Regards, On Tuesday, May 12, 2015 at 6:36:51 AM UTC+7, Mark Walkom wrote: > > What about data volumes, index and search rates? > > On 11 May 2015 at 11:40, haries fajar nugroho <hari...@gmail.com > <javascript:>> wrote: > >> Hi Mark, >> >> Thanks, really appreciate your response. Previously i thought that it is >> better kibana to communicate with master rather than data node, but i was >> wrong. >> >> So this is my setup, hope it answers your question: >> >> Currently, my cluster contain 2 nodes (node a and node b). Node a as >> master node, data note, and also logstash server and node b as data node >> only. Each node has the same disk space, same ram amount, same network >> interface on the same subnet, allocated 2gb as heap size for each nodes, >> using unicast to talk between node, 5 shard and 1 replica, 65536 of >> max_file_descriptors, enable mlockall, and 1.5-3GB of logs/hour that needs >> to be processed, both nodes using elasticsearch version 1.5.2, the logs >> sent by logstash forwarder from 2 servers. >> >> Regards, >> >> On Monday, May 11, 2015 at 4:43:17 AM UTC+7, Mark Walkom wrote: >>> >>> As I said, we don't recommend sending queries, which includes those >>> generated by Kibana, to master only nodes. You would be better off sending >>> them to data nodes. >>> >>> As for your performance problems, that's a multi-layered problem that >>> may not be solved just by adding more nodes. You need to provide more >>> information around your cluster setup. >>> >>> On 10 May 2015 at 22:16, haries fajar nugroho <hari...@gmail.com> wrote: >>> >>>> Hi Mark, >>>> >>>> Thanks for your sharing. It's best if kibana talks to client node. But >>>> which is better, kibana talks to master mode or kibana talks directly to >>>> data nodes. And currently, my query needs 5-6 secs and i want to improve >>>> it >>>> to 1-2 secs. Which is better adding more nodes or adding more ram to >>>> existing node ? >>>> >>>> Regards, >>>> >>>> -- >>>> Please update your bookmarks! We moved to https://discuss.elastic.co/ >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elasticsearc...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/033706f5-4291-43a9-b6f9-e9f523faedf9%40googlegroups.com >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> Please update your bookmarks! We have moved to >> https://discuss.elastic.co/ >> --- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/9d85dfa7-4e97-4788-bc5a-ff3bd65c814e%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/9d85dfa7-4e97-4788-bc5a-ff3bd65c814e%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- Please update your bookmarks! We have moved to https://discuss.elastic.co/ --- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8214f3e9-43bb-42d4-b30b-c27bc381c10c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
