hmm ok.
so it means i better run this metron on separate cluster with my other
apps right?
ok thank you so much for all of your help guys. :)
i'll try it soon after i migrate my other apps.
On 29/03/17 23:52, Simon Elliston Ball wrote:
In general for any ES use case you should be running only ES on the ES
nodes, and yes, you should use min(31, RAM / 2) as heap size. It’s
also worth considering shard counts for indices and disk layout in the
ES nodes.
As Nick points out, you’re also going to find it hard to tune ES for
large numbers of applications running different indexing, so it’s
generally a good idea to build different ES environments for your
multiple use cases.
Simon
On 29 Mar 2017, at 17:02, tkg_cangkul <yuza.ras...@gmail.com
<mailto:yuza.ras...@gmail.com>> wrote:
Well,
actually there are not only metron that running on my cluster. there
are some other apps running on it. i'm afraid it will be crash if i
set it to 50% of my RAM. Are ES heap ideally set to 50% RAM memory to
get the maximum performance?
On 29/03/17 17:18, zeo...@gmail.com wrote:
Right off the bat I would give 31GB heap to each ES node. Normally
you want that number close to 32GB but not exceeding 50% of your
total server RAM so it can be used for disk caching. Let me know if
that helps, I'd be happy to help further.
Jon
On Wed, Mar 29, 2017, 12:56 AM tkg_cangkul <yuza.ras...@gmail.com
<mailto:yuza.ras...@gmail.com>> wrote:
hi Zeolla,
thanks before for your reply.
sorry if i changed the email subject. i think the problem of
kafka topic partition has been solved. but the indexing still
slow for me.
this is my cluster environtment for ES:
i have 5 servers ( 1 master, 4 datanode )
every server there is 64Gb memory on each machine.
i have set 4Gb to my ES Heap Size. 4 Partition of kafka topic
but the indexing still slow.
any suggest to tuning the indexing?
On 28/03/17 08:43, zeo...@gmail.com <mailto:zeo...@gmail.com> wrote:
Can you clarify what you mean by recreate your kafka topics?
Usually what I do to add partitions to a kafka topic in Metron
is something like:
zk=server1:2181;/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh
--zookeeper $zk --alter --topic bro --partitions 4
Once you run this, your bro topic (or whatever you specify next
to topic) will now have multiple partitions. It is effectively
transparent to the rest of Metron.
That said, are you running kafka and ES on the same boxes? You
say you want 4 partitions because you have 4 ES boxes, but it
should really map to the # of boxes/disks you have running
kafka brokers. Would you mind providing a basic layout of your
environment (what servers run where, how many servers, etc.)?
Thanks,
Jon
On Fri, Mar 24, 2017 at 11:56 AM tkg_cangkul
<yuza.ras...@gmail.com <mailto:yuza.ras...@gmail.com>> wrote:
anyone can help me for solving this problem?
On 24/03/17 18:24, tkg_cangkul wrote:
hi,
i wanna try to tuning my ES when indexing data from all
the sensors (bro, yaf, snort).
i've read this article :
https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier
on point 3. Assuming that ES will get behind from time to
time, you may want to increase the indexing kafka topic
size. You should also consider increasing the partitioning
kafka topics in general to distribute the load better and
increase parallelism.
then i try to recreate my kafka topic to 4 partition
because i have 4 datanode on my elastic search. then, i
check the broker node in zookeeper for all the topics that
i create and i've seen there are *[1,2,3]* partition
inside the */brokers/topics/(all_my_topics_name)/partitions*
But when i start the storm topology, i've seen this error
message on my storm logs if there is no node for
*/brokers/topics/(all_my_topics_name)/partitions/0/state
*Why it still read the *0 *partitions.? is the metron just
set for assigned 1 partition only on kafka topic?
then i try to create manually the
*/brokers/topics/(all_my_topics_name)/partitions/0/state
*but i've found another error msg like this:
any suggestion about this?
--
Jon
--
Jon
