hi Zeolla,
thanks before for your reply.
sorry if i changed the email subject. i think the problem of kafka topic
partition has been solved. but the indexing still slow for me.
this is my cluster environtment for ES:
i have 5 servers ( 1 master, 4 datanode )
every server there is 64Gb memory on each machine.
i have set 4Gb to my ES Heap Size. 4 Partition of kafka topic but the
indexing still slow.
any suggest to tuning the indexing?
On 28/03/17 08:43, zeo...@gmail.com wrote:
Can you clarify what you mean by recreate your kafka topics? Usually
what I do to add partitions to a kafka topic in Metron is something like:
zk=server1:2181;/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh
--zookeeper $zk --alter --topic bro --partitions 4
Once you run this, your bro topic (or whatever you specify next to
topic) will now have multiple partitions. It is effectively
transparent to the rest of Metron.
That said, are you running kafka and ES on the same boxes? You say you
want 4 partitions because you have 4 ES boxes, but it should really
map to the # of boxes/disks you have running kafka brokers. Would you
mind providing a basic layout of your environment (what servers run
where, how many servers, etc.)? Thanks,
Jon
On Fri, Mar 24, 2017 at 11:56 AM tkg_cangkul <yuza.ras...@gmail.com
<mailto:yuza.ras...@gmail.com>> wrote:
anyone can help me for solving this problem?
On 24/03/17 18:24, tkg_cangkul wrote:
hi,
i wanna try to tuning my ES when indexing data from all the
sensors (bro, yaf, snort).
i've read this article :
https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier
on point 3. Assuming that ES will get behind from time to time,
you may want to increase the indexing kafka topic size. You
should also consider increasing the partitioning kafka topics in
general to distribute the load better and increase parallelism.
then i try to recreate my kafka topic to 4 partition because i
have 4 datanode on my elastic search. then, i check the broker
node in zookeeper for all the topics that i create and i've seen
there are *[1,2,3]* partition inside the
*/brokers/topics/(all_my_topics_name)/partitions*
But when i start the storm topology, i've seen this error message
on my storm logs if there is no node for
*/brokers/topics/(all_my_topics_name)/partitions/0/state
*Why it still read the *0 *partitions.? is the metron just set
for assigned 1 partition only on kafka topic?
then i try to create manually the
*/brokers/topics/(all_my_topics_name)/partitions/0/state *but
i've found another error msg like this:
any suggestion about this?
--
Jon
