Have you increased the indexing "batch_size"? That is the first knob to start tuning.
https://github.com/apache/metron/tree/master/metron-platform/metron-indexing#sensor-indexing-configuration On Tue, May 14, 2019 at 10:26 AM <[email protected]> wrote: > Hello happy metron users, > > > > I’ve a Metron cluster based on Hortonworks CP, and I’ve setup Kerberos on > the top of it, as you all probably have done since we deal with security J > > > > It seems that everything is working fine, Kerberos, ranger,… but I’m > facing an issue regarding the overall throuput. > > > > I feed my cluster with Nifi, here is what I do: > > Test 1: > > - Send 2 lines of logs to Kafka sensor topic with Nifi > > - Use Kafka CLI consumer to read messages from sensor topic: > response is immediate > > - Use kafka CLI consumer to read messages from enrichment topic: > messages are coming after nearly 20 s > > Test 2: > > - Send 200 lines of logs to Kafka sensor topic with Nifi > > - Use Kafka CLI consumer to read messages from sensor topic: > response is immediate > > - Use kafka CLI consumer to read messages from enrichment topic: > some messages are coming immediately, but it seems they come 10 by 10 > (nearly), with many seconds between each flow > > > > > > It’s probably related to Storm configuration, but I don’t know where to go > now. I’ve tried to change various parameters like > topology.max.spout.pending (currently set to 500), but no improvement > > > > Thanks for your help > > > > Stéphane > > > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > >
