The usual use case of Kafka (or Redis/AMQP) is to sit as the middle man to buffer and queue the material before it goes into logstash and then elastic.
ie. Kafka -> Logstash -> Elastic nodes If you are only pushing the data to Elastic then pushing it via the logstash HTTP input or a load balanced logstash should be suitable to give you some form of buffering and queueing of messages. Or you can skip the intermediate step here and go directly to the event queue based approach start using Kafka/AMQP or even a simple Redis queue, where you would then publish your message to these queues and logstash just pulls them out to populate elastic nodes at the rate it can handle. Cheers, Gerwin On Thu, 26 Nov 2020 at 07:04, Calvin Ellison <[email protected]> wrote: > Is there any best practice or standard method of getting data from > OpenSIPS to Elasticsearch? > > The scenario is an internal call filter that will block or permit a call. > e.g. 503 route advance or 603 stop route. Checks are performed against > pre-populated caches and database queries, and we want the results of those > checks to ultimately reach Elasticsearch. > > I'd prefer to avoid file-based things like syslog + logstash. The > "OpenSIPS and BigData" presentation from 2016 suggested using the > Elasticsearch REST API directly with async(rest_post()) or RabbitMQ. Our > experience with Elasticsearch indicates it will not keep up with our call > volume without a message queue, and now I'm seeing talking about Kafka > being the new standard for this. > > Is this a solved problem? > > > Regards, > > *Calvin Ellison* > Senior Voice Operations Engineer > [email protected] > +1 (213) 285-0555 > > ----------------------------------------------- > *voxox.com <http://www.voxox.com/> * > 5825 Oberlin Drive, Suite 5 > San Diego, CA 92121 > [image: Voxox] > _______________________________________________ > Users mailing list > [email protected] > http://lists.opensips.org/cgi-bin/mailman/listinfo/users >
_______________________________________________ Users mailing list [email protected] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
