I think slow down is not a big deal as eventadmin dispatcher act as a buffer (with timeout).
So the current state is already good. The only drawback is about the number of append / refresh performed on ES. That?s why bulk could be interesting.
Regards
JB
Le mar. 18 f?vr. 2020 ? 15:31, Francois Papon <[email protected]> a ?crit :
Hi,There is some improvement on the ES7 client so may be we could update the ES appender.Another solution is that we could detect slow down ingestion and add a temporization but I also like the bulk usage :) regards,Fran?ois [email protected]Le 18/02/2020 ? 09:04, Jean-Baptiste Onofre a ?crit :
Hi, That?s a good question ;) Currently, Decanter Elasticsearch appender sends each event followed by a refresh. It directly uses Elasticsearch REST native client via performRequest. A possible improvement is to use bulk but it means that we might have some delay between when the event actually occurs and when it?s stored in Elasticsearch. Using bulk would dramatically decrease network usage. I was thinking to introduce a property on the appender to allow user to have the choice between bulk or atomic Elasticsearch update. Regards JBLe 18 f?vr. 2020 ? 08:01, dschulten <[email protected]> a ?crit : Hi, I have combined decanter-collector-log decanter-appender-elasticsearch-rest and decanter sends the log events to my elasticsearch nicely. However, I am a bit concerned about network traffic and blocking rest requests. I am not entirely sure, but the code just seems to make blocking http requests to ES. The ES client benchmark https://www.elastic.co/de/blog/benchmarking- rest-client-transport-client shows that up to 30.000 ingestions/s could be doable under optimal circumstances, that seems not all that much. How does the ES Rest Appender send the log events? What if ES ingestion slows down? Does the Rest Appender do anything to cope with heavy load on the network or reduce the call frequency? Are there any advisable strategies which have been proven to work? Best Dietrich -- Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
