Thank you Jörg for the suggestions. I will continue to test today but here is what I can answer for now:
- The network traffic only hits 2-2.5mb so I think we're good there - Right now we are using the default of 5 shards with 1 replica - We do have predefined mappings for the index - iostat didn't show any issues during our testing but I can verify again today. I see a lot of people talking about the refresh rate, I guess the issue I have with that is this system is supposed to be as close to real time as possible. We are taking log information from Kafka and sending it to Elasticsearch and we want users to be able to search the data as quickly as possible. I guess I can play around with making the refresh rate higher but I don't thinking turning it off is an option right now. We are using the BulkProcessor simply to keep up with the speed our incoming messages are being generated at, not because we are loading large amounts of data once a day or something. Is this the wrong thing to be doing? Again, we're trying to get as close to real time as possible from the time a log message is created to the point when it hits Elasticsearch and can be searched on. On Thursday, December 19, 2013 7:39:43 PM UTC-7, tdjb wrote: > > Hi all, I'm currently working on a project where elasticsearch is our > backend but have been running into issues with insert rates. Some > background is our cluster is four physical boxes, each with 32 CPU cores > and 252 gigs of RAM. Each box runs a data node, a master node and a search > node. On two other machines that have the same hardware specs we have a > java app running that pulls our data from Kafka, does some adjusting of the > data and then inserts it into Elasticsearch. > > In the java app we are using the "node" style client along with the > BulkProcessor class to handle our inserts. Everything is running on > Elasticsearch 0.90.5 with Java 1.7.0_45. The issue we are running into is > we can't seem to be able to get over about 7k inserts per second per java > app (so 14k total since we have two instances of our java app running). It > seems around 6500k-7k the Elasticsearch inserts start to lag behind how > fast we're pulling the data from Kafka. Our initial thoughts were that the > "data adjusting" stage of our app was causing the latency but we've been > able to rule that out by adding some metrics around that part of the app. > Everything is fine until we reach the point where we want to do inserts. My > question is are there any other users out there pushing ~10k inserts per > second (that is our goal) using the Java API? If so would you mind sharing > some of the settings you are using? We've tried adjusting the BulkProcessor > concurrent count and bulk size but nothing seems to really improve it. One > thing I've noticed with our monitoring is that sometimes it seems like our > Elasticsearch client gets backed up or something. We'll see inserts > chugging along at 6k and then just start dropping and then after a few > seconds they start coming back up. No GCs or anything happen during this > time so I'm not sure what would be causing that. > > The health of the boxes while we're running looks fine (both on the ES > nodes as well as where our app lives) and inside of the JVM everything > seems to be ok as well (no huge GCs or anything). I've searched this list > and have found people talking about doing 10k inserts per second so we know > it's totally possible, we just can't seem to get the right setup to get to > that number. Any suggestions or tips would be greatly appreciated! > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/da70d4ee-6f62-400c-a7b0-97e8cfc0b54a%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
