Well, I hit the same error again after 6.7M documents and the increased heap size.
After a restart of the cluster, it's taking a very long time to bring up the replica shards. Until I have about 50% of my shards (primary + replica) initialized, I'm unable to load any more documents. When I try to bulk load, I fail after about another 1000 documents. Here's my current cluster stats from Marvel: Nodes: 6 Indices: 106 Shards: 2342 Data: 1.14 TB CPU: 238% Memory: 44.21 GB / 719.90 GB Up time: 3.4 h Version: 1.1.1 My total number of shards is actually 3124, still waiting on several hundred to come back after three and a half hours. I have the following questions: - Is there an index size to heap size ratio that we should be adhering? - What would prevent us from indexing additional documents? We seem to be no where close to filling the memory allocated to elasticsearch, and the CPU usage has remained low. - I just changed the es.logger.level to DEBUG, restarted the cluster, waited for all the primary shards to initialize, and submitted a document for indexing. It failed with the UnavailableShardsException and nothing appeared in the logs on the node where I submitted the request. Is there somewhere else I should be looking? - I've given no tuning parameters to the indexes or the system as a whole. Is there something I may be missing? I'm sure this is something that could be solved on my side with some change in parameters. Any ideas what I can try? Thanks, -Chris On Friday, June 6, 2014 4:59:49 PM UTC-7, Chris Mildebrandt wrote: > > It looks like the problem is on my end. I misplaced the HEAP size > parameter and was only running with 1GB. After bumping it up to a more > respectable amount, the loading is humming along again. > > -Chris > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f2e4833c-7fc0-4ca6-8a07-81ee63a0cceb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
