Re: SolrCloud OOM Problem

2014-08-13 Thread tuxedomoon
Great info. Can I ask how much data you are handling with that 6G or 7G heap? -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-OOM-Problem-tp4152389p4152712.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud OOM Problem

2014-08-13 Thread tuxedomoon
Have you used a queue to intercept queries and if so what was your implementation? We are indexing huge amounts of data from 7 SolrJ instances which run independently, so there's a lot of concurrent indexing. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-OOM

Re: SolrCloud OOM Problem

2014-08-13 Thread Shawn Heisey
On 8/13/2014 5:34 AM, tuxedomoon wrote: Great info. Can I ask how much data you are handling with that 6G or 7G heap? My dev server is the one with the 7GB heap. My production servers only handle half the index shards, so they have the smaller heap. Here is the index size info from my dev

Re: SolrCloud OOM Problem

2014-08-13 Thread Shawn Heisey
On 8/13/2014 5:42 AM, tuxedomoon wrote: Have you used a queue to intercept queries and if so what was your implementation? We are indexing huge amounts of data from 7 SolrJ instances which run independently, so there's a lot of concurrent indexing. On my setup, the queries come from a java

Re: SolrCloud OOM Problem

2014-08-13 Thread tuxedomoon
20 0 129g 14g 1.9g S2.021.317:40.61 java -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-OOM-Problem-tp4152389p4152753.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud OOM Problem

2014-08-12 Thread Toke Eskildsen
On Tue, 2014-08-12 at 01:27 +0200, dancoleman wrote: My SolrCloud of 3 shard / 3 replicas is having a lot of OOM errors. Here are some specs on my setup: hosts: all are EC2 m1.large with 250G data volumes Is that 3 (each running a primary and a replica shard) or 6 instances? documents:

Re: SolrCloud OOM Problem

2014-08-12 Thread tuxedomoon
/SolrCloud-OOM-Problem-tp4152389p4152585.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud OOM Problem

2014-08-12 Thread Shawn Heisey
On 8/12/2014 3:12 PM, tuxedomoon wrote: I have modified my instances to m2.4xlarge 64-bit with 68.4G memory. Hate to ask this but can you recommend Java memory and GC settings for 90G data and the above memory? Currently I have CATALINA_OPTS=${CATALINA_OPTS} -XX:NewSize=1536m

SolrCloud OOM Problem

2014-08-11 Thread dancoleman
)} status=0 QTime=808 Any ideas on why memory usage is so high are appreciated. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-OOM-Problem-tp4152389.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud OOM Problem

2014-08-11 Thread Shawn Heisey
On 8/11/2014 5:27 PM, dancoleman wrote: My SolrCloud of 3 shard / 3 replicas is having a lot of OOM errors. Here are some specs on my setup: hosts: all are EC2 m1.large with 250G data volumes documents: 120M total zookeeper: 5 external t1.micros snip Linux top command output with no

Re: SolrCloud OOM Problem

2014-08-11 Thread dancoleman
90G is correct, each host is currently holding that much data. Are you saying that 32GB to 96GB would be needed for each host? Assuming we did not add more shards that is. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-OOM-Problem-tp4152389p4152401.html Sent

Re: SolrCloud OOM Problem

2014-08-11 Thread Shawn Heisey
90G is correct, each host is currently holding that much data. Are you saying that 32GB to 96GB would be needed for each host? Assuming we did not add more shards that is. If you want good performance and enough memory to give Solr the heap it will need, yes. Lucene (the search API that