Re: slave index not cleaned
The slave index does indeed grow over a period of time regardless of
restarts. We do run on 1.4 however. We will be updating to 3.6 very
soon however so I will see how that works out. Actually we should be
able to see this on our staging platform.
thanks everyone.
mvg,
Jasper
On Mon, May 14, 2012 at 4:40 PM, Bill Bell billnb...@gmail.com wrote:
This is a known issue in 1.4 especially in Windows. Some of it was resolved
in 3x.
Bill Bell
Sent from mobile
On May 14, 2012, at 5:54 AM, Erick Erickson erickerick...@gmail.com wrote:
Hmmm, replication will require up to twice the space of the
index _temporarily_, just checking if that's what you're seeing
But that should go away reasonably soon. Out of curiosity, what
happens if you restart your server, do the extra files go away?
But it sounds like your index is growing over a longer period of time
than just a single replication, is that true?
Best
Erick
On Fri, May 11, 2012 at 6:03 AM, Jasper Floor jasper.fl...@m4n.nl wrote:
Hi,
On Thu, May 10, 2012 at 5:59 PM, Otis Gospodnetic
otis_gospodne...@yahoo.com wrote:
Hi Jasper,
Sorry, I should've added more technical info wihtout being prompted.
Solr does handle that for you. Some more stuff to share:
* Solr version?
1.4
* JVM version?
1.7 update 2
* OS?
Debian (2.6.32-5-xen-amd64)
* Java replication?
yes
* Errors in Solr logs?
no
* deletion policy section in solrconfig.xml?
missing I would say, but I don't see this on the replication wiki page.
This is what we have configured for replication:
requestHandler name=/replication class=solr.ReplicationHandler
lst name=slave
str
name=masterUrl${solr.master.url}/df-stream-store/replication/str
str name=pollInterval00:20:00/str
str name=compressioninternal/str
str name=httpConnTimeout5000/str
str name=httpReadTimeout1/str
/lst
/requestHandler
We will be updating to 3.6 fairly soon however. To be honest, from
what I've read, the Solr cloud is what we really want in the future
but we will have to be patient for that.
thanks in advance
mvg,
Jasper
You may also want to look at your Index report in SPM
(http://sematext.com/spm) before/during/after replication and share what
you see.
Otis
Performance Monitoring for Solr / ElasticSearch / HBase -
http://sematext.com/spm
- Original Message -
From: Jasper Floor jasper.fl...@m4n.nl
To: solr-user@lucene.apache.org
Cc:
Sent: Thursday, May 10, 2012 9:08 AM
Subject: slave index not cleaned
Perhaps I am missing the obvious but our slaves tend to run out of
disk space. The index sizes grow to multiple times the size of the
master. So I just toss all the data and trigger a replication.
However, can't solr handle this for me?
I'm sorry if I've missed a simple setting which does this for me, but
if its there then I have missed it.
mvg
Jasper
Re: slave index not cleaned
Btw, confirmed that this doesn't happen on our development stage with 3.6.
On Wed, May 16, 2012 at 3:59 PM, Jasper Floor jasper.fl...@m4n.nl wrote:
The slave index does indeed grow over a period of time regardless of
restarts. We do run on 1.4 however. We will be updating to 3.6 very
soon however so I will see how that works out. Actually we should be
able to see this on our staging platform.
thanks everyone.
mvg,
Jasper
On Mon, May 14, 2012 at 4:40 PM, Bill Bell billnb...@gmail.com wrote:
This is a known issue in 1.4 especially in Windows. Some of it was resolved
in 3x.
Bill Bell
Sent from mobile
On May 14, 2012, at 5:54 AM, Erick Erickson erickerick...@gmail.com wrote:
Hmmm, replication will require up to twice the space of the
index _temporarily_, just checking if that's what you're seeing
But that should go away reasonably soon. Out of curiosity, what
happens if you restart your server, do the extra files go away?
But it sounds like your index is growing over a longer period of time
than just a single replication, is that true?
Best
Erick
On Fri, May 11, 2012 at 6:03 AM, Jasper Floor jasper.fl...@m4n.nl wrote:
Hi,
On Thu, May 10, 2012 at 5:59 PM, Otis Gospodnetic
otis_gospodne...@yahoo.com wrote:
Hi Jasper,
Sorry, I should've added more technical info wihtout being prompted.
Solr does handle that for you. Some more stuff to share:
* Solr version?
1.4
* JVM version?
1.7 update 2
* OS?
Debian (2.6.32-5-xen-amd64)
* Java replication?
yes
* Errors in Solr logs?
no
* deletion policy section in solrconfig.xml?
missing I would say, but I don't see this on the replication wiki page.
This is what we have configured for replication:
requestHandler name=/replication class=solr.ReplicationHandler
lst name=slave
str
name=masterUrl${solr.master.url}/df-stream-store/replication/str
str name=pollInterval00:20:00/str
str name=compressioninternal/str
str name=httpConnTimeout5000/str
str name=httpReadTimeout1/str
/lst
/requestHandler
We will be updating to 3.6 fairly soon however. To be honest, from
what I've read, the Solr cloud is what we really want in the future
but we will have to be patient for that.
thanks in advance
mvg,
Jasper
You may also want to look at your Index report in SPM
(http://sematext.com/spm) before/during/after replication and share what
you see.
Otis
Performance Monitoring for Solr / ElasticSearch / HBase -
http://sematext.com/spm
- Original Message -
From: Jasper Floor jasper.fl...@m4n.nl
To: solr-user@lucene.apache.org
Cc:
Sent: Thursday, May 10, 2012 9:08 AM
Subject: slave index not cleaned
Perhaps I am missing the obvious but our slaves tend to run out of
disk space. The index sizes grow to multiple times the size of the
master. So I just toss all the data and trigger a replication.
However, can't solr handle this for me?
I'm sorry if I've missed a simple setting which does this for me, but
if its there then I have missed it.
mvg
Jasper
Re: slave index not cleaned
Hmmm, replication will require up to twice the space of the
index _temporarily_, just checking if that's what you're seeing
But that should go away reasonably soon. Out of curiosity, what
happens if you restart your server, do the extra files go away?
But it sounds like your index is growing over a longer period of time
than just a single replication, is that true?
Best
Erick
On Fri, May 11, 2012 at 6:03 AM, Jasper Floor jasper.fl...@m4n.nl wrote:
Hi,
On Thu, May 10, 2012 at 5:59 PM, Otis Gospodnetic
otis_gospodne...@yahoo.com wrote:
Hi Jasper,
Sorry, I should've added more technical info wihtout being prompted.
Solr does handle that for you. Some more stuff to share:
* Solr version?
1.4
* JVM version?
1.7 update 2
* OS?
Debian (2.6.32-5-xen-amd64)
* Java replication?
yes
* Errors in Solr logs?
no
* deletion policy section in solrconfig.xml?
missing I would say, but I don't see this on the replication wiki page.
This is what we have configured for replication:
requestHandler name=/replication class=solr.ReplicationHandler
lst name=slave
str
name=masterUrl${solr.master.url}/df-stream-store/replication/str
str name=pollInterval00:20:00/str
str name=compressioninternal/str
str name=httpConnTimeout5000/str
str name=httpReadTimeout1/str
/lst
/requestHandler
We will be updating to 3.6 fairly soon however. To be honest, from
what I've read, the Solr cloud is what we really want in the future
but we will have to be patient for that.
thanks in advance
mvg,
Jasper
You may also want to look at your Index report in SPM
(http://sematext.com/spm) before/during/after replication and share what you
see.
Otis
Performance Monitoring for Solr / ElasticSearch / HBase -
http://sematext.com/spm
- Original Message -
From: Jasper Floor jasper.fl...@m4n.nl
To: solr-user@lucene.apache.org
Cc:
Sent: Thursday, May 10, 2012 9:08 AM
Subject: slave index not cleaned
Perhaps I am missing the obvious but our slaves tend to run out of
disk space. The index sizes grow to multiple times the size of the
master. So I just toss all the data and trigger a replication.
However, can't solr handle this for me?
I'm sorry if I've missed a simple setting which does this for me, but
if its there then I have missed it.
mvg
Jasper
Re: slave index not cleaned
This is a known issue in 1.4 especially in Windows. Some of it was resolved in
3x.
Bill Bell
Sent from mobile
On May 14, 2012, at 5:54 AM, Erick Erickson erickerick...@gmail.com wrote:
Hmmm, replication will require up to twice the space of the
index _temporarily_, just checking if that's what you're seeing
But that should go away reasonably soon. Out of curiosity, what
happens if you restart your server, do the extra files go away?
But it sounds like your index is growing over a longer period of time
than just a single replication, is that true?
Best
Erick
On Fri, May 11, 2012 at 6:03 AM, Jasper Floor jasper.fl...@m4n.nl wrote:
Hi,
On Thu, May 10, 2012 at 5:59 PM, Otis Gospodnetic
otis_gospodne...@yahoo.com wrote:
Hi Jasper,
Sorry, I should've added more technical info wihtout being prompted.
Solr does handle that for you. Some more stuff to share:
* Solr version?
1.4
* JVM version?
1.7 update 2
* OS?
Debian (2.6.32-5-xen-amd64)
* Java replication?
yes
* Errors in Solr logs?
no
* deletion policy section in solrconfig.xml?
missing I would say, but I don't see this on the replication wiki page.
This is what we have configured for replication:
requestHandler name=/replication class=solr.ReplicationHandler
lst name=slave
str
name=masterUrl${solr.master.url}/df-stream-store/replication/str
str name=pollInterval00:20:00/str
str name=compressioninternal/str
str name=httpConnTimeout5000/str
str name=httpReadTimeout1/str
/lst
/requestHandler
We will be updating to 3.6 fairly soon however. To be honest, from
what I've read, the Solr cloud is what we really want in the future
but we will have to be patient for that.
thanks in advance
mvg,
Jasper
You may also want to look at your Index report in SPM
(http://sematext.com/spm) before/during/after replication and share what
you see.
Otis
Performance Monitoring for Solr / ElasticSearch / HBase -
http://sematext.com/spm
- Original Message -
From: Jasper Floor jasper.fl...@m4n.nl
To: solr-user@lucene.apache.org
Cc:
Sent: Thursday, May 10, 2012 9:08 AM
Subject: slave index not cleaned
Perhaps I am missing the obvious but our slaves tend to run out of
disk space. The index sizes grow to multiple times the size of the
master. So I just toss all the data and trigger a replication.
However, can't solr handle this for me?
I'm sorry if I've missed a simple setting which does this for me, but
if its there then I have missed it.
mvg
Jasper
Re: slave index not cleaned
Hi,
On Thu, May 10, 2012 at 5:59 PM, Otis Gospodnetic
otis_gospodne...@yahoo.com wrote:
Hi Jasper,
Sorry, I should've added more technical info wihtout being prompted.
Solr does handle that for you. Some more stuff to share:
* Solr version?
1.4
* JVM version?
1.7 update 2
* OS?
Debian (2.6.32-5-xen-amd64)
* Java replication?
yes
* Errors in Solr logs?
no
* deletion policy section in solrconfig.xml?
missing I would say, but I don't see this on the replication wiki page.
This is what we have configured for replication:
requestHandler name=/replication class=solr.ReplicationHandler
lst name=slave
str
name=masterUrl${solr.master.url}/df-stream-store/replication/str
str name=pollInterval00:20:00/str
str name=compressioninternal/str
str name=httpConnTimeout5000/str
str name=httpReadTimeout1/str
/lst
/requestHandler
We will be updating to 3.6 fairly soon however. To be honest, from
what I've read, the Solr cloud is what we really want in the future
but we will have to be patient for that.
thanks in advance
mvg,
Jasper
You may also want to look at your Index report in SPM
(http://sematext.com/spm) before/during/after replication and share what you
see.
Otis
Performance Monitoring for Solr / ElasticSearch / HBase -
http://sematext.com/spm
- Original Message -
From: Jasper Floor jasper.fl...@m4n.nl
To: solr-user@lucene.apache.org
Cc:
Sent: Thursday, May 10, 2012 9:08 AM
Subject: slave index not cleaned
Perhaps I am missing the obvious but our slaves tend to run out of
disk space. The index sizes grow to multiple times the size of the
master. So I just toss all the data and trigger a replication.
However, can't solr handle this for me?
I'm sorry if I've missed a simple setting which does this for me, but
if its there then I have missed it.
mvg
Jasper
slave index not cleaned
Perhaps I am missing the obvious but our slaves tend to run out of disk space. The index sizes grow to multiple times the size of the master. So I just toss all the data and trigger a replication. However, can't solr handle this for me? I'm sorry if I've missed a simple setting which does this for me, but if its there then I have missed it. mvg Jasper
Re: slave index not cleaned
Hi Jasper, Solr does handle that for you. Some more stuff to share: * Solr version? * JVM version? * OS? * Java replication? * Errors in Solr logs? * deletion policy section in solrconfig.xml? * merge policy section in solrconfig.xml? * ... You may also want to look at your Index report in SPM (http://sematext.com/spm) before/during/after replication and share what you see. Otis Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm - Original Message - From: Jasper Floor jasper.fl...@m4n.nl To: solr-user@lucene.apache.org Cc: Sent: Thursday, May 10, 2012 9:08 AM Subject: slave index not cleaned Perhaps I am missing the obvious but our slaves tend to run out of disk space. The index sizes grow to multiple times the size of the master. So I just toss all the data and trigger a replication. However, can't solr handle this for me? I'm sorry if I've missed a simple setting which does this for me, but if its there then I have missed it. mvg Jasper
