Re: Choosing what document to return.
Hi Mei Thank you very much - looks good :-) Noam. -- View this message in context: http://old.nabble.com/Choosing-what-document-to-return.-tp27259310p27293083.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Replication Handler Severe Error: Unable to move index file
On Fri, Jan 22, 2010 at 4:24 AM, Trey solrt...@gmail.com wrote:
Unfortunately, when I went back to look at the logs this morning, the log
file had been blown away... that puts a major damper on my debugging
capabilities - so sorry about that. As a double whammy, we optimize
nightly, so the old index files have completely changed at this point.
I do not remember seeing an exception / stack trace in the logs associated
with the SEVERE *Unable to move file* entry, but we were grepping the
logs, so if it was outputted onto another line it could have possibly been
there. I wouldn't really expect to see anything based upon the code in
SnapPuller.java:
/**
* Copy a file by the File#renameTo() method. If it fails, it is
considered a failure
* p/
* Todo may be we should try a simple copy if it fails
*/
private boolean copyAFile(File tmpIdxDir, File indexDir, String fname,
ListString copiedfiles) {
File indexFileInTmpDir = new File(tmpIdxDir, fname);
File indexFileInIndex = new File(indexDir, fname);
boolean success = indexFileInTmpDir.renameTo(indexFileInIndex);
if (!success) {
LOG.error(Unable to move index file from: + indexFileInTmpDir
+ to: + indexFileInIndex);
for (String f : copiedfiles) {
File indexFile = new File(indexDir, f);
if (indexFile.exists())
indexFile.delete();
}
delTree(tmpIdxDir);
return false;
}
return true;
}
In terms of whether this is an off case: this is the first occurrence of
this I have seen in the logs. We tried to replicate the conditions under
which the exception occurred, but were unable. I'll send along some more
useful info if this happens again.
In terms of the behavior we saw: It appears that a replication occurred and
the Unable to move file error occurred. As a result, it looks like the
ENTIRE index was subsequently replicated again into a temporary directory
(several times, over and over).
The end result was that we had multiple full copies of the index in
temporary index folders on the slave, and the original still couldn't be
updated (the move to ./index wouldn't work). Does Solr ever hold files open
in a manner that would prevent a file in the index directory from being
overridden?
There is a TODO which says manual it try to copy if move (renameTo)
fails. We never did it because we never observed renameTo failing.
2010/1/21 Noble Paul നോബിള് नोब्ळ् noble.p...@corp.aol.com
is it a one off case? do you observerve this frequently?
On Thu, Jan 21, 2010 at 11:26 AM, Otis Gospodnetic
otis_gospodne...@yahoo.com wrote:
It's hard to tell without poking around, but one of the first things I'd
do would be to look for /home/solr/cores/core8/index.20100119103919/_6qv.fnm
- does this file/dir really exist? Or, rather, did it exist when the error
happened.
I'm not looking at the source code now, but is that really the only error
you got? No exception stack trace?
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
- Original Message
From: Trey solrt...@gmail.com
To: solr-user@lucene.apache.org
Sent: Wed, January 20, 2010 11:54:43 PM
Subject: Replication Handler Severe Error: Unable to move index file
Does anyone know what would cause the following error?:
10:45:10 AM org.apache.solr.handler.SnapPuller copyAFile
SEVERE: *Unable to move index file* from:
/home/solr/cores/core8/index.20100119103919/_6qv.fnm to:
/home/solr/cores/core8/index/_6qv.fnm
This occurred a few days back and we noticed that several full copies of
the
index were subsequently pulled from the master to the slave, effectively
evicting our live index from RAM (the linux os cache), and killing our
query
performance due to disk io contention.
Has anyone experienced this behavior recently? I found an old thread
about
this error from early 2009, but it looks like it was patched almost a
year
ago:
http://old.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-td21157722.html
Additional Relevant information:
-We are using the Solr 1.4 official release + a field collapsing patch
from
mid December (which I believe should only affect query side, not
indexing /
replication).
-Our Replication PollInterval for slaves checking the master is very
small
(15 seconds)
-We have a multi-box distributed search with each box possessing
multiple
cores
-We issue a manual (rolling) optimize across the cores on the master
once a
day (occurred ~ 1-2 hours before the above timeline)
-maxWarmingSearchers is set to 1.
--
-
Noble Paul | Systems Architect| AOL | http://aol.com
--
-
Noble Paul | Systems Architect| AOL | http://aol.com
after migration to solr 1.4, queries always return 0 items
We have a working system based on Solr 1.3 that I tried to upgrade to 1.4 with no luck. The new 1.4 Solr server is up and running but always returns 0 results when you query it. It seems to have the correct schema but the number of documents is 0 even though I used a backup of the production data. I created the 1.4 solr server using a few machine and following the instructions on the solr wiki. For the data, I used the 'backup' command to make a copy of the live index and moved to the data directory on the new 1.4 server. When I start the new server, the tomcat catinla logs don't have any errors (only info msgs). The logs show the correct solr home and I can see its reading the schema.xml (acts_as_solr). Finally, I see that the data directory where the index is has new subfolders for spellchecker spellchecker2 so I assume it is looking the correct directory. I'd appreciate any tips on how to debug this. Below are some more details on the setup.. thanks in advance. Athir
Re: after migration to solr 1.4, queries always return 0 items
You omitted the details. Is there an index directory under your data directory? That's the heart of the Lucene index there. Look at Solr's admin, it'll tell you the data directory location it is using, be sure it is set where you think it should be. Erik On Jan 24, 2010, at 8:59 AM, athir nuaimi wrote: We have a working system based on Solr 1.3 that I tried to upgrade to 1.4 with no luck. The new 1.4 Solr server is up and running but always returns 0 results when you query it. It seems to have the correct schema but the number of documents is 0 even though I used a backup of the production data. I created the 1.4 solr server using a few machine and following the instructions on the solr wiki. For the data, I used the 'backup' command to make a copy of the live index and moved to the data directory on the new 1.4 server. When I start the new server, the tomcat catinla logs don't have any errors (only info msgs). The logs show the correct solr home and I can see its reading the schema.xml (acts_as_solr). Finally, I see that the data directory where the index is has new subfolders for spellchecker spellchecker2 so I assume it is looking the correct directory. I'd appreciate any tips on how to debug this. Below are some more details on the setup.. thanks in advance. Athir
Re: after migration to solr 1.4, queries always return 0 items
Hi Eric. Thanks for the quick reply. I though the backup went in /data. I have restored the backup to /data/index and am now getting results. On 2010-01-24, at 9:46 AM, Erik Hatcher wrote: You omitted the details. Is there an index directory under your data directory? That's the heart of the Lucene index there. Look at Solr's admin, it'll tell you the data directory location it is using, be sure it is set where you think it should be. Erik On Jan 24, 2010, at 8:59 AM, athir nuaimi wrote: We have a working system based on Solr 1.3 that I tried to upgrade to 1.4 with no luck. The new 1.4 Solr server is up and running but always returns 0 results when you query it. It seems to have the correct schema but the number of documents is 0 even though I used a backup of the production data. I created the 1.4 solr server using a few machine and following the instructions on the solr wiki. For the data, I used the 'backup' command to make a copy of the live index and moved to the data directory on the new 1.4 server. When I start the new server, the tomcat catinla logs don't have any errors (only info msgs). The logs show the correct solr home and I can see its reading the schema.xml (acts_as_solr). Finally, I see that the data directory where the index is has new subfolders for spellchecker spellchecker2 so I assume it is looking the correct directory. I'd appreciate any tips on how to debug this. Below are some more details on the setup.. thanks in advance. Athir
strategy for snapshottig Solr data directory on EC2
We are running two Solr servers (master/slave) on EC2 and have the solr home directories on EBS drives that we snapshot every 12 hours. While that will mean that we will lose at most 12 hours of data, I wondered if there was a way I could reduce the window of data loss. With our mysql servers, we snapshot every 12 hours but also copy the binary logs to S3 every 5 minutes. We are doing commits every 10 minutes on the master and will be using the built-in java replication (today we are using snapshotting to replicate but are in the process of migrating from 1.3 to 1.4). On a related note, are we doing the right thing in having our slave solr home directory on an EBS volume? If the slave were to die and we had to create a fresh one, will it just resync the entire index from the master? is the reason to have the slave on an EBS volume so that the slave has less data to resync on startup? thanks in advance Athir
Re: strategy for snapshottig Solr data directory on EC2
Our setup on ec2 is as follows: a) mysql master on ebs volume. b) solr master on its own ebs volume c) solr slaves do not use ebs -- but rather use the ephemeral instance stores. There is a small period of time where the solr slave has to re-sync the data from the solr master. Cheers, Bill -- From: athir nuaimi at...@nuaim.com Sent: Sunday, January 24, 2010 7:35 AM To: solr-user@lucene.apache.org Subject: strategy for snapshottig Solr data directory on EC2 We are running two Solr servers (master/slave) on EC2 and have the solr home directories on EBS drives that we snapshot every 12 hours. While that will mean that we will lose at most 12 hours of data, I wondered if there was a way I could reduce the window of data loss. With our mysql servers, we snapshot every 12 hours but also copy the binary logs to S3 every 5 minutes. We are doing commits every 10 minutes on the master and will be using the built-in java replication (today we are using snapshotting to replicate but are in the process of migrating from 1.3 to 1.4). On a related note, are we doing the right thing in having our slave solr home directory on an EBS volume? If the slave were to die and we had to create a fresh one, will it just resync the entire index from the master? is the reason to have the slave on an EBS volume so that the slave has less data to resync on startup? thanks in advance Athir
Default value attribute in RSS DIH
Hey All, Can anyone tell me what the attribute name is for defining a default value in the field tag of the RSS data import handler?? Basically I want to do something like field column=type value=external_source commonField=true/ Any Ideas? Regards, Dave
Retrieve Field Names:
How to retrieve field names from the dynamic schema in the client side program ? Any links or reference would be helpful. (Similar to the schema browser on the SOLR admin screen.) Kumar_/|\_ www.saisk.com ku...@saisk.com making a profound difference with knowledge and creativity...
using bq with standard request handler
Hi all, I am using a query like: http://localhost:8080/solr/select?q=product_category:Groceryfq=in_stock:truedebugQuery=true; sort=display_priority+desc,prop_l_price+asc To first sort on display_priority and then on price.. Is it possible to use display_priority/price fields in bq itself to acheive the same result.. I tried forming some queries that but was unable to get desired results... Any kind of help would be highly appreciated... Regards, Gunjan -- View this message in context: http://old.nabble.com/using-bq-with-standard-request-handler-tp27302951p27302951.html Sent from the Solr - User mailing list archive at Nabble.com.
