Re: Corrupted index in SolrCloud
Thanks for the explanation Erick, that makes sense! Matt On 21/09/2018 15:50, Erick Erickson wrote: The disk corruption is, of course, a red flag and likely the root cause. As for how it replicated let's assume a 2 replica shard (leader + follower). If the follower ever went into full recovery it would use old-style replication to copy down the entire index, corrupted index and all, from the leader. The follower can go into "full recovery" for a number of reasons, from it being shut down for a while and indexing still happening to the leader to communications burps. There's been a lot of work put in to making fewer full recoveries, but much of that only came to fruition in recent Solr releases, especially starting with Solr 7.3. (SOLR-11702) Best, Erick On Fri, Sep 21, 2018 at 7:17 AM Matt Pearce wrote: Hi, We've just been working with a client who had a corruption issue with their SolrCloud install. They're running Solr 5.3.1, with a collection spread across 12 shards. Each shard has a single replica. They were seeing "Index Corruption" errors when running certain queries. We investigated, and narrowed it down to a single shard. Using the Lucene CheckIndex utility, we tested both the primary and replica copies of the data, and found the same issue with both - the first segment, containing the majority of the documents, was reporting corruption. They were able to restore from a backup, but it would be good to get some idea what could have caused the problem in SolrCloud. One of the machines ran out of disk space last week during indexing, which we guess could have been the starting point for the corrupted data files. Our question is: why would the corruption have spread to the replica as well? Could a corrupted document be replicated and cause the replica index to break as well? Thanks, Matt -- Matt Pearce Flax - Open Source Enterprise Search www.flax.co.uk -- Matt Pearce Flax - Open Source Enterprise Search www.flax.co.uk
Re: Corrupted index in SolrCloud
The disk corruption is, of course, a red flag and likely the root cause. As for how it replicated let's assume a 2 replica shard (leader + follower). If the follower ever went into full recovery it would use old-style replication to copy down the entire index, corrupted index and all, from the leader. The follower can go into "full recovery" for a number of reasons, from it being shut down for a while and indexing still happening to the leader to communications burps. There's been a lot of work put in to making fewer full recoveries, but much of that only came to fruition in recent Solr releases, especially starting with Solr 7.3. (SOLR-11702) Best, Erick On Fri, Sep 21, 2018 at 7:17 AM Matt Pearce wrote: > > Hi, > > We've just been working with a client who had a corruption issue with > their SolrCloud install. They're running Solr 5.3.1, with a collection > spread across 12 shards. Each shard has a single replica. > > They were seeing "Index Corruption" errors when running certain queries. > We investigated, and narrowed it down to a single shard. Using the > Lucene CheckIndex utility, we tested both the primary and replica copies > of the data, and found the same issue with both - the first segment, > containing the majority of the documents, was reporting corruption. They > were able to restore from a backup, but it would be good to get some > idea what could have caused the problem in SolrCloud. One of the > machines ran out of disk space last week during indexing, which we guess > could have been the starting point for the corrupted data files. > > Our question is: why would the corruption have spread to the replica as > well? Could a corrupted document be replicated and cause the replica > index to break as well? > > Thanks, > > Matt > > -- > Matt Pearce > Flax - Open Source Enterprise Search > www.flax.co.uk
Corrupted index in SolrCloud
Hi, We've just been working with a client who had a corruption issue with their SolrCloud install. They're running Solr 5.3.1, with a collection spread across 12 shards. Each shard has a single replica. They were seeing "Index Corruption" errors when running certain queries. We investigated, and narrowed it down to a single shard. Using the Lucene CheckIndex utility, we tested both the primary and replica copies of the data, and found the same issue with both - the first segment, containing the majority of the documents, was reporting corruption. They were able to restore from a backup, but it would be good to get some idea what could have caused the problem in SolrCloud. One of the machines ran out of disk space last week during indexing, which we guess could have been the starting point for the corrupted data files. Our question is: why would the corruption have spread to the replica as well? Could a corrupted document be replicated and cause the replica index to break as well? Thanks, Matt -- Matt Pearce Flax - Open Source Enterprise Search www.flax.co.uk
Corrupted Index with custom directory
Hi,
We have a requirement to pre-encrypt an index we are building before it
hits disk. We are doing this by using a wrapper around MMapDirectory that
wraps the input/output streams(I know the general recommendation is to
encrypt the filesystem instead but this option was explicitly rejected by
our security group).
The issue we've been running into is that the once indexes get beyond "very
small", we start seeing corruption issues on some but not all queries.
Running CheckIndex reports any segments that are not compound=false as
corrupt(don't know if this is the actual root cause). While the exact
error messages differ it is always an issue with the .doc segment file.
Also of interest, this doesn't seem to block segment merges at all as
segments reporting as corrupt later get merged succesfully.
We have not yet been able to reliably reproduce the corruption issue in a
simple isolated test. I'm wondering if anyone has any tips on places to
look or tests to run that might help isolate the issue?
One corrupted segment:
2 of 12: name=_7h08 maxDoc=41769
version=5.3.1
id=8f4kteokbievldjcgq7ly7cj3
codec=Lucene53
compound=false
numFiles=10
size (MB)=8.666
diagnostics = {os=Linux, java.vendor=Oracle Corporation,
java.version=1.8.0_65, java.vm.version=25.65-b01, lucene.version=5.3.1,
mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_65-b17,
source=merge, mergeFactor=10, os.version=2.6.32-573.12.1.el6.x86_64,
timestamp=1453492996850}
no deletions
test: open reader.OK [took 0.034 sec]
test: check integrity.OK [took 1.200 sec]
test: check live docs.OK [took 0.000 sec]
test: field infos.OK [24 fields] [took 0.000 sec]
test: field norms.OK [5 fields] [took 0.034 sec]
test: terms, freq, prox...ERROR: java.io.EOFException: at the end of
the file
java.io.EOFException: at the end of the file
at
org.apache.lucene.store.DecryptingMMapIndexInput.ensureNotEOF(DecryptingMMapIndexInput.java:236)
at
org.apache.lucene.store.DecryptingMMapIndexInput.readByte(DecryptingMMapIndexInput.java:216)
at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
at
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.readVIntBlock(Lucene50PostingsReader.java:132)
at
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockPostingsEnum.refillDocs(Lucene50PostingsReader.java:619)
at
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockPostingsEnum.advance(Lucene50PostingsReader.java:716)
at org.apache.lucene.index.CheckIndex.checkFields(CheckIndex.java:1411)
at org.apache.lucene.index.CheckIndex.testPostings(CheckIndex.java:1666)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:700)
at org.apache.lucene.index.CheckIndex.doMain(CheckIndex.java:2354)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2237)
test: stored fields...OK [125307 total field count; avg 3.0 fields
per doc] [took 0.332 sec]
test: term vectorsOK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...OK [0 docvalues fields; 0 BINARY; 0 NUMERIC;
0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] [took 0.000 sec]
FAILED
WARNING: exorciseIndex() would remove reference to this segment; full
exception:
java.lang.RuntimeException: Term Index test failed
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:720)
at org.apache.lucene.index.CheckIndex.doMain(CheckIndex.java:2354)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2237)
Thanks for any help!
Cheers,
Geoff
Re: Fixing corrupted index?
Oh, somehow missed that in your original e-mail. How do you run the checkindex? Do you pass the -fix option? [1] You may want to try luke [2] to open index without opening the IndexReader and run the Tools-Check Index tool from the luke. [1] http://java.dzone.com/news/lucene-and-solrs-checkindex [2] https://github.com/DmitryKey/luke/releases On Mon, Mar 24, 2014 at 10:52 PM, zqzuk ziqizh...@hotmail.co.uk wrote: Hi Thanks. But I am already using CheckIndex and the error is given by the CheckIndex utility: it could not even continue after reporting could not read any segements file in directory. -- View this message in context: http://lucene.472066.n3.nabble.com/Fixing-corrupted-index-tp4126644p4126687.html Sent from the Solr - User mailing list archive at Nabble.com. -- Dmitry Blog: http://dmitrykan.blogspot.com Twitter: http://twitter.com/dmitrykan
Re: Fixing corrupted index?
Thank you. I tried Luke with IndexReader disabled, however it seems the index is compeletely broken, as it complains ERROR: java.lang.Exception: there is no valid Lucene index in this directory. Sounds like I am out of luck, is it so? -- View this message in context: http://lucene.472066.n3.nabble.com/Fixing-corrupted-index-tp4126644p4126830.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Fixing corrupted index?
1. Luke: if you leave the IndexReader on, does the index even open? Can you access the CheckIndex? 2. The command line CheckIndex: what does the CheckIndex -fix do? On Tue, Mar 25, 2014 at 10:54 AM, zqzuk ziqizh...@hotmail.co.uk wrote: Thank you. I tried Luke with IndexReader disabled, however it seems the index is compeletely broken, as it complains ERROR: java.lang.Exception: there is no valid Lucene index in this directory. Sounds like I am out of luck, is it so? -- View this message in context: http://lucene.472066.n3.nabble.com/Fixing-corrupted-index-tp4126644p4126830.html Sent from the Solr - User mailing list archive at Nabble.com. -- Dmitry Blog: http://dmitrykan.blogspot.com Twitter: http://twitter.com/dmitrykan
Re: Fixing corrupted index?
1. No, if IndexReader is on I get the same error message from checkindex 2. It doesnt do any thing but giving that error message I posted before then quit. The full print of the error trace is: Opening index @ E:\...\zookeeper\solr\collec tion1\data\index ERROR: could not read any segments file in directory java.io.FileNotFoundException: E:\...\zookee per\solr\collection1\data\index\segments_b5tb (The system cannot find the file s pecified) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.init(Unknown Source) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:22 3) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:285) at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:347) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfo s.java:783) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfo s.java:630) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:343) at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:383) at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1777) -- View this message in context: http://lucene.472066.n3.nabble.com/Fixing-corrupted-index-tp4126644p4126837.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Fixing corrupted index?
right. If you have cfs files in the index directory, there is a thread discussing the method of regenerating the segment files: http://www.gossamer-threads.com/lists/lucene/java-user/39744 backup before doing changes! source on SO: http://stackoverflow.com/questions/9935177/how-to-repair-corrupted-lucene-index On Tue, Mar 25, 2014 at 11:57 AM, zqzuk ziqizh...@hotmail.co.uk wrote: 1. No, if IndexReader is on I get the same error message from checkindex 2. It doesnt do any thing but giving that error message I posted before then quit. The full print of the error trace is: Opening index @ E:\...\zookeeper\solr\collec tion1\data\index ERROR: could not read any segments file in directory java.io.FileNotFoundException: E:\...\zookee per\solr\collection1\data\index\segments_b5tb (The system cannot find the file s pecified) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.init(Unknown Source) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:22 3) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:285) at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:347) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfo s.java:783) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfo s.java:630) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:343) at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:383) at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1777) -- View this message in context: http://lucene.472066.n3.nabble.com/Fixing-corrupted-index-tp4126644p4126837.html Sent from the Solr - User mailing list archive at Nabble.com. -- Dmitry Blog: http://dmitrykan.blogspot.com Twitter: http://twitter.com/dmitrykan
Fixing corrupted index?
My Lucene index - built with Solr using Lucene4.1 - is corrupted. Upon trying to read the index using the following code I get org.apache.solr.common.SolrException: No such core: collection1 exception: File configFile = new File(cacheFolder + File.separator + solr.xml); CoreContainer container = new CoreContainer(cacheFolder, configFile); SolrServer server = new EmbeddedSolrServer(container, collection1); ModifiableSolrParams params = new ModifiableSolrParams(); params.set(q, idFieldName + : + ClientUtils.escapeQueryChars(queryId)); params.set(fl,idFieldName+,+valueFieldName); QueryResponse response = server.query(params) I used checkindex util to check the integrity of the index and it seems not able to perform the task by throwing the following error: Opening index @ /../solrindex_cache/zookeeper/solr/collection1/data/index ERROR: could not read any segments file in directory java.io.FileNotFoundException: /../solrindex_cache/zookeeper/solr/collection1/data/index/segments_b5tb (No such file or directory) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.init(RandomAccessFile.java:233) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:223) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:285) at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:347) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:630) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:343) at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:383) at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1777) The file segments_b5tb that index checker is looking for is indeed missing in the index folder. The only file that looks similar is segments.gen. However, the index segment files including .si, tip, doc, fdx etc still exist. Is there any way to fix this as it took me 2 weeks to build this index... Many many thanks for your kind advice! -- View this message in context: http://lucene.472066.n3.nabble.com/Fixing-corrupted-index-tp4126644.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Fixing corrupted index?
Hi, Have a look at: http://lucene.apache.org/core/4_1_0/core/org/apache/lucene/index/CheckIndex.html HTH, Dmitry On Mon, Mar 24, 2014 at 8:16 PM, zqzuk ziqizh...@hotmail.co.uk wrote: My Lucene index - built with Solr using Lucene4.1 - is corrupted. Upon trying to read the index using the following code I get org.apache.solr.common.SolrException: No such core: collection1 exception: File configFile = new File(cacheFolder + File.separator + solr.xml); CoreContainer container = new CoreContainer(cacheFolder, configFile); SolrServer server = new EmbeddedSolrServer(container, collection1); ModifiableSolrParams params = new ModifiableSolrParams(); params.set(q, idFieldName + : + ClientUtils.escapeQueryChars(queryId)); params.set(fl,idFieldName+,+valueFieldName); QueryResponse response = server.query(params) I used checkindex util to check the integrity of the index and it seems not able to perform the task by throwing the following error: Opening index @ /../solrindex_cache/zookeeper/solr/collection1/data/index ERROR: could not read any segments file in directory java.io.FileNotFoundException: /../solrindex_cache/zookeeper/solr/collection1/data/index/segments_b5tb (No such file or directory) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.init(RandomAccessFile.java:233) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:223) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:285) at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:347) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:630) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:343) at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:383) at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1777) The file segments_b5tb that index checker is looking for is indeed missing in the index folder. The only file that looks similar is segments.gen. However, the index segment files including .si, tip, doc, fdx etc still exist. Is there any way to fix this as it took me 2 weeks to build this index... Many many thanks for your kind advice! -- View this message in context: http://lucene.472066.n3.nabble.com/Fixing-corrupted-index-tp4126644.html Sent from the Solr - User mailing list archive at Nabble.com. -- Dmitry Blog: http://dmitrykan.blogspot.com Twitter: http://twitter.com/dmitrykan
Re: Fixing corrupted index?
Hi Thanks. But I am already using CheckIndex and the error is given by the CheckIndex utility: it could not even continue after reporting could not read any segements file in directory. -- View this message in context: http://lucene.472066.n3.nabble.com/Fixing-corrupted-index-tp4126644p4126687.html Sent from the Solr - User mailing list archive at Nabble.com.
corrupted index in slave?
(SolrDispatchFilter.java:639)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)\n\tat org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)\n\tat org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)\n\tat org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)\n\tat org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)\n\tat org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)\n\tat org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)\n\tat org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)\n\tat org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)\n\tat org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)\n\tat org.mortbay.jetty.Server.handle(Server.java:326)\n\tat org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)\n\tat org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926)\n\tat org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)\n\tat org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)\n\tat org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)\n\tat org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)\n\tat org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)\n,code:500}} Thanks in advance, regards Victor -- View this message in context: http://lucene.472066.n3.nabble.com/corrupted-index-in-slave-tp4054769.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: corrupted index in slave?
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)\n\tat org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)\n\tat org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)\n\tat org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)\n\tat org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)\n\tat org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)\n\tat org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)\n\tat org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)\n\tat org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)\n\tat org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)\n\tat org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)\n\tat org.mortbay.jetty.Server.handle(Server.java:326)\n\tat org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)\n\tat org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926)\n\tat org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)\n\tat org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)\n\tat org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)\n\tat org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)\n\tat org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)\n,code:500}} Thanks in advance, regards Victor -- View this message in context: http://lucene.472066.n3.nabble.com/corrupted-index-in-slave-tp4054769p4054772.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Avoiding corrupted index
: First, I tried the scripts provided in the Solr distribution without success ... : And that's true : there is no /opt/apache-solr-1.4.1/src/bin/scripts-util : but a /opt/apache-solr-1.4.1/src/scripts/scripts-util : Is this normal to distribute the scripts with a bad path ? it looks like you are trying to run the scripts from the src directory of the distro ... they are ment to be installed in a bin directory in your solr home dir (so they can locate the default data dir, etc...) If you haven't seen them already... http://wiki.apache.org/solr/CollectionDistribution http://wiki.apache.org/solr/SolrCollectionDistributionScripts : Then I discovered that these utility scripts were not distributed anymore : with the version 3.1.0 : were they not reliable ? can we get corrupted : backups with this scripts ? no, as far as i know they work great. they were not included in the *binary* distributions of Solr, but they were most certianly included in the *source* distributions ... i think that was actually an oversight ... 3.1 is hte first time we had a binary distibution, and there's no reason i know of why they shouldn't have been in both. (in general, these scripts have fallen out of favor because they aren't as portable or as easy to test as the java based replication, so they are easy to forget) -Hoss
Avoiding corrupted index
Hi everyone, We are using Solr 1.4.1 in my company and we need to do some backups of the indexes. After some googling, I'm quite confused about the differents ways of backing up the index. First, I tried the scripts provided in the Solr distribution without success : I untarred the apache-solr-1.4.1.tar.gz into /opt; then I launched but I get this error : $ /opt/apache-solr-1.4.1/src/scripts/backup /opt/apache-solr-1.4.1/src/scripts/backup: line 26: /opt/apache-solr-1.4.1/src/bin/scripts-util: No such file or directory And that's true : there is no /opt/apache-solr-1.4.1/src/bin/scripts-util but a /opt/apache-solr-1.4.1/src/scripts/scripts-util Is this normal to distribute the scripts with a bad path ? Then I discovered that these utility scripts were not distributed anymore with the version 3.1.0 : were they not reliable ? can we get corrupted backups with this scripts ? Finally, we found the page about SolrReplication on the Solr wiki also this post http://stackoverflow.com/questions/3083314/solr-incremental-backup-on-real-time-system-with-heavy-indexand in particular the answer advising to use the replication. So we tried to use this replication mecanism (and call the URL on the slave with the query parameters command=backup and location=/backup) but this method requires lots of i/o for big index. Is it the best way to get not corrupted backup of the index ? Is there another way to do the backup with Solr 3.1 ? Thanks in advance for your time. Regards, Laurent
Corrupted Index
Hi all, Our application uses solrj to communicate with our solr servers. We started a fresh index yesterday after upping the maxFieldLength setting in solrconfig. Our task indexes content in batches and all appeared to be well until noonish today, when after 40k docs, I started seeing errors. I've placed three stack traces below, the first occurred once and was the initial error, the second occurred a few times before the third started occurring on each request. I'd really appreciate any insight into what could have caused this, a missing file and then a corrupt index. If you know we'll have to nuke the entire index and start over I'd like to know that too-oddly enough searches against the index appear to be working. Thanks! Jake #1 January 7, 2010 12:10:06 PM CST Caught error; TaskWrapper block 1 January 7, 2010 12:10:07 PM CST solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update January 7, 2010 12:10:07 PM CST solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update org.benetech.exception.WrappedException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424) org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(243) org.apache.solr.client.solrj.request.AbstractUpdateRequest#process(105) org.apache.solr.client.solrj.SolrServer#commit(86) org.apache.solr.client.solrj.SolrServer#commit(75) org.bookshare.search.solr.SolrSearchServerWrapper#add(63) org.bookshare.search.solr.SolrSearchEngine#index(232) org.bookshare.service.task.SearchEngineIndexingTask#initialInstanceLoad(95) org.bookshare.service.task.SearchEngineIndexingTask#run(53) org.bookshare.service.scheduler.TaskWrapper#run(233) java.util.TimerThread#mainLoop(512) java.util.TimerThread#run(462) Caused by: solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update org.apache.solr.common.SolrException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424) org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(243) org.apache.solr.client.solrj.request.AbstractUpdateRequest#process(105) org.apache.solr.client.solrj.SolrServer#commit(86) org.apache.solr.client.solrj.SolrServer#commit(75) org.bookshare.search.solr.SolrSearchServerWrapper#add(63) org.bookshare.search.solr.SolrSearchEngine#index(232) org.bookshare.service.task.SearchEngineIndexingTask#initialInstanceLoad(95) org.bookshare.service.task.SearchEngineIndexingTask#run(53) org.bookshare.service.scheduler.TaskWrapper#run(233) java.util.TimerThread#mainLoop(512) java.util.TimerThread#run(462) #2 January 7, 2010 12:10:10 PM CST Caught error; TaskWrapper block 1 January 7, 2010 12:10:10 PM CST org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update January 7, 2010 12:10:10 PM CST org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.benetech.exception.WrappedException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424) org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(243) org.apache.solr.client.solrj.request.AbstractUpdateRequest#process(105)
Re: Corrupted Index
what version of solr are you running? On Jan 7, 2010, at 3:08 PM, Jake Brownell wrote: Hi all, Our application uses solrj to communicate with our solr servers. We started a fresh index yesterday after upping the maxFieldLength setting in solrconfig. Our task indexes content in batches and all appeared to be well until noonish today, when after 40k docs, I started seeing errors. I've placed three stack traces below, the first occurred once and was the initial error, the second occurred a few times before the third started occurring on each request. I'd really appreciate any insight into what could have caused this, a missing file and then a corrupt index. If you know we'll have to nuke the entire index and start over I'd like to know that too-oddly enough searches against the index appear to be working. Thanks! Jake #1 January 7, 2010 12:10:06 PM CST Caught error; TaskWrapper block 1 January 7, 2010 12:10:07 PM CST solr-home/core0/data/index/ _fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update January 7, 2010 12:10:07 PM CST solr-home/core0/data/index/ _fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update org.benetech.exception.WrappedException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424) org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(243) org .apache.solr.client.solrj.request.AbstractUpdateRequest#process(105) org.apache.solr.client.solrj.SolrServer#commit(86) org.apache.solr.client.solrj.SolrServer#commit(75) org.bookshare.search.solr.SolrSearchServerWrapper#add(63) org.bookshare.search.solr.SolrSearchEngine#index(232) org .bookshare .service.task.SearchEngineIndexingTask#initialInstanceLoad(95) org.bookshare.service.task.SearchEngineIndexingTask#run(53) org.bookshare.service.scheduler.TaskWrapper#run(233) java.util.TimerThread#mainLoop(512) java.util.TimerThread#run(462) Caused by: solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update org.apache.solr.common.SolrException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424) org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(243) org .apache.solr.client.solrj.request.AbstractUpdateRequest#process(105) org.apache.solr.client.solrj.SolrServer#commit(86) org.apache.solr.client.solrj.SolrServer#commit(75) org.bookshare.search.solr.SolrSearchServerWrapper#add(63) org.bookshare.search.solr.SolrSearchEngine#index(232) org .bookshare .service.task.SearchEngineIndexingTask#initialInstanceLoad(95) org.bookshare.service.task.SearchEngineIndexingTask#run(53) org.bookshare.service.scheduler.TaskWrapper#run(233) java.util.TimerThread#mainLoop(512) java.util.TimerThread#run(462) #2 January 7, 2010 12:10:10 PM CST Caught error; TaskWrapper block 1 January 7, 2010 12:10:10 PM CST org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update January 7, 2010 12:10:10 PM CST org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.benetech.exception.WrappedException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424)
RE: Corrupted Index
Yes, that would be helpful to include, sorry, the official 1.4. -Original Message- From: Ryan McKinley [mailto:ryan...@gmail.com] Sent: Thursday, January 07, 2010 2:15 PM To: solr-user@lucene.apache.org Subject: Re: Corrupted Index what version of solr are you running? On Jan 7, 2010, at 3:08 PM, Jake Brownell wrote: Hi all, Our application uses solrj to communicate with our solr servers. We started a fresh index yesterday after upping the maxFieldLength setting in solrconfig. Our task indexes content in batches and all appeared to be well until noonish today, when after 40k docs, I started seeing errors. I've placed three stack traces below, the first occurred once and was the initial error, the second occurred a few times before the third started occurring on each request. I'd really appreciate any insight into what could have caused this, a missing file and then a corrupt index. If you know we'll have to nuke the entire index and start over I'd like to know that too-oddly enough searches against the index appear to be working. Thanks! Jake #1 January 7, 2010 12:10:06 PM CST Caught error; TaskWrapper block 1 January 7, 2010 12:10:07 PM CST solr-home/core0/data/index/ _fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update January 7, 2010 12:10:07 PM CST solr-home/core0/data/index/ _fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update org.benetech.exception.WrappedException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424) org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(243) org .apache.solr.client.solrj.request.AbstractUpdateRequest#process(105) org.apache.solr.client.solrj.SolrServer#commit(86) org.apache.solr.client.solrj.SolrServer#commit(75) org.bookshare.search.solr.SolrSearchServerWrapper#add(63) org.bookshare.search.solr.SolrSearchEngine#index(232) org .bookshare .service.task.SearchEngineIndexingTask#initialInstanceLoad(95) org.bookshare.service.task.SearchEngineIndexingTask#run(53) org.bookshare.service.scheduler.TaskWrapper#run(233) java.util.TimerThread#mainLoop(512) java.util.TimerThread#run(462) Caused by: solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update org.apache.solr.common.SolrException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424) org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(243) org .apache.solr.client.solrj.request.AbstractUpdateRequest#process(105) org.apache.solr.client.solrj.SolrServer#commit(86) org.apache.solr.client.solrj.SolrServer#commit(75) org.bookshare.search.solr.SolrSearchServerWrapper#add(63) org.bookshare.search.solr.SolrSearchEngine#index(232) org .bookshare .service.task.SearchEngineIndexingTask#initialInstanceLoad(95) org.bookshare.service.task.SearchEngineIndexingTask#run(53) org.bookshare.service.scheduler.TaskWrapper#run(233) java.util.TimerThread#mainLoop(512) java.util.TimerThread#run(462) #2 January 7, 2010 12:10:10 PM CST Caught error; TaskWrapper block 1 January 7, 2010 12:10:10 PM CST org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update January 7, 2010 12:10:10 PM CST org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2
Re: Corrupted Index
If you need to fix the index and maybe lose some data (in bad segments), check Lucene's CheckIndex (cmd-line app) Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch - Original Message From: Jake Brownell ja...@benetech.org To: solr-user@lucene.apache.org solr-user@lucene.apache.org Sent: Thu, January 7, 2010 3:08:55 PM Subject: Corrupted Index Hi all, Our application uses solrj to communicate with our solr servers. We started a fresh index yesterday after upping the maxFieldLength setting in solrconfig. Our task indexes content in batches and all appeared to be well until noonish today, when after 40k docs, I started seeing errors. I've placed three stack traces below, the first occurred once and was the initial error, the second occurred a few times before the third started occurring on each request. I'd really appreciate any insight into what could have caused this, a missing file and then a corrupt index. If you know we'll have to nuke the entire index and start over I'd like to know that too-oddly enough searches against the index appear to be working. Thanks! Jake #1 January 7, 2010 12:10:06 PM CST Caught error; TaskWrapper block 1 January 7, 2010 12:10:07 PM CST solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update January 7, 2010 12:10:07 PM CST solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update org.benetech.exception.WrappedException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424) org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(243) org.apache.solr.client.solrj.request.AbstractUpdateRequest#process(105) org.apache.solr.client.solrj.SolrServer#commit(86) org.apache.solr.client.solrj.SolrServer#commit(75) org.bookshare.search.solr.SolrSearchServerWrapper#add(63) org.bookshare.search.solr.SolrSearchEngine#index(232) org.bookshare.service.task.SearchEngineIndexingTask#initialInstanceLoad(95) org.bookshare.service.task.SearchEngineIndexingTask#run(53) org.bookshare.service.scheduler.TaskWrapper#run(233) java.util.TimerThread#mainLoop(512) java.util.TimerThread#run(462) Caused by: solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) solr-home/core0/data/index/_fsk_1uj.del (No such file or directory) request: /core0/update org.apache.solr.common.SolrException org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(424) org.apache.solr.client.solrj.impl.CommonsHttpSolrServer#request(243) org.apache.solr.client.solrj.request.AbstractUpdateRequest#process(105) org.apache.solr.client.solrj.SolrServer#commit(86) org.apache.solr.client.solrj.SolrServer#commit(75) org.bookshare.search.solr.SolrSearchServerWrapper#add(63) org.bookshare.search.solr.SolrSearchEngine#index(232) org.bookshare.service.task.SearchEngineIndexingTask#initialInstanceLoad(95) org.bookshare.service.task.SearchEngineIndexingTask#run(53) org.bookshare.service.scheduler.TaskWrapper#run(233) java.util.TimerThread#mainLoop(512) java.util.TimerThread#run(462) #2 January 7, 2010 12:10:10 PM CST Caught error; TaskWrapper block 1 January 7, 2010 12:10:10 PM CST org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update January 7, 2010 12:10:10 PM CST org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _hug: fieldsReader shows 8 but segmentInfo shows 2 request: /core0/update org.apache.lucene.index.CorruptIndexException: doc counts differ
