Re: Seek past EOF
We are using ext3 on Debian. Noticed today that i only need to reload the core to get it working again…. On 30 November 2011 19:59, Simon Willnauer simon.willna...@googlemail.comwrote: can you give us some details about what filesystem you are using? simon On Wed, Nov 30, 2011 at 3:07 PM, Ruben Chadien ruben.chad...@aspiro.com wrote: Happened again…. I got 3 directories in my index dir 4096 Nov 4 09:31 index.2004083156 4096 Nov 21 10:04 index.2021090440 4096 Nov 30 14:55 index.2029024919 as you can se the first two are old and also empty , the last one from today is and containing 9 files none of the are 0 size and total size 7 GB. The size of the index on the master is 14GB. Any ideas on what to look for ? Thanks Ruben Chadien On 29 November 2011 15:58, Mark Miller markrmil...@gmail.com wrote: Hmm...I've seen a bug like this, but I don't think it would be tickled if you are replicating config files... It def looks related though ... I'll try to dig around. Next time it happens, take a look on the slave for 0 size files - also if the index dir on the slave is plain 'index' or has a timestamp as part of the name (eg timestamp.index). On Tue, Nov 29, 2011 at 9:53 AM, Ruben Chadien ruben.chad...@aspiro.com wrote: Hi, for the moment there are no 0 sized files, but all indexes are working now. I will have to look next time it breaks. Yes, the directory name is index and it replicates the schema and a synonyms file. /Ruben Chadien On 29 November 2011 15:29, Mark Miller markrmil...@gmail.com wrote: Also, on your master, what is the name of the index directory? Just 'index'? And are you replicating config files as well or no? On Nov 29, 2011, at 9:23 AM, Mark Miller wrote: Does the problem index have any 0 size files in it? On Nov 29, 2011, at 2:54 AM, Ruben Chadien wrote: HI all After upgrading tol Solr 3.4 we are having trouble with the replication. The setup is one indexing master with a few slaves that replicate the indexes once every night. The largest index is 20 GB and the master and slaves are on the same DMZ. Almost every night one of the indexes (17 in total) fail after the replication with an EOF file. SEVERE: Error during auto-warming of key:org.apache.solr.search.QueryResultKey@bda006e3 :java.io.IOException: seek past EOF at org.apache.lucene.store.MMapDirectory$MMapIndexInput.seek(MMapDirectory.java:347) at org.apache.lucene.index.SegmentTermEnum.seek(SegmentTermEnum.java:114) at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:203) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:273) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:210) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:507) at org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:309) at org.apache.lucene.search.TermQuery$TermWeight$1.add(TermQuery.java:56) at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:77) at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:82) After a restart the errors are gone, anyone else seen this ? Thanks Ruben Chadien - Mark Miller lucidimagination.com - Mark Miller lucidimagination.com -- *Ruben Chadien *Senior Developer Mobile +47 900 35 371 ruben.chad...@aspiro.com * Aspiro Music AS* Øvre Slottsgate 25, P.O. Box 8710 Youngstorget, N-0028 Oslo Tel +47 452 86 900, fax +47 22 37 36 59 www.aspiro.com/music -- - Mark http://www.lucidimagination.com -- *Ruben Chadien *Senior Developer Mobile +47 900 35 371 ruben.chad...@aspiro.com * Aspiro Music AS* Øvre Slottsgate 25, P.O. Box 8710 Youngstorget, N-0028 Oslo Tel +47 452 86 900, fax +47 22 37 36 59 www.aspiro.com/music -- *Ruben Chadien *Senior Developer Mobile +47 900 35 371 ruben.chad...@aspiro.com * Aspiro Music AS* Øvre Slottsgate 25, P.O. Box 8710 Youngstorget, N-0028 Oslo Tel +47 452 86 900, fax +47 22 37 36 59 www.aspiro.com/music
Re: Seek past EOF
Happened again…. I got 3 directories in my index dir 4096 Nov 4 09:31 index.2004083156 4096 Nov 21 10:04 index.2021090440 4096 Nov 30 14:55 index.2029024919 as you can se the first two are old and also empty , the last one from today is and containing 9 files none of the are 0 size and total size 7 GB. The size of the index on the master is 14GB. Any ideas on what to look for ? Thanks Ruben Chadien On 29 November 2011 15:58, Mark Miller markrmil...@gmail.com wrote: Hmm...I've seen a bug like this, but I don't think it would be tickled if you are replicating config files... It def looks related though ... I'll try to dig around. Next time it happens, take a look on the slave for 0 size files - also if the index dir on the slave is plain 'index' or has a timestamp as part of the name (eg timestamp.index). On Tue, Nov 29, 2011 at 9:53 AM, Ruben Chadien ruben.chad...@aspiro.com wrote: Hi, for the moment there are no 0 sized files, but all indexes are working now. I will have to look next time it breaks. Yes, the directory name is index and it replicates the schema and a synonyms file. /Ruben Chadien On 29 November 2011 15:29, Mark Miller markrmil...@gmail.com wrote: Also, on your master, what is the name of the index directory? Just 'index'? And are you replicating config files as well or no? On Nov 29, 2011, at 9:23 AM, Mark Miller wrote: Does the problem index have any 0 size files in it? On Nov 29, 2011, at 2:54 AM, Ruben Chadien wrote: HI all After upgrading tol Solr 3.4 we are having trouble with the replication. The setup is one indexing master with a few slaves that replicate the indexes once every night. The largest index is 20 GB and the master and slaves are on the same DMZ. Almost every night one of the indexes (17 in total) fail after the replication with an EOF file. SEVERE: Error during auto-warming of key:org.apache.solr.search.QueryResultKey@bda006e3 :java.io.IOException: seek past EOF at org.apache.lucene.store.MMapDirectory$MMapIndexInput.seek(MMapDirectory.java:347) at org.apache.lucene.index.SegmentTermEnum.seek(SegmentTermEnum.java:114) at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:203) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:273) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:210) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:507) at org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:309) at org.apache.lucene.search.TermQuery$TermWeight$1.add(TermQuery.java:56) at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:77) at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:82) After a restart the errors are gone, anyone else seen this ? Thanks Ruben Chadien - Mark Miller lucidimagination.com - Mark Miller lucidimagination.com -- *Ruben Chadien *Senior Developer Mobile +47 900 35 371 ruben.chad...@aspiro.com * Aspiro Music AS* Øvre Slottsgate 25, P.O. Box 8710 Youngstorget, N-0028 Oslo Tel +47 452 86 900, fax +47 22 37 36 59 www.aspiro.com/music -- - Mark http://www.lucidimagination.com -- *Ruben Chadien *Senior Developer Mobile +47 900 35 371 ruben.chad...@aspiro.com * Aspiro Music AS* Øvre Slottsgate 25, P.O. Box 8710 Youngstorget, N-0028 Oslo Tel +47 452 86 900, fax +47 22 37 36 59 www.aspiro.com/music
Re: Seek past EOF
Hi, for the moment there are no 0 sized files, but all indexes are working now. I will have to look next time it breaks. Yes, the directory name is index and it replicates the schema and a synonyms file. /Ruben Chadien On 29 November 2011 15:29, Mark Miller markrmil...@gmail.com wrote: Also, on your master, what is the name of the index directory? Just 'index'? And are you replicating config files as well or no? On Nov 29, 2011, at 9:23 AM, Mark Miller wrote: Does the problem index have any 0 size files in it? On Nov 29, 2011, at 2:54 AM, Ruben Chadien wrote: HI all After upgrading tol Solr 3.4 we are having trouble with the replication. The setup is one indexing master with a few slaves that replicate the indexes once every night. The largest index is 20 GB and the master and slaves are on the same DMZ. Almost every night one of the indexes (17 in total) fail after the replication with an EOF file. SEVERE: Error during auto-warming of key:org.apache.solr.search.QueryResultKey@bda006e3:java.io.IOException: seek past EOF at org.apache.lucene.store.MMapDirectory$MMapIndexInput.seek(MMapDirectory.java:347) at org.apache.lucene.index.SegmentTermEnum.seek(SegmentTermEnum.java:114) at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:203) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:273) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:210) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:507) at org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:309) at org.apache.lucene.search.TermQuery$TermWeight$1.add(TermQuery.java:56) at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:77) at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:82) After a restart the errors are gone, anyone else seen this ? Thanks Ruben Chadien - Mark Miller lucidimagination.com - Mark Miller lucidimagination.com -- *Ruben Chadien *Senior Developer Mobile +47 900 35 371 ruben.chad...@aspiro.com * Aspiro Music AS* Øvre Slottsgate 25, P.O. Box 8710 Youngstorget, N-0028 Oslo Tel +47 452 86 900, fax +47 22 37 36 59 www.aspiro.com/music
Seek past EOF
HI all After upgrading tol Solr 3.4 we are having trouble with the replication. The setup is one indexing master with a few slaves that replicate the indexes once every night. The largest index is 20 GB and the master and slaves are on the same DMZ. Almost every night one of the indexes (17 in total) fail after the replication with an EOF file. SEVERE: Error during auto-warming of key:org.apache.solr.search.QueryResultKey@bda006e3:java.io.IOException: seek past EOF at org.apache.lucene.store.MMapDirectory$MMapIndexInput.seek(MMapDirectory.java:347) at org.apache.lucene.index.SegmentTermEnum.seek(SegmentTermEnum.java:114) at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:203) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:273) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:210) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:507) at org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:309) at org.apache.lucene.search.TermQuery$TermWeight$1.add(TermQuery.java:56) at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:77) at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:82) After a restart the errors are gone, anyone else seen this ? Thanks Ruben Chadien
expungeDeletes on commit in Dataimport
Hi I know this has been discussed before, but is there any way do expungeDeletes=true when the DataImportHandler does the commit. I am using the deleteDocByQuery in a Transformer when doing a delta-import and as discussed before the documents are not deleted until restart. Also, how do i know in a Transformer if its running a Delta or Full Import , i tries looking at Context. currentProcess() but that gives me FULL_DUMP when doing a delta import...? Thanks! Ruben Chadien
Re: DataImport, remove doc when marked as deleted
I have now :-) Thanks , missed that in the Wiki. Ruben On Apr 16, 2009, at 7:10 PM, Noble Paul നോബിള് नोब्ळ् wrote: did you try the deletedPkQuery? On Thu, Apr 16, 2009 at 7:49 PM, Ruben Chadien ruben.chad...@aspiro.com wrote: Hi I am new to Solr, but have been using Lucene for a while. I am trying to rewrite some old lucene indexing code using the Jdbc DataImport i Solr, my problem: I have Entities that can be marked in the db as deleted, these i don't want to index and thats no problem when doing a full-import. When doing a delta- import my deltaQuery will catch Entities that has been marked as deleted since last index, but how do i get it to delete those from the index ? I tried making the deltaImportQuery so that in don't return the Entity if its deleted, that didnt help... Any ideas ? Thanks Ruben -- --Noble Paul
DataImport, remove doc when marked as deleted
Hi I am new to Solr, but have been using Lucene for a while. I am trying to rewrite some old lucene indexing code using the Jdbc DataImport i Solr, my problem: I have Entities that can be marked in the db as deleted, these i don't want to index and thats no problem when doing a full-import. When doing a delta- import my deltaQuery will catch Entities that has been marked as deleted since last index, but how do i get it to delete those from the index ? I tried making the deltaImportQuery so that in don't return the Entity if its deleted, that didnt help... Any ideas ? Thanks Ruben
