Re: Index compatibility 1.4 Vs 3.1 Trunk
: We will only provide a conversion tool that can convert indexes from : the last branch_3x up to this trunk (4.0) release, so they can be : read later, but may not contain terms with all current analyzers, so : people need mostly reindexing. Older indexes will not be able to be : read natively without conversion first (with maybe loss of analyzer : compatibility). ... : i dont see anything suggesting we should support any tools for 2.x indexes! I read the sentence that sayd Older indexes will not be able to be read natively without conversion first sentence and assumed that was talking about ... i dunno, older indexes. I certainly don't pretend to be an expert on an index converstion tool thta doesn't even exist yet, but the point i was trying to make in response to the users question... Is re-indexing my only option or is there a tool of some sort to convert the 1.4 index to 3.1 format Index format versions change over time, and support for older formats is phased out. Solr 3.1 should be able to read your Solr 1.4 index and convert it automaticly. Solr 4.x will *not* be be able to automaticly read indexes created by Solr 1.4, but there will be a process for 'converting' your indexes. Unlike previous instances where the index format version changed when upgrading Solr, starting with Solr 4.0 this conversion will not be automatic, and will require the use of a tool. Does that sound more accurate based on your understanding? that even if the yet-to-be-built-index-conversion-tool for Solr/Lucene 4.0 can't read Solr1.4/Lucene2.9 indexes, there will still be an upgrade process from Solr 1.4 to Solr 4.0 (mainly: upgrade to Solr 3.x, let it auto convert your indexes, and then ultimately use the 4.0 conversation tool to explicitly convert to Solr 4.0) ? ? -Hoss
Re: Index compatibility 1.4 Vs 3.1 Trunk
: : That should still be true in the the official 4.0 release (i really should : have said When 4.0 can no longer read SOlr 1.4 indexes), ... : i havne't been following the detials closely, but i suspect that tool : hasn't been writen yet because there isn't much point until the full : details of the trunk index format are nailed down. : This is news to me? : : File formats are back-compatible between major versions. Version X.N should : be able to read indexes generated by any version after and including version : X-1.0, but may-or-may-not be able to read indexes generated by version : X-2.N. It was a big part of the proposal regarding hte creation of hte 3x branch ... that index format compabtibility between major versions would no longer be supported by silently converted on first write -- instead there there would be a tool for explicit conversion... http://search.lucidimagination.com/search/document/c10057266d3471c6/proposal_about_version_api_relaxation http://search.lucidimagination.com/search/document/c494a78f1ec1bfb5/lucene_3_x_branch_created -Hoss
Re: Index compatibility 1.4 Vs 3.1 Trunk
On Thu, Aug 12, 2010 at 8:29 PM, Chris Hostetter hossman_luc...@fucit.orgwrote: It was a big part of the proposal regarding hte creation of hte 3x branch ... that index format compabtibility between major versions would no longer be supported by silently converted on first write -- instead there there would be a tool for explicit conversion... http://search.lucidimagination.com/search/document/c10057266d3471c6/proposal_about_version_api_relaxation http://search.lucidimagination.com/search/document/c494a78f1ec1bfb5/lucene_3_x_branch_created Hoss, did you actually *read* these documents We will only provide a conversion tool that can convert indexes from the last branch_3x up to this trunk (4.0) release, so they can be read later, but may not contain terms with all current analyzers, so people need mostly reindexing. Older indexes will not be able to be read natively without conversion first (with maybe loss of analyzer compatibility). the fact 4.0 can read 3.x indexes *at all* without a converter tool is only because Mike Mccandless went the extra mile. i dont see anything suggesting we should support any tools for 2.x indexes! -- Robert Muir rcm...@gmail.com
Re: Index compatibility 1.4 Vs 3.1 Trunk
Hello Mr. Horsetter, I again tried the code from trunk ' https://svn.apache.org/repos/asf/lucene/dev/trunk' on solr 1.4 index and it gave me the following IndexFormatTooOldExceptio which in the first place prompted me to think the indexes are incompatible. Any ideas ? java.lang.RuntimeException: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported in file '_1d60.fdx': 1 (needs to be between 2 and 2). This version of Lucene only supports indexes created with release 3.0 and later. at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1067) at org.apache.solr.core.SolrCore.init(SolrCore.java:582) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:453) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:308) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:198) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:123) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:86) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:273) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:385) at org.apache.catalina.core.ApplicationFilterConfig.init(ApplicationFilterConfig.java:119) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4529) at org.apache.catalina.core.StandardContext.start(StandardContext.java:5348) at com.sun.enterprise.web.WebModule.start(WebModule.java:353) at com.sun.enterprise.web.LifecycleStarter.doRun(LifecycleStarter.java:58) at com.sun.appserv.management.util.misc.RunnableBase.runSync(RunnableBase.java:304) at com.sun.appserv.management.util.misc.RunnableBase.run(RunnableBase.java:341) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported in file '_1d60.fdx': 1 (needs to be between 2 and 2). This version of Lucene only supports indexes created with release 3.0 and later. at org.apache.lucene.index.FieldsReader.init(FieldsReader.java:109) at org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:242) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:523) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:494) at org.apache.lucene.index.DirectoryReader.init(DirectoryReader.java:133) at org.apache.lucene.index.ReadOnlyDirectoryReader.init(ReadOnlyDirectoryReader.java:28) at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:98) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:630) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) at org.apache.lucene.index.IndexReader.open(IndexReader.java:415) at org.apache.lucene.index.IndexReader.open(IndexReader.java:294) at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1056) ... 21 more Ravi Kiran Bhaskar On Tue, Aug 3, 2010 at 11:15 AM, Ravi Kiran ravi.bhas...@gmail.com wrote: Hello Mr.Hostetter, Thank you very much for the clarification. I do remember that when I first deployed the solr code from trunk on a test server I couldnt open the index (created via 1.4) even via the solr admin page, It kept giving me corrupted index EOF kind of exception, so I was curious. Let me try it out again and report to you with the exact error. On Mon, Aug 2, 2010 at 4:28 PM, Chris Hostetter hossman_luc...@fucit.orgwrote: : I am trying to use the solr code from ' : https://svn.apache.org/repos/asf/lucene/dev/trunk' as my design warrants use : of PolyType fields. My understanding is that the indexes are incompatible, : am I right ?. I have about a million docs in my index (indexed via solr : 1.4). Is re-indexing my only option or is there a tool of some sort to : convert the 1.4 index to 3.1 format ? a) the trunk is what will ultimately be Solr 4.x, not 3.x ... for the 3.x line there is a 3x branch... http://wiki.apache.org/solr/Solr3.1 http://wiki.apache.org/solr/Solr4.0 b) The 3x branch can read indexes created by Solr 1.4 -- the first time you add a doc and commit the new segments wil automaticly be converted to the new format. I am fairly certian that as of this moment, the 4x trunk can also read indexes created by Solr 1.4, with the same automatic converstion taking place. c) If/When the trunk can no longer read Solr 1.4 indexes, there will be a tool provided
Re: Index compatibility 1.4 Vs 3.1 Trunk
: Hello Mr. Horsetter, Please, call me Hoss. Mr. Horsetter is ... well frankly i have no idea who that is. : I again tried the code from trunk ' : https://svn.apache.org/repos/asf/lucene/dev/trunk' on solr 1.4 index and it Please note my previous comments... : a) the trunk is what will ultimately be Solr 4.x, not 3.x ... for the : 3.x line there is a 3x branch... : : http://wiki.apache.org/solr/Solr3.1 : http://wiki.apache.org/solr/Solr4.0 : : b) The 3x branch can read indexes created by Solr 1.4 -- the first time : you add a doc and commit the new segments wil automaticly be converted to : the new format. I am fairly certian that as of this moment, the 4x trunk : can also read indexes created by Solr 1.4, with the same automatic : converstion taking place. ...aparently i was mistaken about trunk that has already had the code for reading Lucene 2.9 indexes (what's used in Solr 1.4) removed (hence the IndexFormatTooOldException. But that doens't change hte fact that 3.1 will be able to read Solr 1.4 indexes. And 4.0 will be able to read 3.1 indexes. You should, infact, be able to use the 3x branch code today to open your SOlr 1.4 index, add one document to have it convert to a 3x index. then use the trunk code to open that index, add one doucment, andh ave it convert to a trunk index Of course: there is no garuntee that index format in the official 4.0 index format will be the same as what's on trunk right now -- it hasn't been officially released. : c) If/When the trunk can no longer read Solr 1.4 indexes, there will be : a tool provided for upgrading index versions. That should still be true in the the official 4.0 release (i really should have said When 4.0 can no longer read SOlr 1.4 indexes), ... i havne't been following the detials closely, but i suspect that tool hasn't been writen yet because there isn't much point until the full details of the trunk index format are nailed down. -Hoss
Re: Index compatibility 1.4 Vs 3.1 Trunk
On Thu, Aug 5, 2010 at 9:07 PM, Chris Hostetter hossman_luc...@fucit.orgwrote: That should still be true in the the official 4.0 release (i really should have said When 4.0 can no longer read SOlr 1.4 indexes), ... i havne't been following the detials closely, but i suspect that tool hasn't been writen yet because there isn't much point until the full details of the trunk index format are nailed down. This is news to me? File formats are back-compatible between major versions. Version X.N should be able to read indexes generated by any version after and including version X-1.0, but may-or-may-not be able to read indexes generated by version X-2.N. (And personally I think there is stuff in 2.x like modified-utf8 that i would object to adding support for with terms now as byte[]) -- Robert Muir rcm...@gmail.com
Re: Index compatibility 1.4 Vs 3.1 Trunk
Hello Mr.Hostetter, Thank you very much for the clarification. I do remember that when I first deployed the solr code from trunk on a test server I couldnt open the index (careted via 1.4) even via the solr admin page, It kept giving me corrupted index EOF kind of exception, so I was curious. Let me try it out again and report to you with the exact error. On Mon, Aug 2, 2010 at 4:28 PM, Chris Hostetter hossman_luc...@fucit.orgwrote: : I am trying to use the solr code from ' : https://svn.apache.org/repos/asf/lucene/dev/trunk' as my design warrants use : of PolyType fields. My understanding is that the indexes are incompatible, : am I right ?. I have about a million docs in my index (indexed via solr : 1.4). Is re-indexing my only option or is there a tool of some sort to : convert the 1.4 index to 3.1 format ? a) the trunk is what will ultimately be Solr 4.x, not 3.x ... for the 3.x line there is a 3x branch... http://wiki.apache.org/solr/Solr3.1 http://wiki.apache.org/solr/Solr4.0 b) The 3x branch can read indexes created by Solr 1.4 -- the first time you add a doc and commit the new segments wil automaticly be converted to the new format. I am fairly certian that as of this moment, the 4x trunk can also read indexes created by Solr 1.4, with the same automatic converstion taking place. c) If/When the trunk can no longer read Solr 1.4 indexes, there will be a tool provided for upgrading index versions. -Hoss
Re: Index compatibility 1.4 Vs 3.1 Trunk
: I am trying to use the solr code from ' : https://svn.apache.org/repos/asf/lucene/dev/trunk' as my design warrants use : of PolyType fields. My understanding is that the indexes are incompatible, : am I right ?. I have about a million docs in my index (indexed via solr : 1.4). Is re-indexing my only option or is there a tool of some sort to : convert the 1.4 index to 3.1 format ? a) the trunk is what will ultimately be Solr 4.x, not 3.x ... for the 3.x line there is a 3x branch... http://wiki.apache.org/solr/Solr3.1 http://wiki.apache.org/solr/Solr4.0 b) The 3x branch can read indexes created by Solr 1.4 -- the first time you add a doc and commit the new segments wil automaticly be converted to the new format. I am fairly certian that as of this moment, the 4x trunk can also read indexes created by Solr 1.4, with the same automatic converstion taking place. c) If/When the trunk can no longer read Solr 1.4 indexes, there will be a tool provided for upgrading index versions. -Hoss
