Re: [DISCUSS] Opening old indices for reading

2019-03-08 Thread Cassandra Targett
I have a question about Simon’s commit that he discussed in an earlier mail to this thread, found at https://github.com/apache/lucene-solr/commit/0c4c885214ef30627a01e320f9c861dc2521b752 I see the commit diffs and files changed in GitHub at the above URL, but one odd thing about it is that it d

Re: [DISCUSS] Opening old indices for reading

2019-01-31 Thread Adrien Grand
This looks reasonable to me. On Tue, Jan 29, 2019 at 4:23 PM Simon Willnauer wrote: > > thanks folks, > > these are all good points. I created a first cut of what I had in mind > [1] . It's relatively simple and from a java visibility perspective > the only change that a user can take advantage o

Re: [DISCUSS] Opening old indices for reading

2019-01-29 Thread Simon Willnauer
thanks folks, these are all good points. I created a first cut of what I had in mind [1] . It's relatively simple and from a java visibility perspective the only change that a user can take advantage of is this [2] and this [3] respectively. This would allow opening indices back to Lucene 7.0 give

Re: [DISCUSS] Opening old indices for reading

2019-01-25 Thread Michael McCandless
Another example is long ago Lucene allowed pos=-1 to be indexed and it caused all sorts of problems. We also stopped allowing positions close to Integer.MAX_VALUE (https://issues.apache.org/jira/browse/LUCENE-6382). Yet another is allowing negative vInts which are possible but horribly inefficien

Re: [DISCUSS] Opening old indices for reading

2019-01-25 Thread Adrien Grand
Agreed with Michael that setting expectations is going to be important. The thing that I would like to make sure is that we would never refrain from moving Lucene forward because of this feature. In particular, lucene-core should be free to make assumptions that are valid for N and N-1 indices with

Re: [DISCUSS] Opening old indices for reading

2019-01-24 Thread Michael Sokolov
+1 it makes sense to me; real world problems sometimes require messy solutions. I guess the alternative is everybody develops their own suite of tools and it is hard to share. Some caution is warranted though I think; even with misc/experimental caveats, these tools will only be useful if people c

Re: [DISCUSS] Opening old indices for reading

2019-01-23 Thread Erick Erickson
+1, A lot of this was discussed on SOLR-12259, we should probably link any Lucene JIRAs for this back to that one to make an easy trail to follow. One thing I'd thought of is whether we should merge segments during this operation. If we're going to rewrite the entire index anyway, does it make sen

Re: [DISCUSS] Opening old indices for reading

2019-01-23 Thread Andrzej Białecki
+1. I think that even with these caveats (read-only, some data may require re-interpretation) it would still be a great help for accessing legacy data, for which the original source may no longer exist. > On 23 Jan 2019, at 15:11, Simon Willnauer wrote: > > Hey folks, > > tl;dr; I want to be

[DISCUSS] Opening old indices for reading

2019-01-23 Thread Simon Willnauer
Hey folks, tl;dr; I want to be able to open an indexreader on an old index if the SegmentInfo version is supported and all segment codecs are available. Today that's not possible even if I port old formats to current versions. Our BWC policy for quite a while has been N-1 major versions. That's g