Re: upcoming Solr release on Lucene 2.9-dev

2009-05-28 Thread Shai Erera
If by changes you also mean deprecated features, then take a look at LUCENE-1614 - if you have your own Scorers/DISIs, you might want to implement the new methods, since the current ones are deprecated. On Thu, May 28, 2009 at 1:23 AM, Yonik Seeley yo...@lucidimagination.comwrote: We're aiming

[jira] Updated: (LUCENE-1614) Add next() and skipTo() variants to DocIdSetIterator that return the current doc, instead of boolean

2009-05-28 Thread Shai Erera (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shai Erera updated LUCENE-1614: --- Attachment: LUCENE-1614.patch Deprecated SpanScorer.firstTime and removed the use of it. (in

[jira] Commented: (LUCENE-1252) Avoid using positions when not all required terms are present

2009-05-28 Thread Shai Erera (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12713918#action_12713918 ] Shai Erera commented on LUCENE-1252: I must admit that I read the issue briefly, and

Re: upcoming Solr release on Lucene 2.9-dev

2009-05-28 Thread Michael McCandless
The IndexWriter diagnostics (LUCENE-1654: recording Lucene version, Java/OS version, etc into each segment created) also bumped the index file format. And LUCENE-1623 (fixing back-compat issue w/ field names that have non-ascii characters) also bumped the format of the fnm file. Offhand I can't

[jira] Commented: (LUCENE-1252) Avoid using positions when not all required terms are present

2009-05-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12713928#action_12713928 ] Michael McCandless commented on LUCENE-1252: I agree, we would want to do a

[jira] Commented: (LUCENE-1614) Add next() and skipTo() variants to DocIdSetIterator that return the current doc, instead of boolean

2009-05-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12713931#action_12713931 ] Michael McCandless commented on LUCENE-1614: {quote} After I made add()

[jira] Resolved: (LUCENE-1657) Make boolean readOnly a required arg to IndexReader.open

2009-05-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-1657. Resolution: Fixed Make boolean readOnly a required arg to IndexReader.open

[jira] Resolved: (LUCENE-1653) Change DateTools to not create a Calendar in every call to dateToString or timeToString

2009-05-28 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller resolved LUCENE-1653. - Resolution: Fixed Lucene Fields: [New, Patch Available] (was: [New]) Thanks Shai!

Re: upcoming Solr release on Lucene 2.9-dev

2009-05-28 Thread Yonik Seeley
On Thu, May 28, 2009 at 5:35 AM, Michael McCandless luc...@mikemccandless.com wrote: The IndexWriter diagnostics (LUCENE-1654: recording Lucene version, Java/OS version, etc into each segment created) also bumped the index file format. And LUCENE-1623 (fixing back-compat issue w/ field names

Re: upcoming Solr release on Lucene 2.9-dev

2009-05-28 Thread Yonik Seeley
On Thu, May 28, 2009 at 2:56 AM, Shai Erera ser...@gmail.com wrote: If by changes you also mean deprecated features, then take a look at LUCENE-1614 - if you have your own Scorers/DISIs, you might want to implement the new methods, since the current ones are deprecated. Yes, we have our own

[jira] Created: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Renaud Delbru (JIRA)
Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST -- Key: LUCENE-1666 URL:

[jira] Updated: (LUCENE-1542) Lucene can incorrectly set the position of tokens that start a field with positonInc 0.

2009-05-28 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated LUCENE-1542: Summary: Lucene can incorrectly set the position of tokens that start a field with positonInc 0.

[jira] Updated: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Renaud Delbru (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renaud Delbru updated LUCENE-1666: -- Attachment: LUCENE-1666.patch Constants causing NullPointerException when fetching metadata

[jira] Updated: (LUCENE-1542) Lucene can incorrectly set the position of tokens that start a field with positonInc 0.

2009-05-28 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated LUCENE-1542: Attachment: LUCENE-1542.patch something like this to fix Lucene can incorrectly set the

[jira] Commented: (LUCENE-1542) Lucene can incorrectly set the position of tokens that start a field with positonInc 0.

2009-05-28 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12713999#action_12713999 ] Mark Miller commented on LUCENE-1542: - so that appears to fix it - but i'm not sure

Re: upcoming Solr release on Lucene 2.9-dev

2009-05-28 Thread Grant Ingersoll
On May 28, 2009, at 10:24 AM, Yonik Seeley wrote: removing all the deprecated stuff related to Token/TokenStream is for another day... I think Lucene needs to re-look at the TokenStream stuff anyway. I seem to recall talking with Michael B. about changing it at ApacheCon. I'd be

[jira] Assigned: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless reassigned LUCENE-1666: -- Assignee: Michael McCandless Constants causing NullPointerException when

[jira] Resolved: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-1666. Resolution: Fixed Woops, thanks Renaud -- I just committed this. Constants

[jira] Commented: (LUCENE-1542) Lucene can incorrectly set the position of tokens that start a field with positonInc 0.

2009-05-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714020#action_12714020 ] Michael McCandless commented on LUCENE-1542: Alas, this looks like a dup of

[jira] Updated: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1666: -- Attachment: LUCENE-1666.patch Mike, you were a little bit too fast :-) I worked out a better

[jira] Updated: (LUCENE-1542) Lucene can incorrectly set the position of tokens that start a field with positonInc 0.

2009-05-28 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated LUCENE-1542: Attachment: LUCENE-1542.patch with unit test Lucene can incorrectly set the position of tokens

[jira] Updated: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1666: -- Attachment: LUCENE-1666.patch Oh forget the -dev thing... Attached is fixed patch.

[jira] Updated: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1666: -- Attachment: (was: LUCENE-1666.patch) Constants causing NullPointerException when

[jira] Commented: (LUCENE-1542) Lucene can incorrectly set the position of tokens that start a field with positonInc 0.

2009-05-28 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714033#action_12714033 ] Mark Miller commented on LUCENE-1542: - I don't think the fix here needs to disallow

[jira] Commented: (LUCENE-1255) CheckIndex should allow term position = -1

2009-05-28 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714039#action_12714039 ] Mark Miller commented on LUCENE-1255: - why should the first token have an increment of

[jira] Commented: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714046#action_12714046 ] Michael McCandless commented on LUCENE-1666: Much simpler, thanks Uwe! But,

[jira] Updated: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1666: -- Attachment: LUCENE-1666.patch I think the check should be simply if getPackage() returns null

Jonathan Mamou is out of the office.

2009-05-28 Thread Jonathan Mamou
I will be out of the office starting 27/05/2009 and will not return until 31/05/2009. I will respond to your message when I return. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands,

[jira] Commented: (LUCENE-1542) Lucene can incorrectly set the position of tokens that start a field with positonInc 0.

2009-05-28 Thread Shai Erera (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714065#action_12714065 ] Shai Erera commented on LUCENE-1542: Just wanted to say we've had an internal

[jira] Commented: (LUCENE-1654) Include diagnostics per-segment when writing a new segment

2009-05-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714068#action_12714068 ] Uwe Schindler commented on LUCENE-1654: --- I found a bug for indexes with the

[jira] Commented: (LUCENE-1666) Constants causing NullPointerException when fetching metadata Implementation Version in MANIFEST

2009-05-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714074#action_12714074 ] Michael McCandless commented on LUCENE-1666: Excellent -- I just committed

[jira] Issue Comment Edited: (LUCENE-1654) Include diagnostics per-segment when writing a new segment

2009-05-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714068#action_12714068 ] Uwe Schindler edited comment on LUCENE-1654 at 5/28/09 10:50 AM:

[jira] Commented: (LUCENE-1654) Include diagnostics per-segment when writing a new segment

2009-05-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714087#action_12714087 ] Michael McCandless commented on LUCENE-1654: Woops -- thanks Uwe! Include

NRT getReader turnaround on large segments

2009-05-28 Thread Jason Rutherglen
An interesting discussion came up. How do we handle IW.getReader turnaround time on large new segments?

Re: NRT getReader turnaround on large segments

2009-05-28 Thread Michael McCandless
This is exactly why we added IndexReaderWarmer -- it pre-warms a newly merged segment before committing to SegmentInfos. So, while such warming is happening, if getReader() is called, the returned reader will still read the old segments. Mike On Thu, May 28, 2009 at 3:06 PM, Jason Rutherglen

Re: NRT getReader turnaround on large segments

2009-05-28 Thread Jason Rutherglen
And warming a segment in mergeMiddle doesn't block the addition of new segments. On Thu, May 28, 2009 at 12:30 PM, Michael McCandless luc...@mikemccandless.com wrote: This is exactly why we added IndexReaderWarmer -- it pre-warms a newly merged segment before committing to SegmentInfos. So,

Re: NRT getReader turnaround on large segments

2009-05-28 Thread Michael McCandless
Right. If you play w/ this please report back on how it goes! Mike On Thu, May 28, 2009 at 3:38 PM, Jason Rutherglen jason.rutherg...@gmail.com wrote: And warming a segment in mergeMiddle doesn't block the addition of new segments. On Thu, May 28, 2009 at 12:30 PM, Michael McCandless

Re: NRT getReader turnaround on large segments

2009-05-28 Thread Jason Rutherglen
Also important to note that only with ConcurrentMergeScheduler will the segmentReader warming occur in the background. When using SerialMergeScheduler, the warming will happen in the foreground. On Thu, May 28, 2009 at 12:40 PM, Michael McCandless luc...@mikemccandless.com wrote: Right. If

[jira] Created: (LUCENE-1667) ConcurrentMergeScheduler use a thread pool (per directory)

2009-05-28 Thread Jason Rutherglen (JIRA)
ConcurrentMergeScheduler use a thread pool (per directory) -- Key: LUCENE-1667 URL: https://issues.apache.org/jira/browse/LUCENE-1667 Project: Lucene - Java Issue Type: Improvement

IndexReaderFactory for IndexWriter LUCENE-1516

2009-05-28 Thread Jason Rutherglen
Some folks may want to have IW.getReader (LUCENE-1516) use custom readers underneath, we can have IW support an IndexReaderFactory?

Re: NRT getReader turnaround on large segments

2009-05-28 Thread Yonik Seeley
On Thu, May 28, 2009 at 3:30 PM, Michael McCandless luc...@mikemccandless.com wrote: This is exactly why we added IndexReaderWarmer -- it pre-warms a newly merged segment before committing to SegmentInfos. So, while such warming is happening, if getReader() is called, the returned reader will

Re: NRT getReader turnaround on large segments

2009-05-28 Thread Michael McCandless
Newly added docs are still free to make new segments, and be reopened, while this warming is taking place. So, getReader() will wait for newly added/deleted docs to be flushed reopened, but will not wait for any running merges to complete (finish writing the files finish warming a SegmentReader

Re: NRT getReader turnaround on large segments

2009-05-28 Thread Yonik Seeley
On Thu, May 28, 2009 at 4:18 PM, Michael McCandless luc...@mikemccandless.com wrote: Newly added docs are still free to make new segments, and be reopened, while this warming is taking place. So, getReader() will wait for newly added/deleted docs to be flushed reopened, but will not wait for

More efficient loading of terms dictionary

2009-05-28 Thread Jason Rutherglen
This is for Marvin who previously mentioned loading the term dictionary directly from the filesystem (rather than load every Nth term into Java heap) which could improve latency of opening new readers. Were you able to take this idea any further?

[jira] Commented: (LUCENE-1629) contrib intelligent Analyzer for Chinese

2009-05-28 Thread Otis Gospodnetic (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714285#action_12714285 ] Otis Gospodnetic commented on LUCENE-1629: -- I just got to look at this code and I

[jira] Commented: (LUCENE-1629) contrib intelligent Analyzer for Chinese

2009-05-28 Thread Xiaoping Gao (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714293#action_12714293 ] Xiaoping Gao commented on LUCENE-1629: -- I think the algorithm of Hidden Markov Model

[jira] Commented: (LUCENE-1614) Add next() and skipTo() variants to DocIdSetIterator that return the current doc, instead of boolean

2009-05-28 Thread Shai Erera (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12714298#action_12714298 ] Shai Erera commented on LUCENE-1614: Mike, I think we may have relaxed the policy of