Build failed in Hudson: Lucene-trunk #651

2008-11-19 Thread Apache Hudson Server
See http://hudson.zones.apache.org/hudson/job/Lucene-trunk/651/changes -- [...truncated 146 lines...] AU src/test/org/apache/lucene/index/TestPositionBasedTermVectorMapper.java AUsrc/test/org/apache/lucene/index/index.22.cfs.zip AUsrc

[jira] Commented: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer

2008-11-19 Thread Jason Rutherglen (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649219#action_12649219 ] Jason Rutherglen commented on LUCENE-1316: -- I looked at 2.4 and am surprised this

[jira] Commented: (LUCENE-1461) Cached filter for a single term field

2008-11-19 Thread Tim Sturge (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649195#action_12649195 ] Tim Sturge commented on LUCENE-1461: Thanks Paul. This solved a nasty performance itc

[jira] Commented: (LUCENE-831) Complete overhaul of FieldCache API/Implementation

2008-11-19 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649191#action_12649191 ] Mark Miller commented on LUCENE-831: You've tried the patch? Awesome! How long did it

[jira] Commented: (LUCENE-831) Complete overhaul of FieldCache API/Implementation

2008-11-19 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649188#action_12649188 ] Mark Miller commented on LUCENE-831: Also - your reopen time will vary greatly dependin

[jira] Reopened: (LUCENE-1439) Inconsistent API

2008-11-19 Thread Ivan.S (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan.S reopened LUCENE-1439: No one seems to care about this issue since it is closed, so I dare to reopen it (after I created a Wiki page

[jira] Commented: (LUCENE-831) Complete overhaul of FieldCache API/Implementation

2008-11-19 Thread Alex Vigdor (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649180#action_12649180 ] Alex Vigdor commented on LUCENE-831: Another useful feature that seems like it would be

Re: [jira] Commented: (LUCENE-1458) Further steps towards flexible indexing

2008-11-19 Thread Michael McCandless
I think we wouldn't do any term compression for the btree, at least for the parts loaded in RAM (we don't today, ie, we create the full Term or String as an array). For the parts left on disk we should be able to do something similar to what we do today, eg for child nodes only encode the

Re: [jira] Updated: (LUCENE-1458) Further steps towards flexible indexing

2008-11-19 Thread Michael McCandless
MutableString looks cool but totally different from flexible indexing. Mike Jason Rutherglen wrote: On a side note, and I have not looked at the flexible indexing API enough to know if there is some equivalent but are we moving to something like MG4J's MutableString http://mg4j.dsi.unimi.i

Re: [jira] Updated: (LUCENE-1458) Further steps towards flexible indexing

2008-11-19 Thread Michael McCandless
Flexible indexing doesn't try to address the real-time updates; it only tries to make index writing & reading modular so that you can plug in your own codecs for your own format. Also, so far, I've only worked on the postings lists. I think for column-stride fields it should be easier to

Re: Allow IndexReader to take ownership of Directory

2008-11-19 Thread Michael McCandless
WeakReference doesn't fit here. We need a real reference. I don't like falling back to GC to do "close" in case there are "precious" resources that the Directory needs to release. Mike robert engels wrote: Why not create new lightweight references to the the directory, and using WeakRef

[jira] Closed: (LUCENE-1463) Cannot set an indexDictionary, perhaps the IndexWriter API changed?

2008-11-19 Thread Jason Porter (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Porter closed LUCENE-1463. Resolution: Invalid Looks like I already had an old version of Lucene on the classpath in an uber

[jira] Created: (LUCENE-1463) Cannot set an indexDictionary, perhaps the IndexWriter API changed?

2008-11-19 Thread Jason Porter (JIRA)
Cannot set an indexDictionary, perhaps the IndexWriter API changed? --- Key: LUCENE-1463 URL: https://issues.apache.org/jira/browse/LUCENE-1463 Project: Lucene - Java Issue Type

[jira] Issue Comment Edited: (LUCENE-1461) Cached filter for a single term field

2008-11-19 Thread Paul Elschot (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649143#action_12649143 ] [EMAIL PROTECTED] edited comment on LUCENE-1461 at 11/19/08 11:33 AM: --

[jira] Commented: (LUCENE-1461) Cached filter for a single term field

2008-11-19 Thread Paul Elschot (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649143#action_12649143 ] Paul Elschot commented on LUCENE-1461: -- This is a nice tradeoff: reduced space (the e

Re: Option to fsync files

2008-11-19 Thread robert engels
It is not just database by the way, any journaling file system would be pointless... On Nov 19, 2008, at 12:55 PM, robert engels wrote: The "utility" referenced no longer exists... and its no wonder. If is most likely that the tester did not have the drives configured properly. In almost

Re: Option to fsync files

2008-11-19 Thread robert engels
I don't believe it - unless the older drives had no cache, in which case it wouldn't matter. It is also doubtful at the OS level, as system integrity would be hopelessly compromised... On Nov 19, 2008, at 12:11 PM, Mark Miller wrote: robert engels wrote: There is an option on some hard dr

Re: Option to fsync files

2008-11-19 Thread robert engels
The "utility" referenced no longer exists... and its no wonder. If is most likely that the tester did not have the drives configured properly. In almost all cases, if the drive did this, you could not run a database system with any resiliency. They would also have problems with shutdown -

Re: Option to fsync files

2008-11-19 Thread Mark Miller
robert engels wrote: There is an option on some hard drives that is off by default (under windows) called lazy writing. You would need to go into the driver and turn this on. Only very specialized systems would ever do this. I see evidence that older mac and linux system do do this by default.

Re: Option to fsync files

2008-11-19 Thread Mark Miller
Ummm...well durability isnt really corruption is it. I guess I'm convert there. Mark Miller wrote: Thanks Jason...had just found h2 again myself - that guy seems to say even with write caching disabled on the OS, its still a problem (though probably less of a problem than without sync? Jason

Re: Option to fsync files

2008-11-19 Thread Mark Miller
Thanks Jason...had just found h2 again myself - that guy seems to say even with write caching disabled on the OS, its still a problem (though probably less of a problem than without sync? Jason Rutherglen wrote: http://www.h2database.com/html/advanced.html#durability_problems http://hardware.s

Re: Option to fsync files

2008-11-19 Thread Jason Rutherglen
http://www.h2database.com/html/advanced.html#durability_problems http://hardware.slashdot.org/article.pl?sid=05/05/13/0529252 On Wed, Nov 19, 2008 at 9:23 AM, Mark Miller <[EMAIL PROTECTED]> wrote: > /A look at the slashdot article seems to indicate the OS may have been at > fault in his case (an

Re: [jira] Commented: (LUCENE-1458) Further steps towards flexible indexing

2008-11-19 Thread Jason Rutherglen
Michael B: Are you interested in making column stride fields realtime and use the btree for the terms? This is an idea I started on I called tag index where the postings are divided into blocks. The blocks can then be replaced in memory with periodic flush to disk as the in ram postings grows. M

Re: Option to fsync files

2008-11-19 Thread robert engels
There is an option on some hard drives that is off by default (under windows) called lazy writing. You would need to go into the driver and turn this on. Only very specialized systems would ever do this. On Nov 19, 2008, at 11:13 AM, Mark Miller wrote: Okay, I'll admit I am trusting some el

Re: Option to fsync files

2008-11-19 Thread Mark Miller
/A look at the slashdot article seems to indicate the OS may have been at fault in his case (an update he posted in response to some slashdot flames). Thats interesting. Wasnt there the last time this topic came up. Can't remember what java db had the other info...but looking... /

Re: Option to fsync files

2008-11-19 Thread Mark Miller
Okay, I'll admit I am trusting some else testing this. One of the java database implementations (hsql or something?) talks about it and tests it it to be the case also with derby and other transaction systems. Corruption didn't appear that hard for him to lure out at all. Also, if you google sl

Re: Option to fsync files

2008-11-19 Thread robert engels
I would really like to see some PROOF of these drives "lying". If that were the case, no database system would ever be reliable on these drives ! And data corruption would be happening all over the place ! On Nov 19, 2008, at 10:56 AM, Mark Miller wrote: Michael McCandless wrote: Mark

Re: Option to fsync files

2008-11-19 Thread Mark Miller
Michael McCandless wrote: Mark Miller wrote: It is dangerous, but probably not any more dangerous than using a consumer hard drive that lies to sync (don't know the numbers, but I have to assume some/many are doing this with Lucene - in which case you pay perf for a false sense of security).

Re: Option to fsync files

2008-11-19 Thread Michael McCandless
Mark Miller wrote: It is dangerous, but probably not any more dangerous than using a consumer hard drive that lies to sync (don't know the numbers, but I have to assume some/many are doing this with Lucene - in which case you pay perf for a false sense of security). Well, if the consumer

Re: Option to fsync files

2008-11-19 Thread Mark Miller
It is dangerous, but probably not any more dangerous than using a consumer hard drive that lies to sync (don't know the numbers, but I have to assume some/many are doing this with Lucene - in which case you pay perf for a false sense of security). Not a real suggestion at this point though. Ju

Re: Option to fsync files

2008-11-19 Thread Michael McCandless
Well... because it's quite dangerous to turn off. A simple way to disable it entirely is to subclass FSDir and override sync() to be a noop. Mike Mark Miller wrote: What was the reason not to make this toggle-able? - To

Option to fsync files

2008-11-19 Thread Mark Miller
What was the reason not to make this toggle-able? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]

[jira] Commented: (LUCENE-1458) Further steps towards flexible indexing

2008-11-19 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649028#action_12649028 ] Michael McCandless commented on LUCENE-1458: {quote} In KS and Lucy, at least

[jira] Created: (LUCENE-1462) Instantiated/IndexWriter discrepanies

2008-11-19 Thread Karl Wettin (JIRA)
Instantiated/IndexWriter discrepanies - Key: LUCENE-1462 URL: https://issues.apache.org/jira/browse/LUCENE-1462 Project: Lucene - Java Issue Type: Bug Components: contrib/* Affects Versions:

[jira] Commented: (LUCENE-1458) Further steps towards flexible indexing

2008-11-19 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648971#action_12648971 ] Michael McCandless commented on LUCENE-1458: bq. So something like a B+Tree wo