Re: (SOLR-1868) Cutover to flex APIs
Hi Koji, Contributions are very welcome :) Unfortunately that wiki page is somewhat stale -- it hasn't been updated based on what's been done on flex. But I think big change for Solr is to cutover to FieldsEnum, TermsEnum, DocsEnum, DocsAndPositionsEnum instead of TermEnum, TermDocs, TermPositions. At some point (lower priority I think) we should make it possible for an app to pick the postings codec, probably per-field? EG the pulsing codec would be good for highly unique fields (eg the id field), but is very experimental right now. Mike On Thu, Apr 8, 2010 at 12:17 PM, Koji Sekiguchi wrote: > Michael McCandless (JIRA) wrote: >> Cutover to flex APIs >> >> >> Key: SOLR-1868 >> URL: https://issues.apache.org/jira/browse/SOLR-1868 >> Project: Solr >> Issue Type: Bug >> Reporter: Michael McCandless >> Fix For: 3.1 >> >> >> We need to fix Solr to use flex APIs! >> >> > Hello, > > I'm a latecomer on the issue of "flex", but I'd like to know it > and if possible, contribute something. But I chicken out > when I see LUCENE-1458 and its friend issues: > > https://issues.apache.org/jira/browse/LUCENE/fixforversion/12314439 > > I think I should read FlexibleIndexing on wiki, but I'd appreciate > it if someone recommend pointers for flex latecomers, if any. > > Thank you! > > Koji > > -- > http://www.rondhuit.com/en/ > >
[jira] Created: (SOLR-1868) Cutover to flex APIs
Cutover to flex APIs Key: SOLR-1868 URL: https://issues.apache.org/jira/browse/SOLR-1868 Project: Solr Issue Type: Bug Reporter: Michael McCandless Fix For: 3.1 We need to fix Solr to use flex APIs! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Welcome Uwe Schindler to the Lucene PMC
Welcome Uwe!! Mike On Thu, Apr 1, 2010 at 7:05 AM, Grant Ingersoll wrote: > I'm pleased to announce that the Lucene PMC has voted to add Uwe Schindler to > the PMC. Uwe has been doing a lot of work in Lucene and Solr, including > several of the last releases in Lucene. > > Please join me in extending congratulations to Uwe! > > -Grant Ingersoll > PMC Chair > - > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org > >
Re: lucene and solr trunk
All tests pass for me :) Mike On Thu, Mar 18, 2010 at 12:27 PM, Mark Miller wrote: > Alight, so we have implemented Hoss' suggestion here on the lucene/solr > merged dev branch at lucene/solr/branches/newtrunk. > > Feel free to check it out and give some feedback. > > We also roughly have Solr running on Lucene trunk - eg compiling Solr will > first compile lucene and run off those compiled class files. Running dist or > example in Solr will grab Lucene's jars and put them in the war. This still > needs further love, but it works. > > There is also a top level build.xml with two targets: clean, and test. Clean > will clean both Lucene and Solr, and test will run tests for both Lucene and > Solr. > > Thanks to everyone that contributed to getting all this working! > > -- > - Mark > > http://www.lucidimagination.com > > > > On 03/17/2010 12:40 PM, Mark Miller wrote: >> >> Okay, so this looks good to me (a few others seemed to like it - though >> Lucene-Dev was somehow dropped earlier) - lets try this out on the branch? >> (then we can get rid of that "horrible" branch name ;) ) >> >> Anyone on the current branch object to having to do a quick svn switch? >> >> On 03/16/2010 06:46 PM, Chris Hostetter wrote: >>> >>> : Otis, yes, I think so, eventually. But that's gonna take much more >>> discussion. >>> : >>> : I don't think this initial cutover should try to "solve" how modules >>> : will be organized, yet... we'll get there, eventually. >>> >>> But we should at least consider it, and not move in a direction that's >>> distinct from the ultimate goal of better refactoring (especailly since >>> that was one of the main goals of unifying development efforts) >>> >>> Here's my concrete suggestion that could be done today (for simplicity: >>> $svn = https://svn.apache.org/repos/asf/lucene)... >>> >>> svn mv $svn/java/trunk $svn/java/tmp-migration >>> svn mkdir $svn/java/trunk >>> svn mv $svn/solr/trunk $svn/java/trunk/solr >>> svn mv $svn/java/tmp-migration $svn/java/trunk/core >>> >>> At which point: >>> >>> 0. People who want to work only on "Lucene-Java" can start checking out >>> $svn/java/trunk/core (i'm pretty sure existing checkouts will continue to >>> work w/o any changes, the svn info should just update itself) >>> >>> 1. build files can be added to (the new) $svn/java/trunk to build ./core >>> followed by ./solr >>> >>> 2. the build files in $svn/java/trunk/solr can be modified to look at >>> ../core/ to find lucene jars >>> >>> 3. people who care about Solr (including all committers) should start >>> checking out and building all of $svn/java/trunk >>> >>> 4. Long term, we could choose to branch all of $svn/java/trunk >>> for releases ... AND/OR we could choose to branch specific modules >>> (ie: solr) independently (with modifications to the build files on those >>> branches to pull in their dependencies from alternate locations >>> >>> 5. Long term, we can start refactoring additional "modules" out of >>> $svn/java/trunk/solr and $svn/java/trunk/core (like >>> $svn/java/trunk/core/contrib) into their own directory in $svn/java/trunk >>> >>> 6. Long term, people who want to work on more then just "core" but don't >>> care about certain "modules" (like solr) can do a simple non-recursive >>> checkout of $svn/java/trunk and then do full checkouts of whatever >>> modules >>> they care about >>> >>> >>> (Please note: I'm just trying to list things we *could* do if we go this >>> route, i'm not advocating that we *should* do any of these things) >>> >>> I can't think of any objections people have raised to any of the previous >>> suggestions which apply to this suggestion. Is there anything people can >>> think of that would be useful, but not possible, if we go this route? >>> >>> >>> -Hoss >>> >> >> > > > > - > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org > >
Re: rough outline of where Solr's going
Ahh, OK. Meaning Solr will have to remove deprecated support, which means Solr's next released version would be a major release? Ie 2.0? Mike On Thu, Mar 18, 2010 at 11:26 AM, Robert Muir wrote: > On Thu, Mar 18, 2010 at 11:33 AM, Michael McCandless > wrote: >> On version numbering... my inclination would be to let Solr and Lucene >> use their own version numbers (don't sync them up). I know it'd >> simplify our lives to have the same version across the board, but >> these numbers are really for our users, telling them when big changes >> were made, back compat broken, etc. I think that trumps dev >> convenience. > > Be sure to consider the deprecations removal, its not possible for > Solr to move to Lucene's trunk without this. > > Here are two examples of necessary deprecation removals in the branch > so that Solr can use Lucene's trunk: > https://issues.apache.org/jira/browse/SOLR-1820 > http://www.lucidimagination.com/search/document/f07da8e4d69f5bfe/removal_of_deprecated_htmlstrip_tokenizer_factories > > It seems to be the consensus that people want a major version change > number when this is done. > > So this is an example where the version numbers of Solr really do > relate to Lucene, if we want them to share the same trunk. > > > -- > Robert Muir > rcm...@gmail.com >
Re: rough outline of where Solr's going
On Thu, Mar 18, 2010 at 8:20 AM, Marvin Humphrey wrote: > I think the concern is what happens if Solr migrates a bunch of stuff into > Lucene, ceding control over crucial functionality, and then Solr wants to > release but Lucene does not. That would pose a problem for Solr, no? But, I don't think we'd ever do this -- ie when we release Solr we'll also release Lucene. Think about it... if for some exotic reason Lucene is unreleasable, then, presumably we would not up and release Solr until we fixed whatever was broken with Lucene, since it'd also break Solr. It is conceivable we could release only Lucene (yes, this was an explicit part of the vote, take 2), but I expect in practice that'll be the exception not the rule... it remains to be seen. On version numbering... my inclination would be to let Solr and Lucene use their own version numbers (don't sync them up). I know it'd simplify our lives to have the same version across the board, but these numbers are really for our users, telling them when big changes were made, back compat broken, etc. I think that trumps dev convenience. Mike
Re: lucene and solr trunk
Git, Maven, Hg, etc., all sound great for the future, but let's focus now on the baby step (how to re-org svn), today, so we can land the Solr upgrade work now being done on a branch... Hoss's side-by-side proposal sounds great... and his concrete steps "that could be done today" look good (I'm hoping Uwe "the policeman who wears many hats" volunteers ;). Mike On Wed, Mar 17, 2010 at 8:05 AM, Otis Gospodnetic wrote: > +1 for this structure and this set of steps. > Otis > > > > > - Original Message >> From: Chris Hostetter >> To: solr-dev@lucene.apache.org >> Sent: Tue, March 16, 2010 6:46:19 PM >> Subject: Re: lucene and solr trunk >> >> : Otis, yes, I think so, eventually. But that's gonna take much more >> discussion. > : > : I don't think this initial cutover should try to "solve" >> how modules > : will be organized, yet... we'll get there, >> eventually. > > But we should at least consider it, and not move in a >> direction that's > distinct from the ultimate goal of better refactoring >> (especailly since > that was one of the main goals of unifying development >> efforts) > > Here's my concrete suggestion that could be done today (for >> simplicity: > $svn = >> target=_blank >https://svn.apache.org/repos/asf/lucene)... > > svn >> mv $svn/java/trunk $svn/java/tmp-migration > svn mkdir >> $svn/java/trunk > svn mv $svn/solr/trunk $svn/java/trunk/solr > >> svn mv $svn/java/tmp-migration $svn/java/trunk/core > > At which >> point: > > 0. People who want to work only on "Lucene-Java" can start >> checking out > $svn/java/trunk/core (i'm pretty sure existing checkouts will >> continue to > work w/o any changes, the svn info should just update >> itself) > > 1. build files can be added to (the new) $svn/java/trunk to build >> ./core > followed by ./solr > > 2. the build files in $svn/java/trunk/solr >> can be modified to look at > ../core/ to find lucene jars > > 3. people who >> care about Solr (including all committers) should start > checking out and >> building all of $svn/java/trunk > > 4. Long term, we could choose to branch >> all of $svn/java/trunk > for releases ... AND/OR we could choose to branch >> specific modules > (ie: solr) independently (with modifications to the build >> files on those > branches to pull in their dependencies from alternate >> locations > > 5. Long term, we can start refactoring additional "modules" out >> of > $svn/java/trunk/solr and $svn/java/trunk/core (like >> > $svn/java/trunk/core/contrib) into their own directory in >> $svn/java/trunk > > 6. Long term, people who want to work on more then just >> "core" but don't > care about certain "modules" (like solr) can do a simple >> non-recursive > checkout of $svn/java/trunk and then do full checkouts of >> whatever modules > they care about > > > (Please note: I'm just trying to >> list things we *could* do if we go this > route, i'm not advocating that we >> *should* do any of these things) > > I can't think of any objections people >> have raised to any of the previous > suggestions which apply to this >> suggestion. Is there anything people can > think of that would be >> useful, but not possible, if we go this route? > > > -Hoss >
Re: lucene and solr trunk
Otis, yes, I think so, eventually. But that's gonna take much more discussion. I don't think this initial cutover should try to "solve" how modules will be organized, yet... we'll get there, eventually. Mike On Tue, Mar 16, 2010 at 4:57 PM, Otis Gospodnetic wrote: > Hi, > > Check out the dir structure mentioned here: > http://markmail.org/message/gwpmaevw7tavqqge > > Isn't that what we want? > > Otis > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > Hadoop ecosystem search :: http://search-hadoop.com/ > > > > - Original Message >> From: Mark Miller >> To: solr-dev@lucene.apache.org >> Sent: Mon, March 15, 2010 11:43:48 PM >> Subject: Re: lucene and solr trunk >> >> On 03/15/2010 11:28 PM, Yonik Seeley wrote: >> So, we have a few options on >> where to put Solr's new trunk: >> >> >> Solr moves to Lucene's >> trunk: >> /java/trunk, /java/trunk/sol > +1. With the goal of >> merged dev, merged tests, this looks the best to > me. Simple to do patches >> that span both, simple to setup > Solr to use Lucene trunk rather than jars. >> Short paths. Simple. I like it. > > -- > - Mark > > >> href="http://www.lucidimagination.com"; target=_blank >> >http://www.lucidimagination.com >
Free live video streaming of ApacheCon US 2009
Team, For those Lucene fanatics not in Oakland this week for ApacheCon US, don't miss the FREE live video streaming, starting today: http://streaming.linux-magazin.de/en/program-apachecon-us-2009.htm Note that there are many talks available, covering Apache Hadoop, Apache HTTPD, Lucene, as well as the Apache Pioneer's Panel and keynote presentations. Lucene's track is this Friday (NOTE these times are UTC -- use http://www.timeanddate.com to map to your time zone): 17:00 Implementing an Information Retrieval Framework for an Organizational Repository, Sithu D Sudarsan 18:00 Apache Mahout - Going from raw data to information Isabel Drost 19:15 MIME Magic with Apache Tika Jukka Zitting 20:15 Keynote: How Open Source Developers Can (Still!) Save The World Brian Behlendorf 22:00 Building Intelligent Search Applications with the Lucene Ecosystem, Ted Dunning 23:00 Realtime Search Jason Rutherglen Happy viewing, Mike
[jira] Commented: (SOLR-940) TrieRange support
[ https://issues.apache.org/jira/browse/SOLR-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724286#action_12724286 ] Michael McCandless commented on SOLR-940: - OK try again? Maybe 3rd time's the charm... > TrieRange support > - > > Key: SOLR-940 > URL: https://issues.apache.org/jira/browse/SOLR-940 > Project: Solr > Issue Type: New Feature >Reporter: Yonik Seeley >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: SOLR-940-LUCENE-1602.patch, SOLR-940-LUCENE-1602.patch, > SOLR-940-LUCENE-1701.patch, SOLR-940-newTrieAPI.patch, > SOLR-940-newTrieAPI.patch, SOLR-940-rangequery.patch, > SOLR-940-rangequery.patch, SOLR-940-test.patch, SOLR-940.patch, > SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, > SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch > > > We need support in Solr for the new TrieRange Lucene functionality. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-940) TrieRange support
[ https://issues.apache.org/jira/browse/SOLR-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724255#action_12724255 ] Michael McCandless commented on SOLR-940: - Sigh. I'll go reopen LUCENE-1630! > TrieRange support > - > > Key: SOLR-940 > URL: https://issues.apache.org/jira/browse/SOLR-940 > Project: Solr > Issue Type: New Feature >Reporter: Yonik Seeley >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: SOLR-940-LUCENE-1602.patch, SOLR-940-LUCENE-1602.patch, > SOLR-940-LUCENE-1701.patch, SOLR-940-newTrieAPI.patch, > SOLR-940-newTrieAPI.patch, SOLR-940-rangequery.patch, > SOLR-940-rangequery.patch, SOLR-940-test.patch, SOLR-940.patch, > SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, > SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch > > > We need support in Solr for the new TrieRange Lucene functionality. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-940) TrieRange support
[ https://issues.apache.org/jira/browse/SOLR-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724080#action_12724080 ] Michael McCandless commented on SOLR-940: - OK I just committed a fix for LUCENE-1630 that should fix that exception. > TrieRange support > - > > Key: SOLR-940 > URL: https://issues.apache.org/jira/browse/SOLR-940 > Project: Solr > Issue Type: New Feature >Reporter: Yonik Seeley >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: SOLR-940-LUCENE-1602.patch, SOLR-940-LUCENE-1602.patch, > SOLR-940-LUCENE-1701.patch, SOLR-940-newTrieAPI.patch, > SOLR-940-newTrieAPI.patch, SOLR-940-rangequery.patch, > SOLR-940-rangequery.patch, SOLR-940-test.patch, SOLR-940.patch, > SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, > SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch > > > We need support in Solr for the new TrieRange Lucene functionality. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-940) TrieRange support
[ https://issues.apache.org/jira/browse/SOLR-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723975#action_12723975 ] Michael McCandless commented on SOLR-940: - Shalin I think that exception you got is a break in back-compat. Sorry :( I'm reopening LUCENE-1630 to fix it... > TrieRange support > - > > Key: SOLR-940 > URL: https://issues.apache.org/jira/browse/SOLR-940 > Project: Solr > Issue Type: New Feature >Reporter: Yonik Seeley >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: SOLR-940-LUCENE-1602.patch, SOLR-940-LUCENE-1602.patch, > SOLR-940-LUCENE-1701.patch, SOLR-940-newTrieAPI.patch, > SOLR-940-newTrieAPI.patch, SOLR-940-rangequery.patch, > SOLR-940-rangequery.patch, SOLR-940-test.patch, SOLR-940.patch, > SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, > SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch, SOLR-940.patch > > > We need support in Solr for the new TrieRange Lucene functionality. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1221) Change Solr Highlighting to use the SpanScorer with MultiTerm expansion by default
[ https://issues.apache.org/jira/browse/SOLR-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720285#action_12720285 ] Michael McCandless commented on SOLR-1221: -- +1 ;) Solr's defaults are important, too. > Change Solr Highlighting to use the SpanScorer with MultiTerm expansion by > default > -- > > Key: SOLR-1221 > URL: https://issues.apache.org/jira/browse/SOLR-1221 > Project: Solr > Issue Type: Improvement > Components: highlighter >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 1.4 > > > To improve the out of the box experience of Solr 1.4, I really think we > should make this change. You will still be able to turn both off. > Comments? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Welcome new Solr committers Mark Miller and Noble Paul
Welcome aboard Mark and Noble! Mike On Thu, Apr 30, 2009 at 1:41 PM, Yonik Seeley wrote: > I'm pleased to announce that Mark Miller and Noble Paul have accepted > invitations to become Solr committers! > Welcome Mark & Noble, and thanks for all your great work on Solr! > > -Yonik > > ps: following recent custom, why don't you guys tell us a little about > yourselves (in addition to adding yourself to > http://lucene.apache.org/solr/who.html when you get the karma) >
Re: lucene 2.9 migration issues
This is LUCENE-1573 (properly responding to Thread.interrupt()), which is new in 2.9. Do you know what would have interrupted the thread in your context? Mike On Fri, Apr 24, 2009 at 2:25 PM, Ryan McKinley wrote: > thanks! I'm not sure this is related to 2.9, but I have not seen this > before either > > 2009-04-24 14:19:53,024 ERROR org.apache.solr.core.SolrCore - > java.lang.RuntimeException: java.lang.InterruptedException: sleep > interrupted > at > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:620) > at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:289) > at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1456) > at org.apache.lucene.index.IndexWriter.(IndexWriter.java:1295) > at > org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:159) > at > org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123) > at > org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:170) > at > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:217) > at > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:60) > at > org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:140) > at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69) > at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1330) > at > org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:139) > at > org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:259) > at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:63) > > Any thoughts what could cause that? This is using the embedded solr > server... > > > On Apr 24, 2009, at 12:10 PM, Shalin Shekhar Mangar wrote: > >> On Fri, Apr 24, 2009 at 9:07 PM, Ryan McKinley wrote: >> >>> Yes, that would be great! the changes we need are in rev 768275: >>> http://svn.apache.org/viewvc?view=rev&revision=768275 >>> >> >> Done. I upgraded to r768336. >> >> -- >> Regards, >> Shalin Shekhar Mangar. > >
Re: faster example schema
+1 I think appropriate selection of defaults, biasing for performance in this case, is extremely important yet often overlooked. The out-of-the-box first-impression experience that people see, with Solr or Lucene, often decides whether Solr or Lucene should be considered for an application. Mike Yonik Seeley wrote: I've occasionally run across people going with another search engine because it was faster at indexing. The example schema that people may be using as a base to do their benchmarking (with perhaps minimal modifications) is slow. There are many people out there that check what's fastest first, and *then* check if it is satisfactory to meet their needs in other areas. With very simple synthetic test documents (just a few fields each) and the CSV loader, I've personally seen the indexing rate go from ~330/sec to ~3000/sec, when I removed the default field values, term vectors, copyFields, etc. The default example schema should still be able to show how something can be done, but that doesn't mean it needs to be enabled by default. So what do people think about speeding up the default/example schema before 1.4? -Yonik http://www.lucidimagination.com
Re: trunk test failure
Except... the only reason (I think) that the Lucene build failed was because I had switched the back-compat test to check out an https:// URL instead of http:// (the build got hung asking whether you trust that host because its cert was suspect) so I switched it back... and I'm not sure why that Solr test failed! Hmmm... Mike Yonik Seeley wrote: On Thu, Jan 29, 2009 at 12:07 PM, Mark Miller wrote: Mark Miller wrote: Anyone else seeing the function query test failing on trunk? - Mark Okay, I tried building trunk lucene myself, and after putting that in, the function query test passed. Perhaps an odd Lucene build went in? The previous lucene nightly had failed (on an svn tag lookup), so I built lucene myself (and all tests did pass). I've tried multiple times, and I can't get that same test to fail (WinXP). Very strange. Anyway, since the latest lucene nightly did pass, I'll update to that. Hopefully that should fix things. -Yonik
[jira] Commented: (SOLR-342) Add support for Lucene's new Indexing and merge features (excluding Document/Field/Token reuse)
[ https://issues.apache.org/jira/browse/SOLR-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570327#action_12570327 ] Michael McCandless commented on SOLR-342: - Super, I'm glad to hear that! > Add support for Lucene's new Indexing and merge features (excluding > Document/Field/Token reuse) > --- > > Key: SOLR-342 > URL: https://issues.apache.org/jira/browse/SOLR-342 > Project: Solr > Issue Type: Improvement > Components: update >Reporter: Grant Ingersoll >Assignee: Grant Ingersoll >Priority: Minor > Attachments: copyLucene.sh, SOLR-342.patch, SOLR-342.patch, > SOLR-342.patch, SOLR-342.tar.gz > > > LUCENE-843 adds support for new indexing capabilities using the > setRAMBufferSizeMB() method that should significantly speed up indexing for > many applications. To fix this, we will need trunk version of Lucene (or > wait for the next official release of Lucene) > Side effect of this is that Lucene's new, faster StandardTokenizer will also > be incorporated. > Also need to think about how we want to incorporate the new merge scheduling > functionality (new default in Lucene is to do merges in a background thread) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-342) Add support for Lucene's new Indexing and merge features (excluding Document/Field/Token reuse)
[ https://issues.apache.org/jira/browse/SOLR-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569845#action_12569845 ] Michael McCandless commented on SOLR-342: - Will, one more thing to try is to on assertions for org.apache.lucene.*; this may catch an issue sooner. > Add support for Lucene's new Indexing and merge features (excluding > Document/Field/Token reuse) > --- > > Key: SOLR-342 > URL: https://issues.apache.org/jira/browse/SOLR-342 > Project: Solr > Issue Type: Improvement > Components: update >Reporter: Grant Ingersoll >Assignee: Grant Ingersoll >Priority: Minor > Attachments: copyLucene.sh, SOLR-342.patch, SOLR-342.patch, > SOLR-342.patch, SOLR-342.tar.gz > > > LUCENE-843 adds support for new indexing capabilities using the > setRAMBufferSizeMB() method that should significantly speed up indexing for > many applications. To fix this, we will need trunk version of Lucene (or > wait for the next official release of Lucene) > Side effect of this is that Lucene's new, faster StandardTokenizer will also > be incorporated. > Also need to think about how we want to incorporate the new merge scheduling > functionality (new default in Lucene is to do merges in a background thread) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-342) Add support for Lucene's new Indexing and merge features (excluding Document/Field/Token reuse)
[ https://issues.apache.org/jira/browse/SOLR-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569525#action_12569525 ] Michael McCandless commented on SOLR-342: - Will, did you create a new index in your test? Also make sure you are using this URL to checkout the 2.3 branch sources: https://svn.apache.org/repos/asf/lucene/java/branches/lucene_2_3 You should see 7 issues listed in the CHANGES.txt under "Bug fixes"? > Add support for Lucene's new Indexing and merge features (excluding > Document/Field/Token reuse) > --- > > Key: SOLR-342 > URL: https://issues.apache.org/jira/browse/SOLR-342 > Project: Solr > Issue Type: Improvement > Components: update >Reporter: Grant Ingersoll >Assignee: Grant Ingersoll >Priority: Minor > Attachments: copyLucene.sh, SOLR-342.patch, SOLR-342.patch, > SOLR-342.patch, SOLR-342.tar.gz > > > LUCENE-843 adds support for new indexing capabilities using the > setRAMBufferSizeMB() method that should significantly speed up indexing for > many applications. To fix this, we will need trunk version of Lucene (or > wait for the next official release of Lucene) > Side effect of this is that Lucene's new, faster StandardTokenizer will also > be incorporated. > Also need to think about how we want to incorporate the new merge scheduling > functionality (new default in Lucene is to do merges in a background thread) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: switch to native locks by default?
Chris Hostetter wrote: : Ah, I hadn't realized that they might not be supported everywhere... I I'm just trusting the javadoc for NativeFSLockFactory ... i have no idea if it's accurate or not. Hi! I had added the caveat about native locks based on my dicey experience getting them working over NFS (NFS locking was not "turned on" by default in my setup; and, frustratingly, it would take ~35 seconds for a timeout to tell me this). I don't have any specific evidence that other OS/filesystems are problematic but then again I haven't done much research to understand overall portability of Java's native lock interface. It would not surprise me if other OS/filesystems had issues. I was hoping by getting the NativeFSLockFactory out there that it would get some healthy testing first and then we could use that feedback to decide whether benefits outweigh the risks of making it the default. It's not clear how many people have actually tested it at this point, though! : The current locking can also guard against mistakes though (multiple : instances of Solr trying to write to the same dir, someone opening a : Luke index on it, etc). right ... but it's only useful if all of the potential clients are using the same locking mechanism ... right now it's only safe to do any of those things if all the apps use SimpleFSLockFactory. all the more reason to make the factory and the lockDir configurable in Solr i guess. Mike
Re: Solr, new index format (segments_N)
Yonik Seeley wrote:
On 11/28/06, Manuel Albela Miranda <[EMAIL PROTECTED]> wrote:
Thanks for your answer. Yes, i'm using Sun's JDK 1.5.0 too. Can you tell
me the step you followed to get solr running with the lockless commits?.
I compile the latest java lucene and then put the jar files generated in
the lib directory of the solr distribution. Finally running 'ant' it
compiles without any problem. But when trying the example with the new
index format, it fails.
These are the steps I did (with Sun's JDK 1.5.0):
* Checked out the current trunk of Solr (from svn)
* Checked out the current trunk of Lucene (from svn)
* Built the Lucene JARs ("ant jar-core")
* Copied jar to Solr dir:
"cp build/lucene-core-2.1-dev.jar
../solr/lib/lucene-core-nightly.jar"
* Then ran "ant example" in solr dir (which creates solr.war under
example/webapps).
* Start jetty (cd example; java -jar start.jar)
* Submit all the xml files (cd exampledocs; sh post.sh *.xml)
* Verified that index (example/solr/data/index/*) shows segments_N
file (not segments), to confirm lockless is in fact running.
* Ran some searches, did some more of the examples (using curl to
delete & commit, etc.), and they all seem to be working fine.
Mike
Re: Solr, new index format (segments_N)
Manuel Albela Miranda wrote: Hello everybody, I wanted to know what can i do to make solr manage the new index format that comes with last svn version of Lucene, i mean, now there are not a segments file but segments_1, segments_2 and so on, and a segments.gen file. I have tried to compile solr with the last .jar files generated by the compilation of the lucene sources but i'm having the following errors: Hello! I was able to recompile the Solr example (solr.war) using the current trunk version of Lucene (with lockless commits). I was also able to walk through the initial steps of the example (update, delete, searching) running under Jetty and everything seems to work correctly. I'm using Sun's JDK 1.5 on Linux. From your exception, it looks like your JRE can't find the "matches" native method for a java.lang.String, which is very strange. This method has been available since 1.4. I'm not sure why this would happen. Maybe double-check to make sure you're running >= JRE 1.4? Mike
Re: [jira] Updated: (SOLR-52) Lazy Field loading
Yonik Seeley wrote: On 11/15/06, Mike Klaas <[EMAIL PROTECTED]> wrote: Any objections to sync'ing solr with lucene trunk? It might be nice from an impact perspective to do so before lockless commits are committed. +1, my thoughts as well. Hi! Should I hold off on committing lockless until you've done a sync for Solr to the Lucene trunk? Is this done now? Mike
