Re: newbie questions
: 1. It seems that Solr creates/searches one big index, at least logically : - for the current release, does it ever create multiple smaller indexes : behind the scene (except for the secondary, temporary ones used for : update)? Solr only creates one physical lucene index on disk. : 2. The snapshooter makes a "complete" copy of the index, right? When a : snapshot is pulled from master to slave, is it making a complete copy? : If snapshots are "fronzen" copies, how does the rsyncd's "copying only : the files that actually changed" help, what (e.g., a snapshot?) is the : source, and what is the destination? This is explained fairly in depth on the wiki... http://wiki.apache.org/solr/CollectionDistribution ...shapshots aren't copies of the index, they are hardlinks, so no additional disk space is used. when the snapshot is pulled to the slave, only the changed parts are pulled because of the way rsync is used. -Hoss
Re: SV: Re: Simple Web Interface
On Mar 20, 2007, at 8:55 PM, Erik Hatcher wrote: Yes, basically. The plan will be to take what's currently in the Flare Rails application and distill the heart of it into a Rails plugin, so you wouldn't need to do much but provide a skeleton controller that would tap into Flare. Perhaps something like this: class BrowseController < ApplicationController flare # subject to change :) end And that'd take care of everything you currently see in the BrowseController, letting the FlareContext come into play automatically. Ok, done. It was pretty trivial to convert what we had in Rails into a plugin and then use that plugin within the app-formerly-known- as-"flare" :) (maybe we'll rename the Rails app itself to "flare- app" or something like that. I still want to keep the current Rails application functioning as it currently does, it'll just have all of its code pushed down into a plugin and look much leaner and cleaner. Make sense? And, in fact this is exactly how it works after moving it to a plugin. No change in the application behavior at all. This will still evolve from here, as I figure out how I want the FlareContext to work within both the "flare-app" and in the other applications I'm building around flare. Antonio - all still well? Comments? Erik
Re: schema field properties
On 3/20/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: I'm revisiting SOLR-162 (the luke stuff) and run into a few questions about field specification... What is the difference between: SchemaField.isTokenized() and FieldType.isTokenized() Is the lucene Field tokenized if either is true? One should imply the other, unless the SchemaField was programmatically-constructed in a strange way. During indexing, only the FieldType method is used (see FieldType.java:206) How should I ask if a field is binary? (SchemaField.isBinary() is package protected and never called) Good question. I believe it is impossible to create a binary field in Solr currently. Is there a per field way to specify "isLazy"? If "enableLazyFieldLoading" is set to true, are all stored fields lazy? "lazy" is purely a property of reading fields--it is not inherent to the field type (that particular option enables some possible optimizations when reading a subset of stored fields). It is possibly misdocumented. If our most prolific contributor is confused, everyone else must be completely lost :) -Mike
schema field properties
I'm revisiting SOLR-162 (the luke stuff) and run into a few questions about field specification... What is the difference between: SchemaField.isTokenized() and FieldType.isTokenized() Is the lucene Field tokenized if either is true? How should I ask if a field is binary? (SchemaField.isBinary() is package protected and never called) Is there a per field way to specify "isLazy"? If "enableLazyFieldLoading" is set to true, are all stored fields lazy? thanks ryan
Re: SV: Re: Simple Web Interface
On Mar 20, 2007, at 5:18 PM, Antonio Eggberg wrote: Erik Hatcher <[EMAIL PROTECTED]> skrev: Faceting only appears in Flare when there are "*_facet" fields in your index. Flare is going to undergo another spurt of evolution over the next couple of weeks as I tease it apart into a Rails plugin, making it easy to incorporate into an existing Rails application (rather than being the Rails application itself). hmm.. Just so I understand correctly. Solr-Ruby is already a rails plugin and a ruby gem .. True, but I think it deserves more explanation. solr-ruby is simply a Ruby library that can be packaged as a gem, but even that is unnecessary. All you need is the solr-ruby lib directory in your Ruby load path. The Rails "plugin" capability with solr-ruby is really just a trick to get the library into the Rails load path. and the plan now is to make Flare a plugin That's always been the plan. Currently Flare is just an ugly hack as a Rails application that consists of a couple of Rails controllers and some view templates. Nothing fancy, and its not reusable in other Rails applications as-is. (i.e. DSL like streamlined) correct? .. so there will be "ways" to have user models (i.e flares parent application) using flare rails plugin to do tags/folksonomy, saved, faceted search etc. correct? Yes, basically. The plan will be to take what's currently in the Flare Rails application and distill the heart of it into a Rails plugin, so you wouldn't need to do much but provide a skeleton controller that would tap into Flare. Perhaps something like this: class BrowseController < ApplicationController flare # subject to change :) end And that'd take care of everything you currently see in the BrowseController, letting the FlareContext come into play automatically. I have to start re-thinking about the current app that i am developing. How so? How are you currently using Flare? You're about the only one I know of, so I'm quite curious how you've made use of the little hack that it currently is :) I still want to keep the current Rails application functioning as it currently does, it'll just have all of its code pushed down into a plugin and look much leaner and cleaner. Make sense? Will you be doing lot of changes? I'll be reorganizing the whole thing - again, it ain't much, so it shouldn't cause you or anyone else pain since the main Rails application will function basically the same as it does now, but the pieces will be reusable for other Rails applications. The intent was always to make a plugin, but I didn't want to tackle it myself until some real work demands it, which is now. If you could fill us in more on what you're doing, we'd be happy to take your use cases into account as we revamp it. Erik
SV: Re: Simple Web Interface
Erik Hatcher <[EMAIL PROTECTED]> skrev: Faceting only appears in Flare when there are "*_facet" fields in your index. Flare is going to undergo another spurt of evolution over the next couple of weeks as I tease it apart into a Rails plugin, making it easy to incorporate into an existing Rails application (rather than being the Rails application itself). hmm.. Just so I understand correctly. Solr-Ruby is already a rails plugin and a ruby gem .. and the plan now is to make Flare a plugin (i.e. DSL like streamlined) correct? .. so there will be "ways" to have user models (i.e flares parent application) using flare rails plugin to do tags/folksonomy, saved, faceted search etc. correct? ok.. This means I have to start re-thinking about the current app that i am developing. Will you be doing lot of changes? Thanks for the heads up .. In general I think its a good idea. Regards - Stava rätt! Stava lätt! Yahoo! Mails stavkontroll tar hand om tryckfelen och mycket mer! Få den på http://se.mail.yahoo.com
Re: Thank you...
Heh - it used to be in my sig ... my bad. Thanks, all. :) http://www.stubhub.com On 3/20/07, bruce <[EMAIL PROTECTED]> wrote: hey cass... anyway you could let us know the site/app that we're powering!!! always good to see what's going on in the world! thanks -Original Message- From: Cass Costello [mailto:[EMAIL PROTECTED] Sent: Tuesday, March 20, 2007 12:58 PM To: solr-user@lucene.apache.org; java-user@lucene.apache.org Subject: Thank you... ...to everyone who helps make Lucene and Solr such fantastic tools. I'm the Platform Architect for a leading online event ticket after-marketplace (think eBay for tickets), and we've just completed a 12 month project to rewrite the Browse and Search components of our customer-facing site. Both rely heavily on Solr and a smattering of custom Lucene API calls throughout the code base. As of this morning (2 weeks after our final-stage push to production), we've reduced our hardware needs over 50% and have improved page response times by 30-60% as compared to the old architecture (a mix of very old CF code for the browse pieces, and a couple GSAs for the seach pages). More importantly, we've established new patterns and flex points for future growth, and I have utmost confidence that we'll be leveraging the Lucene/Solr combo for years to come as we grow. The devs are proud, the execs are happy, our customers have noticed the speed improvements, and we all learned many positive lessons along the way. These forums and various online resources (the Solr wiki first and foremost) have been tremendously helpful. And so, thank you, all, for pointing the way to a simpler, faster, better solution for our needs. Cass Costello -- A complex system that works is invariably found to have evolved from a simple system that works. - John Gaule -- A complex system that works is invariably found to have evolved from a simple system that works. - John Gaule
Re: Simple Web Interface
On Mar 20, 2007, at 9:36 AM, thomas arni wrote: Thanks for you hint. I looked at the features of Flare. I'm wondering if there is only a user interface for Rails. It looks like Flare mostly focus on faceted browsing. Faceted browsing is not my first priority. I'm developing a full-text search, which use only one "lucene field". I index all tokens from my heterogeneous documents into one field. That is the reason I have no need for faceted browsing. Instead I'm focusing on: - paging - high-lighting of search terms - summary of documents Flare does all of those, but also has faceting. Currently Flare is a very simple (thus easily hackable to your own needs) Rails application, not much code at all. The heart of the interaction with Solr is via the solr-ruby API, not in the Flare code itself. Faceting only appears in Flare when there are "*_facet" fields in your index. Flare is going to undergo another spurt of evolution over the next couple of weeks as I tease it apart into a Rails plugin, making it easy to incorporate into an existing Rails application (rather than being the Rails application itself). Whether Flare is the right fit for you depends on whether Rails is a good fit. Seems like not since you asked about JSP/servlets also (ahem... JRuby!) Can someone recommend other solutions? It really is trivial to communicate with Solr and use the results in an application. So maybe you'd be better off creating some custom UI code that speaks to Solr. I really am aiming Flare to be what you're asking for here though, but it might not currently be in a state that its a good fit for you to adopt/adapt. Erik
Thank you...
...to everyone who helps make Lucene and Solr such fantastic tools. I'm the Platform Architect for a leading online event ticket after-marketplace (think eBay for tickets), and we've just completed a 12 month project to rewrite the Browse and Search components of our customer-facing site. Both rely heavily on Solr and a smattering of custom Lucene API calls throughout the code base. As of this morning (2 weeks after our final-stage push to production), we've reduced our hardware needs over 50% and have improved page response times by 30-60% as compared to the old architecture (a mix of very old CF code for the browse pieces, and a couple GSAs for the seach pages). More importantly, we've established new patterns and flex points for future growth, and I have utmost confidence that we'll be leveraging the Lucene/Solr combo for years to come as we grow. The devs are proud, the execs are happy, our customers have noticed the speed improvements, and we all learned many positive lessons along the way. These forums and various online resources (the Solr wiki first and foremost) have been tremendously helpful. And so, thank you, all, for pointing the way to a simpler, faster, better solution for our needs. Cass Costello -- A complex system that works is invariably found to have evolved from a simple system that works. - John Gaule
newbie questions
Hi, I have a few general questions: 1. It seems that Solr creates/searches one big index, at least logically - for the current release, does it ever create multiple smaller indexes behind the scene (except for the secondary, temporary ones used for update)? 2. The snapshooter makes a "complete" copy of the index, right? When a snapshot is pulled from master to slave, is it making a complete copy? If snapshots are "fronzen" copies, how does the rsyncd's "copying only the files that actually changed" help, what (e.g., a snapshot?) is the source, and what is the destination? Thanks! Xiaocheng TV dinner still cooling? Check out "Tonight's Picks" on Yahoo! TV. http://tv.yahoo.com/
Re: Simple Web Interface
Thanks for you hint. I looked at the features of Flare. I'm wondering if there is only a user interface for Rails. It looks like Flare mostly focus on faceted browsing. Faceted browsing is not my first priority. I'm developing a full-text search, which use only one "lucene field". I index all tokens from my heterogeneous documents into one field. That is the reason I have no need for faceted browsing. Instead I'm focusing on: - paging - high-lighting of search terms - summary of documents Has someone already built a simple query interface with JSP/Servlets? Can someone recommend other solutions? Otis Gospodnetic wrote: If you can run your app on Rails, look at Flare: http://wiki.apache.org/solr/Flare It would be gool to gave a Grails version of Flareif anyone has extra time on their hands...hah. Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share - Original Message From: thomas arni <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Monday, March 19, 2007 9:47:51 AM Subject: Simple Web Interface Hello I would like to build a simple web front end (e.g. google.com) for the users of solr. Which is the best way to archive this goal? Has someone already built a simple query interface (gui)? My query interface should have: - paging - summary of documents with the relevant search terms - high-lighting of search terms Any sample code is highly appreciated. Thanks for you help Thomas
