Re: crawler feed?

2007-02-07 Thread Thorsten Scherler
On Wed, 2007-02-07 at 11:09 +0100, rubdabadub wrote: Hi: Are there relatively stand-alone crawler that are suitable/customizable for Solr? has anyone done any trials.. I have seen some discussion about coocon crawler.. was that successfull? http://wiki.apache.org/solr/SolrForrest I am

Re: crawler feed?

2007-02-07 Thread rubdabadub
Thorsten: Thank you very much for the update. On 2/7/07, Thorsten Scherler [EMAIL PROTECTED] wrote: On Wed, 2007-02-07 at 11:09 +0100, rubdabadub wrote: Hi: Are there relatively stand-alone crawler that are suitable/customizable for Solr? has anyone done any trials.. I have seen some

facet optimizing

2007-02-07 Thread Gunther, Andrew
Any suggestions on how to optimize the loading of facets? My index is roughly 35,000 and I am asking solr to return 6 six facet fields on every query. On large result sets with facet params set to false searching is zippy, but when set to true, and facet fields designated, it takes some time to

Re: crawler feed?

2007-02-07 Thread rubdabadub
Hi: Just want to say that my tiny experiment with Sami's Solr/Nutch integration worked :-!) Super thanks for the pointer. Which leads me to write the following.. It would be great if I could use this in my current project. This way I can eliminate my current python based aggregator/crawler

Re: facet optimizing

2007-02-07 Thread Mike Klaas
On 2/7/07, Gunther, Andrew [EMAIL PROTECTED] wrote: Yes most all terms are multi-valued which I can't avoid. Since the data is coming from a library catalogue I am translating a subject field to make a subject facet. That facet alone is the biggest, hovering near 39k. If I remove this

Re: facet optimizing

2007-02-07 Thread Chris Hostetter
: Andrew, I haven't yet found a successful way to implement the SOLR : faceting for library catalog data. I developed my own system, so for Just to clarify: the out of hte box faceting support Solr has at the moment is very deliberately refered to as SimpleFacets ... it's intended to solve

Re: cache warming optmization

2007-02-07 Thread karl wettin
7 feb 2007 kl. 19.04 skrev Erik Hatcher: I'm interested in improving my existing custom cache warming by being selective about what updates rather than rebuilding completely. I know it is not Solr, but I've made great progress on my cache that updates affected results only, on insert and

Re: facet optimizing

2007-02-07 Thread Yonik Seeley
On 2/7/07, Gunther, Andrew [EMAIL PROTECTED] wrote: Any suggestions on how to optimize the loading of facets? My index is roughly 35,000 35,000 documents? That's not that big. and I am asking solr to return 6 six facet fields on every query. On large result sets with facet params set to

Re: crawler feed?

2007-02-07 Thread Thorsten Scherler
On Wed, 2007-02-07 at 18:03 +0200, Sami Siren wrote: rubdabadub wrote: Hi: Are there relatively stand-alone crawler that are suitable/customizable for Solr? has anyone done any trials.. I have seen some discussion about coocon crawler.. was that successfull? There's also integration

Re: facet optimizing

2007-02-07 Thread Ryan McKinley
Are there any simple automatic test we can run to see what fields would support fast faceting? Is it just that the cache size needs to be bigger then the number of distinct values for a field? If so, it would be nice to add an /admin page that lists each field, the distinct value count and a

Re: facet optimizing

2007-02-07 Thread Chris Hostetter
: Is it just that the cache size needs to be bigger then the number of : distinct values for a field? basically yes, but the cache is going to be used for all filters -- not just those for a single facet (so your cache might be big enough that faceting on fieldA or fieldB is fine, but if you

RE: facet optimizing

2007-02-07 Thread Binkley, Peter
In the library subject heading context, I wonder if a layered approach would bring performance into the acceptable range. Since Library of Congress Subject Headings break into standard parts, you could have first-tier facets representing the main heading, second-tier facets with the main heading

Re: facet optimizing

2007-02-07 Thread Yonik Seeley
On 2/7/07, Binkley, Peter [EMAIL PROTECTED] wrote: In the library subject heading context, I wonder if a layered approach would bring performance into the acceptable range. Since Library of Congress Subject Headings break into standard parts, you could have first-tier facets representing the

Re: facet optimizing

2007-02-07 Thread Erik Hatcher
Yonik - I like the way you think Yeah! It's turtles (err, trees) all the way down. Erik /me Pulling the Algorithms book off my shelf so I can vaguely follow along. On Feb 7, 2007, at 8:22 PM, Yonik Seeley wrote: On 2/7/07, Binkley, Peter [EMAIL PROTECTED] wrote: In the

Re: facet optimizing

2007-02-07 Thread Yonik Seeley
On 2/7/07, Erik Hatcher [EMAIL PROTECTED] wrote: Yonik - I like the way you think Yeah! It's turtles (err, trees) all the way down. Heh... I'm still thinking/brainstorming about it... it only helps if you can effectively prune though. Each node in the tree could also keep the max