Re: Null Pointer Exception while indexing

2010-09-15 Thread Yonik Seeley
On Wed, Sep 15, 2010 at 1:12 PM, andrewdps mstpa...@gmail.com wrote: What could be possible error for 14-Sep-10 4:28:47 PM org.apache.solr.common.SolrException log SEVERE: java.util.concurrent.ExecutionException: java.lang.NullPointerException   at

Re: Facet Field Value truncation

2010-09-14 Thread Yonik Seeley
On Tue, Sep 14, 2010 at 3:35 PM, Niall O'Connor ocon...@jimmy.harvard.edu wrote: Has anyone come across a situation where they have seen their facet field values wrap into a new facet entry when the value exceeds 256 characters? Yes, for indexed string fields, there currently is a limit of 256

Re: solr.DateField: org.apache.solr.common.SolrException: Error while creating field

2010-09-14 Thread Yonik Seeley
On Tue, Sep 14, 2010 at 4:54 PM, h00kpub...@gmail.com h00kpub...@googlemail.com wrote: SEVERE: org.apache.solr.common.SolrException: Error while creating field 'metadata_last_modified{type=date,properties=indexed,stored,omitNorms}' from value '2010-09-14T22:29:24+0200' Different timezones are

Re: solr / lucene engineering positions in Boston, MA USA @ the Echo Nest

2010-09-10 Thread Yonik Seeley
On Fri, Sep 10, 2010 at 9:18 AM, Brian Whitman br...@echonest.com wrote: Hi all, brief message to let you know that we're in heavy hire mode at the Echo Nest. As many of you know we are very heavy solr/lucene users (~1bn documents across many many servers) and a lot of our staff have been

Re: SEVERE: java.io.IOException: The specified network name is no longer available

2010-09-10 Thread Yonik Seeley
On Fri, Sep 10, 2010 at 2:12 PM, brian519 bpear...@desire2learn.com wrote: Once we see the error, it is persistent.  Restarting Tomcat makes the error stop.  This is happening across a variety of deployments and networks, so I don't think there is an actual network problem.  Many other apps

Re: Null Pointer Exception with shardsfacets where some shards have no values for some facets.

2010-09-10 Thread Yonik Seeley
On Fri, Sep 10, 2010 at 7:21 PM, Ron Mayer r...@0ape.com wrote: Ron Mayer wrote: Yes, looks good now. Thanks! Great, thanks for the report! -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8

Re: Solr, c/s type ?

2010-09-09 Thread Yonik Seeley
On Thu, Sep 9, 2010 at 1:20 AM, Jonathan Rochkind rochk...@jhu.edu wrote: You _could_ use SolrJ with EmbeddedSolrServer.  But personally I wouldn't unless there's a reason to.  There's no automatic reason not to use the ordinary Solr HTTP api, even for an in-house application which is not a

Re: Null Pointer Exception with shardsfacets where some shards have no values for some facets.

2010-09-08 Thread Yonik Seeley
On Tue, Sep 7, 2010 at 8:31 PM, Ron Mayer r...@0ape.com wrote: Short summary:  * Mixing Facets and Shards give me a NullPointerException    when not all docs have all facets. https://issues.apache.org/jira/browse/SOLR-2110 I believe the underlying real issue stemmed from your use of a complex

Re: Re: Invariants on a specific fq value

2010-09-08 Thread Yonik Seeley
2010 at 1:32 PM, Markus Jelsma markus.jel...@buyways.nl wrote: Interesting! I haven't met the appends method before and i'll be sure to give it a try tomorrow. Try, the wiki [1] is not very clear on what it really does. Here's a comment from the example solrconfig.xml: !-- In addition to

Re: Null Pointer Exception with shardsfacets where some shards have no values for some facets.

2010-09-08 Thread Yonik Seeley
? Can you try trunk again now? -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8 On Wed, Sep 8, 2010 at 6:28 PM, Ron Mayer r...@0ape.com wrote: Yonik Seeley wrote: On Tue, Sep 7, 2010 at 8:31 PM, Ron Mayer r...@0ape.com wrote: Short summary:  * Mixing Facets and Shards

[ANN] Webinar, Sep 15: Mastering the Power of Faceted Search

2010-09-08 Thread Yonik Seeley
Folks, here's an upcoming Solr webinar sponsored by my employer. It's Hoss on faceting, so it should be good! -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8 --- Webinar Details Join us for a free

Re: Null Pointer Exception with shardsfacets where some shards have no values for some facets.

2010-09-07 Thread Yonik Seeley
Thanks for the report Ron, can you open a JIRA issue? What version of Solr is this? -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8 On Tue, Sep 7, 2010 at 8:31 PM, Ron Mayer r...@0ape.com wrote: Short summary:  * Mixing Facets and Shards give me a

Re: anyone use hadoop+solr?

2010-09-06 Thread Yonik Seeley
On Mon, Sep 6, 2010 at 8:37 AM, MitchK mitc...@web.de wrote: 10 % numShards(10) -  1 - doc 10 will be indexed at shard 1... and what about the older version at shard 2? I am no expert when it comes to cloudComputing and the other stuff. If you can point me to one or another reference where I

Re: anyone use hadoop+solr?

2010-09-06 Thread Yonik Seeley
On Mon, Sep 6, 2010 at 9:47 AM, MitchK mitc...@web.de wrote: are there any discussions about SolrCloud-indexing? Not recently - personally I've been sidetracked by other stuff. Mapping docs to shards is the easy part... take a hash of the id, and then I imagine the shard id (the label for the

Re: SolrCloud distributed indexing (Re: anyone use hadoop+solr?)

2010-09-06 Thread Yonik Seeley
On Mon, Sep 6, 2010 at 10:18 AM, MitchK mitc...@web.de wrote: [...consistent hashing...] But it doesn't solve the problem at all, correct me if I am wrong, but: If you add a new server, let's call him IP3-1, and IP3-1 is nearer to the current ressource X, than doc x will be indexed at IP3-1 -

Re: How to enable Unicode Support in Solr

2010-09-06 Thread Yonik Seeley
On Mon, Sep 6, 2010 at 10:30 AM, Walter Underwood wun...@wunderwood.org wrote: On Sep 6, 2010, at 1:49 AM, Lance Norskog wrote: 1) The XML file must include the UTF-8 encoding metadata in the first line. If it requires that, it isn't a legal XML parser. The encoding declaration is optional

Re: How to retrieve the full corpus

2010-09-06 Thread Yonik Seeley
On Mon, Sep 6, 2010 at 10:52 AM, Roland Villemoes r...@alpha-solutions.dk wrote: How can I retrieve all words from a Solr core? I need a list of all the words and how often they occur in the index. http://wiki.apache.org/solr/TermsComponent It doesn't currently stream though, so requesting

Re: false matches with ReversedWildcardFilterFactory

2010-09-03 Thread Yonik Seeley
On Thu, Sep 2, 2010 at 1:10 PM, Landon Kuhn landon9...@gmail.com wrote: Hello, I am using the ReversedWildcardFilterFactory, and I am wondering if there is a way to prevent false matches when a query token matches the reversed indexed token. For instance, the query *zemog* matches documents

Re: Slow facet sorting - lex vs count

2010-08-25 Thread Yonik Seeley
On Wed, Aug 25, 2010 at 7:22 AM, Eric Grobler impalah...@googlemail.com wrote: There is a huge difference doing facet sorting on lex vs count The strange thing is that count sorting is fast when setting a small limit. I realize I can do sorting in the client, but I am just curious why this is.

Re: Regd WSTX EOFException

2010-08-25 Thread Yonik Seeley
On Wed, Aug 25, 2010 at 6:41 AM, Pooja Verlani pooja.verl...@gmail.com wrote: Hi, Sometimes while indexing to solr, I am getting  the following exception. com.ctc.wstx.exc.WstxEOFException: Unexpected end of input block in end tag I think its some configuration issue. Kindly suggest. I have

Re: Slow facet sorting - lex vs count

2010-08-25 Thread Yonik Seeley
Conference, Boston Oct 7-8 Regards Eric On Wed, Aug 25, 2010 at 3:28 PM, Yonik Seeley yo...@lucidimagination.com wrote: On Wed, Aug 25, 2010 at 7:22 AM, Eric Grobler impalah...@googlemail.com wrote: There is a huge difference doing facet sorting on lex vs count The strange thing

Re: Solr searching performance issues, using large documents (now 1MB documents)

2010-08-25 Thread Yonik Seeley
On Wed, Aug 25, 2010 at 11:29 AM, Peter Spam ps...@mac.com wrote: So, I went through all the effort to break my documents into max 1 MB chunks, and searching for hello still takes over 40 seconds (searching across 7433 documents):        8 results (41980 ms) What is going on???  (scroll

Re: Slow facet sorting - lex vs count

2010-08-25 Thread Yonik Seeley
On Wed, Aug 25, 2010 at 10:55 AM, Eric Grobler impalah...@googlemail.com wrote: Thanks for the technical explanation. I will in general try to use lex and sort by count in the client if there are not too many rows. I just developed a patch that may help this scenario:

Re: Slow facet sorting - lex vs count

2010-08-25 Thread Yonik Seeley
On Wed, Aug 25, 2010 at 2:50 PM, Yonik Seeley yo...@lucidimagination.com wrote: On Wed, Aug 25, 2010 at 10:55 AM, Eric Grobler impalah...@googlemail.com wrote: Thanks for the technical explanation. I will in general try to use lex and sort by count in the client if there are not too many rows

Re: Slow facet sorting - lex vs count

2010-08-25 Thread Yonik Seeley
On Wed, Aug 25, 2010 at 7:22 AM, Eric Grobler impalah...@googlemail.com wrote: Hi Solr experts, There is a huge difference doing facet sorting on lex vs count The strange thing is that count sorting is fast when setting a small limit. I realize I can do sorting in the client, but I am just

Re: Solr searching performance issues, using large documents (now 1MB documents)

2010-08-25 Thread Yonik Seeley
On Wed, Aug 25, 2010 at 2:34 PM, Peter Spam ps...@mac.com wrote: This is a very small number of documents (7000), so I am surprised Solr is having such a hard time with it!! I do facet on 3 terms. Subsequent hello searches are faster, but still well over a second.  This is a very fast Mac

Re: Proper Escaping of Ampersands

2010-08-23 Thread Yonik Seeley
in this mail...) On 20.08.2010 16:57, Yonik Seeley wrote On Thu, Aug 19, 2010 at 11:33 AM, Nikolas Tautenhahn nik_s...@livinglogic.de wrote: But when I search for q=at%26s (=ats), I get nothing. That's the correct encoding if you're typing it directly into a browser address box. http

Re: Proper Escaping of Ampersands

2010-08-20 Thread Yonik Seeley
On Thu, Aug 19, 2010 at 11:33 AM, Nikolas Tautenhahn nik_s...@livinglogic.de wrote: Hi, I have a problem with, for example, company names like ATS. A Job is sending data to the solr 1.4 (also tested it with 1.4.1) index via python in XML, everything is escaped properly ( becomes amp;). When

Re: Fun with Spatial (Haversine formula)

2010-08-20 Thread Yonik Seeley
Lance, have you figured out what the issue is? Anyone know if this is a haversine limitation, or a bug? -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8 On Wed, Aug 18, 2010 at 1:54 AM, Lance Norskog goks...@gmail.com wrote: The Haversine formula in

Re: Fun with Spatial (Haversine formula)

2010-08-20 Thread Yonik Seeley
On Fri, Aug 20, 2010 at 11:05 PM, Lance Norskog goks...@gmail.com wrote: are latitudes equidistant on the surface of the sphere? Yes - each degree of latitude is ~69 miles. There is also a slight variation due to the earth not being a perfect sphere. -Yonik http://lucenerevolution.org

Re: tire fields and sortMissingLast

2010-08-19 Thread Yonik Seeley
On Thu, Aug 19, 2010 at 12:28 PM, harish.agarwal harish.agar...@gmail.com wrote: Just curious if there has been any progress on implementing sortMissingLast on TrieFields? Not yet - that info is not available from the lucene FieldCache. -Yonik http://www.lucidimagination.com

Re: queryResultCache has no hits for date boost function

2010-08-18 Thread Yonik Seeley
On Tue, Aug 17, 2010 at 6:29 PM, Peter Karich peat...@yahoo.de wrote: my queryResultCache has no hits. But if I am removing one line from the bf section in my dismax handler all is fine. Here is the line: recip(ms(NOW,date),3.16e-11,1,1) NOW has millisecond resolution, so it's actually a

Re: sort order of missing items

2010-08-18 Thread Yonik Seeley
On Tue, Aug 17, 2010 at 4:10 PM, Brad Dewar bde...@stfx.ca wrote: When items are sorted, are all the docs with the sort field missing considered tied in terms of their sort order, or are they indeterminate, or do they have some arbitrary order imposed on them (e.g. _docid_)? If it's a

Re: Jetty rerturning HTTP error code 413

2010-08-18 Thread Yonik Seeley
Yep, or you can submit the query via POST, which has a much bigger limit on the size of the body. -Yonik http://www.lucidimagination.com On Wed, Aug 18, 2010 at 3:58 PM, didier deshommes dfdes...@gmail.com wrote: Hi Alexandre, Have you tried setting a higher headerBufferSize?  Look in

Re: queryResultCache has no hits for date boost function

2010-08-18 Thread Yonik Seeley
On Wed, Aug 18, 2010 at 4:34 PM, Peter Karich peat...@yahoo.de wrote: Thanks a lot Yonik! Rounding makes sense. Is there a date math for the 'LAST_COMMIT'? No - but it's an interesting idea! -Yonik http://www.lucidimagination.com

Re: diacritics on query string

2010-08-15 Thread Yonik Seeley
On Fri, Aug 13, 2010 at 3:20 AM, Andrea Gazzarini andrea.gazzar...@atcult.it wrote:  Hi, I have a problem regarding a diacritic character on my query string : *q=intertestualità * which is encoded in *q=intertestualit%E0 The correct encoding is q=intertestualit%C3%A0 But I can see how

Re: diacritics on query string

2010-08-15 Thread Yonik Seeley
Update: it looks like this (what I consider buggy) behavior is common to both firefox and IE. Both correctly encode the path part of the URL, but neither encode the query string part in UTF-8 (I believe for back compat with old buggy websites). Chrome does use UTF-8 for both. It's easy to verify

Re: edismax pf2 and ps

2010-08-13 Thread Yonik Seeley
Perhaps a ps2 parameter to match pf2? -Yonik http://www.lucidimagination.com On Fri, Aug 13, 2010 at 2:11 PM, Ron Mayer r...@0ape.com wrote: Jayendra Patil wrote: We pretty much had the same issue, ended up customizing the ExtendedDismax code. In your case its just a change of a single line

Re: edismax pf2 and ps

2010-08-13 Thread Yonik Seeley
On Fri, Aug 13, 2010 at 2:38 PM, Ron Mayer r...@0ape.com wrote: Yonik Seeley wrote: Perhaps a ps2 parameter to match pf2? That might be nice. I could try to put together such a patch if people were interested. One more thing I've been contemplating is if my results might be even better

Re: how to update solr to older 1.5 builds instead of to trunk

2010-08-12 Thread Yonik Seeley
Another option is the 3x branch - that should still be able to read indexes from Solr 1.4/Lucene 2.9 I personally don't expect a 1.5 release to ever materialize. There will eventually be a Lucene/Solr 3.1 release off of the 3x branch, and a Lucene/Solr 4.0 release off of trunk. -Yonik

Re: how to update solr to older 1.5 builds instead of to trunk

2010-08-12 Thread Yonik Seeley
On Thu, Aug 12, 2010 at 12:24 PM, solr-user solr-u...@hotmail.com wrote: Thanks Yonik but http://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/solr/CHANGES.txt says that the lucene index has changed Right - but it will be able to read your older index. Do you need Solr 1.4 to be able

Re: Solr 1.4 - stats page slow

2010-08-11 Thread Yonik Seeley
FYI, I opened https://issues.apache.org/jira/browse/SOLR-2036 for this. -Yonik http://www.lucidimagination.com On Tue, Aug 10, 2010 at 8:35 PM, entdeveloper cameron.develo...@gmail.com wrote: Apologies if this was resolved, but we just deployed Solr 1.4.1 and the stats page takes over a

Re: Scoring on multi-valued fields

2010-08-03 Thread Yonik Seeley
On Tue, Aug 3, 2010 at 2:42 PM, oleg.gnatovskiy crooke...@gmail.com wrote: Oh sorry guys, I didn't correctly submit my original post to the mailing list. The original message was this: Hello all. We are having some trouble with queries similar to the type shown below: name: pizza OR

Re: Scoring on multi-valued fields

2010-08-03 Thread Yonik Seeley
On Tue, Aug 3, 2010 at 3:16 PM, oleg.gnatovskiy crooke...@gmail.com wrote: Sorry guess I messed up my example query. The query should look like this: name:pizza AND id:(10 OR 20 OR 30) Thus if I do name:pizza^10 AND id:(10 OR 20 OR 30)^0 wouldn't a document that has all the ids (10,20, and

Re: Good list of English words that get butchered by Porter Stemmer

2010-07-30 Thread Yonik Seeley
On Fri, Jul 30, 2010 at 4:41 PM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: I'm looking for a list of English  words that, when stemmed by Porter stemmer, end up in the same stem as  some similar, but unrelated words.  Below are some examples: # this gets stemmed to iron, so if you

Re: Querying throws java.util.ArrayList.RangeCheck

2010-07-29 Thread Yonik Seeley
On Thu, Jul 29, 2010 at 6:37 AM, Michael McCandless luc...@mikemccandless.com wrote: Can you run CheckIndex on the index and post the output? One of these days we need to get around to adding support for this in Solr's admin interface. http://issues.apache.org/jira/browse/SOLR-566 -Yonik

Re: myField:value does not seem to work

2010-07-29 Thread Yonik Seeley
Is membername an indexed field in the schema, and was it populated with something that would match Khai? If so, what is the fieldType in the schema for the membername field? -Yonik http://www.lucidimagination.com On Thu, Jul 29, 2010 at 3:17 PM, Khai Doan khaitd...@gmail.com wrote: Hello, My

Re: java GC overhead limit exceeded

2010-07-26 Thread Yonik Seeley
On Mon, Jul 26, 2010 at 7:17 PM, Jonathan Rochkind rochk...@jhu.edu wrote: I am now occasionally getting a Java GC overhead limit exceeded error in my Solr. This may or may not be related to recently adding much better (and more) warming querries. When memory gets tight, the JVM kicks of a

Re: Querying throws java.util.ArrayList.RangeCheck

2010-07-26 Thread Yonik Seeley
Do you have any custom code, or is this stock solr (and which version, and what is the request)? -Yonik http://www.lucidimagination.com On Tue, Jul 27, 2010 at 12:30 AM, Manepalli, Kalyan kalyan.manepa...@orbitz.com wrote: Hi,   I am stuck at this weird problem during querying. While querying

Re: Solr 3.1 dev

2010-07-23 Thread Yonik Seeley
On Fri, Jul 23, 2010 at 6:09 AM, Eric Grobler impalah...@googlemail.com wrote: I have a few questions :-) a) Will the next release of solr be 3.0 (instead of 1.5)? The next release will be 3.1 (matching the next lucene version off of the 3x branch). Trunk is 4.0-dev b) How stable/mature is

Re: a bug of solr distributed search

2010-07-23 Thread Yonik Seeley
On Fri, Jul 23, 2010 at 2:23 PM, MitchK mitc...@web.de wrote: why do we do not send the output of TermsComponent of every node in the cluster to a Hadoop instance? Since TermsComponent does the map-part of the map-reduce concept, Hadoop only needs to reduce the stuff. Maybe we even do not need

Re: a bug of solr distributed search

2010-07-23 Thread Yonik Seeley
On Fri, Jul 23, 2010 at 2:40 PM, MitchK mitc...@web.de wrote: That only works if the docs are exactly the same - they may not be. Ahm, what? Why? If the uniqueID is the same, the docs *should* be the same, don't they? Documents aren't supposed to be duplicated across shards... so the presence

Re: a bug of solr distributed search

2010-07-22 Thread Yonik Seeley
As the comments suggest, it's not a bug, but just the best we can do for now since our priority queues don't support removal of arbitrary elements. I guess we could rebuild the current priority queue if we detect a duplicate, but that will have an obvious performance impact. Any other

Re: limiting the total number of documents matched

2010-07-16 Thread Yonik Seeley
On Wed, Jul 14, 2010 at 5:46 PM, Paul p...@nines.org wrote: I thought of another way to do it, but I still have one thing I don't know how to do. I could do the search without sorting for the 50th page, then look at the relevancy score on the first item on that page, then repeat the search,

Re: Nested Function Query Syntax

2010-07-15 Thread Yonik Seeley
On Thu, Jul 15, 2010 at 11:51 AM, Rodrigo Rezende rcreze...@gmail.com wrote: I solved the problem. The correct syntax is: http://localhost:8983/solr/articles.0/select/?q={!func}query({!query v='hello'})fl=Document.title,score,debugQuery=on query() causes a new QParser to be created. so does

Re: Nested Function Query Syntax

2010-07-15 Thread Yonik Seeley
On Thu, Jul 15, 2010 at 12:49 PM, Rodrigo Rezende rcreze...@gmail.com wrote: Yeah, it is redundant, but I am using that to use the solr query response as input of a plugin function: http://localhost:8983/solr/articles.0/select/?q={!func}myFunction(query({!query v='the query string here'}))

Re: range query on TrieLongField - strange result?

2010-07-15 Thread Yonik Seeley
Yikes... confirmed! Something is very wrong here. -Yonik http://www.lucidimagination.com On Thu, Jul 15, 2010 at 8:47 PM, Yonik Seeley yo...@lucidimagination.com wrote: Hmmm, I'll try and duplicate. -Yonik http://www.lucidimagination.com 2010/7/15 Koji Sekiguchi k...@r.email.ne.jp: I see

Re: Problem with Wildcard searches in Solr

2010-07-12 Thread Yonik Seeley
On Mon, Jul 12, 2010 at 4:39 PM, imranak imranak...@gmail.com wrote: A general search like 'computer' returns results but 'com*er' doesn't return any results. This is due to issues with wildcards and stemming. computer is indexed and searched as comput... but it's not generally possible to stem

Re: Modifications to AbstractSubTypeFieldType

2010-07-07 Thread Yonik Seeley
On Wed, Jul 7, 2010 at 8:15 AM, Grant Ingersoll gsing...@apache.org wrote: Originally, I had intended that it was just for one Field Sub Type, thinking that if we ever wanted multiple sub types, that a new, separate class would be needed Right - this was my original thinking too.

Re: Modifications to AbstractSubTypeFieldType

2010-07-02 Thread Yonik Seeley
On Fri, Jul 2, 2010 at 9:51 AM, Mark Allan mark.al...@ed.ac.uk wrote: [...] The changes to AbstractSubTypeFieldType do not have any adverse effects on the solr.PointType class, so I'd quite like to suggest it gets included in the main solr source code.  Where can I send a patch for someone to

Re: SolrJ: BinaryRequestWriter with StreamingUpdateSolrServer

2010-07-01 Thread Yonik Seeley
There's an issue open for this: https://issues.apache.org/jira/browse/SOLR-1565 I'm not sure off the top of my head how much is involved in making it happen though. -Yonik http://www.lucidimagination.com On Thu, Jul 1, 2010 at 9:25 AM, Jan Høydahl / Cominvent jan@cominvent.com wrote: Hi,

Re: ValueSource/Function questions

2010-07-01 Thread Yonik Seeley
On Thu, Jul 1, 2010 at 1:02 PM, Blargy zman...@hotmail.com wrote: Can someone explain what the createWeight methods should do? Its primary function is to add Searcher context to anything that needs it (such as weighting a query). If you're not dealing with relevancy-type queries, value sources

Re: REST calls

2010-06-30 Thread Yonik Seeley
Solr's APIs are described as REST-like, and probably do qualify as restful the way the term is commonly used. I'm personally much more interested in making our APIs more powerful and easier to use, regardless of any REST purity tests. -Yonik http://www.lucidimagination.com

Re: OOM on uninvert field request

2010-06-30 Thread Yonik Seeley
On Tue, Jun 29, 2010 at 7:32 PM, Robert Petersen rober...@buy.com wrote: Hello I am trying to find the right max and min settings for Java 1.6 on 20GB index with 8 million docs, running 1.6_018 JVM with solr 1.4, and am currently have java set to an even 4GB (export JAVA_OPTS=-Xmx4096m

Re: REST calls

2010-06-30 Thread Yonik Seeley
On Wed, Jun 30, 2010 at 4:55 PM, Lance Norskog goks...@gmail.com wrote: Apparently this is not ReStFuL It is IMVHO insane. Patches welcome... -Yonik http://www.lucidimagination.com

Re: OOM on uninvert field request

2010-06-30 Thread Yonik Seeley
.123_contentAttributeToken.facet.method=fc -Yonik http://www.lucidimagination.com -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Wednesday, June 30, 2010 1:38 PM To: solr-user@lucene.apache.org Subject: Re: OOM on uninvert field request On Tue, Jun 29, 2010

Re: How to wait for StreamingUpdateSolrServer to finish?

2010-06-29 Thread Yonik Seeley
On Tue, Jun 22, 2010 at 9:38 AM, Stephen Duncan Jr stephen.dun...@gmail.com wrote: I'm prototyping using StreamingUpdateSolrServer.  I want to send a commit (or optimize) after I'm done adding all of my docs, rather than wait for the autoCommit to kick in.  However, since

Re: Similarity

2010-06-24 Thread Yonik Seeley
On Thu, Jun 24, 2010 at 3:17 PM, Blargy zman...@hotmail.com wrote: Can someone explain how I can override the default behavior of the tf contributing a higher score for documents with repeated words? For example: Query: foo Doc1: foo bar score 1.0 Doc2: foo foo bar score 1.1 Doc2

Re: Similarity

2010-06-24 Thread Yonik Seeley
On Thu, Jun 24, 2010 at 4:20 PM, Blargy zman...@hotmail.com wrote: Yonik Seeley-2-2 wrote: Depends on the larger context of what you are trying to do. Do you still want the idf and length norm relevancy factors?  If not, use a filter, or boost the particular clause with 0. I do want

Re: LocalParams, quotes, bug?

2010-06-16 Thread Yonik Seeley
On Wed, Jun 16, 2010 at 3:27 PM, Jonathan Rochkind rochk...@jhu.edu wrote: {!dismax qf=$some_qf}   = no problem, and debugQuery reveals it is indeed using the qf I desire. {!dismax qf='$some_qf'}  = Solr throws undefined field $some_qf. Is this a bug in Solr? Nope, it's by design. Parameter

Re: Questions about hsin and dist

2010-06-14 Thread Yonik Seeley
On Mon, Jun 14, 2010 at 3:35 PM, Chris Hostetter hossman_luc...@fucit.org wrote: fq={!frange l=0 u=1}hsin(XXX,44.0,73.0,latitude,longitude,true) ...where XXX is the radius of hte earth in miles (i didn't bother to look it up) That's what the docs say, but it doesn't really work in my

Re: Faceted Search Slows Down as index gets larger

2010-06-06 Thread Yonik Seeley
On Sun, Jun 6, 2010 at 7:38 AM, Furkan Kuru furkank...@gmail.com wrote: facet.limit = default value 100 facet.minCount is 1 The document count that matches the query is 8-10K in average. I did not calculate the terms (maybe using using facet.limit=-1 and facet.minCount=1) My index entirely

Re: Faceted Search Slows Down as index gets larger

2010-06-06 Thread Yonik Seeley
On Sun, Jun 6, 2010 at 1:12 PM, Furkan Kuru furkank...@gmail.com wrote: We try to provide real-time search. So the index is changing almost in every minute. We commit for every 100 documents received. The facet search is executed every 5 mins. OK, that's the problem - pretty much every

Re: Faceted Search Slows Down as index gets larger

2010-06-04 Thread Yonik Seeley
Faceting on a full-text field is hard. What version of Solr are you using? If it's 1.4 or later, try setting facet.method=enum And to use the filterCache less, try facet.enum.cache.minDf=100 -Yonik http://www.lucidimagination.com On Fri, Jun 4, 2010 at 10:31 AM, Furkan Kuru

Re: Faceted Search Slows Down as index gets larger

2010-06-04 Thread Yonik Seeley
On Fri, Jun 4, 2010 at 7:33 PM, Andy angelf...@yahoo.com wrote: Yonik, Just curious why does using enum improve the facet performance. Furkan was faceting on a text field with each word being a facet value. I'd imagine that'd mean there's a large number of facet values. According to the

Re: nested querries, and LocalParams syntax

2010-06-01 Thread Yonik Seeley
It's not clear if you're just trying to figure it all out, or get something specific to work. If you can give a specific example, we might be able to suggest easier ways to achieve it rather than going escape crazy :-) -Yonik http://www.lucidimagination.com On Tue, Jun 1, 2010 at 5:06 PM,

Re: nested querries, and LocalParams syntax

2010-06-01 Thread Yonik Seeley
my interest in figuring out proper escaping. Jonathan Yonik Seeley wrote: It's not clear if you're just trying to figure it all out, or get something specific to work. If you can give a specific example, we might be able to suggest easier ways to achieve it rather than going escape crazy

Re: Solr trunk and Jetty threadpool implementation problem

2010-05-29 Thread Yonik Seeley
David, if it's fast for you to reproduce, would it be possible for you to try the latest Jetty 6.1.24 and see if the issue still exists? http://dist.codehaus.org/jetty/ Seems like we should upgrade to 6.1.24 anyway (there were quite a few fixes in 6.1.23)

Re: matching only empty fields

2010-05-29 Thread Yonik Seeley
On Sat, May 29, 2010 at 10:01 AM, Lukas Kahwe Smith m...@pooteeweet.org wrote: Hi, I have implemented a facet search, where users essentially select what should not be included. I do this by constructing an fq filter where I match for the deselected items that I then negate:            $fq

Re: Solr trunk and Jetty threadpool implementation problem

2010-05-28 Thread Yonik Seeley
Wow, thanks for the heads-up David! This probably got inadvertently changed when Jetty was upgraded... sounds like we should prob change back to BoundedThreadPool as a default! -Yonik http://www.lucidimagination.com On Thu, May 27, 2010 at 5:43 PM, Smiley, David W. dsmi...@mitre.org wrote: I'd

Re: Generic questions

2010-05-27 Thread Yonik Seeley
On Thu, May 27, 2010 at 12:48 PM, Blargy zman...@hotmail.com wrote: Can someone explain to be what the state of Solr/Lucene is... didn't they recently combine? Yes, it started in March. Development is combined (committers, dev list, etc), but separate downloads and user lists will remain. I

Re: Generic questions

2010-05-27 Thread Yonik Seeley
On Thu, May 27, 2010 at 2:12 PM, Blargy zman...@hotmail.com wrote: What version is (http://svn.apache.org/repos/asf/lucene/dev/trunk/)? Im guessing its 4.0-dev Yes. but then where does 3.1 fit in? http://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/ Say I am running 1.4 and want

Re: fl and nulls

2010-05-26 Thread Yonik Seeley
On Wed, May 26, 2010 at 6:12 AM, dan sutton danbsut...@gmail.com wrote: In Solr 1.3 it looks like null fields were returned if requested with the fl param,, whereas with solr 1.4, nulls are omitted entirely. Can you elaborate on what you mean by null? Is this a string field with a zero length

Re: nested querries, and LocalParams syntax

2010-05-26 Thread Yonik Seeley
Have you seen http://wiki.apache.org/solr/LocalParams It may answer some of the questions, such as stating that backslash escaping works within quoted strings. I'd encourage you to try things out with the example server and adding debugQuery=true to your requests... it's the easiest way to

Re: Random Field

2010-05-01 Thread Yonik Seeley
On Sat, May 1, 2010 at 10:23 AM, Blargy zman...@hotmail.com wrote: Can someone explain a useful case for the RandomSortField? People sometimes have requirements to show different results to everyone (essentially randomly shuffling matches per person). -Yonik Apache Lucene Eurocon 2010 18-21 May

Re: Random Field

2010-05-01 Thread Yonik Seeley
On Sat, May 1, 2010 at 12:32 PM, Static Void static.void@gmail.com wrote: What would be more useful would be randomizing closely related hits. IE hits within 5% of each other This is not the use case I've encountered multiple times in the past, but it should also be doable by using the

Re: Trouble with parenthesis

2010-04-30 Thread Yonik Seeley
Pure negatives in lucene syntax don't match anything (solr currently only fixes this for you if it's a pure negative at the top-level, not embeded). Try changing (NOT periodicite:annuel) to (*:* NOT periodicite:annuel) But the second version below where you just removed the parens will be more

Re: How is DeletionPolicy supposed to work?

2010-04-30 Thread Yonik Seeley
Simply use what the default was in the example solrconfig.xml... there is no need to modify that unless you are doing something advanced. In the config below, you show maxOptimizedCommitsToKeep=1, which will increase index size by always keeping around one optimized commit point. -Yonik Apache

Re: StreamingUpdateSolrServer hangs

2010-04-30 Thread Yonik Seeley
On Thu, Apr 29, 2010 at 7:51 PM, Yonik Seeley yo...@lucidimagination.com wrote: I'm trying to reproduce now... single thread adding documents to a multithreaded client, StreamingUpdateSolrServer(addr,32,4) I'm currently at the 2.5 hour mark and 100M documents - no issues so far. I let it go

Re: StreamingUpdateSolrServer hangs

2010-04-29 Thread Yonik Seeley
On Fri, Apr 16, 2010 at 1:34 PM, Sascha Szott sz...@zib.de wrote: In my case the whole application hangs and never recovers (CPU utilization goes down to near 0%). Interestingly, the problem reproducibly occurs only if SUSS is created with *more than 2* threads. Is your application also using

Re: StreamingUpdateSolrServer hangs

2010-04-29 Thread Yonik Seeley
On Thu, Apr 29, 2010 at 6:04 PM, Lance Norskog goks...@gmail.com wrote: In solrconfig.xml, there is a parameter controlling remote streaming:   requestDispatcher handleSelect=true      !--Make sure your system has some authentication before enabling remote streaming!  --      requestParsers

Re: StreamingUpdateSolrServer hangs

2010-04-29 Thread Yonik Seeley
, Yonik Seeley yo...@lucidimagination.com wrote: On Fri, Apr 16, 2010 at 1:34 PM, Sascha Szott sz...@zib.de wrote: In my case the whole application hangs and never recovers (CPU utilization goes down to near 0%). Interestingly, the problem reproducibly occurs only if SUSS is created with *more than

Re: Replicate cores from master to slave

2010-04-28 Thread Yonik Seeley
On Tue, Apr 27, 2010 at 9:56 PM, Chris Hostetter hossman_luc...@fucit.org wrote: but as i understand the new cloud stuff (by which i mean: i don't understand the new cloud stuff, but i've heard rumors) this will be possible with that functionality. Yeah, that should be the goal. The

Re: Replicate cores from master to slave

2010-04-28 Thread Yonik Seeley
On Wed, Apr 28, 2010 at 10:14 AM, Jason Rutherglen jason.rutherg...@gmail.com wrote: I guess I didn't explain it properly. I want to create a core on the master, and then have N slaves also (aka replicate) create those new core(s) on the slave servers, then of course, begin to replicate (yeah,

Re: Solr Cloud Gossip Protocols

2010-04-28 Thread Yonik Seeley
On Wed, Apr 28, 2010 at 2:23 PM, Jon Baer jonb...@gmail.com wrote: From what I understand Cassandra uses a generic gossip protocol for node discovery (custom), will the Solr-Cloud have something similar? SolrCloud uses zookeeper, so node discovery is a simple matter of looking there. Nodes

Re: StreamingUpdateSolrServer hangs

2010-04-16 Thread Yonik Seeley
, Sascha Szott sz...@zib.de wrote: Hi Yonik, Yonik Seeley wrote: Stephen, were you running stock Solr 1.4, or did you apply any of the SolrJ patches? I'm trying to figure out if anyone still has any problems, or if this was fixed with SOLR-1711: I'm using the latest trunk version (rev. 934846

Re: Handling missing date fields in a date-oriented function query

2010-04-16 Thread Yonik Seeley
On Fri, Apr 16, 2010 at 4:42 PM, Chris Harris rygu...@gmail.com wrote: The docs dated exactly *at* the epoch, though, are trouble, because I can't tell those docs apart from the undated docs in my function query. Neither can Solr currently... it's a Lucene FieldCache limitation. The other

Re: Displaying fieldValueCache stats in Solr 1.4 admin/stats page

2010-04-13 Thread Yonik Seeley
This is an implicit cache (if you don't define it, it will still exist and show up on stats.jsp). Can you be more specific about FieldValueCache stats are not getting displayed If you start the example server, go to the stats page, and search for fieldValueCache, is it there? Or do you mean that

Re: Solr giving 500's

2010-04-09 Thread Yonik Seeley
Looks like you're missing one of the index files... segments_n It points to all the other index files. -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague On Fri, Apr 9, 2010 at 6:20 AM, william pink will.p...@gmail.com wrote: Hi, I was seeing this error from Solr this morning

Re: json.nl=arrarr does not work with facet.date

2010-04-09 Thread Yonik Seeley
On Fri, Apr 9, 2010 at 1:04 PM, fabritw fabr...@gmail.com wrote: Apologies for the second post, I noticed the json.nl=arrarr does work with facet.field but not with facet.date? Hmmm, this is because date faceting uses a SimpleOrderedMap instead of a NamedList (implying that access-like-a-map

<    7   8   9   10   11   12   13   14   15   16   >