Analyzing CSV phrase fields

Hey all, Very basic question.. I want to index fields of comma separated values: Example document: id: 1 title: Football Teams keywords: philadelphia eagles, cleveland browns, new york jets id: 2 title: Baseball Teams keywords:"philadelphia phillies", "new york yankees", "cleveland indians" A q

Re: new faceting algorithm

And if you want to verify that the new faceting code has indeed kicked in, some statistics are logged, like: Nov 24, 2008 11:14:32 PM org.apache.solr.request.UnInvertedField uninvert INFO: UnInverted multi-valued field features, memSize=14584, time=47, phase1=47, nTerms=285, bigTerms=99, termInst

Solr/PHP/Java job opportunity

Lead PHP Developer Successful Holland, Michigan based ecommerce retailer seeks a talented senior developer to improve and build upon our existing customer facing systems. The role is very hands on and PHP intensive but will involve a growing leadership presence as the company expands. A minimum o

Re: Facet Query (fq) and Query (q)

First, terminology clarification. fq is *filter* query. facet.query is facet query. On Nov 24, 2008, at 9:55 AM, Jae Joo wrote: I am having some trouble to utilize the facet Query. As I know that the facet Query has better performance that simple query (q). The performance is (about?) t

Re: broken socket in Jetty causing invalid XML ?

I thought the Jetty maxFormContentSize was only for form data (not for the POST body). Does increasing this param help? -Yonik On Mon, Nov 24, 2008 at 2:45 PM, Anoop Bhatti <[EMAIL PROTECTED]> wrote: > Hello Solr Community, > > I'm getting the stracktrace below when adding docs using the > Commo

new faceting algorithm

A new faceting algorithm has been committed to the development version of Solr, and should be available in the next nightly test build (will be dated 11-25). This change should generally improve field faceting where the field has many unique values but relatively few values per document. This new

Re: New user questions

: I've gotten Solr up and running, I can ingest the demo objects and : query them via the admin tool, so far so good. Now, how do I ingest : some basic XML, how can I pull from an existing MySQL database, what : about pulling records in via OAI? I'm assuming I need to write some : schemas for th

RE: Sorting and JVM heap size ....

Thanks Yonik. It explains. Regards, Sourav -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley Sent: Monday, November 24, 2008 7:07 PM To: solr-user@lucene.apache.org Subject: Re: Sorting and JVM heap size On Mon, Nov 24, 2008 at 9:19 PM, s

Re: Sorting and JVM heap size ....

On Mon, Nov 24, 2008 at 9:19 PM, souravm <[EMAIL PROTECTED]> wrote: > Hi Yonik, > > Thanks again for the detail input. > > Let me try to re-confirm my understanding - > > 1. What you say is - if sorting is asked for a field, the same field from all > documents, which are indexed, would be put in a

Re: a question about solr phrasequery process.

2008/11/24 finy finy <[EMAIL PROTECTED]>: > thanks for your help > *** > set the position increment of all of the > tokens you get to 0... this should have the effect of an "OR". > *** > how to do this? in solr solrconfig.xml?

Re: Phrase query search with stopwords

Ack! I tried it too, and it failed for me also. The analysis page indicates that the tokens are all in the same positions... need to look into this deeper. Could you open up a JIRA issue? -Yonik On Mon, Nov 24, 2008 at 5:58 PM, Robert Haschart <[EMAIL PROTECTED]> wrote: > Yonik, > > I did make s

RE: Sorting and JVM heap size ....

Hi Yonik, Thanks again for the detail input. Let me try to re-confirm my understanding - 1. What you say is - if sorting is asked for a field, the same field from all documents, which are indexed, would be put in a memory in an un-inverted form. So given this if I have a field of String type w

Re: solr.WordDelimiterFilterFactory

On Thu, Nov 20, 2008 at 9:20 AM, Daniel Rosher <[EMAIL PROTECTED]> wrote: > I'm trying to index some content that has things like 'java/J2EE' but with > solr.WordDelimiterFilterFactory and parameters [generateWordParts="1" > generateNumberParts="0" catenateWords="0" catenateNumbers="0" > catenateAl

Re: Sorting and JVM heap size ....

On Mon, Nov 24, 2008 at 8:48 PM, souravm <[EMAIL PROTECTED]> wrote: > I have around 200M documents in index. The field I'm sorting on is a date > string (containing date and time in dd-mmm- hh:mm:yy format) and the > field is part of the search criteria. > > Also please note that the number

Re: Using Solr for indexing emails

On Tue, 2008-11-25 at 12:20 +1100, Norberto Meijome wrote: > > Store the per-mailbox highest indexed UID in a new unique field created > > like "//". Always update it by deleting the > > old one first and then adding the new one. > > you mean delete, commit, add, commit? if you replace the record,

RE: Sorting and JVM heap size ....

Hi Yonik, Thanks for the reply. I have around 200M documents in index. The field I'm sorting on is a date string (containing date and time in dd-mmm- hh:mm:yy format) and the field is part of the search criteria. Also please note that the number of documents returned by the search criteri

Re: Sorting and JVM heap size ....

On Mon, Nov 24, 2008 at 6:26 PM, souravm <[EMAIL PROTECTED]> wrote: > I have indexed data of size around 20GB. My JVM memory is 1.5GB. > > For this data if I do a query with sort flag on (for a single field) I always > get java out of memory exception even if the number of hit is 0. With no > sor

unsubscribe me

Charlie Alswiti eBusiness SW Development NORTEL Information Technology 3500 Carling Avenue | Ottawa, Ontario | K2H-8E9 Tel +1 (613) 763 7371 ESN 393 [EMAIL PROTECTED]

Re: Querying Ranges Problem

Ensure that the fieldType maps back to solr.SortableIntField rather than solr.IntField -Yonik On Mon, Nov 24, 2008 at 5:19 PM, Jake Conk <[EMAIL PROTECTED]> wrote: > I have the following query: > > > q=(+thread_title_t:test OR +posts_t_ns_mv:test) AND locked_i:0 AND > replies_i:[50 TO *] > > > I

Re: Using Solr for indexing emails

On Mon, 24 Nov 2008 20:21:17 +0200 Timo Sirainen <[EMAIL PROTECTED]> wrote: > I think I gave enough reasons above for why I don't like this > solution. :) I also don't like adding new shared global state databases > just for Solr. Solr should be the one shared global state database.. fair enough

RE: facet sort by ranking

: We having 100 category and each category having it own internal ranking. : Let consider if I search for any product and its fall under 30 categories : and we are showing top 10 categories in filter so that user can filter there : results. the faceting solr provides out of the box is explicitly

Re: a question about solr phrasequery process.

thanks for your help *** set the position increment of all of the tokens you get to 0... this should have the effect of an "OR". *** how to do this? in solr solrconfig.xml? or schema.xml? please help me. 2008/11/24 Yonik Seele

Re: Solr Core Admin

Jeff: would you mind opening a Jira issue about this? at a minum there should be a better error message -- but i think we could probably also improve the way the file is updated to not require directory write perms. : Date: Thu, 20 Nov 2008 08:59:26 -0800 : From: Jeff Newburn <[EMAIL PROTECTE

Re: Boosting by field contents

: text field:value : : I want to return all documents with 'text'. Documents where 'field = value' : boosted over documents where 'field = some other value'. : : This query does it: : : (text field:value)^100 (text -field:value) : : Is there a simpler way? (I might have left out crucia

Re: Question about dismax 'mm' - give boost to searches by location

: of those 4 words. So whats happening is last will and testament from all : states are returned although user specifically asked for florida will. I : don't want to alter the 'mm' either because its working fine for other : searches. Just for the search terms with a 'location' , i want to be able

Re: solr.WordDelimiterFilterFactory

: I'm trying to index some content that has things like 'java/J2EE' but with : solr.WordDelimiterFilterFactory and parameters [generateWordParts="1" : generateNumberParts="0" catenateWords="0" catenateNumbers="0" : catenateAll="0" splitOnCaseChange="0"] this ends up tokenized as : 'java','j','2',E

Re: Phrase query search with stopwords

: Subject: Phrase query search with stopwords : In-Reply-To: <[EMAIL PROTECTED]> http://people.apache.org/~hossman/#threadhijack Thread Hijacking on Mailing Lists When starting a new discussion on a mailing list, please do not reply to an existing message, instead start a fresh email. Even if

Re: Please Help !! Question about Query Phrase Slop (qs) in dismax

: Subject: Re: Please Help !! Question about Query Phrase Slop (qs) in dismax : : : Please help someone...i've been waiting for an answer for the last couple of : days & no one seems to be helping out here. I did search the wiki & this Please don't send messages like this. This is a voluntee

Re: Question about Query Phrase Slop (qs) in dismax

: >From the solr wiki, it sounded like if qs is set to 5 for example, & if the : search term is 'child custody', only docs with 'child' & 'custody' within 5 : words of one another would be returned in results. Is this correct? If so, No. as explained on the wiki... >> Amount of slop on phrase q

Sorting and JVM heap size ....

Hi, I have indexed data of size around 20GB. My JVM memory is 1.5GB. For this data if I do a query with sort flag on (for a single field) I always get java out of memory exception even if the number of hit is 0. With no sorting (or default sorting with score) the query works perfectly fine.` I

Re: Query for Distributed search -

: Subject: Query for Distributed search - : In-Reply-To: <[EMAIL PROTECTED]> http://people.apache.org/~hossman/#threadhijack Thread Hijacking on Mailing Lists When starting a new discussion on a mailing list, please do not reply to an existing message, instead start a fresh email. Even if yo

Schema Design Guidance

Hi, and apologies in advance for the lengthy question! I'm looking to use Solr to power searching & browsing over a large set of product data stored in a relational db. I'm wandering what the most appropriate schema design strategy to use is. A simplified view of the relational data is: Shop (~10

Re: Phrase query search with stopwords

Yonik, I did make sure enablePositionIncrements="true" for both indexing and queries and just did a test where I re-indexed a couple of test record sets, and submitted a query from the solr admin page, this time searching for title_text:"gone with the wind" which should return three hits,

Querying Ranges Problem

I have the following query: q=(+thread_title_t:test OR +posts_t_ns_mv:test) AND locked_i:0 AND replies_i:[50 TO *] I have replies_i which is an integer field set to return me back documents that have a value 50 or greater but the problem is I'm getting back results with the replied_i field colu

broken socket in Jetty causing invalid XML ?

Hello Solr Community, I'm getting the stracktrace below when adding docs using the CommonsHttpSolrServer.add(Collection docs) method. The server doesn't seem to be able to recover from this error. We are adding a collection with 1,000 SolrInputDocument's at a time. I'm using Solr 1.3.0 and Java 1

Re: [VOTE] Community Logo Preferences

https://issues.apache.org/jira/secure/attachment/12394282/solr2_maho_impression.png https://issues.apache.org/jira/secure/attachment/12394266/apache_solr_b_red.jpg thanks to everyone who contributed, rob On Sun, Nov 23, 2008 at 11:59 AM, Ryan McKinley <[EMAIL PROTECTED]> wrote: > Please submit yo

Re: Phrase query search with stopwords

Robert, I've reproduced (sort of) this bad behavior with the example schema. There was an example configuration "bug" introduced in SOLR-521 where enablePositionIncrements="true" was only set on the index analyzer but not the query analyzer for the "text" fieldType. A query on the example data of

Re: port of Nutch CommonGrams to Solr for help with slow phrase queries

This technique was used at Infoseek in 1996, and is very effective. It also gives a relevance improvement, because you have an estimate of IDF for phrases (exact for two-word phrases). The terms "the" and "who" will be very common, but "the who" is quite rare and will have a big IDF. wunder On 1

port of Nutch CommonGrams to Solr for help with slow phrase queries

Hello all, We are having problems with extremely slow phrase queries when the phrase query contains a common words. We are reluctant to just use stop words due to various problems with false hits and some things becoming impossible to search with stop words turned on. (For example "to be or not to

Re: Using Solr for indexing emails

On Mon, 2008-11-24 at 14:25 +1100, Norberto Meijome wrote: > > The main problem is that before doing the search, I first have to check > > if there are any unindexed messages and then add them to Solr. This is > > done using a query like: > > - fl=uid > > - rows=1 > > - sort=uid desc > > - q=ui

Re: [VOTE] Community Logo Preferences

1. https://issues.apache.org/jira/secure/attachment/12394263/apache_solr_a_blue.jpg 2. https://issues.apache.org/jira/secure/attachment/12394282/solr2_maho_impression.png Thanks for your time! Matthew Runo Software Engineer, Zappos.com [EMAIL PROTECTED] - 702-943-7833 On Nov 23, 2008, at 8:59

Re: Very bad performance

Great news! I’ll try to test your patch tomorrow. Thanks again, Cédric Yonik Seeley wrote: > > On Mon, Nov 24, 2008 at 9:19 AM, Cedric Houis <[EMAIL PROTECTED]> > wrote: > >> You've said that someone will work to improve faceted search. Could you >> tell >> me where I can tract those evol

Re: Score always 0.0

Thanks a lot Yonik. Effectively, I had something like that in my code: SchemaField sfield = _schema.getFieldOrNull(fieldName); if (sfield != null) doc.add(sfield.createField(nodeValue, 0)); I set the boost value to 1 and it works now… Thanks again, Cédric Yonik Seeley wrote: > > Looks l

Third Hadoop Get Together @ Berlin

The third German Hadoop get together is going to take place at 9th of December at newthinking store in Berlin: http://upcoming.yahoo.com/event/1383706/?ps=6 You can order drinks directly at the bar in the newthinking store. As this Get Together takes place in December - Christmas time - there

Lucene 2.3.1 vs 2.4 benchmarks using LuSql

I have some simple indexing benchmarks comparing Lucene 2.3.1 with 2.4: http://zzzoot.blogspot.com/2008/11/lucene-231-vs-24-benchmarks-using-lusql.html In the next couple of days I will be running benchmarks comparing Solr's DataImportHandler/JdbcDataSource indexing performance with LuSql and wil

RE: Query for Distributed search -

Hi, I understand your point on how do I do it myself in my Java code. However, I'm more interested to know how the default behaviour of DistributedSearch work when I issue a command like "curl 'http://localhost:8983/solr/select?shards=localhost:8983/solr,localhost:7574/solr&indent=true&q=ipod+

Re: [VOTE] Community Logo Preferences

https://issues.apache.org/jira/secure/attachment/12393951/sslogo-solr-classic.png https://issues.apache.org/jira/secure/attachment/12394291/apache_solr_contour.png https://issues.apache.org/jira/secure/attachment/12394376/solr_sp.png

Re: [VOTE] Community Logo Preferences

https://issues.apache.org/jira/secure/attachment/12394263/apache_solr_a_blue.jpg

Re: Score always 0.0

Looks like the norm for the doc for that field is 0... did you perhaps boost the field or document by 0 when indexing? -Yonik On Mon, Nov 24, 2008 at 9:59 AM, Cedric Houis <[EMAIL PROTECTED]> wrote: > > Hi Solr team. > > I've got a question about scoring; when we make a search like this: "bush >

Re: [VOTE] Community Logo Preferences

https://issues.apache.org/jira/secure/attachment/12394314/apache_soir_001.jpg https://issues.apache.org/jira/secure/attachment/12394366/solr3_maho.png https://issues.apache.org/jira/secure/attachment/12394353/solr.s5.jpg https://issues.apache.org/jira/secure/attachment/12394376/solr_sp.png https:/

Re: [VOTE] Community Logo Preferences

https://issues.apache.org/jira/secure/attachment/12394282/solr2_maho_impression.png

RE: [VOTE] Community Logo Preferences

https://issues.apache.org/jira/secure/attachment/12392306/apache_solr_sun.png https://issues.apache.org/jira/secure/attachment/12391946/apache_solr_burning.png https://issues.apache.org/jira/secure/attachment/12394267/apache_solr_c_blue.jpg https://issues.apache.org/jira/secure/attachment/1239407

Re: Very bad performance

On Mon, Nov 24, 2008 at 9:19 AM, Cedric Houis <[EMAIL PROTECTED]> wrote: > I've made the test with the latest nightly build of Solr. Performances are > similar. Yep, see http://www.nabble.com/NIO-not-working-yet-to20468152.html#a20468152 > You've said that someone will work to improve faceted sea

Re: AND query on multivalue text

>On Nov 24, 2008, at 8:52 AM, Erik Hatcher wrote: > > On Nov 24, 2008, at 8:37 AM, David Santamauro wrote: i need to search something as myText:billion AND guarantee i need to be extracted only the record where the words exists in the same value (in this case only th

Re: a question about solr phrasequery process.

2008/11/24 finy finy <[EMAIL PROTECTED]>: > maybe you have make a misunderstanding about what i say, > > another example: > > the keyword "oneworldonedream" in myself analyzer will be analyzed about > four token： > one world one dream OK, this is just the way the Lucene and Solr query parsers wor

Re: a question about solr phrasequery process.

maybe you have make a misunderstanding about what i say, another example: the keyword "oneworldonedream" in myself analyzer will be analyzed about four token： one world one dream when i use myself analyzer in solr, i input "oneworldonedream", solr give the follow result: *-*

Score always 0.0

Hi Solr team. I’ve got a question about scoring; when we make a search like this: “bush obama^9”, we always have a 0.0 score. Should we not have a higher score we the document contains both “bush” and “obama”? Our default search filed is defined like this: Here is debug result: bush ob

Facet Query (fq) and Query (q)

I am having some trouble to utilize the facet Query. As I know that the facet Query has better performance that simple query (q). Here is the example. http://localhost:8080/test_solr/select?q=*:*&facet=true&fq=state:CA&facet.mincount=1&facet.field=city&facet.field=sector&facet.limit=-1&sort=score+

Re: AND query on multivalue text

On Nov 24, 2008, at 8:52 AM, Erik Hatcher wrote: On Nov 24, 2008, at 8:37 AM, David Santamauro wrote: i need to search something as myText:billion AND guarantee i need to be extracted only the record where the words exists in the same value (in this case only the first record) because in

solr internationalization support

hi, 1)Which languages solr supports out-of-the box other than english? 2)What are the analyzers(stemmer,synonym,tokenizer etc) it provides for each language? 3)Shall we create our own analyzers for any languages?(If possible explain how?) thanx in advance -- View this message in context: htt

Re: Very bad performance

Hi Yonik. I’ve made the test with the latest nightly build of Solr. Performances are similar. You’ve said that someone will work to improve faceted search. Could you tell me where I can tract those evolutions? Thanks in advance, Regards, Cédric Yonik Seeley wrote: > > On Fri, Nov 7, 2008

Re: AND query on multivalue text

On Nov 24, 2008, at 8:37 AM, David Santamauro wrote: i need to search something as myText:billion AND guarantee i need to be extracted only the record where the words exists in the same value (in this case only the first record) because in the 2nd record the two words are in different value

Re: a question about solr phrasequery process.

On Mon, Nov 24, 2008 at 3:55 AM, finy finy <[EMAIL PROTECTED]> wrote: > hello everyone: > > i use solr 1.2, i find a problem about solr1.2, > > when i search some keyword, i use myself analyzer, i find that solr > consider my terms as PhraseQuery, > > for example,solr parser's result is:

Re: AND query on multivalue text

Hello all, I'm new to the list but want to say great work! ... see comment below On Nov 24, 2008, at 7:59 AM, Erik Hatcher wrote: On Nov 24, 2008, at 6:12 AM, Antonio Zippo wrote: is it possible to have and AND query on a multivalue text? i need to search something as myText:billion AND

Re: AND query on multivalue text

On Nov 24, 2008, at 6:12 AM, Antonio Zippo wrote: is it possible to have and AND query on a multivalue text? i need to extract the record only if the words are contained inside the same value for example 1st record: The U.S. government has announced a massive rescue package for Citigrou

Re: DateField Problem

On Nov 24, 2008, at 8:03 AM, Peer Allan wrote: I am trying to update a system that uses solr to use version 1.3, but have stumbled across a problem I can’t seem to fix. Solr is throwing errors on a date and I don't know why. Here is the request XML: Movie 1 Movie:1 Napoleon

DateField Problem

Hello all, I am trying to update a system that uses solr to use version 1.3, but have stumbled across a problem I can¹t seem to fix. Solr is throwing errors on a date and I don't know why. Here is the request XML: Movie 1 Movie:1 Napoleon Dynamite Cool movie about a goof

AND query on multivalue text

hi all, is it possible to have and AND query on a multivalue text? i need to extract the record only if the words are contained inside the same value for example 1st record: The U.S. government has announced a massive rescue package for Citigroup, saying it would guarantee more than $300 bi

Problem generating summaries for redirected url´s

Hello everyone, I am using Nutch with the Solr plugin, and I am having a problem indexing redirected url´s. While Solr generates its fields just fine, as if they belonged to the redirected url, Nutch leaves the summary field empty. It seems as if Nutch tries to generate the summary of the original

I: Highlighting wildcards

q=tele?* seems to not work the query is ok... but the highlight returns records without the highlighted text (only the uniqueKey in highlight section) > To do it now, you'd have to switch the query parser to using the old style wildcard (and/or prefix) query, which is slower on large indexe

Re: Query for Distributed search -

Up to your solr client. On Mon, Nov 24, 2008 at 1:24 PM, souravm <[EMAIL PROTECTED]> wrote: > Hi, > > Looking for some insight on distributed search. > > Say I have an index distributed in 3 boxes and the index contains time and > text data (typical log file). Each box has index for different tim

Re: a question about solr queryparser

i have plugged in my custom Analyzer (correctly) in the solrconfig.xml, my analyzer result is : one world one dream but when i input : oneworld onedream, solr parse this into : PhraseQquery(title:"one world one dream") , solr consider this as a PhraseQuery, but this is not what i want to get. i

a question about solr phrasequery process.

hello everyone: i use solr 1.2, i find a problem about solr1.2, when i search some keyword, i use myself analyzer, i find that solr consider my terms as PhraseQuery, for example,solr parser's result is: PhraseQuery( title:"i am good man"), but i want to get the query: title:i title:

Re: Query for Distributed search -

If you for instance use SolrJ and the HttpSolrServer, you could for instance add logic to your querying making your searches more efficient! That is partially the idea of sharding, right? :) So if the user wants to search for a log file in June, your application knows that June logs are sto