RE: Ensuring stable timestamp ordering

2010-11-02 Thread Toke Eskildsen
Dennis Gearon [gear...@sbcglobal.net] wrote: how about a timrstamp with either a GUID appended on the end of it? Since long (8 bytes) is the largest atomic type supported by Java, this would have to be represented as a String (or rather BytesRef) and would take up 4 + 32 bytes + 2 * 4 bytes

using HebMorph

2010-11-02 Thread mark peleus
Hi I'm trying to use HebMorph, a new Hebrew analyzer. http://github.com/itaifrenkel/HebMorph/tree/master/java/ The instructions says: 1. Download the code from herehttp://github.com/synhershko/HebMorph/tree/master/java/ . 2. Use the hebmorph ant

Solr MySQL Adding new column to table

2010-11-02 Thread nitin.vanaku...@gmail.com
Hello Techies, I am new to Solr, i am using it with mysql. Suppose i have table called person in mysql with two columns name, age and i have configured mysql in solr. now i have added a new column to person table called phoneNumber, is it possible for solr to recognize new column dynamically ?

RE: Solr MySQL Adding new column to table

2010-11-02 Thread sivaprasad
You have to change the old configuration for the newly added field.Or you can use dynamic fields concept. Go through the link http://wiki.apache.org/solr/SchemaXml -Original Message- From: nitin.vanaku...@gmail.com [via Lucene] ml-node+1826759-1041834398-225...@n3.nabble.com Sent:

Re: Phrase Query Problem?

2010-11-02 Thread Tod
On 11/1/2010 11:14 PM, Ken Stanley wrote: On Mon, Nov 1, 2010 at 10:26 PM, Todlistac...@gmail.com wrote: I have a number of fields I need to do an exact match on. I've defined them as 'string' in my schema.xml. I've noticed that I get back query results that don't have all of the words I'm

RE: Solr MySQL Adding new column to table

2010-11-02 Thread nitin.vanaku...@gmail.com
Hi Sivaprasad, first of all thanks for your kind response. i gone through that link, if i use the dynamicField concept,still i need to alter the query in data-config.xml right! thanks Nitin -- View this message in context:

RE: Solr MySQL Adding new column to table

2010-11-02 Thread Ephraim Ofir
Not if you use 'SELECT * FROM person' Ephraim Ofir -Original Message- From: nitin.vanaku...@gmail.com [mailto:nitin.vanaku...@gmail.com] Sent: Tuesday, November 02, 2010 11:19 AM To: solr-user@lucene.apache.org Subject: RE: Solr MySQL Adding new column to table Hi Sivaprasad, first

Dynamically create new core

2010-11-02 Thread samarth s
Hi, I have a requirement of dynamically creating new cores(master). Each core should have a replicated slave core. I am working with Java and using SolrJ as my solr client. I came across CoreAdminRequest class and looks like the way to go. CoreAdminRequest.createCore(NewCore1, NewCore1,

RE: Solr MySQL Adding new column to table

2010-11-02 Thread nitin.vanaku...@gmail.com
ok. i have one more issue. i am getting following exception can you please explore on it INFO: Creating a connection for entity person with URL: jdbc:mysql://localhost:3306/example Nov 2, 2010 3:34:11 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call INFO: Time taken for

RE: Solr MySQL Adding new column to table

2010-11-02 Thread Ephraim Ofir
Your uniqueKey field is defined as id (in schema.xml) and your query doesn't return an id field. Ephraim Ofir -Original Message- From: nitin.vanaku...@gmail.com [mailto:nitin.vanaku...@gmail.com] Sent: Tuesday, November 02, 2010 12:10 PM To: solr-user@lucene.apache.org Subject: RE: Solr

Re: How to use polish stemmer - Stempel - in schema.xml?

2010-11-02 Thread Jakub Godawa
Thank you Bernd! I couldn't make it run though. Here is my problem: 1. There is a file ~/apache-solr-1.4.1/ifaq/lib/stempel-1.0.jar 2. In ~/apache-solr-1.4.1/ifaq/solr/conf/solrconfig.xml there is a directive: lib path=../lib/stempel-1.0.jar / 3. In ~/apache-solr-1.4.1/ifaq/solr/conf/schema.xml

Re: Phrase Query Problem?

2010-11-02 Thread Erick Erickson
That's not the response I get when I try your query, so I suspect something's not quite right with your test... But you could also try putting parentheses around the words, like mykeywords:(Compliance+With+Conduct+Standards) Best Erick On Tue, Nov 2, 2010 at 5:25 AM, Tod listac...@gmail.com

how to get TermVectorComponent using xml , vs. SOLR-949

2010-11-02 Thread Will Milspec
Hi all, This seems a basic question: what's the best way to get TermVectorComponents. from the Solr XmL response? SolrJ does not include TermVectorComponents in its api; the SOLR-949 patch adds this ability, but after 2 years it's still not in the mainline. (And doesn't patch cleanly to the

Re: Disk usage per-field

2010-11-02 Thread Muneeb Ali
Hi, I am currently benchmarking solr index with different fields to see the impact on its size/ search speed etc. A feature to find the disk usage per field of index would be really handy and save me alot of time. Do we have any updates on this? Has anyone tried writing custom code for it ? -

Re: How to use polish stemmer - Stempel - in schema.xml?

2010-11-02 Thread Jakub Godawa
Erick I've put the jar files like that before. I also added the directive and put the file in instanceDir/lib What is still a problem is that even the files are loaded: 2010-11-02 13:20:48 org.apache.solr.core.SolrResourceLoader replaceClassLoader INFO: Adding

Re: How to use polish stemmer - Stempel - in schema.xml?

2010-11-02 Thread Bernd Fehling
Hi Jakub, if you unzip your stempel-1.0.jar do you have the required directory structure and file in there? org/getopt/stempel/lucene/StempelFilter.class Regards, Bernd Am 02.11.2010 13:54, schrieb Jakub Godawa: Erick I've put the jar files like that before. I also added the directive and put

Re: Problem with phrase matches in Solr

2010-11-02 Thread Moazzam Khan
I will. Thanks Darren -Moazzam On Mon, Nov 1, 2010 at 1:15 PM, dar...@ontrenet.com wrote: Take a look at term proximity and phrase query. http://wiki.apache.org/solr/SolrRelevancyCookbook Hey guys, I have a solr index where i store information about experts from various fields. The

Re: How to use polish stemmer - Stempel - in schema.xml?

2010-11-02 Thread Jakub Godawa
This is what stempel-1.0.jar consist of after jar -xf: jgod...@ubuntu:~/apache-solr-1.4.1/ifaq/lib$ ls -R org/ org/: egothor getopt org/egothor: stemmer org/egothor/stemmer: Cell.class Diff.classGener.class MultiTrie2.class Optimizer2.class Reduce.classRow.class

Re: Phrase Query Problem?

2010-11-02 Thread Ken Stanley
On Tue, Nov 2, 2010 at 8:19 AM, Erick Erickson erickerick...@gmail.comwrote: That's not the response I get when I try your query, so I suspect something's not quite right with your test... But you could also try putting parentheses around the words, like

Re: How to use polish stemmer - Stempel - in schema.xml?

2010-11-02 Thread Bernd Fehling
So you call org.getopt.solr.analysis.StempelTokenFilterFactory. In this case I would assume a file StempelTokenFilterFactory.class in your directory org/getopt/solr/analysis/. And a class which extends the BaseTokenFilterFactory rigth? ... public class StempelTokenFilterFactory extends

Re: How to use polish stemmer - Stempel - in schema.xml?

2010-11-02 Thread Jakub Godawa
Sorry, I am not Java programmer at all. I would appreciate more verbose (or step by step) help. 2010/11/2 Bernd Fehling bernd.fehl...@uni-bielefeld.de: So you call org.getopt.solr.analysis.StempelTokenFilterFactory. In this case I would assume a file StempelTokenFilterFactory.class in your

Highlighting and maxBooleanClauses limit

2010-11-02 Thread Ken Stanley
By default, the solrconfig.xml has maxBooleanClauses set to 1024, which in my opinion should be more than enough clauses in general. Recently, we have been noticing errors in our Catalina log: SEVERE: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 2048. As a

Slave replication with custom dataDir

2010-11-02 Thread Kura
Hey guys, I have 2 instances of Solr running, one as a master, one as a slave. Both have dataDir/var/lib/solr/data/dataDir The master works fine, the slave dies with a huge set of stack traces. The Solr wiki says that replication must match the dataDir if it's custom, but how do I actually

Re: Slave replication with custom dataDir

2010-11-02 Thread Kura
This is a log dump, please be aware that this only appears in my log if I have the following enabled in config. dataDir/var/lib/solr/data/dataDir ... snip ... requestHandler name=/replication class=solr.ReplicationHandler lst name=slave str

Query question

2010-11-02 Thread kenf_nc
I can't seem to find the right formula for this. I have a need to build a query where one of the fields should boost the score, but not affect the query if there isn't a match. For example, if I have documents with restaurants, name, address, cuisine, description, etc. I want to search on, say,

Re: Query question

2010-11-02 Thread Jonathan Rochkind
I think you'll find the dismax request handler helpful in general, it supports more flexible query wrangling like that. With the dismax request handler, I think the bq (boost query) parameter will do what you need, eg: bq=city:Chicago^5.0 The ^5.0 is how much boost you want, you can play

Re: Phrase Query Problem?

2010-11-02 Thread Tod
On 11/2/2010 9:21 AM, Ken Stanley wrote: On Tue, Nov 2, 2010 at 8:19 AM, Erick Ericksonerickerick...@gmail.comwrote: That's not the response I get when I try your query, so I suspect something's not quite right with your test... But you could also try putting parentheses around the words,

Re: Query question

2010-11-02 Thread Erick Erickson
Do you want something like (Romantic AND View) OR city:Chicago^10? Best Erick On Tue, Nov 2, 2010 at 10:45 AM, kenf_nc ken.fos...@realestate.com wrote: I can't seem to find the right formula for this. I have a need to build a query where one of the fields should boost the score, but not

Re: Dynamically create new core

2010-11-02 Thread Marc Sturlese
To create the core, the folder with the confs must already exist and has to be placed in the proper place (inside the solr home). Once you run the create core action, this core will we added to solr.xml and dinamically loaded. -- View this message in context:

Re: Highlighting and maxBooleanClauses limit

2010-11-02 Thread Koji Sekiguchi
(10/11/02 23:14), Ken Stanley wrote: I've noticed in the stack trace that this exception occurs when trying to build the query for the highlighting; I've confirmed this by copying the params and changing hl=true to hl=false. Unfortunately, when using debugQuery=on, I do not see any details on

Re: Phrase Query Problem?

2010-11-02 Thread Jonathan Rochkind
Indeed something doesn't seem right about that, quotes are for phrases, you are right, and I get confused even thinking about what happens when you try to escape spaces like that. I think there's something odd going on with your URI-escaping in general. Here's what the string should actually

Re: Query question

2010-11-02 Thread kenf_nc
Jonathan, Dismax is something I've been meaning to look into, and bq does seem to fit the bill, although I'm worried about this line in the wiki :TODO: That latter part is deprecated behavior but still works. It can be problematic so avoid it. It still seems to be the closest to what I want

Re: Query question

2010-11-02 Thread Jonathan Rochkind
Don't worry about that line. It just means that one particular kind of 'default' behavior in bq shouldn't be relied upon, if you don't entirely understand that behavior they're saying is deprecated (as I don't either!) anyway, don't worry about it, just supply an explicit boost in your bq.

Re: Highlighting and maxBooleanClauses limit

2010-11-02 Thread Markus Jelsma
Hmm, i'm not sure it's the highlighter alone. Depending on the query it can also get triggered by the spellcheck component. See below what happens with a maxBoolean = 16. HTTP ERROR: 500 maxClauseCount is set to 16 org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to

Re: Highlighting and maxBooleanClauses limit

2010-11-02 Thread Ken Stanley
On Tue, Nov 2, 2010 at 11:26 AM, Koji Sekiguchi k...@r.email.ne.jp wrote: (10/11/02 23:14), Ken Stanley wrote: I've noticed in the stack trace that this exception occurs when trying to build the query for the highlighting; I've confirmed this by copying the params and changing hl=true to

IndexableBinaryStringTools (was FieldCache)

2010-11-02 Thread Mathias Walter
Hi, [...] I tried to use IndexableBinaryStringTools to re-encode my 11 byte array. The size was increased to 7 characters (= 14 bytes) which is still a gain of more than 50 percent compared to the UTF8 encoding. BTW: I found no sample how to use the IndexableBinaryStringTools class

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Simon Wistow
On Mon, Nov 01, 2010 at 05:42:51PM -0700, Lance Norskog said: You should query against the indexer. I'm impressed that you got 5s replication to work reliably. That's our current solution - I was just wondering if there was anything I was missing. Thanks!

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Yonik Seeley
On Tue, Nov 2, 2010 at 12:32 PM, Simon Wistow si...@thegestalt.org wrote: On Mon, Nov 01, 2010 at 05:42:51PM -0700, Lance Norskog said: You should query against the indexer. I'm impressed that you got 5s replication to work reliably. That's our current solution - I was just wondering if there

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Jonathan Rochkind
It's definitely a known 'issue' that you can't replicate (or do any other kind of index change, including a commit) at a faster frequency than your warming queries take to complete, or you'll wind up with something like you've seen. It's in some documentation somewhere I saw, for sure. The

Solr like for autocomplete field?

2010-11-02 Thread PeterKerk
I have a city field. Now when a user starts typing in a city textbox I want to return found matches (like Google). So for example, user types new, and I will return new york, new hampshire etc. my schema.xml field name=city type=string indexed=true stored=true/ my current url:

Re: Solr like for autocomplete field?

2010-11-02 Thread Matthew Hall
We used the filters talked about at Lucid Imagination for our site, it seems to work pretty well: http://www.lucidimagination.com/blog/2009/09/08/auto-suggest-from-popular-queries-using-edgengrams/ Your mileage might vary, but its a pretty good place to start. Matt On 11/2/2010 1:56 PM,

Querying Solr using dismax, requested field not showing up in debug score boosts

2010-11-02 Thread zakuhn
I'm storing a set of products in solr as ducuments. I'm separating out the name, description, keywords, and product category name into separate fields so that I can boost them independently using the dismax handler. All the fields are stored as text in the same way. I'm passing these four fields

Re: Query question

2010-11-02 Thread Erick Erickson
I... Need... more... coffee. On Tue, Nov 2, 2010 at 11:31 AM, kenf_nc ken.fos...@realestate.com wrote: Jonathan, Dismax is something I've been meaning to look into, and bq does seem to fit the bill, although I'm worried about this line in the wiki :TODO: That latter part is deprecated

Re: Solr like for autocomplete field?

2010-11-02 Thread Erick Erickson
Also, you might want to consider TermsComponent, see: http://wiki.apache.org/solr/TermsComponent Also, note that there's an autosuggestcomponent, that's recently been committed. Best Erick On Tue, Nov 2, 2010 at 1:56 PM, PeterKerk vettepa...@hotmail.com wrote: I have a city field. Now when

Re: Querying Solr using dismax, requested field not showing up in debug score boosts

2010-11-02 Thread Erick Erickson
First, you should show us the query, as well as the debug output, it often helps to have a second set of eyes... Where are you specifying the qf? Under any circumstance it would be helpful to see the definition of the request handler you're using. Because as it stands, the best I can say is that

Updating last_modified field when using DIH

2010-11-02 Thread Juan Manuel Alvarez
Hello everyone! I would like to ask you a question about DIH and delta import. I am trying to sync Solr with a PostgreSQL database and I have a field ent_lastModified of type timestamp without timezone. Here is my xml file: dataConfig dataSource name=jdbc driver=org.postgresql.Driver

RE: Stored or indexed?

2010-11-02 Thread Olson, Ron
Thanks for the great info! I appreciate everybody's help in getting started with Solr, hopefully I'll be able to get my stuff working and move on to more difficult questions. :) -Original Message- From: Elizabeth L. Murnane [mailto:emurn...@architexa.com] Sent: Friday, October 29, 2010

Re: Query question

2010-11-02 Thread Ahmet Arslan
Erick, that query would return all restaurants in Chicago, whether they matched Romantic View or not. Although the scores should sort relevant results to the top, the results would still contain a lot of things I wasn't interested in. How about this one? +(city:Chicago^1000 OR (*:*

Re: Querying Solr using dismax, requested field not showing up in debug score boosts

2010-11-02 Thread zakuhn
Ok, here is the query cleaned up a bit: solr/select/?q=mattress q.op=AND

Re: Stored or indexed?

2010-11-02 Thread Erick Erickson
IMO, the very, very best way to increase your grasp of all things Solr is to try to answer questions on this list. Folks are pretty gentle about correcting mistaken posts. And I certainly remember any advice I've given that's been corrected G. Besides, if you try to answer the things you *do*

Re: Influencing scores on values in multiValue fields

2010-11-02 Thread Imran
Thanks Mike for your suggestion. It did take me down the correct route. I basically created another multiValue field of type 'string' and boosted that. To get the partial matches to avoid the length normalisation I had the 'text' type multiValue field to omitNorms. The results look as per expected

Re: Query question

2010-11-02 Thread Michael Sokolov
My impression was that city:Chicago^10 +Romantic +View would do what you want (with the standard lucene query parser and default operator OR), and I'm not sure about this, but I have a feeling that the version with Boolean operators AND/OR and parens might actually net out to the same thing,

Re: xpath processing

2010-11-02 Thread pghorpade
?xml version=1.0 encoding=UTF-8? mods:mods xmlns:mods=http://www.loc.gov/mods/v3; xmlns:xlink=http://www.w3.org/1999/xlink; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=http://www.loc.gov/mods/v3 http://www.loc.gov/standards/mods/v3/mods-3-0.xsd;

Re: Ensuring stable timestamp ordering

2010-11-02 Thread Dennis Gearon
memory's cheap! (I know processing it is not' though ) Dennis Gearon Signature Warning It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from

Re: using HebMorph

2010-11-02 Thread Lance Norskog
I don't know the paths in the Solr package for Ubuntu. In the Solr apache release, you go to the example/ directory. The example/solr directory needs a new lib directory, and you copy the jars to there. Then run 'java -jar start.jar' still in the example/ directory. Solr should start. Now, you

Re: how to get TermVectorComponent using xml , vs. SOLR-949

2010-11-02 Thread Lance Norskog
TVC is in Solr 1.4 onwards. It is configured in example/solr/conf/solrconfig.xml as 'tvrh'. This is not a solr/url thing, so you have to say solr/select?q=word'qt=tvrh' and look at the bottom of the xml. On Tue, Nov 2, 2010 at 5:34 AM, Will Milspec will.mils...@gmail.com wrote: Hi all, This

Re: Disk usage per-field

2010-11-02 Thread Lance Norskog
The Lucene CheckIndex program opens an index and reads many types of data from it. It's easy to start with it and change that to count up the space used by terms and store data for field X. On Tue, Nov 2, 2010 at 5:51 AM, Muneeb Ali muneeba...@hotmail.com wrote: Hi, I am currently

Re: How to use polish stemmer - Stempel - in schema.xml?

2010-11-02 Thread Lance Norskog
Here's the problem: Solr is a little dumb about these Filter classes, and so you have to make a Factory object for the Stempel Filter. There are a lot of other FilterFactory classes. You would have to just copy one and change the names to Stempel and it might actually work. This will take some

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Lance Norskog
Isn't that what this code does? onDeckSearchers++; if (onDeckSearchers 1) { // should never happen... just a sanity check log.error(logid+ERROR!!! onDeckSearchers is + onDeckSearchers); onDeckSearchers=1; // reset } else if (onDeckSearchers

Re: Solr like for autocomplete field?

2010-11-02 Thread Lance Norskog
And the SpellingComponent. There's nothing to help you with phrases. On Tue, Nov 2, 2010 at 11:21 AM, Erick Erickson erickerick...@gmail.com wrote: Also, you might want to consider TermsComponent, see: http://wiki.apache.org/solr/TermsComponent Also, note that there's an

Re: xpath processing

2010-11-02 Thread Lance Norskog
The XPathEP has the option to run a real XSL script at some point in its processing chain. I guess you could make an XSL that pulls your fields out into a simpler XML in the /a/b/c format that the XPath parser supports. On Tue, Nov 2, 2010 at 5:37 PM, pghorp...@ucla.edu wrote: ?xml