date:20081229

apparently the local solr expects everything as string. I guess we
must raise an issue with them.
meanwhile you can convert those double fields to String using
TemplateTransformer


  



On Tue, Dec 30, 2008 at 5:49 AM, MapYours  wrote:
>
> I have my data import handler set up correctly it connects and streams
> results returned from the database (sqlserver 2005) using JTDS driver.
>
> However, no documents are indexed:
> Dec 29, 2008 2:03:07 PM org.apache.solr.handler.dataimport.SolrWriter upload
> WARNING: Error creating document :
> SolrInputDocumnt[{incident_num=incident_num(1.0)={08518194 },
> incident_date=incident_date(1.0)={2008-12-29 00:00:00.0},
> location=location(1.0)={4300BLK SW 173RD AVE
> }, lat=lat(1.0)={45.4882}, lng=lng(1.0)={-122.85455}}]
> java.lang.ClassCastException: java.lang.Double cannot be cast to
> java.lang.String
>at
> com.pjaol.search.solr.update.LocalUpdaterProcessor.processAdd(LocalUpdateProcessorFactory.java:136)
>at
> org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:69)
>at
> org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:288)
>at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:319)
>at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
>at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
>at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
>at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>
> Here is my schema:
> 
>   
>   
>   
>
> 
>   
>   
>
> stored="true"/>
>  
>
>  
>  incident_num
>
>  
>  location
>
> --
> View this message in context: 
> http://www.nabble.com/Local-Solr---Error-creating-document-with-DIH-tp21211133p21211133.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



-- 
--Noble Paul

Re: DataImportHandler full-import: SolrException: Document [null] missing required field: id

The column names are case sensitive.
USER_ID != user_id

On Tue, Dec 30, 2008 at 2:12 AM, Rakesh Sinha  wrote:
> Hi -
>   My dataconfig.xml looks as follows.
>
>
> 
>user="username" password="password" />
>
>query="select user_id, firstname from user where user_id < 5">
>
>
>
>
> 
>
> When I do a full-import with this revised schema ( where the primary
> key of the table is not id , but user_id ), I am getting the following
> error.
>
> WARNING: Error creating document : SolrInputDocument[{}]
> org.apache.solr.common.SolrException: Document [null] missing required field: 
> id
>at 
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:292)
>at 
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:59)
>at 
> org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:70)
>at 
> org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:275)
>at 
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:328)
>at 
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:183)
>at 
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:134)
>at 
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:323)
>at 
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:385)
>
>
> I am trying to understand this since I had defined a mapping for a
> field above as < field column="USER_ID" name="id" /> .
>
> Any idea - what is missing here for the indexing. Also - why would
> Document [null] be null since my query does give appropriate results.
>



-- 
--Noble Paul

Re: DataImportHandler - The field :xyz present in DataConfig does not have a counterpart in Solr Schema

These are warning messages . They do not stop anything in DIH
It is very common that users make typos and that may be hard to debug.
So the messages are spit out

On Tue, Dec 30, 2008 at 12:06 AM, Rakesh Sinha  wrote:
> Hi -
>  I am testing around with the full - import functionality of Data
> Import Handler.  My dataconfig file looks as follows.
>
>
> 
>user="username" password="password" />
>
>query="select id, firstname, lastname from user">
>
>
>
>
>
> 
>
> In solrconfig.xml - I am setting the access for DIH as follows.
>
>   class="org.apache.solr.handler.dataimport.DataImportHandler">
>
>  data-config.xml
>
>  
>
>
> When I try to access the deployed web-app ( even before hitting
> full-import functionality using command ) - I am getting the following
> sequence of errors.
>
> The field :lastname present in DataConfig does not have a counterpart
> in Solr Schema
> The field :firstname present in DataConfig does not have a counterpart
> in Solr Schema
>
> The config file is very similar to what is given in the DIH wiki.
>
> Curious, what gives ?
>



-- 
--Noble Paul

Re: can i reuse the index built with 1.3 with current solr from trunk

Vinay,

Please email only one list - solr-user.
The answer is positive, but it's always best to back up your index just in case.


Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: vinay kumar kaku 
> To: solr-...@lucene.apache.org; solr-user@lucene.apache.org
> Sent: Monday, December 29, 2008 7:38:57 PM
> Subject: can i reuse the index built with 1.3 with current solr from trunk
> 
> 
> Hi,  I have an index thats built using 1.3 (before official release). can i 
> re-use same data with current solr i get from trunk? what are the things that 
> i 
> need to keep in mind.vinay
> _
> Send e-mail anywhere. No map, no compass.
> http://windowslive.com/oneline/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008

Re: how large can the index be?

What you have below is not really what we call "Distributed Search", but more 
of "Query Load Balancing".  Yes, the diagram below will work IF a single Solr 
box (A or B) can really handle a full 50M doc index.  Of course "handle" can be 
fuzzy.  That is, you could have a large index on a Solr box and it will 
"handle" it - nothing will crash, nothing will die, it's just that it may not 
be able to handle it "well enough" - that is, the queries may take longer than 
you'd like.

NFS mounting an index directory is a separate story and very often a bad idea, 
again because of performance.


Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Antonio Eggberg 
> To: solr-user@lucene.apache.org
> Sent: Monday, December 29, 2008 4:19:23 PM
> Subject: Re: how large can the index be?
> 
> Thanks you very much for your answer.
> 
> I was afraid of that the each document has about 20 fields.. As you pointed 
> out 
> it will slow down. Anyway I am thinking is it not possible to do the 
> following:
> 
> Load Balancer 
>  |
> Solr A, Solr B, ...
>  |
>   one index
> 
> So I send 50% query to Solr A, 50% to Solr B and so forth.. is this not good? 
> Also to add The index will be like a mounted drive to the solr boxes... On 
> the 
> above do I really need to worry about Solr Master, Solr Slave? It probably 
> solve 
> my load but I think query speed will be slow...
> 
> Just curious anyone using distributed search in production?
> 
> Cheers
> 
> 
> 
> --- Den mån 2008-12-29 skrev Otis Gospodnetic :
> 
> > Från: Otis Gospodnetic 
> > Ämne: Re: how large can the index be?
> > Till: solr-user@lucene.apache.org
> > Datum: måndag 29 december 2008 21.53
> > Hi Antonio,
> > 
> > Besides thinking in terms of documents, you also need to
> > think in terms of index size on the file system vs. the
> > amount of RAM your search application/server can use.  50M
> > documents may be doable on a single server if those
> > documents are not too large and you have sufficient RAM.  It
> > gets even better if your index doesn't change very often
> > and if you can get decent hit ratios on the various Solr
> > caches.
> > 
> > If you are indexing largish documents, or even something as
> > small as an average web page, 50M docs may be too much on a
> > "commodity box" (say dual core 8 GB RAM box)
> > 
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > 
> > 
> > 
> > - Original Message 
> > > From: Antonio Eggberg 
> > > To: solr-user@lucene.apache.org
> > > Sent: Monday, December 29, 2008 3:41:48 PM
> > > Subject: how large can the index be?
> > > 
> > > Hi,
> > > 
> > > We are running successfully a solr index of 3 million
> > docs. I have just been 
> > > informed that our index size will increase to 50
> > million. I been going through 
> > > the doc 
> > > 
> > > http://wiki.apache.org/solr/DistributedSearch
> > > 
> > > Seems like we will loose out on the date facet and
> > some more other stuff that we 
> > > use. which is important to us. So far we been using 1
> > index and 1 machine. 
> > > 
> > > Can I still stick with my 1 index but have many query
> > servers? We don't update 
> > > our index so often this are rather static data. Over
> > the past year we have 
> > > updated the index data a total of 3 times and about
> > 300 records :)
> > > 
> > > Can someone provide some idea how/what should I do to
> > deal with new datasets?.
> > > 
> > > Thanks for your help.
> > > 
> > > 
> > >  
> > __
> > > Går det långsamt? Skaffa dig en snabbare
> > bredbandsuppkoppling. 
> > > Sök och jämför priser hos Kelkoo.
> > >
> > http://www.kelkoo.se/c-100015813-bredband.html?partnerId=96914325
> 
> 
>   __
> Låna pengar utan säkerhet. Jämför vilkor online hos Kelkoo.
> http://www.kelkoo.se/c-100390123-lan-utan-sakerhet.html?partnerId=96915014

Re: http internal error if i enable debugQuery=on

Hi,

Do you have q=XXX parameter in your URL as well?  Try specifying it.


Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: vinay kumar kaku 
> To: solr-user@lucene.apache.org
> Sent: Monday, December 29, 2008 6:09:24 PM
> Subject: http internal error if i enable debugQuery=on
> 
> 
> Hi,  i got solr on 23rd dec and using for my application. i have a filter 
> &fq=-filtername[1+TO+*] all works fine until i say &debugQuery=on i get below 
> error from my tomcat catalina.out logfile. if i change that filter to say 
> &fq=-filtername[1+TO+10]&debugQuery=on it works fine.HTTP Status 500 - 
> java.lang.NullPointerException: value cannot be null 
> java.lang.RuntimeException: 
> java.lang.NullPointerException: value cannot be null at 
> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:469) at 
> org.apache.solr.handler.component.DebugComponent.process(DebugComponent.java:75)
>  
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:179)
>  
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>  
> at 
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:233)
>  
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1325) at 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>  
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>  
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>  
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>  
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>  
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
>  
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) 
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568) 
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>  
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) 
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) 
> at 
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>  
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at 
> java.lang.Thread.run(Thread.java:619) Caused by: 
> java.lang.NullPointerException: 
> value cannot be null at org.apache.lucene.document.Field.(Field.java:275) 
> at org.apache.lucene.document.Field.(Field.java:251) at 
> org.apache.solr.search.QueryParsing.writeFieldVal(QueryParsing.java:306) at 
> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:338) at 
> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:401) at 
> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:466) ... 20 
> more 
> thanks,vinay
> _
> Send e-mail anywhere. No map, no compass.
> http://windowslive.com/oneline/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008

can i reuse the index built with 1.3 with current solr from trunk

2008-12-29 Thread vinay kumar kaku


Hi,  I have an index thats built using 1.3 (before official release). can i 
re-use same data with current solr i get from trunk? what are the things that i 
need to keep in mind.vinay
_
Send e-mail anywhere. No map, no compass.
http://windowslive.com/oneline/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008

Local Solr - Error creating document with DIH

2008-12-29 Thread MapYours


I have my data import handler set up correctly it connects and streams
results returned from the database (sqlserver 2005) using JTDS driver.

However, no documents are indexed:
Dec 29, 2008 2:03:07 PM org.apache.solr.handler.dataimport.SolrWriter upload
WARNING: Error creating document :
SolrInputDocumnt[{incident_num=incident_num(1.0)={08518194 },
incident_date=incident_date(1.0)={2008-12-29 00:00:00.0},
location=location(1.0)={4300BLK SW 173RD AVE 
}, lat=lat(1.0)={45.4882}, lng=lng(1.0)={-122.85455}}]
java.lang.ClassCastException: java.lang.Double cannot be cast to
java.lang.String
at
com.pjaol.search.solr.update.LocalUpdaterProcessor.processAdd(LocalUpdateProcessorFactory.java:136)
at
org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:69)
at
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:288)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:319)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

Here is my schema:

   
   
   
   

   
   
   

 

 
 incident_num

 
 location

-- 
View this message in context: 
http://www.nabble.com/Local-Solr---Error-creating-document-with-DIH-tp21211133p21211133.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Dismax Minimum Match/Stopwords Bug

2008-12-29 Thread Matthew Runo

Hmm, that makes sense to me - however I still think that even if we
have mm set to "2" and we have "the 7449078" it should still match
7449078 in a productId field (it does not: http://zeta.zappos.com/search?department=&term=the+7449078)
. This seems like it works against the way one would reasonably expect
it to - that stopwords shouldn't impact the counts for mm (so, "the
7449078" would count as 1 term for mm since "the" is a stopword).

Would there be a way around this? Could we possibly get it reworked?
What would the downside to that be?

We have people asking for "the north" to return results from a brand
called "the north face" - but it doesn't, and can't, because of this
mm issue.

Thanks for your time helping us with this issue =)

Matthew Runo
Software Engineer, Zappos.com
mr...@zappos.com - 702-943-7833

On Dec 20, 2008, at 10:45 AM, Chris Hostetter wrote:

: Would this mean that, for example, if we wanted to search
productId (long)
: we'd need to make a field type that had stopwords in it rather
than simply

: using (long)?

not really ... that's kind of a special usecase. if someone
searches for
a productId that's usually *all* they search for (1 "chunk" of input
fro

mthe query parser) so it's mandatory and produces a clause across all
fields. It doesn't matter if the other fields have stopwords --
even if

the productId happens to be a stop word, that just means it doesn't
produce a clause on those "stop worded" fields, but it will will on
your

productId field.

The only case where you might get into trouble is if someone
searches for

"the 123456" ... now you have two chunks of input, so the mm param
comes into play you have no stopwords on your productId field so both
"the" and "123456" produce clauses, but "the" isn't going to be
found in

your productId field, and because of stopwords it doens't exist in the
other fields at all ... so you don't match anything.

FWIW: if i remember right if you want to put numeric fields in the
qf, i
think you need *all* of them to be numeric and all of your input
needs to

be numeric, or you get exceptions from the FieldType (not the dismax
parser) when people search for normal words. i always copyField
productId into a productId_str field for purposes like this.

-Hoss

http internal error if i enable debugQuery=on

2008-12-29 Thread vinay kumar kaku


Hi,  i got solr on 23rd dec and using for my application. i have a filter 
&fq=-filtername[1+TO+*] all works fine until i say &debugQuery=on i get below 
error from my tomcat catalina.out logfile. if i change that filter to say 
&fq=-filtername[1+TO+10]&debugQuery=on it works fine.HTTP Status 500 - 
java.lang.NullPointerException: value cannot be null 
java.lang.RuntimeException: java.lang.NullPointerException: value cannot be 
null at org.apache.solr.search.QueryParsing.toString(QueryParsing.java:469) at 
org.apache.solr.handler.component.DebugComponent.process(DebugComponent.java:75)
 at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:179)
 at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
 at 
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:233)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1325) at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) 
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
 at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
 at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) 
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568) at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) 
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) 
at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at 
java.lang.Thread.run(Thread.java:619) Caused by: 
java.lang.NullPointerException: value cannot be null at 
org.apache.lucene.document.Field.(Field.java:275) at 
org.apache.lucene.document.Field.(Field.java:251) at 
org.apache.solr.search.QueryParsing.writeFieldVal(QueryParsing.java:306) at 
org.apache.solr.search.QueryParsing.toString(QueryParsing.java:338) at 
org.apache.solr.search.QueryParsing.toString(QueryParsing.java:401) at 
org.apache.solr.search.QueryParsing.toString(QueryParsing.java:466) ... 20 more 
thanks,vinay
_
Send e-mail anywhere. No map, no compass.
http://windowslive.com/oneline/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008

Re: how large can the index be?

2008-12-29 Thread Antonio Eggberg

Thanks you very much for your answer.

I was afraid of that the each document has about 20 fields.. As you pointed out 
it will slow down. Anyway I am thinking is it not possible to do the following:

Load Balancer 
 |
Solr A, Solr B, ...
 |
  one index

So I send 50% query to Solr A, 50% to Solr B and so forth.. is this not good? 
Also to add The index will be like a mounted drive to the solr boxes... On the 
above do I really need to worry about Solr Master, Solr Slave? It probably 
solve my load but I think query speed will be slow...

Just curious anyone using distributed search in production?

Cheers



--- Den mån 2008-12-29 skrev Otis Gospodnetic :

> Från: Otis Gospodnetic 
> Ämne: Re: how large can the index be?
> Till: solr-user@lucene.apache.org
> Datum: måndag 29 december 2008 21.53
> Hi Antonio,
> 
> Besides thinking in terms of documents, you also need to
> think in terms of index size on the file system vs. the
> amount of RAM your search application/server can use.  50M
> documents may be doable on a single server if those
> documents are not too large and you have sufficient RAM.  It
> gets even better if your index doesn't change very often
> and if you can get decent hit ratios on the various Solr
> caches.
> 
> If you are indexing largish documents, or even something as
> small as an average web page, 50M docs may be too much on a
> "commodity box" (say dual core 8 GB RAM box)
> 
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> 
> 
> 
> - Original Message 
> > From: Antonio Eggberg 
> > To: solr-user@lucene.apache.org
> > Sent: Monday, December 29, 2008 3:41:48 PM
> > Subject: how large can the index be?
> > 
> > Hi,
> > 
> > We are running successfully a solr index of 3 million
> docs. I have just been 
> > informed that our index size will increase to 50
> million. I been going through 
> > the doc 
> > 
> > http://wiki.apache.org/solr/DistributedSearch
> > 
> > Seems like we will loose out on the date facet and
> some more other stuff that we 
> > use. which is important to us. So far we been using 1
> index and 1 machine. 
> > 
> > Can I still stick with my 1 index but have many query
> servers? We don't update 
> > our index so often this are rather static data. Over
> the past year we have 
> > updated the index data a total of 3 times and about
> 300 records :)
> > 
> > Can someone provide some idea how/what should I do to
> deal with new datasets?.
> > 
> > Thanks for your help.
> > 
> > 
> >  
> __
> > Går det långsamt? Skaffa dig en snabbare
> bredbandsuppkoppling. 
> > Sök och jämför priser hos Kelkoo.
> >
> http://www.kelkoo.se/c-100015813-bredband.html?partnerId=96914325


  __
Låna pengar utan säkerhet. Jämför vilkor online hos Kelkoo.
http://www.kelkoo.se/c-100390123-lan-utan-sakerhet.html?partnerId=96915014

Re: how large can the index be?

Hi Antonio,

Besides thinking in terms of documents, you also need to think in terms of 
index size on the file system vs. the amount of RAM your search 
application/server can use.  50M documents may be doable on a single server if 
those documents are not too large and you have sufficient RAM.  It gets even 
better if your index doesn't change very often and if you can get decent hit 
ratios on the various Solr caches.

If you are indexing largish documents, or even something as small as an average 
web page, 50M docs may be too much on a "commodity box" (say dual core 8 GB RAM 
box)

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Antonio Eggberg 
> To: solr-user@lucene.apache.org
> Sent: Monday, December 29, 2008 3:41:48 PM
> Subject: how large can the index be?
> 
> Hi,
> 
> We are running successfully a solr index of 3 million docs. I have just been 
> informed that our index size will increase to 50 million. I been going 
> through 
> the doc 
> 
> http://wiki.apache.org/solr/DistributedSearch
> 
> Seems like we will loose out on the date facet and some more other stuff that 
> we 
> use. which is important to us. So far we been using 1 index and 1 machine. 
> 
> Can I still stick with my 1 index but have many query servers? We don't 
> update 
> our index so often this are rather static data. Over the past year we have 
> updated the index data a total of 3 times and about 300 records :)
> 
> Can someone provide some idea how/what should I do to deal with new datasets?.
> 
> Thanks for your help.
> 
> 
>   __
> Går det långsamt? Skaffa dig en snabbare bredbandsuppkoppling. 
> Sök och jämför priser hos Kelkoo.
> http://www.kelkoo.se/c-100015813-bredband.html?partnerId=96914325

DataImportHandler full-import: SolrException: Document [null] missing required field: id

2008-12-29 Thread Rakesh Sinha

Hi -
   My dataconfig.xml looks as follows.












When I do a full-import with this revised schema ( where the primary
key of the table is not id , but user_id ), I am getting the following
error.

WARNING: Error creating document : SolrInputDocument[{}]
org.apache.solr.common.SolrException: Document [null] missing required field: id
at 
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:292)
at 
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:59)
at 
org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:70)
at 
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:275)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:328)
at 
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:183)
at 
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:134)
at 
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:323)
at 
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:385)


I am trying to understand this since I had defined a mapping for a
field above as < field column="USER_ID" name="id" /> .

Any idea - what is missing here for the indexing. Also - why would
Document [null] be null since my query does give appropriate results.

how large can the index be?

2008-12-29 Thread Antonio Eggberg

Hi,

We are running successfully a solr index of 3 million docs. I have just been 
informed that our index size will increase to 50 million. I been going through 
the doc 

http://wiki.apache.org/solr/DistributedSearch

Seems like we will loose out on the date facet and some more other stuff that 
we use. which is important to us. So far we been using 1 index and 1 machine. 

Can I still stick with my 1 index but have many query servers? We don't update 
our index so often this are rather static data. Over the past year we have 
updated the index data a total of 3 times and about 300 records :)

Can someone provide some idea how/what should I do to deal with new datasets?.

Thanks for your help.


  __
Går det långsamt? Skaffa dig en snabbare bredbandsuppkoppling. 
Sök och jämför priser hos Kelkoo.
http://www.kelkoo.se/c-100015813-bredband.html?partnerId=96914325

Re: DataImportHandler - The field :xyz present in DataConfig does not have a counterpart in Solr Schema

2008-12-29 Thread Rakesh Sinha

Oops. The fields were out of sync with those in schema.xml .

Looking at the dynamic field name configuration in schema.xml - my
dataconfig.xml file looks as follows.











 

The naming of fields with suffix ( _s ) , as per the dynamic field
naming conventions fixed the issue.



On Mon, Dec 29, 2008 at 1:36 PM, Rakesh Sinha  wrote:
> Hi -
>  I am testing around with the full - import functionality of Data
> Import Handler.  My dataconfig file looks as follows.
>
>
> 
>user="username" password="password" />
>
>query="select id, firstname, lastname from user">
>
>
>
>
>
> 
>
> In solrconfig.xml - I am setting the access for DIH as follows.
>
>   class="org.apache.solr.handler.dataimport.DataImportHandler">
>
>  data-config.xml
>
>  
>
>
> When I try to access the deployed web-app ( even before hitting
> full-import functionality using command ) - I am getting the following
> sequence of errors.
>
> The field :lastname present in DataConfig does not have a counterpart
> in Solr Schema
> The field :firstname present in DataConfig does not have a counterpart
> in Solr Schema
>
> The config file is very similar to what is given in the DIH wiki.
>
> Curious, what gives ?
>

Re: Hi w.r.t solr 1.4

Nightly build, please.


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: vinay kumar kaku 
> To: solr-user@lucene.apache.org
> Sent: Monday, December 29, 2008 2:05:42 PM
> Subject: Hi w.r.t solr 1.4
> 
> 
> Hi,  is there a stable build of solr 1.4 that i could grab from? vinay
> _
> Send e-mail faster without improving your typing skills.
> http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_speed_122008

Hi w.r.t solr 1.4

2008-12-29 Thread vinay kumar kaku


Hi,  is there a stable build of solr 1.4 that i could grab from? vinay
_
Send e-mail faster without improving your typing skills.
http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_speed_122008

DataImportHandler - The field :xyz present in DataConfig does not have a counterpart in Solr Schema

2008-12-29 Thread Rakesh Sinha

Hi -
  I am testing around with the full - import functionality of Data
Import Handler.  My dataconfig file looks as follows.













In solrconfig.xml - I am setting the access for DIH as follows.

  

  data-config.xml

  


When I try to access the deployed web-app ( even before hitting
full-import functionality using command ) - I am getting the following
sequence of errors.

The field :lastname present in DataConfig does not have a counterpart
in Solr Schema
The field :firstname present in DataConfig does not have a counterpart
in Solr Schema

The config file is very similar to what is given in the DIH wiki.

Curious, what gives ?

Re: Custom Search Results

Johnny,

There are lots of examples of SearchComponents in Solr itself.  Have a look:

o...@lesina:~/workspace/asf-solr$ ffjg -l "extends SearchComponent"
./src/java/org/apache/solr/handler/component/FacetComponent.java
./src/java/org/apache/solr/handler/component/QueryComponent.java
./src/java/org/apache/solr/handler/component/MoreLikeThisComponent.java
./src/java/org/apache/solr/handler/component/HighlightComponent.java
./src/java/org/apache/solr/handler/component/SpellCheckComponent.java
./src/java/org/apache/solr/handler/component/TermVectorComponent.java
./src/java/org/apache/solr/handler/component/DebugComponent.java
./src/java/org/apache/solr/handler/component/QueryElevationComponent.java
./src/java/org/apache/solr/handler/component/StatsComponent.java
./src/java/org/apache/solr/handler/component/TermsComponent.java
./src/java/org/apache/solr/core/SolrCore.java


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Johnny X 
> To: solr-user@lucene.apache.org
> Sent: Saturday, December 27, 2008 2:23:32 PM
> Subject: Re: Custom Search Results
> 
> 
> Cheers Otis, that seems to be what I'm looking for.
> 
> While the Solr pages look helpful, are there any implementation examples of
> SearchComponents available that I could base what I'm doing this on, or are
> there any tutorials about?
> 
> I'm not really familiar with Java either, but I suppose if I had something
> to base what I need to do on I could pick up what I need for the
> SearchComponent from that if you see what I mean.
> 
> Thanks again!
> 
> 
> 
> Otis Gospodnetic wrote:
> > 
> > Hi Johnny,
> > 
> > It sounds like you wish to post-process and analyze results and categorize
> > each document in the result set based on presence and positioning of
> > certain key words in the document.  If that's so, I think what you are
> > after is a custom SearchComponent:
> > http://wiki.apache.org/solr/SearchComponent
> > 
> > 
> > Please do note that post-processing large result sets can get expensive
> > and will hurt performance.
> >  Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > 
> > 
> > 
> > - Original Message 
> >> From: Johnny X 
> >> To: solr-user@lucene.apache.org
> >> Sent: Thursday, December 25, 2008 9:29:15 PM
> >> Subject: Custom Search Results
> >> 
> >> 
> >> Hi there,
> >> 
> >> 
> >> N00b here, so if you could explain in simple terms I'd appreciate it.
> >> 
> >> I'm trying to create what I want to call a 'filter' to search through a
> >> bunch of indexed documents. The idea of this filter is search for very
> >> specific things in each of the indexed documents to catagorise them into
> >> a
> >> particular field if they're relevant.
> >> 
> >> In this case the search is going to involve a lot of specific information
> >> about the occurance of certain words, their placement in the document,
> >> other
> >> words around them etc so I don't know if I need to develop this using
> >> some
> >> form of custom advanced query or produce some form of new componenet
> >> (analyzer/filter???).
> >> 
> >> Thanks for any help!
> >> -- 
> >> View this message in context: 
> >> http://www.nabble.com/Custom-Search-Results-tp21171303p21171303.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> > 
> > 
> > 
> 
> -- 
> View this message in context: 
> http://www.nabble.com/Custom-Search-Results-tp21171303p21186849.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Multiple language support

Hi,

The problem is that a single document (and even a field in your case) is 
multilingual.  Ideally you'd detect different languages within a document and 
apply a different tokenizer/filter to different parts of the field.  So the 
first part would be handled as EN, and the second part as Chinese.  At search 
time you would have to find the language of the query one way or the other, and 
again apply the appropriate analyzer.  If the right analyzer is applied, you 
could match even this multilingual field.  None of the existing 
Analyzers/tokenizers/filters are capable of handling a single piece of text in 
multiple languages, so you will have to create a custom analyzer that is smart 
enough to do that.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: "Deshpande, Mukta" 
> To: solr-user@lucene.apache.org
> Sent: Monday, December 29, 2008 4:52:19 AM
> Subject: Multiple language support 
> 
> Hi All,
> 
> I have a multiple language supporting schema in which there is a separate 
> field 
> for every language.
> 
> I have a field "product_name" to store product name and its description that 
> can 
> be in any user preferred language. 
> This can be stored in fields product_name_EN if user prefers English 
> language, 
> product_name_SCH if user prefers Simplified Chinese language.
> The WhitespaceTokenizerFactory and filter EnglishPorterFilterFactory are 
> applied 
> on product_name_EN.
> The CJKAnalyzer and CJKTokenizer are applied on product_name_SCH.
> 
> e.g. Value can be : ElectrolyticCapacitor - 被对立的电容器以价值220µF
> 
> Now my problem is: Which field do I store the above value?
> product_name_EN OR product_name_SCH OR should it be something else?
> 
> How do I find out which analyzers should get applied for this field.
> 
> Did any one face a similar situation before. 
> Please help ASAP.
> 
> Thanks,
> ~Mukta

Re: High response times after snapshot install

Hello Victor,

The high response times could be caused by long and intense warming of the new 
searcher(s), which could be causing high IO (it sounds like all instances share 
the same index on the same file system) and high CPU usage (lots of cache items 
being copied from the old searcher).


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Victor Hogemann 
> To: Solr-User 
> Sent: Monday, December 29, 2008 7:53:44 AM
> Subject: High response times after snapshot install
> 
> Greetings,
> 
> Before anything I'd like to thank the Solr team for such great piece of 
> software! Thank you.
> 
> Now... back to my problem. I'm experiencing slowdowns on a Solr 1.3 multicore 
> installation after a snapshot install, response times go up from 5ms to above 
> 5sec, and don't go down unless we restart the servers.
> 
> Our index has more than 20 fields, and the slave servers got some 50hits/sec. 
> We're running it on Weblogic, with 6 Weblogic Nodes/Solr instances per 
> machine, 
> sharing the same index on the filesystem.
> 
> Any insight on what is happening?
> 
> Thanks.
> 
> Victor Hogemann - Ideais Tecnologia
> 
> Tel (RJ) +55 +21 3553-1301 (R217)
> Tel (SP) +55 +11 4063-0443 (R217)
> Fax +55 +21 3553-1302
> 
> Rua da Assembléia, nº 98, 9º andar
> Centro | Rio de Janeiro | Brasil
> CEP 20011-000

Re: Units for maxFieldLength

Hi Alan,

That number represents the number of tokens.  Extra long field values get 
silently truncated.

 
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Alan May 
> To: solr-user@lucene.apache.org
> Sent: Monday, December 29, 2008 1:09:37 PM
> Subject: Units for maxFieldLength
> 
> Hi,
> 
> I'd love to get more details about the maxFieldLength attribute in the
> solrConfig.xml.
> 
> Is this length in characters or bytes or some other measure?  If chars, how
> are double byte characters handled?  If an add request is sent with a field
> that exceeds this limit what is the behavior?  (Is it truncated or does it
> throw an exception?)
> 
> Kind Regards,
> Alan

Units for maxFieldLength

2008-12-29 Thread Alan May

Hi,

I'd love to get more details about the maxFieldLength attribute in the
solrConfig.xml.

Is this length in characters or bytes or some other measure?  If chars, how
are double byte characters handled?  If an add request is sent with a field
that exceeds this limit what is the behavior?  (Is it truncated or does it
throw an exception?)

Kind Regards,
Alan

Re: Delta DataImport is not picking the modified value in DB

On Mon, Dec 29, 2008 at 5:11 PM, Manupriya  wrote:
>
> Hi Shalin,
>
> I am using apache-solr-1.3.0 in my project. I thought it to be the latest
> release. Does it not support delta import?
It should, 1.3 of solr was the first realease of DIH. A lot of
features got added to DIH since then. The nightly build of DIH is
compatible with Solr 1.3. BTW 1.3 supports delta import.

>
> Thanks,
> Manu
>
>
> Shalin Shekhar Mangar wrote:
>>
>> Which version of Solr are you using? The deltaImportQuery feature was
>> added
>> recently, you'd need a nightly build for it to work.
>>
>> On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
>> wrote:
>>
>>>
>>> Hi Noble,
>>>
>>> I tried with the deltaImportQuery as well. But still I am getting the
>>> same
>>> exception on the server console -
>>>
>>> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
>>> doDeltaImport
>>>
>>> SEVERE: Delta Import Failed
>>> java.lang.NullPointerException
>>>at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>>at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>>at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>>at
>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>>at
>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>>at
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>>at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>>at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>>at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>
>>> Another thing, I do have proper data in the DB.
>>>
>>> Thanks,
>>> Manu
>>>
>>>
>>> Noble Paul നോബിള്‍ नोब्ळ् wrote:
>>> >
>>> > The same example with deltaImportQuery would look as follows
>>> > 
>>> >>> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>>> />
>>> >
>>> >>> >deltaQuery="select id from item where last_modified >
>>> > '${dataimporter.last_index_time}'"
>>> > deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>>> >>
>>> >
>>> >
>>> >
>>> >
>>> >>> >query="select DESCRIPTION from FEATURE where
>>> > ITEM_ID='${item.ID}'"
>>> >deltaQuery="select ITEM_ID from FEATURE where
>>> > last_modified > '${dataimporter.last_index_time}'"
>>> >parentDeltaQuery="select ID from item where
>>> > ID=${feature.ITEM_ID}">
>>> >
>>> >
>>> >
>>> >>> >query="select CATEGORY_ID from item_category where
>>> > ITEM_ID='${item.ID}'"
>>> >deltaQuery="select ITEM_ID, CATEGORY_ID from
>>> > item_category where last_modified > '${dataimporter.last_index_time}'"
>>> >parentDeltaQuery="select ID from item where
>>> > ID=${item_category.ITEM_ID}">
>>> >>> >query="select DESCRIPTION from category where ID
>>> =
>>> > '${item_category.CATEGORY_ID}'"
>>> >deltaQuery="select ID from category where
>>> > last_modified > '${dataimporter.last_index_time}'"
>>> >parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>>> from
>>> > item_category where CATEGORY_ID=${category.ID}">
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > 
>>> >
>>> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
>>> >  wrote:
>>> >> did you put in the data?
>>> >>
>>> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya 
>>> >> wrote:
>>> >>>
>>> >>> Hi Shalin,
>>> >>>
>>> >>> I am actually trying out the example provided at
>>> >>> http://wiki.apache.org/solr/DataImportHandler.
>>> >>>
>>> >>> My db-data-config.xml is as following -
>>> >>>
>>> >>> 
>>> >>>>> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>>> />
>>> >>>
>>> >>>>> >>>deltaQuery="select id from item where last_modified >
>>> >>> '${dataimporter.last_index_time}'">
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>>> >>>query="select DESCRIPTION from FEATURE where
>>> >>> ITEM_ID='${item.ID}'"
>>> >>>deltaQuery="select ITEM_ID from FEATURE where
>>> >>> last_modified > '${dataimporter.last_index_time}'"
>>> >>>parentDeltaQuery="select ID from item where
>>> >>> ID=${feature.ITEM_ID}">
>>> >>>
>>> >>>
>>> >>>
>>> >>>>> >>>query="select CATEGORY_ID from item_category where
>>> >>> ITEM_ID='${item.ID}'"
>>>

Re: Retrieve documents that contain max value for a field

2008-12-29 Thread Sushil Vegad


This looks useful, but I am not sure how to use the component. Could you
please elaborate?

Also, this is not available in Solr 1.3. Any equivalent of it in 1.3?

Thanks,
Sushil


ryantxu wrote:
> 
> not exactly what you are asking for, but check:
> http://wiki.apache.org/solr/StatsComponent
> 
> this will at least tell you the max/min versionId...   right now it  
> only works with numeric values, so it won't help for timestamp.
> 
> ryan
> 

-- 
View this message in context: 
http://www.nabble.com/Retrieve-documents-that-contain-max-value-for-a-field-tp21175643p21203697.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Any new python libraries?

2008-12-29 Thread Ed Summers

Jacob,

If you are interested in contributing any of your code to the solrpy
project [1] please let us know, either on here or on the solrpy
discussion list [2].

One of the motivations for putting the code up at code.google.com was
to make it easy for people to quickly contribute enhancements/fixes
separate from the normal release cycle of Solr proper.

//Ed

[1] http://code.google.com/p/solrpy/
[2] http://groups.google.com/group/solrpy

Re: Please help me integrate Nutch with Solr

2008-12-29 Thread Andrzej Bialecki


Tony Wang wrote:

Thanks Otis.

I've just downloaded
NUTCH-442_v8.patchfrom
https://issues.apache.org/jira/browse/NUTCH-442, but the patching process
gave me lots errors, see below:


This patch will be integrated within a couple days - please monitor this 
issue, and when it's done just download the patched code.



--
Best regards,
Andrzej Bialecki <><
 ___. ___ ___ ___ _ _   __
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

High response times after snapshot install

2008-12-29 Thread Victor Hogemann


Greetings,

Before anything I'd like to thank the Solr team for such great piece  
of software! Thank you.


Now... back to my problem. I'm experiencing slowdowns on a Solr 1.3  
multicore installation after a snapshot install, response times go up  
from 5ms to above 5sec, and don't go down unless we restart the servers.


Our index has more than 20 fields, and the slave servers got some  
50hits/sec. We're running it on Weblogic, with 6 Weblogic Nodes/Solr  
instances per machine, sharing the same index on the filesystem.


Any insight on what is happening?

Thanks.

Victor Hogemann - Ideais Tecnologia

Tel (RJ) +55 +21 3553-1301 (R217)
Tel (SP) +55 +11 4063-0443 (R217)
Fax +55 +21 3553-1302

Rua da Assembléia, nº 98, 9º andar
Centro | Rio de Janeiro | Brasil
CEP 20011-000

Re: Delta DataImport is not picking the modified value in DB


Hi Shalin,

I am using apache-solr-1.3.0 in my project. I thought it to be the latest
release. Does it not support delta import?

Thanks,
Manu


Shalin Shekhar Mangar wrote:
> 
> Which version of Solr are you using? The deltaImportQuery feature was
> added
> recently, you'd need a nightly build for it to work.
> 
> On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
> wrote:
> 
>>
>> Hi Noble,
>>
>> I tried with the deltaImportQuery as well. But still I am getting the
>> same
>> exception on the server console -
>>
>> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
>> doDeltaImport
>>
>> SEVERE: Delta Import Failed
>> java.lang.NullPointerException
>>at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>at
>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>at
>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>at
>>
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>at
>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>at
>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>
>> Another thing, I do have proper data in the DB.
>>
>> Thanks,
>> Manu
>>
>>
>> Noble Paul നോബിള്‍ नोब्ळ् wrote:
>> >
>> > The same example with deltaImportQuery would look as follows
>> > 
>> >> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>> />
>> >
>> >> >deltaQuery="select id from item where last_modified >
>> > '${dataimporter.last_index_time}'"
>> > deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>> >>
>> >
>> >
>> >
>> >
>> >> >query="select DESCRIPTION from FEATURE where
>> > ITEM_ID='${item.ID}'"
>> >deltaQuery="select ITEM_ID from FEATURE where
>> > last_modified > '${dataimporter.last_index_time}'"
>> >parentDeltaQuery="select ID from item where
>> > ID=${feature.ITEM_ID}">
>> >
>> >
>> >
>> >> >query="select CATEGORY_ID from item_category where
>> > ITEM_ID='${item.ID}'"
>> >deltaQuery="select ITEM_ID, CATEGORY_ID from
>> > item_category where last_modified > '${dataimporter.last_index_time}'"
>> >parentDeltaQuery="select ID from item where
>> > ID=${item_category.ITEM_ID}">
>> >> >query="select DESCRIPTION from category where ID
>> =
>> > '${item_category.CATEGORY_ID}'"
>> >deltaQuery="select ID from category where
>> > last_modified > '${dataimporter.last_index_time}'"
>> >parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>> from
>> > item_category where CATEGORY_ID=${category.ID}">
>> >
>> >
>> >
>> >
>> >
>> > 
>> >
>> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
>> >  wrote:
>> >> did you put in the data?
>> >>
>> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya 
>> >> wrote:
>> >>>
>> >>> Hi Shalin,
>> >>>
>> >>> I am actually trying out the example provided at
>> >>> http://wiki.apache.org/solr/DataImportHandler.
>> >>>
>> >>> My db-data-config.xml is as following -
>> >>>
>> >>> 
>> >>>> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>> />
>> >>>
>> >>>> >>>deltaQuery="select id from item where last_modified >
>> >>> '${dataimporter.last_index_time}'">
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>> >>>query="select DESCRIPTION from FEATURE where
>> >>> ITEM_ID='${item.ID}'"
>> >>>deltaQuery="select ITEM_ID from FEATURE where
>> >>> last_modified > '${dataimporter.last_index_time}'"
>> >>>parentDeltaQuery="select ID from item where
>> >>> ID=${feature.ITEM_ID}">
>> >>>
>> >>>
>> >>>
>> >>>> >>>query="select CATEGORY_ID from item_category where
>> >>> ITEM_ID='${item.ID}'"
>> >>>deltaQuery="select ITEM_ID, CATEGORY_ID from
>> >>> item_category where last_modified >
>> '${dataimporter.last_index_time}'"
>> >>>parentDeltaQuery="select ID from item where
>> >>> ID=${item_category.ITEM_ID}">
>> >>>> >>>query="select DESCRIPTION from category where
>> ID
>> >>> =
>> >>> '${item_category.C

Re: Delta DataImport is not picking the modified value in DB

2008-12-29 Thread Shalin Shekhar Mangar

Which version of Solr are you using? The deltaImportQuery feature was added
recently, you'd need a nightly build for it to work.

On Mon, Dec 29, 2008 at 4:57 PM, Manupriya wrote:

>
> Hi Noble,
>
> I tried with the deltaImportQuery as well. But still I am getting the same
> exception on the server console -
>
> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
>
> SEVERE: Delta Import Failed
> java.lang.NullPointerException
>at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>at
>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>at
>
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>at
>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>at
>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>
> Another thing, I do have proper data in the DB.
>
> Thanks,
> Manu
>
>
> Noble Paul നോബിള്‍ नोब्ळ् wrote:
> >
> > The same example with deltaImportQuery would look as follows
> > 
> > > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
> >
> > >deltaQuery="select id from item where last_modified >
> > '${dataimporter.last_index_time}'"
> > deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
> >>
> >
> >
> >
> >
> > >query="select DESCRIPTION from FEATURE where
> > ITEM_ID='${item.ID}'"
> >deltaQuery="select ITEM_ID from FEATURE where
> > last_modified > '${dataimporter.last_index_time}'"
> >parentDeltaQuery="select ID from item where
> > ID=${feature.ITEM_ID}">
> >
> >
> >
> > >query="select CATEGORY_ID from item_category where
> > ITEM_ID='${item.ID}'"
> >deltaQuery="select ITEM_ID, CATEGORY_ID from
> > item_category where last_modified > '${dataimporter.last_index_time}'"
> >parentDeltaQuery="select ID from item where
> > ID=${item_category.ITEM_ID}">
> > >query="select DESCRIPTION from category where ID =
> > '${item_category.CATEGORY_ID}'"
> >deltaQuery="select ID from category where
> > last_modified > '${dataimporter.last_index_time}'"
> >parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> > item_category where CATEGORY_ID=${category.ID}">
> >
> >
> >
> >
> >
> > 
> >
> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
> >  wrote:
> >> did you put in the data?
> >>
> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya 
> >> wrote:
> >>>
> >>> Hi Shalin,
> >>>
> >>> I am actually trying out the example provided at
> >>> http://wiki.apache.org/solr/DataImportHandler.
> >>>
> >>> My db-data-config.xml is as following -
> >>>
> >>> 
> >>> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
> />
> >>>
> >>> >>>deltaQuery="select id from item where last_modified >
> >>> '${dataimporter.last_index_time}'">
> >>>
> >>>
> >>>
> >>>
> >>> >>>query="select DESCRIPTION from FEATURE where
> >>> ITEM_ID='${item.ID}'"
> >>>deltaQuery="select ITEM_ID from FEATURE where
> >>> last_modified > '${dataimporter.last_index_time}'"
> >>>parentDeltaQuery="select ID from item where
> >>> ID=${feature.ITEM_ID}">
> >>>
> >>>
> >>>
> >>> >>>query="select CATEGORY_ID from item_category where
> >>> ITEM_ID='${item.ID}'"
> >>>deltaQuery="select ITEM_ID, CATEGORY_ID from
> >>> item_category where last_modified > '${dataimporter.last_index_time}'"
> >>>parentDeltaQuery="select ID from item where
> >>> ID=${item_category.ITEM_ID}">
> >>> >>>query="select DESCRIPTION from category where ID
> >>> =
> >>> '${item_category.CATEGORY_ID}'"
> >>>deltaQuery="select ID from category where
> >>> last_modified > '${dataimporter.last_index_time}'"
> >>>parentDeltaQuery="select ITEM_ID, CATEGORY_ID
> >>> from
> >>> item_category where CATEGORY_ID=${category.ID}">
> >>>
> >>>
> >>>

Re: Delta DataImport is not picking the modified value in DB


Hi Noble,

I tried with the deltaImportQuery as well. But still I am getting the same
exception on the server console - 

Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport

SEVERE: Delta Import Failed
java.lang.NullPointerException
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

Another thing, I do have proper data in the DB.

Thanks,
Manu


Noble Paul നോബിള്‍ नोब्ळ् wrote:
> 
> The same example with deltaImportQuery would look as follows
> 
> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
>
>deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'"
> deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>>
>
>
>
> 
>query="select DESCRIPTION from FEATURE where
> ITEM_ID='${item.ID}'"
>deltaQuery="select ITEM_ID from FEATURE where
> last_modified > '${dataimporter.last_index_time}'"
>parentDeltaQuery="select ID from item where
> ID=${feature.ITEM_ID}">
>
>
> 
>query="select CATEGORY_ID from item_category where
> ITEM_ID='${item.ID}'"
>deltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where last_modified > '${dataimporter.last_index_time}'"
>parentDeltaQuery="select ID from item where
> ID=${item_category.ITEM_ID}">
>query="select DESCRIPTION from category where ID =
> '${item_category.CATEGORY_ID}'"
>deltaQuery="select ID from category where
> last_modified > '${dataimporter.last_index_time}'"
>parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where CATEGORY_ID=${category.ID}">
>
>
>
>
>
> 
> 
> On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
>  wrote:
>> did you put in the data?
>>
>> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya 
>> wrote:
>>>
>>> Hi Shalin,
>>>
>>> I am actually trying out the example provided at
>>> http://wiki.apache.org/solr/DataImportHandler.
>>>
>>> My db-data-config.xml is as following -
>>>
>>> 
>>>>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
>>>
>>>>>deltaQuery="select id from item where last_modified >
>>> '${dataimporter.last_index_time}'">
>>>
>>>
>>>
>>>
>>>>>query="select DESCRIPTION from FEATURE where
>>> ITEM_ID='${item.ID}'"
>>>deltaQuery="select ITEM_ID from FEATURE where
>>> last_modified > '${dataimporter.last_index_time}'"
>>>parentDeltaQuery="select ID from item where
>>> ID=${feature.ITEM_ID}">
>>>
>>>
>>>
>>>>>query="select CATEGORY_ID from item_category where
>>> ITEM_ID='${item.ID}'"
>>>deltaQuery="select ITEM_ID, CATEGORY_ID from
>>> item_category where last_modified > '${dataimporter.last_index_time}'"
>>>parentDeltaQuery="select ID from item where
>>> ID=${item_category.ITEM_ID}">
>>>>>query="select DESCRIPTION from category where ID
>>> =
>>> '${item_category.CATEGORY_ID}'"
>>>deltaQuery="select ID from category where
>>> last_modified > '${dataimporter.last_index_time}'"
>>>parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>>> from
>>> item_category where CATEGORY_ID=${category.ID}">
>>>
>>>
>>>
>>>
>>>
>>> 
>>> ---
>>>
>>> My DB structure is as following -
>>>
>>> DROP TABLE IF EXISTS solr_demo_db.item;
>>> CREATE TABLE `item` (
>>>  `ID` int(2) NOT NULL auto_increment,
>>>  `name` varchar(100) default NULL,
>>>  `manu` varchar(20) default NULL,
>>>  `weight` varchar(20) default NULL,
>>>  `price` varchar(20) default NULL,
>>>

Re: Delta DataImport is not picking the modified value in DB

The same example with deltaImportQuery would look as follows

   
   
   
   
   
   

   
   
   

   
   
   
   
   
   
   


On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
 wrote:
> did you put in the data?
>
> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya  wrote:
>>
>> Hi Shalin,
>>
>> I am actually trying out the example provided at
>> http://wiki.apache.org/solr/DataImportHandler.
>>
>> My db-data-config.xml is as following -
>>
>> 
>>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
>>
>>>deltaQuery="select id from item where last_modified >
>> '${dataimporter.last_index_time}'">
>>
>>
>>
>>
>>>query="select DESCRIPTION from FEATURE where
>> ITEM_ID='${item.ID}'"
>>deltaQuery="select ITEM_ID from FEATURE where
>> last_modified > '${dataimporter.last_index_time}'"
>>parentDeltaQuery="select ID from item where
>> ID=${feature.ITEM_ID}">
>>
>>
>>
>>>query="select CATEGORY_ID from item_category where
>> ITEM_ID='${item.ID}'"
>>deltaQuery="select ITEM_ID, CATEGORY_ID from
>> item_category where last_modified > '${dataimporter.last_index_time}'"
>>parentDeltaQuery="select ID from item where
>> ID=${item_category.ITEM_ID}">
>>>query="select DESCRIPTION from category where ID =
>> '${item_category.CATEGORY_ID}'"
>>deltaQuery="select ID from category where
>> last_modified > '${dataimporter.last_index_time}'"
>>parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
>> item_category where CATEGORY_ID=${category.ID}">
>>
>>
>>
>>
>>
>> 
>> ---
>>
>> My DB structure is as following -
>>
>> DROP TABLE IF EXISTS solr_demo_db.item;
>> CREATE TABLE `item` (
>>  `ID` int(2) NOT NULL auto_increment,
>>  `name` varchar(100) default NULL,
>>  `manu` varchar(20) default NULL,
>>  `weight` varchar(20) default NULL,
>>  `price` varchar(20) default NULL,
>>  `popularity` varchar(2) default NULL,
>>  `includes` varchar(10) default NULL,
>>  `last_modified` datetime default NULL,
>>  PRIMARY KEY  (`ID`)
>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>>
>> DROP TABLE IF EXISTS solr_demo_db.feature;
>> CREATE TABLE `feature` (
>>  `DESCRIPTION` varchar(100) default NULL,
>>  `ITEM_ID` int(2) NOT NULL,
>>  `last_modified` datetime default NULL,
>>  PRIMARY KEY  (`ITEM_ID`)
>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>
>> DROP TABLE IF EXISTS solr_demo_db.category;
>> CREATE TABLE `category` (
>>  `ID` int(2) NOT NULL auto_increment,
>>  `DESCRIPTION` varchar(100) default NULL,
>>  `last_modified` date default NULL,
>>  PRIMARY KEY  (`ID`)
>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>>
>> DROP TABLE IF EXISTS solr_demo_db.item_category;
>> CREATE TABLE `item_category` (
>>  `ITEM_ID` int(2) NOT NULL default '0',
>>  `CATEGORY_ID` int(2) NOT NULL default '0',
>>  `last_modified` date default NULL,
>>  PRIMARY KEY  (`ITEM_ID`,`CATEGORY_ID`)
>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>
>> Am I doing something wrong here?
>>
>> Thanks,
>> Manu
>>
>>
>> Shalin Shekhar Mangar wrote:
>>>
>>> What does your data-config look like? especially the delta query part.
>>>
>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>>> wrote:
>>>

 Hi,

 I am using Delta DataImport feature to partially refresh the indexes.

 Note - full-import is working perfectly fine. But I need to do
 delta-import
 as I do not want to rebuild all the indexes.

 My database structure is same as mentioned in the example at
 http://wiki.apache.org/solr/DataImportHandler.

 http://www.nabble.com/file/p21200498/example-schema.png
 ---
 The dataimport.properties has the value as -

 #Mon Dec 29 14:08:12 IST 2008
 last_index_time=2008-12-29 14\:08\:12
 ---
 I have changed the last_modified for the 'tem' table as current
 timestamp.
 And when I query the DB, I get the following result -

 http://www.nabble.com/file/p21200498/data.jpg

 -

 But when I call delta data-import, it returns me the following response -

 Server Response:

 {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importRespon

Re: Delta DataImport is not picking the modified value in DB

did you put in the data?

On Mon, Dec 29, 2008 at 3:07 PM, Manupriya  wrote:
>
> Hi Shalin,
>
> I am actually trying out the example provided at
> http://wiki.apache.org/solr/DataImportHandler.
>
> My db-data-config.xml is as following -
>
> 
> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
>
>deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'">
>
>
>
>
>query="select DESCRIPTION from FEATURE where
> ITEM_ID='${item.ID}'"
>deltaQuery="select ITEM_ID from FEATURE where
> last_modified > '${dataimporter.last_index_time}'"
>parentDeltaQuery="select ID from item where
> ID=${feature.ITEM_ID}">
>
>
>
>query="select CATEGORY_ID from item_category where
> ITEM_ID='${item.ID}'"
>deltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where last_modified > '${dataimporter.last_index_time}'"
>parentDeltaQuery="select ID from item where
> ID=${item_category.ITEM_ID}">
>query="select DESCRIPTION from category where ID =
> '${item_category.CATEGORY_ID}'"
>deltaQuery="select ID from category where
> last_modified > '${dataimporter.last_index_time}'"
>parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where CATEGORY_ID=${category.ID}">
>
>
>
>
>
> 
> ---
>
> My DB structure is as following -
>
> DROP TABLE IF EXISTS solr_demo_db.item;
> CREATE TABLE `item` (
>  `ID` int(2) NOT NULL auto_increment,
>  `name` varchar(100) default NULL,
>  `manu` varchar(20) default NULL,
>  `weight` varchar(20) default NULL,
>  `price` varchar(20) default NULL,
>  `popularity` varchar(2) default NULL,
>  `includes` varchar(10) default NULL,
>  `last_modified` datetime default NULL,
>  PRIMARY KEY  (`ID`)
> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>
> DROP TABLE IF EXISTS solr_demo_db.feature;
> CREATE TABLE `feature` (
>  `DESCRIPTION` varchar(100) default NULL,
>  `ITEM_ID` int(2) NOT NULL,
>  `last_modified` datetime default NULL,
>  PRIMARY KEY  (`ITEM_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>
> DROP TABLE IF EXISTS solr_demo_db.category;
> CREATE TABLE `category` (
>  `ID` int(2) NOT NULL auto_increment,
>  `DESCRIPTION` varchar(100) default NULL,
>  `last_modified` date default NULL,
>  PRIMARY KEY  (`ID`)
> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>
> DROP TABLE IF EXISTS solr_demo_db.item_category;
> CREATE TABLE `item_category` (
>  `ITEM_ID` int(2) NOT NULL default '0',
>  `CATEGORY_ID` int(2) NOT NULL default '0',
>  `last_modified` date default NULL,
>  PRIMARY KEY  (`ITEM_ID`,`CATEGORY_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>
> Am I doing something wrong here?
>
> Thanks,
> Manu
>
>
> Shalin Shekhar Mangar wrote:
>>
>> What does your data-config look like? especially the delta query part.
>>
>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>> wrote:
>>
>>>
>>> Hi,
>>>
>>> I am using Delta DataImport feature to partially refresh the indexes.
>>>
>>> Note - full-import is working perfectly fine. But I need to do
>>> delta-import
>>> as I do not want to rebuild all the indexes.
>>>
>>> My database structure is same as mentioned in the example at
>>> http://wiki.apache.org/solr/DataImportHandler.
>>>
>>> http://www.nabble.com/file/p21200498/example-schema.png
>>> ---
>>> The dataimport.properties has the value as -
>>>
>>> #Mon Dec 29 14:08:12 IST 2008
>>> last_index_time=2008-12-29 14\:08\:12
>>> ---
>>> I have changed the last_modified for the 'tem' table as current
>>> timestamp.
>>> And when I query the DB, I get the following result -
>>>
>>> http://www.nabble.com/file/p21200498/data.jpg
>>>
>>> -
>>>
>>> But when I call delta data-import, it returns me the following response -
>>>
>>> Server Response:
>>>
>>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>>> response format is experimental.  It is likely to change in the future.}
>>>
>>>
>>> ---
>>> And when I query for data for '*:*', I get the following -
>>> Note - Data returned by query doesnt reflect the correct timestamp from
>>> database.
>>>
>>> SolrDocument(1)
>>> Field Name & Value: includes=12
>>> Field Name & Val

Multiple language support

2008-12-29 Thread Deshpande, Mukta

Hi All,
 
I have a multiple language supporting schema in which there is a separate field 
for every language.
 
I have a field "product_name" to store product name and its description that 
can be in any user preferred language. 
This can be stored in fields product_name_EN if user prefers English language, 
product_name_SCH if user prefers Simplified Chinese language.
The WhitespaceTokenizerFactory and filter EnglishPorterFilterFactory are 
applied on product_name_EN.
The CJKAnalyzer and CJKTokenizer are applied on product_name_SCH.
 
e.g. Value can be : ElectrolyticCapacitor - 被对立的电容器以价值220µF
 
Now my problem is: Which field do I store the above value?
product_name_EN OR product_name_SCH OR should it be something else?
 
How do I find out which analyzers should get applied for this field.
 
Did any one face a similar situation before. 
Please help ASAP.
 
Thanks,
~Mukta

Re: Delta DataImport is not picking the modified value in DB


Hi Shalin,

I am actually trying out the example provided at
http://wiki.apache.org/solr/DataImportHandler.

My db-data-config.xml is as following - 





















---

My DB structure is as following - 

DROP TABLE IF EXISTS solr_demo_db.item;
CREATE TABLE `item` (
  `ID` int(2) NOT NULL auto_increment,
  `name` varchar(100) default NULL,
  `manu` varchar(20) default NULL,
  `weight` varchar(20) default NULL,
  `price` varchar(20) default NULL,
  `popularity` varchar(2) default NULL,
  `includes` varchar(10) default NULL,
  `last_modified` datetime default NULL,
  PRIMARY KEY  (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;

DROP TABLE IF EXISTS solr_demo_db.feature;
CREATE TABLE `feature` (
  `DESCRIPTION` varchar(100) default NULL,
  `ITEM_ID` int(2) NOT NULL,
  `last_modified` datetime default NULL,
  PRIMARY KEY  (`ITEM_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

DROP TABLE IF EXISTS solr_demo_db.category;
CREATE TABLE `category` (
  `ID` int(2) NOT NULL auto_increment,
  `DESCRIPTION` varchar(100) default NULL,
  `last_modified` date default NULL,
  PRIMARY KEY  (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;

DROP TABLE IF EXISTS solr_demo_db.item_category;
CREATE TABLE `item_category` (
  `ITEM_ID` int(2) NOT NULL default '0',
  `CATEGORY_ID` int(2) NOT NULL default '0',
  `last_modified` date default NULL,
  PRIMARY KEY  (`ITEM_ID`,`CATEGORY_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Am I doing something wrong here?

Thanks,
Manu


Shalin Shekhar Mangar wrote:
> 
> What does your data-config look like? especially the delta query part.
> 
> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
> wrote:
> 
>>
>> Hi,
>>
>> I am using Delta DataImport feature to partially refresh the indexes.
>>
>> Note - full-import is working perfectly fine. But I need to do
>> delta-import
>> as I do not want to rebuild all the indexes.
>>
>> My database structure is same as mentioned in the example at
>> http://wiki.apache.org/solr/DataImportHandler.
>>
>> http://www.nabble.com/file/p21200498/example-schema.png
>> ---
>> The dataimport.properties has the value as -
>>
>> #Mon Dec 29 14:08:12 IST 2008
>> last_index_time=2008-12-29 14\:08\:12
>> ---
>> I have changed the last_modified for the 'tem' table as current
>> timestamp.
>> And when I query the DB, I get the following result -
>>
>> http://www.nabble.com/file/p21200498/data.jpg
>>
>> -
>>
>> But when I call delta data-import, it returns me the following response -
>>
>> Server Response:
>>
>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>> response format is experimental.  It is likely to change in the future.}
>>
>>
>> ---
>> And when I query for data for '*:*', I get the following -
>> Note - Data returned by query doesnt reflect the correct timestamp from
>> database.
>>
>> SolrDocument(1)
>> Field Name & Value: includes=12
>> Field Name & Value: features=[demo feature, demo feature 1]
>> Field Name & Value: price=12.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=manu-12
>> Field Name & Value: sku=1
>> Field Name & Value: id=1
>> Field Name & Value: popularity=1
>> Field Name & Value: weight=12.0
>> SolrDocument(2)
>> Field Name & Value: includes=43
>> Field Name & Value: features=demo feature 2
>> Field Name & Value: price=10.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=Demo - 12
>> Field Name & Value: sku=2
>> Field Name & Value: id=2
>> Field Name & Value: popularity=5
>> Field Name & Value: weight=12.0
>> SolrDocument(3)
>> Field Name & Value: includes=1
>> Field Name & Value: features=demo -3
>> Field Name & Value: price=1.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=manu - 36
>> Field Name & Value: sku=3
>> Field Name & Value: id=3
>> Field Name & Value: popularity=1
>> Field Name & Value: weight=1.0
>> SolrDocument(4)
>> Field Name & Value: includes=2
>> Field Name & Value: features=demo - 4
>> Field Name & Value: price=2.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 46
>> Field Name & Value: sku=4
>> Field Name & Value: id=4
>> Field Name & Value: popularity=2
>> Field Name & Value: weight=2.0

Re: Delta DataImport is not picking the modified value in DB

This may happen if your deltaQuery has a problem

Try using the deltaImportQuery also.

--

On Mon, Dec 29, 2008 at 2:57 PM, Shalin Shekhar Mangar
 wrote:
> What does your data-config look like? especially the delta query part.
>
> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya wrote:
>
>>
>> Hi,
>>
>> I am using Delta DataImport feature to partially refresh the indexes.
>>
>> Note - full-import is working perfectly fine. But I need to do delta-import
>> as I do not want to rebuild all the indexes.
>>
>> My database structure is same as mentioned in the example at
>> http://wiki.apache.org/solr/DataImportHandler.
>>
>> http://www.nabble.com/file/p21200498/example-schema.png
>> ---
>> The dataimport.properties has the value as -
>>
>> #Mon Dec 29 14:08:12 IST 2008
>> last_index_time=2008-12-29 14\:08\:12
>> ---
>> I have changed the last_modified for the 'tem' table as current timestamp.
>> And when I query the DB, I get the following result -
>>
>> http://www.nabble.com/file/p21200498/data.jpg
>>
>> -
>>
>> But when I call delta data-import, it returns me the following response -
>>
>> Server Response:
>>
>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>> response format is experimental.  It is likely to change in the future.}
>>
>>
>> ---
>> And when I query for data for '*:*', I get the following -
>> Note - Data returned by query doesnt reflect the correct timestamp from
>> database.
>>
>> SolrDocument(1)
>> Field Name & Value: includes=12
>> Field Name & Value: features=[demo feature, demo feature 1]
>> Field Name & Value: price=12.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=manu-12
>> Field Name & Value: sku=1
>> Field Name & Value: id=1
>> Field Name & Value: popularity=1
>> Field Name & Value: weight=12.0
>> SolrDocument(2)
>> Field Name & Value: includes=43
>> Field Name & Value: features=demo feature 2
>> Field Name & Value: price=10.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=Demo - 12
>> Field Name & Value: sku=2
>> Field Name & Value: id=2
>> Field Name & Value: popularity=5
>> Field Name & Value: weight=12.0
>> SolrDocument(3)
>> Field Name & Value: includes=1
>> Field Name & Value: features=demo -3
>> Field Name & Value: price=1.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=manu - 36
>> Field Name & Value: sku=3
>> Field Name & Value: id=3
>> Field Name & Value: popularity=1
>> Field Name & Value: weight=1.0
>> SolrDocument(4)
>> Field Name & Value: includes=2
>> Field Name & Value: features=demo - 4
>> Field Name & Value: price=2.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 46
>> Field Name & Value: sku=4
>> Field Name & Value: id=4
>> Field Name & Value: popularity=2
>> Field Name & Value: weight=2.0
>> SolrDocument(5)
>> Field Name & Value: includes=3
>> Field Name & Value: features=demo - 5
>> Field Name & Value: price=3.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 56
>> Field Name & Value: sku=5
>> Field Name & Value: id=5
>> Field Name & Value: popularity=3
>> Field Name & Value: weight=3.0
>> SolrDocument(6)
>> Field Name & Value: includes=4
>> Field Name & Value: features=demo - 66
>> Field Name & Value: price=4.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 66
>> Field Name & Value: sku=6
>> Field Name & Value: id=6
>> Field Name & Value: popularity=4
>> Field Name & Value: weight=4.0
>>
>>
>> --
>> My Java code that calls delta import is as follows -
>>
>> public static void main(String[] args) throws IOException,
>> SolrServerException {
>> CommonsHttpSolrServer server = new
>> CommonsHttpSolrServer("http://localhost:8983/solr";);
>> BinaryResponseParser parser = new BinaryResponseParser();
>> server.setParser(parser);
>>
>> ModifiableSolrParams params = new ModifiableSolrParams();
>> params.set("qt", "/dataimport");
>> params.set("command", "delta-import");
>> params.set("commit", "true");
>> params.set("wt","json");
>>
>> try{
>>
>>QueryResponse response = server.query(params);
>>server.commit();
>>System.out.println("Server Response: " + response);
>>
>>SolrQuery query = new SolrQuery("*:*");
>>QueryResponse results = server.query(query);
>>
>>SolrDocumentList list = results.getResults();
>>
>>int index = 1;
>>for (SolrDoc

Re: Delta DataImport is not picking the modified value in DB

2008-12-29 Thread Shalin Shekhar Mangar

What does your data-config look like? especially the delta query part.

On Mon, Dec 29, 2008 at 2:35 PM, Manupriya wrote:

>
> Hi,
>
> I am using Delta DataImport feature to partially refresh the indexes.
>
> Note - full-import is working perfectly fine. But I need to do delta-import
> as I do not want to rebuild all the indexes.
>
> My database structure is same as mentioned in the example at
> http://wiki.apache.org/solr/DataImportHandler.
>
> http://www.nabble.com/file/p21200498/example-schema.png
> ---
> The dataimport.properties has the value as -
>
> #Mon Dec 29 14:08:12 IST 2008
> last_index_time=2008-12-29 14\:08\:12
> ---
> I have changed the last_modified for the 'tem' table as current timestamp.
> And when I query the DB, I get the following result -
>
> http://www.nabble.com/file/p21200498/data.jpg
>
> -
>
> But when I call delta data-import, it returns me the following response -
>
> Server Response:
>
> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
> response format is experimental.  It is likely to change in the future.}
>
>
> ---
> And when I query for data for '*:*', I get the following -
> Note - Data returned by query doesnt reflect the correct timestamp from
> database.
>
> SolrDocument(1)
> Field Name & Value: includes=12
> Field Name & Value: features=[demo feature, demo feature 1]
> Field Name & Value: price=12.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> Field Name & Value: manu=manu-12
> Field Name & Value: sku=1
> Field Name & Value: id=1
> Field Name & Value: popularity=1
> Field Name & Value: weight=12.0
> SolrDocument(2)
> Field Name & Value: includes=43
> Field Name & Value: features=demo feature 2
> Field Name & Value: price=10.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> Field Name & Value: manu=Demo - 12
> Field Name & Value: sku=2
> Field Name & Value: id=2
> Field Name & Value: popularity=5
> Field Name & Value: weight=12.0
> SolrDocument(3)
> Field Name & Value: includes=1
> Field Name & Value: features=demo -3
> Field Name & Value: price=1.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> Field Name & Value: manu=manu - 36
> Field Name & Value: sku=3
> Field Name & Value: id=3
> Field Name & Value: popularity=1
> Field Name & Value: weight=1.0
> SolrDocument(4)
> Field Name & Value: includes=2
> Field Name & Value: features=demo - 4
> Field Name & Value: price=2.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> Field Name & Value: manu=manu - 46
> Field Name & Value: sku=4
> Field Name & Value: id=4
> Field Name & Value: popularity=2
> Field Name & Value: weight=2.0
> SolrDocument(5)
> Field Name & Value: includes=3
> Field Name & Value: features=demo - 5
> Field Name & Value: price=3.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> Field Name & Value: manu=manu - 56
> Field Name & Value: sku=5
> Field Name & Value: id=5
> Field Name & Value: popularity=3
> Field Name & Value: weight=3.0
> SolrDocument(6)
> Field Name & Value: includes=4
> Field Name & Value: features=demo - 66
> Field Name & Value: price=4.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> Field Name & Value: manu=manu - 66
> Field Name & Value: sku=6
> Field Name & Value: id=6
> Field Name & Value: popularity=4
> Field Name & Value: weight=4.0
>
>
> --
> My Java code that calls delta import is as follows -
>
> public static void main(String[] args) throws IOException,
> SolrServerException {
> CommonsHttpSolrServer server = new
> CommonsHttpSolrServer("http://localhost:8983/solr";);
> BinaryResponseParser parser = new BinaryResponseParser();
> server.setParser(parser);
>
> ModifiableSolrParams params = new ModifiableSolrParams();
> params.set("qt", "/dataimport");
> params.set("command", "delta-import");
> params.set("commit", "true");
> params.set("wt","json");
>
> try{
>
>QueryResponse response = server.query(params);
>server.commit();
>System.out.println("Server Response: " + response);
>
>SolrQuery query = new SolrQuery("*:*");
>QueryResponse results = server.query(query);
>
>SolrDocumentList list = results.getResults();
>
>int index = 1;
>for (SolrDocument solrDocument : list) {
>
>System.out.println("SolrDocument("+index+")");
>Iterator> iterator = solrDocument.iterator();
>
>while(iterator.hasNext()){
>System.out.println("Field Name & Value: "+iterator.next());
>}
>
>index++;
>

Delta DataImport is not picking the modified value in DB