Hi Alex , you may be able to use CachedSqlEntityprocessor. you can do
delta-import using full-import
http://wiki.apache.org/solr/DataImportHandlerFaq#fullimportdelta
the inner entity can use a CachedSqlEntityProcessor
On Thu, Mar 26, 2009 at 1:45 AM, AlexxelA <alexandre.boudrea...@canoe.ca> wrote:
>
> Yes my database is remote, mysql 5 and i'm using connector/J 5.1.7. My index
> has 20000 documents. When i try to do lets say 14 updates it takes about 18
> sec total. Here's the resulting log of the operation :
>
> 2009-03-25 15:53:57 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
> INFO: Time taken for getConnection(): 411
> 2009-03-25 15:53:59 org.apache.solr.handler.dataimport.DocBuilder
> collectDelta
> INFO: Completed ModifiedRowKey for Entity: profil rows obtained : 14
> 2009-03-25 15:53:59 org.apache.solr.handler.dataimport.DocBuilder
> collectDelta
> INFO: Completed DeletedRowKey for Entity: profil rows obtained : 0
> 2009-03-25 15:53:59 org.apache.solr.handler.dataimport.DocBuilder
> collectDelta
> INFO: Completed parentDeltaQuery for Entity: profil
> 2009-03-25 15:54:00 org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=1
>
> commit{dir=/home/solr-tomcat/solr/data/index,segFN=segments_sb,version=1237322897338,generation=1019,filenames=[_uj.frq,
> _uj.fdx, _uj.tii, _uj.nrm, _uj.tis, _uj.fnm, _uj.prx, segments_sb, _uj.fdt]
> 2009-03-25 15:54:00 org.apache.solr.core.SolrDeletionPolicy updateCommits
> INFO: last commit = 1237322897338
> 2009-03-25 15:54:13 org.apache.solr.handler.dataimport.DocBuilder doDelta
> INFO: Delta Import completed successfully BOTTLE NECK
> 2009-03-25 15:54:13 org.apache.solr.handler.dataimport.DocBuilder commit
> INFO: Full Import completed successfully
> 2009-03-25 15:54:13 org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true)
> 2009-03-25 15:54:15 org.apache.solr.core.SolrDeletionPolicy onCommit
> INFO: SolrDeletionPolicy.onCommit: commits:num=2
>
> commit{dir=/home/solr-tomcat/solr/data/index,segFN=segments_sb,version=1237322897338,generation=1019,filenames=[_uj.frq,
> _uj.fdx, _uj.tii, _uj.nrm, _uj.tis, _uj.fnm, _uj.prx, segments_sb, _uj.fdt]
>
> commit{dir=/home/solr-tomcat/solr/data/index,segFN=segments_sc,version=1237322897339,generation=1020,filenames=[_ul.prx,
> _ul.fnm, _ul.tii, _ul.fdt, _ul.nrm, _ul.fdx, _ul.tis, _ul.frq, segments_sc]
> 2009-03-25 15:54:15 org.apache.solr.core.SolrDeletionPolicy updateCommits
> INFO: last commit = 1237322897339
> 2009-03-25 15:54:15 org.apache.solr.search.SolrIndexSearcher <init>
> INFO: Opening searc...@3da850 main
>
> When i do a full-import it is much faster. Take about 1 min to index 20000
> documents. I tried to play a bit with the config but nothing seems to work
> for the moment.
>
> What i want to do is pretty interactive, my production db has 1,2M documents
> and must be able to delta-import around 2k update every 5min. Is it
> possible with the dataimporthandle to reach those kinda of number ?
>
>
>
> Shalin Shekhar Mangar wrote:
>>
>> On Wed, Mar 25, 2009 at 2:25 AM, AlexxelA
>> <alexandre.boudrea...@canoe.ca>wrote:
>>
>>>
>>> Ok i'm ok with the fact the solr gonna do X request to database for X
>>> update.. but when i try to run the delta-import command with 20000 row to
>>> update is it normal that its kinda really slow ~ 1 document fetched / sec
>>> ?
>>>
>>>
>> Not really, I've seen 1000x faster. Try firing a few of those queries on
>> the
>> database directly. Are they slow? Is the database remote?
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Delta-import-tp22663196p22710222.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
