Re: SOLR Index Time Running Optimization
How long does the query take when it is run directly, without Solr? For our DIH queries, Solr was not the slow part. It took 90 minutes directly or with DIH. With our big cluster, I’ve seen indexing rates of one million docs per minute. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Sep 26, 2018, at 9:44 AM, Jan Høydahl wrote: > > With DIH you are doing indexing single-threaded. You should be able to > configure multiple DIH's on the same collection and then partition the data > between them, issuing slightly different SQL to each. But I don't exactly > know what that would look like. > > -- > Jan Høydahl, search solution architect > Cominvent AS - www.cominvent.com > >> 26. sep. 2018 kl. 14:30 skrev Susheel Kumar : >> >> Also are you using Solr data import? That will be much slower compare to if >> you write our own little indexer which does indexing in batches and with >> multiple threads. >> >> On Wed, Sep 26, 2018 at 8:00 AM Vincenzo D'Amore wrote: >> >>> Hi, I know this is the shortest way but, had you tried to add more core or >>> CPU to your solr instances? How big is you collection in terms of GB and >>> number of documents? >>> >>> Ciao, >>> Vincenzo >>> >>> On 26 Sep 2018, at 08:36, Krizelle Mae Hernandez < >>> krizellemae.marti...@sas.com> wrote: Hi. Our SOLR currently is running approximately 39hours for Full and Delta >>> Import. I would like to ask for your assistance on how can we shorten the >>> 39hours run time in any possible solution? For SOLR version, we are using solr 5.3.1. Regards, Krizelle Mae M. Hernandez >>> >
Re: SOLR Index Time Running Optimization
With DIH you are doing indexing single-threaded. You should be able to configure multiple DIH's on the same collection and then partition the data between them, issuing slightly different SQL to each. But I don't exactly know what that would look like. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 26. sep. 2018 kl. 14:30 skrev Susheel Kumar : > > Also are you using Solr data import? That will be much slower compare to if > you write our own little indexer which does indexing in batches and with > multiple threads. > > On Wed, Sep 26, 2018 at 8:00 AM Vincenzo D'Amore wrote: > >> Hi, I know this is the shortest way but, had you tried to add more core or >> CPU to your solr instances? How big is you collection in terms of GB and >> number of documents? >> >> Ciao, >> Vincenzo >> >> >>> On 26 Sep 2018, at 08:36, Krizelle Mae Hernandez < >> krizellemae.marti...@sas.com> wrote: >>> >>> Hi. >>> >>> Our SOLR currently is running approximately 39hours for Full and Delta >> Import. I would like to ask for your assistance on how can we shorten the >> 39hours run time in any possible solution? >>> For SOLR version, we are using solr 5.3.1. >>> >>> Regards, >>> Krizelle Mae M. Hernandez >>
Re: SOLR Index Time Running Optimization
Also are you using Solr data import? That will be much slower compare to if you write our own little indexer which does indexing in batches and with multiple threads. On Wed, Sep 26, 2018 at 8:00 AM Vincenzo D'Amore wrote: > Hi, I know this is the shortest way but, had you tried to add more core or > CPU to your solr instances? How big is you collection in terms of GB and > number of documents? > > Ciao, > Vincenzo > > > > On 26 Sep 2018, at 08:36, Krizelle Mae Hernandez < > krizellemae.marti...@sas.com> wrote: > > > > Hi. > > > > Our SOLR currently is running approximately 39hours for Full and Delta > Import. I would like to ask for your assistance on how can we shorten the > 39hours run time in any possible solution? > > For SOLR version, we are using solr 5.3.1. > > > > Regards, > > Krizelle Mae M. Hernandez >
Re: SOLR Index Time Running Optimization
Hi, I know this is the shortest way but, had you tried to add more core or CPU to your solr instances? How big is you collection in terms of GB and number of documents? Ciao, Vincenzo > On 26 Sep 2018, at 08:36, Krizelle Mae Hernandez > wrote: > > Hi. > > Our SOLR currently is running approximately 39hours for Full and Delta > Import. I would like to ask for your assistance on how can we shorten the > 39hours run time in any possible solution? > For SOLR version, we are using solr 5.3.1. > > Regards, > Krizelle Mae M. Hernandez
SOLR Index Time Running Optimization
Hi. Our SOLR currently is running approximately 39hours for Full and Delta Import. I would like to ask for your assistance on how can we shorten the 39hours run time in any possible solution? For SOLR version, we are using solr 5.3.1. Regards, Krizelle Mae M. Hernandez
