On 4/13/2018 10:11 AM, Jesus Olivan wrote:
> we're trying to launch a full import of 375 millions of docs aprox. from a
> MySQL database to our solrcloud cluster. Until now, this full import
> process takes around 24/27 hours to finish due to an huge import query
> (several group bys, left joins, etc), but after another import query
> modification (adding more complexity), we're unable to execute this full
> import from MySQL.
> We've done some research about migrating to PostgreSQL, but this option is
> now a real option at this time, because it implies a big refatoring from
> several dev teams.
> Is there some alternative ways to perform successfully this full import
> process?

DIH is a capable tool, and for what it does, it's remarkably efficient.

It can't really be made any faster, because it's single threaded.  To
get increased index speed with Solr, you must index documents from
several sources/processes/threads at the same time.  Writing custom
software that can retrieve information from your source, build the
documents you require, and send several update requests simultaneously
will yield the best results.  The source itself may be a bottleneck
though -- this is frequently the case, and Solr is often MUCH faster
than the information source.

You said that you're unable to execute an updated import from MySQL. 
What exactly happens when you try?  Are there any errors in your solr

I'm not going to debate whether MySQL or PostgreSQL is the better
solution.  For my indexes, my source data is in MySQL.  It works well,
but full rebuilds using DIH are slower than I would like -- because it's
single-threaded.  Our overall system architecture would probably be
improved by a switch to PostgreSQL, but it would be an extremely
time-consuming transition process.  We aren't having any real issues
with MySQL, so we have no incentive to spend the required effort.


Reply via email to