Jesus, Usually zipper join (aka external merge in old ETL world) and explicit partitioning is able to boost import. https://lucene.apache.org/solr/guide/6_6/uploading-structured-data-store-data-with-the-data-import-handler.html#entity-processors
On Fri, Apr 13, 2018 at 7:11 PM, Jesus Olivan <jesus.oli...@letgo.com> wrote: > Hi! > > we're trying to launch a full import of 375 millions of docs aprox. from a > MySQL database to our solrcloud cluster. Until now, this full import > process takes around 24/27 hours to finish due to an huge import query > (several group bys, left joins, etc), but after another import query > modification (adding more complexity), we're unable to execute this full > import from MySQL. > > We've done some research about migrating to PostgreSQL, but this option is > now a real option at this time, because it implies a big refatoring from > several dev teams. > > Is there some alternative ways to perform successfully this full import > process? > > Any ideas are welcome :) > > Thanks in advance! > -- Sincerely yours Mikhail Khludnev
