On 4/13/2018 11:03 AM, Jesus Olivan wrote: > thanks for your answer. It happens that when we launch full import process > didn't finished (we wait for more than 60 hours last time, and we cancelled > it, because this is not an acceptable time for us) There weren't any errors > in solr logfile simply because it was working fine. The problem is that it > lasted eternally and didn't finish. We tried it on Aurora cluster under > AWS, and after 20 hours of work, it failed due to lack of space in Aurora > tmp folder.
375 million documents importing from MySQL with one DIH import is going to take quite a while. The last full rebuild I did of my main index took 21.61 hours. This is an index where six large shards build simultaneously, using DIH, each one having more than 30 million documents. If I were to build it as a single 180 million document import, it would probably take 5 days, maybe longer. We had another index (since retired) that had more than 400 million total documents, built similarly with multiple shards at the same time. The last rebuild I can remember on that index took about two days. Thanks, Shawn
