Hello,
We use ManifoldCF v2.10, with postgresql (9.6) to crawl our websites.
this represents approximately 1.2 million documents.
We split the crawl into 4 jobs that distribute their results on 3 SOLR
collections.
The crawl is powerful up to 500000 documents (25000 to 30000 docs /
hour) then the performance decreases strongly in progress, we observe
freezes very very long, you might think that the crawl is stopped.
We suspect a reindexing, noticeably of the intrinsiclink table which is
very important 85 Million lines.
Is it possible to prohibit re-indexing controlled by manifoldCF?
An other idea ?
best Regards
LIROT daniel
--
