[
https://issues.apache.org/jira/browse/SOLR-9591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552489#comment-15552489
]
Kevin Risden commented on SOLR-9591:
------------------------------------
Another thought: I looked at some of the files and they are say 15MB in size.
That would mean that 500*15MB is 7.5 GB of heap just to load them into memory.
I would try the following and see if the error goes away:
* disable the autocommit for now
* index 50 files at once
* commit without opening a new searcher
* repeat index/commit without opening a new search
* after done indexing, commit with a new searcher so that results are visible.
My guess is you are hitting JVM garbage collection since it looks like Solr is
disconnecting from ZK. There are also errors in there about ZK fsyncing log:
fsync-ing the write ahead log in SyncThread:0 took 3445ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
> Shards and replicas go down when indexing large number of files
> ---------------------------------------------------------------
>
> Key: SOLR-9591
> URL: https://issues.apache.org/jira/browse/SOLR-9591
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
> Affects Versions: 5.5.2
> Reporter: Khalid Alharbi
> Attachments: solr_log_20161002_1504
>
>
> Solr shards and replicas go down when indexing a large number of text files
> using the default [extracting request
> handler|https://cwiki.apache.org/confluence/x/c4DxAQ].
> {code}
> curl
> 'http://localhost:8983/solr/myCollection/update/extract?literal.id=someId' -F
> "myfile=/data/file1.txt"
> {code}
> and committing after indexing 5,000 files using:
> {code}
> curl 'http://localhost:8983/solr/myCollection/update?commit=true&wt=json'
> {code}
> This was on Solr (SolrCloud) version 5.5.2 with an external zookeeper cluster
> of five nodes. I also tried this on a single node SolrCloud with the embedded
> ZooKeeper but the collection went down as well. In both cases the error
> message is always "ERROR null DistributedUpdateProcessor ClusterState says we
> are the leader, but locally we don't think so"
> I managed to come up with a work around that helped me index over 400K files
> without getting replicas down with that error message. The work around is to
> index 5K files, restart Solr, wait for shards and replicas to get active,
> then index the next 5K files, and repeat the previous steps.
> If this is not enough to investigate this issue, I will be happy to provide
> more details regarding this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
