I know it’s not much help if you’re stuck with Solr 6.1 … but Solr 7.5 comes with an alternative strategy for SPLITSHARD that doesn’t consume as much memory and nearly doesn’t consume additional disk space on the leader. This strategy can be turned on by “splitMethod=link” parameter.
> On 4 Oct 2018, at 10:23, Atita Arora <atitaar...@gmail.com> wrote: > > Hi Edwin, > > Thanks for following up on this. > > So here are the configs : > > Memory - 30G - 20 G to Solr > Disk - 1TB > Index = ~ 500G > > and I think that it possibly is due to the reason why this could be > happening is that during split shard, the unsplit index + split index > persists on the instance and may be causing this. > I actually tried splitshard on another instance with index size 64G and it > went through without any issues. > > I would appreciate if you have additional information to enlighten me on > this issue. > > Thanks again. > > Regards, > > Atita > > On Thu, Oct 4, 2018 at 9:47 AM Zheng Lin Edwin Yeo <edwinye...@gmail.com> > wrote: > >> Hi Atita, >> >> What is the amount of memory that you have in your system? >> And what is your index size? >> >> Regards, >> Edwin >> >> On Tue, 25 Sep 2018 at 22:39, Atita Arora <atitaar...@gmail.com> wrote: >> >>> Hi, >>> >>> I am working on a test setup with Solr 6.1.0 cloud with 1 collection >>> sharded across 2 shards with no replication. When triggered a SPLITSHARD >>> command it throws "java.lang.OutOfMemoryError: Java heap space" >> everytime. >>> I tried this with multiple heap settings of 8, 12 & 20G but every time it >>> does create 2 sub-shards but then fails eventually. >>> I know the issue => https://jira.apache.org/jira/browse/SOLR-5214 has >> been >>> resolved but the trace looked very similar to this one. >>> Also just to ensure that I do not run into exceptions due to merge as >>> reported in this ticket, I also tried running optimize before proceeding >>> with splitting the shard. >>> I issued the following commands : >>> >>> 1. >>> >>> >> http://localhost:8983/solr/admin/collections?collection=testcollection&shard=shard1&action=SPLITSHARD >>> >>> This threw java.lang.OutOfMemoryError: Java heap space >>> >>> 2. >>> >>> >> http://localhost:8983/solr/admin/collections?collection=testcollection&shard=shard1&action=SPLITSHARD&async=1000 >>> >>> Then I ran with async=1000 and checked the status. Every time It's >> creating >>> the sub shards, but not splitting the index. >>> >>> Is there something that I am not doing correctly? >>> >>> Please guide. >>> >>> Thanks, >>> Atita >>> >> — Andrzej Białecki
