Re: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Manan Sheth
anuary 10, 2017 10:51 AM To: solr-user Subject: Re: Help needed in breaking large index file into smaller ones Hi Erick, Its due to some past issues observed with Joins on Solr 4, which got OOM on joining to large indexes after optimization/compaction, if those are stored as smaller files those ge

Re: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Manan Sheth
r-user Subject: Re: Help needed in breaking large index file into smaller ones Why do you have a requirement that the indexes be < 4G? If it's arbitrarily imposed why bother? Or is it a non-negotiable requirement imposed by the platform you're on? Because just splitting the files into a sm

Re: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Erick Erickson
Why do you have a requirement that the indexes be < 4G? If it's arbitrarily imposed why bother? Or is it a non-negotiable requirement imposed by the platform you're on? Because just splitting the files into a smaller set won't help you if you then start to index into it, the merge process will

Re: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Anshum Gupta
Can you provide more information about: - Are you using Solr in standalone or SolrCloud mode? What version of Solr? - Why do you want this? Lack of disk space? Uneven distribution of data on shards? - Do you want this data together i.e. as part of a single collection? You can check out the

Re: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Mikhail Khludnev
Perhaps you can copy this index into a separate location. Remove odd and even docs into former and later indexes consequently, and then force merge to single segment in both locations separately. Perhaps shard splitting in SolrCloud does something like that. On Mon, Jan 9, 2017 at 1:12 PM,

RE: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Moenieb Davids
mpetus.co.in> > wrote: > >> Is this really works for lucene index files? >> >> Thanks, >> Manan Sheth >> >> From: Moenieb Davids <moenieb.dav...@gpaa.gov.za> >> Sent: Monday, January 9, 2017 7:36

Re: Help needed in breaking large index file into smaller ones

2017-01-09 Thread billnbell
this really works for lucene index files? >> >> Thanks, >> Manan Sheth >> >> From: Moenieb Davids <moenieb.dav...@gpaa.gov.za> >> Sent: Monday, January 9, 2017 7:36 PM >> To: solr-user@lucene.apache.org

Re: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Yago Riveiro
___ > > From: Moenieb Davids <moenieb.dav...@gpaa.gov.za > > Sent: Monday, January 9, 2017 7:36 PM > > To: solr-user@lucene.apache.org > > Subject: RE: Help needed in breaking large index file into smaller ones > > > > Hi, > >

Re: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Narsimha Reddy CHALLA
, > Manan Sheth > > From: Moenieb Davids <moenieb.dav...@gpaa.gov.za> > Sent: Monday, January 9, 2017 7:36 PM > To: solr-user@lucene.apache.org > Subject: RE: Help needed in breaking large index file into smaller ones > > Hi, &g

Re: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Manan Sheth
Is this really works for lucene index files? Thanks, Manan Sheth From: Moenieb Davids <moenieb.dav...@gpaa.gov.za> Sent: Monday, January 9, 2017 7:36 PM To: solr-user@lucene.apache.org Subject: RE: Help needed in breaking large index file into smalle

RE: Help needed in breaking large index file into smaller ones

2017-01-09 Thread Moenieb Davids
Hi, Try split on linux or unix split -l 100 originalfile.csv this will split a file into 100 lines each see other options for how to split like size -Original Message- From: Narsimha Reddy CHALLA [mailto:chnredd...@gmail.com] Sent: 09 January 2017 12:12 PM To: