Thanks Toke, now I am splitting file before indexing. 

Shalin, thanks for the details. Even this fixed in 5.5 and 6.0 is there any 
threshold value. 
Please suggest me which is best way to index(multithreaded) if your input 
format is text/csv (file).

Thanks,
Prabaharan

-----Original Message-----
From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] 
Sent: 28 June 2016 16:06
To: solr-user@lucene.apache.org; Toke Eskildsen
Subject: Re: SimplePostTool: FATAL: IOException while posting data: 
java.io.IOException: too many bytes written

This was fixed in 5.5 and 6.0. You can upload files larger than 2GB with the 
simple post tool however I don't recommend it because it uses a single indexing 
thread.

On Tue, Jun 28, 2016 at 3:55 PM, Toke Eskildsen <t...@statsbiblioteket.dk>
wrote:

> On Mon, 2016-06-27 at 13:24 +0000, Rajendran, Prabaharan wrote:
> > I am trying to index a text file about 4.2 GB in size. [...]
> >
> > SimplePostTool: FATAL: IOException while posting data:
> java.io.IOException: too many bytes written
>
> SimplePostTool uses
> HttpUrlConnection.setFixedLengthStreamingMode(file_size)
> where file_size is an integer.
>
> Unfortunately there is no check for overflow (which happens with files 
> > 2GB), so there is no sane error message up front and you only get 
> the error you pasted after some bytes has been sent. With a 4.2GB 
> input file, I would guess after about 200MB (4.2GB % 2GB).
>
>
> Long story short: Keep your posts below 2GB.
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
>


--
Regards,
Shalin Shekhar Mangar.

Reply via email to