Thanks Toke, now I am splitting file before indexing. Shalin, thanks for the details. Even this fixed in 5.5 and 6.0 is there any threshold value. Please suggest me which is best way to index(multithreaded) if your input format is text/csv (file).
Thanks, Prabaharan -----Original Message----- From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] Sent: 28 June 2016 16:06 To: solr-user@lucene.apache.org; Toke Eskildsen Subject: Re: SimplePostTool: FATAL: IOException while posting data: java.io.IOException: too many bytes written This was fixed in 5.5 and 6.0. You can upload files larger than 2GB with the simple post tool however I don't recommend it because it uses a single indexing thread. On Tue, Jun 28, 2016 at 3:55 PM, Toke Eskildsen <t...@statsbiblioteket.dk> wrote: > On Mon, 2016-06-27 at 13:24 +0000, Rajendran, Prabaharan wrote: > > I am trying to index a text file about 4.2 GB in size. [...] > > > > SimplePostTool: FATAL: IOException while posting data: > java.io.IOException: too many bytes written > > SimplePostTool uses > HttpUrlConnection.setFixedLengthStreamingMode(file_size) > where file_size is an integer. > > Unfortunately there is no check for overflow (which happens with files > > 2GB), so there is no sane error message up front and you only get > the error you pasted after some bytes has been sent. With a 4.2GB > input file, I would guess after about 200MB (4.2GB % 2GB). > > > Long story short: Keep your posts below 2GB. > > - Toke Eskildsen, State and University Library, Denmark > > > -- Regards, Shalin Shekhar Mangar.
