Let's back up quite a ways here. Where did the 20G file come from? Indexing files in JSON requires that they follow a very specific format, Solr doesn't index arbitrary JSON files.
With that out of the way, yes, 20G is unlikely to work without tweaking some parameters in both solrconfig.xml (there's an upload file limit there) and your communications layer is unlikely to accept uploads of that size. I'd really recomend breaking it up into smaller chunks and/or using SolrJ to parse the file with a sax-like parser and index docs in smaller chunks (I often use 1,000 docs at a time, but it depends on how big they are). Best, Erick On Tue, Sep 22, 2015 at 12:43 PM, Siddhartha Singh Sandhu <sandhus...@gmail.com> wrote: > Hi, > > I am relatively new to Solr and have usage query. I have a 20 GB JSON file > which I want to upload into my solr. Do I have to form smaller chunks or is > there a way to upload the whole thing in one go? > > I am getting the following error with bin/post: > > Entering auto mode. File endings considered are > xml,json,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log > POSTing file 20150401.json (application/json) to [base] > > stderr: Exception in thread "main" java.lang.IllegalArgumentException: > invalid content length > at > java.net.HttpURLConnection.setFixedLengthStreamingMode(HttpURLConnection.java:149) > at > org.apache.solr.util.SimplePostTool.postData(SimplePostTool.java:887) > at > org.apache.solr.util.SimplePostTool.postFile(SimplePostTool.java:794) > at > org.apache.solr.util.SimplePostTool.postFiles(SimplePostTool.java:515) > at > org.apache.solr.util.SimplePostTool.postFiles(SimplePostTool.java:435) > at > org.apache.solr.util.SimplePostTool.doFilesMode(SimplePostTool.java:311) > at > org.apache.solr.util.SimplePostTool.execute(SimplePostTool.java:177) > at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java:166) > > Thank you for your help! > > Regards, > > Sid.
