[
https://issues.apache.org/jira/browse/SOLR-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241608#comment-15241608
]
ASF subversion and git services commented on SOLR-8937:
-------------------------------------------------------
Commit 037a40316c897034c060041730ecefff4eca816a in lucene-solr's branch
refs/heads/master from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=037a403 ]
SOLR-8937: bin/post (SimplePostTool) should tell JDK to stream stdin instead of
fully buffer
> bin/post (SimplePostTool) should stream/chunk if size not known
> ---------------------------------------------------------------
>
> Key: SOLR-8937
> URL: https://issues.apache.org/jira/browse/SOLR-8937
> Project: Solr
> Issue Type: Improvement
> Components: SimplePostTool
> Reporter: David Smiley
> Assignee: David Smiley
> Attachments: SOLR_8937_post_streaming.patch
>
>
> {{SimplePostTool.postData()}} takes a {{Long length}} arg that is sometimes
> null -- for stdin and for processing some web pages for links. If it's null,
> we should this on the underlying HttpURLConnection:
> {{urlc.setChunkedStreamingMode(-1);}}. I have found that I can't use
> bin/post to stream large files from stdin since it the JDK's implementation
> wants to put the whole thing in memory first. Sadly this is true of {{curl}}
> as well. When I set chunk mode, it works. (FYI -1 tells it to use its
> default chunk size -- 4k).
> FYI I'm using stdin because I've got massive JSON files zipped up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
