[jira] [Commented] (SOLR-12312) Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC
[ https://issues.apache.org/jira/browse/SOLR-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466326#comment-16466326 ] ASF subversion and git services commented on SOLR-12312: Commit 93f9cc71b127b84c432bf40e115fd4afe689bd8a in lucene-solr's branch refs/heads/branch_7x from [~dsmiley] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=93f9cc7 ] SOLR-12312: Replication's IndexFetcher buf size should be initialized to an amount no greater than the size of the file being transferred. (cherry picked from commit 81f6112) > Replication's IndexFetcher buf size can be initialized smarter to not waste > RAM/GC > -- > > Key: SOLR-12312 > URL: https://issues.apache.org/jira/browse/SOLR-12312 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: replication (java) >Reporter: David Smiley >Assignee: David Smiley >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > IndexFetcher's constructor knows the size of the file it's going to transfer. > As-such, it ought to initialize the "buf" field to no larger than this size. > This has been shown to waste Java heap/GC in an environment with lots of > cores of small indexes and thus small files. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12312) Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC
[ https://issues.apache.org/jira/browse/SOLR-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466321#comment-16466321 ] ASF subversion and git services commented on SOLR-12312: Commit 81f611209c9b71b7538a46d047631ea164dc2a2c in lucene-solr's branch refs/heads/master from [~dsmiley] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=81f6112 ] SOLR-12312: Replication's IndexFetcher buf size should be initialized to an amount no greater than the size of the file being transferred. > Replication's IndexFetcher buf size can be initialized smarter to not waste > RAM/GC > -- > > Key: SOLR-12312 > URL: https://issues.apache.org/jira/browse/SOLR-12312 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: replication (java) >Reporter: David Smiley >Assignee: David Smiley >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > IndexFetcher's constructor knows the size of the file it's going to transfer. > As-such, it ought to initialize the "buf" field to no larger than this size. > This has been shown to waste Java heap/GC in an environment with lots of > cores of small indexes and thus small files. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12312) Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC
[
https://issues.apache.org/jira/browse/SOLR-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464394#comment-16464394
]
David Smiley commented on SOLR-12312:
-
While I look at this a bit more in my IDE, I wonder if we even need this buf at
all. Notice that {{fetchPackets}} accepts {{FastInputStream fis}} and that's
what we read from. FIS contains an accessible {{byte[]}} internally; it's
partially why FIS exists in the first place (vs some generic InputStream).
fetchPackets could be recoded to have a loop within the packet reading to use
FIS's buffer directly. That would not be an abuse; we're just sending data
along to other places (to a file and some checksum calculator thingy). The net
effect is one less buffer to allocate and copy data to.
That said, I so easily become guilty of scope creep and this is extra work with
more intricacies than the fairly simple change we have on sizing the buf
correctly.
> Replication's IndexFetcher buf size can be initialized smarter to not waste
> RAM/GC
> --
>
> Key: SOLR-12312
> URL: https://issues.apache.org/jira/browse/SOLR-12312
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: replication (java)
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> IndexFetcher's constructor knows the size of the file it's going to transfer.
> As-such, it ought to initialize the "buf" field to no larger than this size.
> This has been shown to waste Java heap/GC in an environment with lots of
> cores of small indexes and thus small files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12312) Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC
[ https://issues.apache.org/jira/browse/SOLR-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464372#comment-16464372 ] David Smiley commented on SOLR-12312: - Latest patch looks good [~millerjeff0]; I like how you identified the re-use of an existing constant for this that the writer is using. I'll probably reword a comment since I can be obsessive about trivialities. I'll commit "soon". > Replication's IndexFetcher buf size can be initialized smarter to not waste > RAM/GC > -- > > Key: SOLR-12312 > URL: https://issues.apache.org/jira/browse/SOLR-12312 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: replication (java) >Reporter: David Smiley >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > IndexFetcher's constructor knows the size of the file it's going to transfer. > As-such, it ought to initialize the "buf" field to no larger than this size. > This has been shown to waste Java heap/GC in an environment with lots of > cores of small indexes and thus small files. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
