[
https://issues.apache.org/jira/browse/IO-468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311334#comment-14311334
]
Bernd Hopp commented on IO-468:
-------------------------------
this is my software setup:
mvn -version
Apache Maven 3.2.2 (NON-CANONICAL_2014-06-19T11:19:24_mockbuild;
2014-06-19T11:19:24+00:00)
Maven home: /usr/share/maven
Java version: 1.8.0_31, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.31-3.b13.fc21.x86_64/jre
Default locale: de_DE, platform encoding: UTF-8
OS name: "linux", version: "3.18.3-201.fc21.x86_64", arch: "amd64", family:
"unix"
The PerfTest.java runnable does not include setup costs, randomizing and
initialization are done in the constructor and not included in time measuring.
now, could anybody please show me one testcase where performance or memory
consumption is repeatedly worse with threadbuffers? So I know what we're
talking about? Excuse my french, but I begin to wonder wether we're in a
cock-block situation. Are we? And if we are, can we get out of it please, since
we're in the same team?
The only scenario where memory consumption could theoretically go up, would be
where a thread would call copy() and then live on for a long time. In this
case, unfreeable memory would be the 4096 times the current count of threads
that called copy() at least once. Allright, so we are talking about a system
where memory saving is so crucial that, say, a hundred kb make a difference.
But that system at the same time uses an extraordinary big number of threads
that all take a very long time to finish or are being used as part of a thread
pool. Now tell me, what plausible real-life scenario are we talking about?
Servers use thread-pools, but the number of threads is typically small and
memory is not THAT crucial.
Also I would argue that if memory is that limited on your system and you
allocate a new byte buffer every time you call copy, your garbage collector
will propably run quite frequently.
Sebb, you argue that users are free to implement TL on theyr own and said it
would be an easy workaround (is it really?). But then again, why would users
use commons-io in the first place? If you have to do the performance
optimization on your own, why use the library to begin with?
How about an overloading for the copy method, a boolean called useThreadLocals,
that defaults to true and gives users total control without asking them to
implement performance optimizations on theyr own? This way, the vast majority
gets additional performance and the people who really need it can save a few kb
( although I remain sceptical to the existence of these people ).
> Avoid allocating memory for method internal buffers, use threadlocal memory
> instead
> -----------------------------------------------------------------------------------
>
> Key: IO-468
> URL: https://issues.apache.org/jira/browse/IO-468
> Project: Commons IO
> Issue Type: Improvement
> Components: Utilities
> Affects Versions: 2.4
> Environment: all environments
> Reporter: Bernd Hopp
> Priority: Minor
> Labels: newbie, performance
> Fix For: 2.5
>
> Attachments: PerfTest.java, monitoring_with_threadlocals.png,
> monitoring_without_threadlocals.png, performancetest.ods
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> In a lot of places, we allocate new buffers dynamically via new byte[]. This
> is a performance drawback since many of these allocations could be avoided if
> we would use threadlocal buffers that can be reused. For example, consider
> the following code from IOUtils.java, ln 2177:
> return copyLarge(input, output, inputOffset, length, new
> byte[DEFAULT_BUFFER_SIZE]);
> This code allocates new memory for every copy-process, that is not used
> outside of the method and could easily and safely reused, as long as is is
> thread-local. So instead of allocating new memory, a new utility-class could
> provide a thread-local bytearray like this:
> byte[] buffer = ThreadLocalByteArray.ofSize(DEFAULT_BUFFER_SIZE);
> return copyLarge(input, output, inputOffset, length, buffer);
> I have not measured the performance-benefits yet, but I would expect them to
> be significant, especially when the streams itself are not the performance
> bottleneck.
> Git PR is at https://github.com/apache/commons-io/pull/6/files
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
