GitHub user vincent-grosbois opened a pull request:
https://github.com/apache/spark/pull/22024
[SPARK-25034][CORE] Remove allocations in onBlockFetchSuccess
This method is only transferring a ManagedBuffer to the caller,
so there is no reason why it should allocate 2 (!) intermediate data
buffers in order to do so.
In this commit I'm removing the conversion from any kind of managed buffer
besides FileSegment to a NioManagedBuffer.
However if you check the only calling method getRemoteBytes(), you will
see that here we either:
- do a memory-map if we have a FileSegmentManagedBuffer
- try again to call the nioByteBuffer() method otherwise
So in any case the conversion will occur later.
## What changes were proposed in this pull request?
Remove needless temporary allocations
## How was this patch tested?
Tested this change with a few jobs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vincent-grosbois/spark no-alloc-onfetchsuccess
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22024.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22024
----
commit 2c182b3c93c7cc70f042d7dcd82520ac2adece1c
Author: Vincent Grosbois <v.grosbois@...>
Date: 2018-08-07T12:34:32Z
[SPARK-25034][CORE] Remove allocations in onBlockFetchSuccess
This method is only transferring a ManagedBuffer to the caller,
so there is no reason why it should allocate 2 (!) intermediate data
buffers in order to do so.
In this commit I'm removing the conversion from any kind of managed buffer
besides FileSegment to a NioManagedBuffer.
However if you check the only calling method getRemoteBytes(), you will
see that here we either:
- do a memory-map if we have a FileSegmentManagedBuffer
- try again to call the nioByteBuffer() method otherwise
So in any case the conversion will occur later.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
