Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/43#discussion_r10188851
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager,
diskManager: DiskBlockManage
override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
val segment = diskManager.getBlockLocation(blockId)
val channel = new RandomAccessFile(segment.file, "r").getChannel()
- val buffer = try {
- channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
- } finally {
- channel.close()
+
+ val buffer =
+ // For small files, directly read rather than memory map
+ if (segment.length < 2 * 4096) {
--- End diff --
We need to limit the number of random configuration parameters - just
because something could possible have some minor performance gain if it's
tweaked does not mean it should be exposed.
This just needs to be set to equal a small number of page sizes. Since
memory maps are aligned on page boundaries, then on average you expect to have
1/2 of a page of extra memory use. If the block size is 2 pages or less, then
this overhead is really significant. Sure, maybe if we made it 3 or 5 instead
of 2, but the range of useful values here is small and I don't think any user
would ever want to tune this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
