[
https://issues.apache.org/jira/browse/ARROW-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16257257#comment-16257257
]
ASF GitHub Bot commented on ARROW-1795:
---------------------------------------
pcmoritz commented on issue #1327: ARROW-1795: [Plasma] Create flag to make
Plasma store use a single memory-mapped file.
URL: https://github.com/apache/arrow/pull/1327#issuecomment-345306499
Hey, this following instruction will make sure that the big mmap memory file
will be reused and no new one will be created:
https://github.com/apache/arrow/blob/cacbacd439919742a0b6fbec27ee73b5af29347f/cpp/src/plasma/store.cc#L825
If dlmemalign sees that there is not enough space and the mmap file is
already at system_memory, it will return a null pointer and then we call the
eviction policy to evict something (see the loop around dlmemalign).
If hypothetically dlfree was called on the last object in the store, the
mmap file might be unmapped and remapped when a new object is dlmemaligned;
this would only happen in corner cases (i.e. if there is only one or very few
large objects that are already released and they need to be evicted). We could
prevent it by mallocing a very small object and the beginning and not freeing
it. If you are concerned about this and can show it actually happens and want
to submit a PR please go ahead (in this case it makes sense to increase the
system_memory by the size of the small object), but it is not even clear that
dlmalloc would unmap the last memory mapped file. And even if it happens, the
behavior would still be correct...
Hope that helps!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> [Plasma C++] change evict policy
> --------------------------------
>
> Key: ARROW-1795
> URL: https://issues.apache.org/jira/browse/ARROW-1795
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Plasma (C++)
> Reporter: Lu Qi
> Assignee: Robert Nishihara
> Priority: Minor
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> case 1.say, we have total free memory 8 G , we have input 5G data, then comes
> another 6G data,
> if we choose to evict space 6G , it will throw exception saying that
> no object can be free. This is because we didn't count the 3G remaining free
> space .If we count this remaining 3G , we need to ask only 3G,thus
> we can evict the 5G data and we are still alive .
> case 2. another situation is : if we have free memory 10G , we input 1.5G
> data ,then comes another
> 9G data , if we use 10*20% = 2G data to evict ,then we will crash . In this
> situation we need to
> use 9+1.5-10 = 0.5G data to evict
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
