[ 
https://issues.apache.org/jira/browse/ARROW-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548761#comment-16548761
 ] 

Philipp Moritz commented on ARROW-2879:
---------------------------------------

Thanks for your message!

Can you try if you can use the whole memory if you use the '-f' flag when 
starting the plasma_store (see 
https://github.com/apache/arrow/blob/5063b333cf130d51a79f229d6b987c192773ab07/cpp/src/plasma/store.cc#L894)?

The reason why I think this is happening is because we allocate memory using a 
number of memory-mapped files, and double the size of the file each time we 
allocate a new one. So it could be that the next one to be allocated would put 
the memory requirements above the 1GB you allocated and therefore you can only 
use half of the specified memory.

The -f flag allocates the whole memory upfront into a single memory mapped file 
an should thus allow you to use the full memory.

If this is indeed the problem, we can try to fix it in the future by using 
[https://github.com/apache/arrow/pull/2205] once that's ready.

> [Python]Arrow plasma can only use a small part of specified shared memory
> -------------------------------------------------------------------------
>
>                 Key: ARROW-2879
>                 URL: https://issues.apache.org/jira/browse/ARROW-2879
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: chineking
>            Priority: Major
>
> Hi, thanks for the great job of arrow, it helps us a lot.
> However, we encounter a problem when we were using plasma.
> The sample code:
> {code:python}
> import numpy as np
> import pyarrow as pa
> import pyarrow.plasma as plasma
> client = plasma.connect("/tmp/plasma", "", 0)
> puts = []
> nbytes = 0
> while True:
>     a = np.ones((1000, 1000))
>     try:
>         oid = client.put(a)
>         puts.append(client.get(oid))
>         nbytes += a.nbytes
>     except pa.lib.PlasmaStoreFull:
>         print('use nbytes', nbytes)
>         break
> {code}
> We start a plasma store with 1G memory, but the nbytes output above is only 
> 496000000, which cannot even reach half of the memory we specified.
> I cannot figure out why plasma can only use such a small part of shared 
> memory. Could anybody help me? Thanks a lot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to