Great, good luck on your use case! On Thu, Feb 6, 2020 at 11:35 AM Justin Polchlopek <[email protected]> wrote:
> Great. That works. I should have checked something like this first. > Thanks. > > On Thu, Feb 6, 2020 at 1:34 PM Jacques Nadeau <[email protected]> wrote: > >> The jvm imposes a limit as well. The value you pass into the allocators >> only matters if it is below the jvm setting. You'll need to change the jvm >> setting via -XX:MaxDirectMemorySize=<size> >> >> https://www.eclipse.org/openj9/docs/xxmaxdirectmemorysize/ >> >> On Thu, Feb 6, 2020 at 9:56 AM Justin Polchlopek <[email protected]> >> wrote: >> >>> Hi, >>> >>> I've been using Arrow Vectors as the underlying storage of some objects >>> to assist in the transmission of those objects, but as I am scaling up my >>> examples, I'm very quickly running into OOM errors. Specifically >>> >>> Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to >>> allocate 16777216 byte(s) of direct memory (used: 939524096, max: 954728448) >>> >>> I'm a bit surprised. The allocator managing the vectors is given by >>> >>> val allocator = new RootAllocator(Long.MaxValue) >>> >>> (I'm using the Java libs via Scala.) I would expect to have more than 1GB >>> available given that construction, but it looks like we're substantially >>> memory-constrained. >>> >>> What can I do to increase the available pool of memory? >>> >>> Thanks >>> >>> -Justin >>> >>>
