You have to specify the size for directmemory with the jvm option:
-XX:MaxDirectMemorySize=4096M
the default is taking the value of -Xmx. Hopefully heap should be small
(only the keys are in heap) and direct memory should be large.
I'll point it out to an example using a different serializer.
Ciao,
R
On Thu, Nov 8, 2012 at 9:21 AM, Jan Kotek <[email protected]> wrote:
> Hi,
>
>
> > 30 buffers.
> JVM kept crashing if I increased memory too much. This was only solution
> that gave me lot of memory and did not crash JVM. Not sure why
>
> > 0.2 snapshot
> You have outdated documentation then. It shows 0.1 (and fragment names are
> wrong BTW).
>
> http://directmemory.apache.**org/simple-usage.html<http://directmemory.apache.org/simple-usage.html>
>
> > Serialization
> How can I use better serialization?
>
> > JVM settings
> I left all at default, except increasing max heap (-Xmx4G for DM, -Xmx13G
> for TreeMap).
> Are there better settings I should use?
>
> Jan
>
>
>
> On 08/11/12 06:44, Raffaele P. Guidi wrote:
>
>> Looking at the code I see Jan implemented against v0.1 which is a lot
>> outdated. I would strongly suggest Jan to use 0.2-SNAPSHOT (yeah, I know
>> snapshots suck...) and us to work on a new release (it's about time)
>> focusing a bit more on performance instead of features. What do you guys
>> think?
>>
>> Ciao,
>> R
>>
>>
>> On Thu, Nov 8, 2012 at 6:16 AM, Raffaele P. Guidi <
>> [email protected]> wrote:
>>
>> Oh, sorry I didn't notice the github link (it's 5 'o clock in the morning
>>> here :-D ). Also why thirty 1mb buffers?
>>> Il giorno 08/nov/2012 06:11, "Christoph Engelbert" <[email protected]
>>> >
>>> ha scritto:
>>>
>>> Unsafe allocator alone won't speed it up that much because I guess he
>>>
>>>> relied on standard Serialization for his test.
>>>>
>>>>
>>>>
>>>> "Raffaele P. Guidi" <[email protected]> schrieb:
>>>>
>>>> I don't believe integers or even 1kb buffers are the right case for
>>>>> directmemory. Small keys, large payloads (+4kb) are. In any case: how
>>>>> were
>>>>> the heap and xxmaxdirectmemory settings? And did you try the Unsafe
>>>>> storage
>>>>> as well? It would be helpful if you could share the benchmark code.
>>>>>
>>>>> Thanks,
>>>>> R
>>>>> Il giorno 08/nov/2012 02:49, "Jan Kotek" <[email protected]> ha
>>>>> scritto:
>>>>>
>>>>> Hi,
>>>>>>
>>>>>> MapDB now has DirectMemory storage.
>>>>>> I made little test to compare our two libraries + java.util.TreeMap
>>>>>>
>>>>>> All tests are on 64bit Linux, 16GB RAM, JDK6. Default JVM settings.
>>>>>> Source code is here: https://github.com/jankotek/**
>>>>>>
>>>>>> mapdb-junk/tree/master/src/****main/java/****directMemoryBenchmark<
>>>>>
>>>> https://github.com/jankotek/**mapdb-junk/tree/master/src/**main/java/**
>>>> directMemoryBenchmark<https://github.com/jankotek/mapdb-junk/tree/master/src/main/java/directMemoryBenchmark>
>>>>
>>>>>
>>>>>> First is performance test. How long does it take to insert and fetch
>>>>>>
>>>>> 1
>>>>>
>>>>>> million records, each 1KB large.
>>>>>> - DM - 6901 ms
>>>>>> - MapDB - 2045 ms
>>>>>> - java.util.TreeMap - 940 ms
>>>>>>
>>>>>> Second is overhead test. It inserts pair of integers in cycle, until
>>>>>>
>>>>> JVM
>>>>>
>>>>>> runs out of memory.
>>>>>> It should measure how much memory library each consumes for its own
>>>>>> structures.
>>>>>> - DM - inserts 9 000 000 records and then it becomes very slow.
>>>>>>
>>>>> Profiler
>>>>>
>>>>>> shows high GC activity.
>>>>>> - MapDB - works about 10 minutes and than it runs out of memory with
>>>>>>
>>>>> 217
>>>>>
>>>>>> 000 000 records.
>>>>>> - java.util.TreeMap - crashes after 99 000 000 records , it consumed
>>>>>> about 12GB heap.
>>>>>>
>>>>>>
>>>>>> I am not sure I configured DM correctly, JVM kept crashing when I
>>>>>> increased size too much.
>>>>>> I ended with this setting:
>>>>>>
>>>>>> CacheService<Object, Object> m = new DirectMemory<Object,
>>>>>>
>>>>> Object>()
>>>>>
>>>>>> .setNumberOfBuffers( 30 )
>>>>>> .setSize((int) 1e8)
>>>>>> .setInitialCapacity( 10000 )
>>>>>> .setConcurrencyLevel( 4 )
>>>>>> .setDisposalTime(1000000)
>>>>>> .newCacheService();
>>>>>>
>>>>>>
>>>>>> Hope this helps.
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>>
>>>>>> --
>>>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>>>> gesendet.
>>>>
>>>
>>>
>
