>>>> DirectMemory could make good use of mapdb to serialize
least frequently used items to disk
>> The only problem may be that MapDB currently does not support concurrent
transactions (it has only one single global transaction). Not sure if it
could be a problem. However it implements ConcurrentMap, so it is possible
to swap items atomically
We should investigate better on the concurrency theme - thing is that we
have a background thread that does cache eviction and, should we overflow
to disk through jdbm this thread should be the only "writer" - and it would
work perfectly. Of course items would be already serialized (as they are
already off-heap), Is this is allowed by MapDB
>>>>> We could merge our serialization efforts [...]
>> My only condition is that lighting is distributed in separate JAR. I
like minimal dependencies.
Indeed lighting has a separate distribution (and maven artifacts).
>> I would prefer MapDB to stay on GitHub [...] JDBM3 (previous version)
nearly become ApacheDS subproject, but on last moment I decided otherwise.
We would greatly appreciate your contribution in any case and would be
happy to contribute back as we can :)
Ciao,
R
On Wed, Nov 7, 2012 at 9:49 AM, Jan Kotek <[email protected]> wrote:
> Hi,
>
> 1. DirectMemory could make good use of mapdb to serialize least
>>
>> frequently used items to disk and free memory
>> 2. DirectMemory could implement a MapDB disk based store in addition
>> to
>>
>> the bytebuffer and unsafe ones
>>
> The only problem may be that MapDB currently does not support concurrent
> transactions (it has only one single global transaction).
> Not sure if it could be a problem.
>
> However it implements ConcurrentMap, so it is possible to swap items
> atomically
>
> 3. MapDB could take advantage of DM's componentization approach to
>>
>> support multiple serializers (we believe each one has its advantages
>> in
>> different scenarios)
>>
> MapDB already supports alternative serializers. User can supply their own
> on Map (similar to table) creation.
> I would love to integrate stuff from lightning serializer.
>
> 4. MapDB could use DM to write items to an off-heap before writing to
>>
>> disk (asynchronously) to improve speed
>>
> Not sure it would be practical. MapDB already uses memory mapped files so
> effect would be very similar. My tests shows that there is only 50%
> performance difference between inMemory store and onDisk store.
>
> Currently MapDB has only heap based inMemory store. But implementing off
> heap memory store is trivial and I will do it soon.
>
> 5. We could merge our serialization efforts (I believe lightning is
>> very
>>
>> fast and worth to be considered) and provide an even better solution
>> or two
>> alternative implementations
>>
>
> 100% agree. I will check lightning sources and see if I could contribute
> my stuff. MapDB serialization is very space-efficiency oriented and it can
> contribute a lot.
>
> My only condition is that lighting is distributed in separate JAR. I like
> minimal dependencies.
>
>
> In both cases we would be open to contribution in different forms - just
>> contributing patches or with you to join us and the ASF as module or
>> subproject (the latter options have to undergo a formal vote by all
>> project
>> members, of course) as I strongly believe that merging efforts would bring
>> to a better and more complete product.
>>
> I would prefer MapDB to stay on GitHub. I find it more comfortable to
> use.
> JDBM3 (previous version) nearly become ApacheDS subproject, but on last
> moment I decided otherwise.
>
>
> Jan
>
