Vadim,
how do you run the LevelDB journal, with native = on or off? Please note
that native = off (LevelDB Java port) is only for testing purposes and
doesn't do data compression. With the native LevelDB journal I end up
with 40M disk space after 1M messages and 200M and 5M messages (whereas
the LevelDB Java port uses much more disk space, about factor 5).
more inline ...
On 10.01.14 17:37, Vadim Bobrov wrote:
Martin,
I understand. But here is what I am trying to achieve - I have an app
that processes potentially large amounts of incoming data (hundreds of
thousands of messages per sec). I want to allow the app to crash
anytime and not take any data with it (write-ahead log). Given the
amount of data I cannot afford keeping the entire history in leveldb,
so I was thinking of taking snapshots periodically as they are there
specifically for that purpose and then truncate all messages prior to
the snapshot. How do I do it? It is only after successful snapshot
saving that I can delete messages prior to it
Yes, after the processor received a SaveSnapshotSuccess(metadata)
message, you can delete all messages up to metadata.sequenceNr (using
deleteMessages(metadata.sequenceNr)).
I still find it suspicious that the database shrinks when messages are
deleted one-by-one and doesn't shrink when they are deleted en masse
(with the same frequency). How can it possibly be explained?
I meanwhile improved the deleteMessages impl, so that it doesn't
redundantly write (LevelDB internal) deletion markers. The additional
overhead of a deleteMessages call is now a seek operation before a
range-delete (as done by deleteMessages), resulting in a strongly
improved overall performance for your use case. This seek overhead is
minimal after compactions. With this new implementation I now also see a
shrink of LevelDB. I'll include that improvement into 2.3-RC1.
Thanks
Vadim
--
Martin Krasser
blog: http://krasserm.blogspot.com
code: http://github.com/krasserm
twitter: http://twitter.com/mrt1nz
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://akka.io/faq/
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.
