I have done something quite similar, only that the structure consumed
hundred of GB's of RAM and was the main bottleneck of the application.
Attempting it on the jvm (either java or clojure) resulted in a lot of
wasted bits per bits and GC hangs, so I tried to use ByteBuffers (Direct,
Mapped,
Vertigo might be worth a look?
https://github.com/ztellman/vertigo
On Thursday, June 26, 2014 9:04:34 PM UTC-7, Vjeran Marcinko wrote:
Hi,
I am planning to play with implementing some giant in-memory index that is
basically tree-like structure containing counters on certain tree nodes,
When you get to very large size the efficiency of your data
representation is more likely to be the defining factor that the data
representation of the language that you are using. So, I don't think
that there is a specific answer, other than general advice for
optimisation. Try it first, and
If this is going to be the main performance bottleneck in your application,
I would probably do it in Java (possibly implementing relevant Clojure
interfaces to make it easy to use from Clojure). Ultimately, this gives you
more control over the memory representation and the ability to do
Hi,
I am planning to play with implementing some giant in-memory index that is
basically tree-like structure containing counters on certain tree nodes,
and can aggregate billion data points and will probably consume tens of GBs
of RAM.
Since space (memory)-efficiency is crucial here, I was
