Hi Tim,
Thanks for testing the new Store Module.
The fluctuations in memory consumption and the long lags mainly
indicate that the garbage collector is pretty busy. Some data
structures will be much smaller if serialized to disk, so I assume
your amount of data is simply too large to be efficiently organized in
main memory. If you need all data in practce, you could think about
only preserving meta data and index structures in memory.
Feel free to provide us with test cases if you feel that the behavior
is suspicious enough.
Cheers,
Christian
On Tue, Nov 8, 2022 at 5:53 AM Tim Thompson wrote:
>
> Hello,
>
> I've been experimenting with the new Store module, which seems very handy. I
> have a custom store that is about 2GB in size. In the GUI (BaseX 10.3 beta),
> when I try to load it into memory with store:read(), the reported memory
> usage appears to max out (with 32GB of available memory):
>
> Total Memory: 32 GB
> Reserved Memory: 32 GB
> Used Memory: 32 GB
>
> However, monitoring memory usage with htop shows a smaller increase in memory
> usage: from 5GB at baseline, it peaks around 15GB, then goes down to 9GB, all
> while the BaseX GUI appears to be frozen.
>
> After a couple of minutes, the store does load, but GUI performance seems to
> be degraded, and it continues to report max memory usage.
>
> Any thoughts?
>
> Tim
>
>
> --
> Tim A. Thompson (he, him)
> Librarian for Applied Metadata Research
> Yale University Library
> www.linkedin.com/in/timathompson
>
