I also tried your snippet, but I get:
Q{java:java.lang.System}gc()
Stopped at , 1/29:
Unknown command: Q{java:java.lang.System}gc(). Try HELP.
On 04.11.2017 19:02, Christian Grün wrote:
Fine. One more question: How do you measure the "memory leak" on
command-line, and are you sure that this value is comparable to the
value that is shown in the bottom bar of the BaseX GUI?
Am 04.11.2017 5:58 nachm. schrieb "Dinu Marina" <dinumar...@gmail.com
<mailto:dinumar...@gmail.com>>:
Indeed, I use the create function from the GUI, I just assumed
it's the same 2 separate operations.
Indeed, with CREATE DB it doesn't get out of memory at 1G. And it
also gets GC'ed and returned to system afterwards with no
additional intervention, after CREATE DB memory shrinks
immediately back to ~30M.
So confirmed, huge memory usage and memory "leak" (or whatever it
is) is linked to ADD only.
Thanks,
Dinu
On 04.11.2017 18 <tel:04.11.2017%2018>:46, Christian Grün wrote:
Hi Dinu,
yes, I have downloaded the file.
Just one more question:
2) using basexclient:
CHECK somedb
ADD /path/to/1_feed.zip
If you use the GUI, do you really add your zip file to an existing
database, or do you specify it as initial input when creating
a new
database? The latter option is definitely more efficient, and the
command-line equivalent would be
CREATE DB somedb /path/to/1_feed.zip
For adding resources to existing databases, enabling ADDCACHE
can help [1].
Cheers,
Christian
[1] http://docs.basex.org/wiki/Options#ADDCACHE
<http://docs.basex.org/wiki/Options#ADDCACHE>
Result:
Out of Main Memory.
To reproduce 2), start server with -Xmx2048m, repeat
operations, then drop
db, close client, check server memory usage.
Thanks,
Dinu
On 04.11.2017 18 <tel:04.11.2017%2018>:18, Christian Grün
wrote:
The fact is, the GUI runs with no problem with
-Xmx512M to do the same
thing, while basexclient fails without -Xmx2048M.
That’s surprising indeed – mostly because I would have
expected the
BaseX client to always consume a small and constant
amount of memory
(the BaseX server instance should be the process to
consume all the
memory). I did some quick tests with large zipped
input, but I failed
to reproduce the behavior you described. Feel free to
provide me with
a step-by-step guide.
I will try that, thanks, but shouldn't this be the
case automatically?
Since
I assume BaseX does free references to data
structures, at least to a
dropped DB?
Absolutely. Anything that’s reproducible is welcome.
On 04.11.2017 18 <tel:04.11.2017%2018>:00,
Christian Grün wrote:
Hi Dinu,
Question 1:
Memory consumption of the BaseX GUI is similar
as on command-line, but
it may be due to garbage collection that some
memory will be freed.
How do you add documents outside the GUI?
Question 2:
If a certain amount of memory is reserved by
Java’s virtual machine,
it may still be used by other applications on
your system (provided
that the memory can be freed by garbage
collection). You can enforce
some GC calls by running the following XQuery
expression (this should
only be done for testing purposes):
(1 to 5) ! Q{java:java.lang.System}gc()
Best,
Christian
After the data is extracted, it's no
longer needed and I DROP the DB;
also
connection is closed. But memory (the huge
2G mentioned above) is never
returned to the system.
The script I use to run BaseX is:
export BASEX_JVM="-Xmx2048m
-XX:MinHeapFreeRatio=10
-XX:MaxHeapFreeRatio=20
-XX:+UseSerialGC -Dorg.basex.LOG=false
-Dorg.basex.DBPATH=/var/basex/data
-Dorg.basex.REPOPATH=/var/basex/repo"
BaseX/bin/basexserver -S
So basically I tried specifying
MaxHeapFreeRatio and SerialGC for java,
but
it's no improvement and it doesn't help so
I assume the memory isn't
hogged
in java... is there a way to free up the
memory once operations
complete
(like mentioned above, "complete" means
created DB is dropped,
connection
closed, waiting for another batch to start
over).
Thanks,
Dinu