It wasn’t clear to me from the OP whether the issue of *slowness* persisted after raising max heap to 2 GB, or whether the issue of *not being able to allocate this amount of space at all* persisted.

If Bram set max heap to more than 1.5 GB and immediately received the message “Could not reserve enough space …”, this could be an indication that he is running a 32-bit Java on his 64-bit machine.

Bram, what is the output of 'java -version'? Mine is:

java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

If it does not contain '64-Bit', it’s 32 bit, and you need to install a different binary from Oracle’s site.

Gerrit

On 16.08.2017 07:53, Kirsten, Dirk wrote:
Hi Bram

How did you set the java heap space? Usual way would be by using the command line and setting e.g.

   java -Xmx2g

However, the object which throws the error is already nearly 1,5GB in size, so if other stuff has to be allocates as well it could simply be that it is still not enough. Try increasing it to at least 4GB.

Indicating the memory leak: Well, it could certainly be, but I personally doubt it. First of all, having millions of databases is a lot, so some performance impact is expected. And given the scale of your databases I wouldn't call 600ms "terribly slow", you just seem to have a lot of data. When you create a new database there is also stuff which has to be checked (I would guess for example that BaseX checks that the database not already exists, which runtime obviously depends on the number of databases you already have), so I would guess the performance is somewhat expected. Of course, it might be possible to optimize this and try to make BaseX more performant for your given scenario.

Cheers
Dirk

**
Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - Amtsgericht Frankfurt am Main - Reg.-Nr.: HRB 105546 Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: Daniel Grözinger


-----Ursprüngliche Nachricht-----
Von: basex-talk-boun...@mailman.uni-konstanz.de [mailto:basex-talk-boun...@mailman.uni-konstanz.de] Im Auftrag von Bram Vanroy
Gesendet: Dienstag, 15. August 2017 22:22
An: basex-talk@mailman.uni-konstanz.de
Betreff: [basex-talk] Could not reserve enough space for object heap

Hi all

I'm running into an issue with many databases. I.e. one server instance with millions of databases. When creating all of these, I found that the more databases are included on the instance, the slower further database generation got. For instance, I could see in the logs that in the first < 10.000 databases the creation happened smoothly with around 50ms per file of 1-4kB. However, when having more and more databases for this server instance, things got very slow: for an XML file of 1-4kB the logs show ~600ms. This is terribly slow, as you can imagine.

At first I thought something was wrong with my hardware, but I checked on another system and the same issues arises. Then I thought maybe Java is doing something strange, so I figured I'd reboot and see if that cleared some stuff up. But now when I try to launch 'basex' or 'basexserver', I get the following message:

      Could not reserve enough space for 1433600KB object heap

I googled the issue, and it was suggested that I added a JAVA option to my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the memory it could use. I set that to 2048MB. But still the same issue persists.

I have contacted the list before, with issues of generating millions of database with the same server instance, and this seems another one related to the problem. I am no expert AT ALL, but isn't it possible there is some sort of micro memory leak that only becomes apparent when creating an amount of databases of this magnitude? If not, other ideas are welcome as well. At least on how to get rid of the Java error mentioned above.


Kind regards

Bram Vanroy


--
Gerrit Imsieke
Geschäftsführer / Managing Director
le-tex publishing services GmbH
Weissenfelser Str. 84, 04229 Leipzig, Germany
Phone +49 341 355356 110, Fax +49 341 355356 510
gerrit.imsi...@le-tex.de, http://www.le-tex.de

Registergericht / Commercial Register: Amtsgericht Leipzig
Registernummer / Registration Number: HRB 24930

Geschäftsführer: Gerrit Imsieke, Svea Jelonek,
Thomas Schmidt, Dr. Reinhard Vöckler

Reply via email to