Re: [basex-talk] Could not reserve enough space for object heap

[Christian Grün]

> I am running 64 bit, a x86-x64 JVM so I'd thought I could assign a larger
> amount of memory to it.

Hm. You could try to specify -d64 to ensure that the 64bit JVM is used.

> Why does it need that much memory to just launch? Is it
> trying to load some thing of all the existing databases in memory?

As the error is returned by Java, not by BaseX, it’s most probably
returnend before any databases are touched. Maybe it helps to consult
some other Java forum pages and compare the configuration with yours?

Hope this helps,
Christian



> Citeren Christian Grün :
>
>
>> Hi Bram,
>>
>> As was indicated before, the JVM is restricted to appr. 1.4 -1.5,
>> sometimes 1.6 GB on 32 bit Windows systems.
>>
>> Generally (and as you have impressively proven), it is indeed possible
>> to have millions of databases in a single BaseX instance. In practice,
>> it is recommendable to find a tradeoff between number of databases and
>> number of documents/XML nodes per database. If there is no obvious way
>> how to distribute your documents, an additional mapping database might
>> be reasonable in order to look up where a document has gone.
>>
>> Hope this helps,
>> Christian
>>
>>
>> On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy 
>> wrote:
>>>
>>> Hi all
>>>
>>> I'm running into an issue with many databases. I.e. one server instance
>>> with
>>> millions of databases. When creating all of these, I found that the more
>>> databases are included on the instance, the slower further database
>>> generation got. For instance, I could see in the logs that in the first <
>>> 10.000 databases the creation happened smoothly with around 50ms per file
>>> of
>>> 1-4kB. However, when having more and more databases for this server
>>> instance, things got very slow: for an XML file of 1-4kB the logs show
>>> ~600ms. This is terribly slow, as you can imagine.
>>>
>>> At first I thought something was wrong with my hardware, but I checked on
>>> another system and the same issues arises. Then I thought maybe Java is
>>> doing something strange, so I figured I'd reboot and see if that cleared
>>> some stuff up. But now when I try to launch 'basex' or 'basexserver', I
>>> get
>>> the following message:
>>>
>>> Could not reserve enough space for 1433600KB object heap
>>>
>>> I googled the issue, and it was suggested that I added a JAVA option to
>>> my
>>> system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the
>>> memory it could use. I set that to 2048MB. But still the same issue
>>> persists.
>>>
>>> I have contacted the list before, with issues of generating millions of
>>> database with the same server instance, and this seems another one
>>> related
>>> to the problem. I am no expert AT ALL, but isn't it possible there is
>>> some
>>> sort of micro memory leak that only becomes apparent when creating an
>>> amount
>>> of databases of this magnitude? If not, other ideas are welcome as well.
>>> At
>>> least on how to get rid of the Java error mentioned above.
>>>
>>>
>>> Kind regards
>>>
>>> Bram Vanroy
>>>
>
>
>

Re: [basex-talk] Could not reserve enough space for object heap

[Bram Vanroy]

Hi Christian

Thanks again for the quick reply.

I am running 64 bit, a x86-x64 JVM so I'd thought I could assign a  
larger amount of memory to it.


In any case, I am confused as to what the error message means in a  
BaseX context. I am simply trying to *start* the basex server, and I  
immediately get that error. Why does it need that much memory to just  
launch? Is it trying to load some thing of all the existing databases  
in memory?


Thanks

Bram


Citeren Christian Grün :


Hi Bram,

As was indicated before, the JVM is restricted to appr. 1.4 -1.5,
sometimes 1.6 GB on 32 bit Windows systems.

Generally (and as you have impressively proven), it is indeed possible
to have millions of databases in a single BaseX instance. In practice,
it is recommendable to find a tradeoff between number of databases and
number of documents/XML nodes per database. If there is no obvious way
how to distribute your documents, an additional mapping database might
be reasonable in order to look up where a document has gone.

Hope this helps,
Christian


On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy  wrote:

Hi all

I'm running into an issue with many databases. I.e. one server instance with
millions of databases. When creating all of these, I found that the more
databases are included on the instance, the slower further database
generation got. For instance, I could see in the logs that in the first <
10.000 databases the creation happened smoothly with around 50ms per file of
1-4kB. However, when having more and more databases for this server
instance, things got very slow: for an XML file of 1-4kB the logs show
~600ms. This is terribly slow, as you can imagine.

At first I thought something was wrong with my hardware, but I checked on
another system and the same issues arises. Then I thought maybe Java is
doing something strange, so I figured I'd reboot and see if that cleared
some stuff up. But now when I try to launch 'basex' or 'basexserver', I get
the following message:

Could not reserve enough space for 1433600KB object heap

I googled the issue, and it was suggested that I added a JAVA option to my
system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the
memory it could use. I set that to 2048MB. But still the same issue
persists.

I have contacted the list before, with issues of generating millions of
database with the same server instance, and this seems another one related
to the problem. I am no expert AT ALL, but isn't it possible there is some
sort of micro memory leak that only becomes apparent when creating an amount
of databases of this magnitude? If not, other ideas are welcome as well. At
least on how to get rid of the Java error mentioned above.


Kind regards

Bram Vanroy

Re: [basex-talk] Could not reserve enough space for object heap

[Christian Grün]

Hi Bram,

As was indicated before, the JVM is restricted to appr. 1.4 -1.5,
sometimes 1.6 GB on 32 bit Windows systems.

Generally (and as you have impressively proven), it is indeed possible
to have millions of databases in a single BaseX instance. In practice,
it is recommendable to find a tradeoff between number of databases and
number of documents/XML nodes per database. If there is no obvious way
how to distribute your documents, an additional mapping database might
be reasonable in order to look up where a document has gone.

Hope this helps,
Christian


On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy  wrote:
> Hi all
>
> I'm running into an issue with many databases. I.e. one server instance with
> millions of databases. When creating all of these, I found that the more
> databases are included on the instance, the slower further database
> generation got. For instance, I could see in the logs that in the first <
> 10.000 databases the creation happened smoothly with around 50ms per file of
> 1-4kB. However, when having more and more databases for this server
> instance, things got very slow: for an XML file of 1-4kB the logs show
> ~600ms. This is terribly slow, as you can imagine.
>
> At first I thought something was wrong with my hardware, but I checked on
> another system and the same issues arises. Then I thought maybe Java is
> doing something strange, so I figured I'd reboot and see if that cleared
> some stuff up. But now when I try to launch 'basex' or 'basexserver', I get
> the following message:
>
> Could not reserve enough space for 1433600KB object heap
>
> I googled the issue, and it was suggested that I added a JAVA option to my
> system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the
> memory it could use. I set that to 2048MB. But still the same issue
> persists.
>
> I have contacted the list before, with issues of generating millions of
> database with the same server instance, and this seems another one related
> to the problem. I am no expert AT ALL, but isn't it possible there is some
> sort of micro memory leak that only becomes apparent when creating an amount
> of databases of this magnitude? If not, other ideas are welcome as well. At
> least on how to get rid of the Java error mentioned above.
>
>
> Kind regards
>
> Bram Vanroy
>

Re: [basex-talk] Could not reserve enough space for object heap

[Kirsten, Dirk]

Indeed, now I googled this 
(https://stackoverflow.com/questions/1434779/maximum-java-heap-size-of-a-32-bit-jvm-on-a-64-bit-os)
 and learned that 32-bit windows is only able to allocate 1.4GB to 1.6GB in 
practice.
I was under the impression that at least 2GB should be usable. So if this is 
root cause of the problem it would of course be the best solution to switch to 
64bit or otherwise (and besides that...) you might want to use a proper OS 
which doesn't restrict your usable heap size so much ;)

Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - Amtsgericht 
Frankfurt am Main - Reg.-Nr.: HRB 105546
Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: Daniel 
Gr?zinger

-Urspr?ngliche Nachricht-
Von: basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] Im Auftrag von Imsieke, 
Gerrit, le-tex
Gesendet: Mittwoch, 16. August 2017 08:04
An: basex-talk@mailman.uni-konstanz.de
Betreff: Re: [basex-talk] Could not reserve enough space for object heap

It wasn't clear to me from the OP whether the issue of *slowness* persisted 
after raising max heap to 2 GB, or whether the issue of *not being able to 
allocate this amount of space at all* persisted.

If Bram set max heap to more than 1.5 GB and immediately received the message 
"Could not reserve enough space ...", this could be an indication that he is 
running a 32-bit Java on his 64-bit machine.

Bram, what is the output of 'java -version'? Mine is:

java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit 
Server VM (build 25.144-b01, mixed mode)

If it does not contain '64-Bit', it's 32 bit, and you need to install a 
different binary from Oracle's site.

Gerrit

On 16.08.2017 07:53, Kirsten, Dirk wrote:
> Hi Bram
>
> How did you set the java heap space? Usual way would be by using the
> command line and setting e.g.
>
>java -Xmx2g
>
> However, the object which throws the error is already nearly 1,5GB in
> size, so if other stuff has to be allocates as well it could simply be
> that it is still not enough. Try increasing it to at least 4GB.
>
> Indicating the memory leak: Well, it could certainly be, but I
> personally doubt it. First of all, having millions of databases is a
> lot, so some performance impact is expected. And given the scale of
> your databases I wouldn't call 600ms "terribly slow", you just seem to
> have a lot of data. When you create a new database there is also stuff
> which has to be checked (I would guess for example that BaseX checks
> that the database not already exists, which runtime obviously depends
> on the number of databases you already have), so I would guess the
> performance is somewhat expected. Of course, it might be possible to
> optimize this and try to make BaseX more performant for your given scenario.
>
> Cheers
> Dirk
>
> **
> Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - Amtsgericht
> Frankfurt am Main - Reg.-Nr.: HRB 105546
> Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender:
> Daniel Gr?zinger
>
>
> -Urspr?ngliche Nachricht-
> Von: basex-talk-boun...@mailman.uni-konstanz.de
> [mailto:basex-talk-boun...@mailman.uni-konstanz.de] Im Auftrag von
> Bram Vanroy
> Gesendet: Dienstag, 15. August 2017 22:22
> An: basex-talk@mailman.uni-konstanz.de
> Betreff: [basex-talk] Could not reserve enough space for object heap
>
> Hi all
>
> I'm running into an issue with many databases. I.e. one server
> instance with millions of databases. When creating all of these, I
> found that the more databases are included on the instance, the slower
> further database generation got. For instance, I could see in the logs
> that in the first < 10.000 databases the creation happened smoothly
> with around 50ms per file of 1-4kB. However, when having more and more
> databases for this server instance, things got very slow: for an XML
> file of 1-4kB the logs show ~600ms. This is terribly slow, as you can imagine.
>
> At first I thought something was wrong with my hardware, but I checked
> on another system and the same issues arises. Then I thought maybe
> Java is doing something strange, so I figured I'd reboot and see if
> that cleared some stuff up. But now when I try to launch 'basex' or
> 'basexserver', I get the following message:
>
>   Could not reserve enough space for 1433600KB object heap
>
> I googled the issue, and it was suggested that I added a JAVA option
> to my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4)
> indicating the memory it could use. I set that to 2048MB. But still
> the same issue persists.
>
> I have contacted the list before, with issues of generating millions
> of database with the same server instance, and this seems another one
> related to the problem. I am no expert AT ALL, but isn't it possible
> there is some sort of micro memory leak that only becomes apparent
> when creating an amount of databases of this magnitude? If 

Re: [basex-talk] Could not reserve enough space for object heap

[Imsieke, Gerrit, le-tex]

It wasn’t clear to me from the OP whether the issue of *slowness* 
persisted after raising max heap to 2 GB, or whether the issue of *not 
being able to allocate this amount of space at all* persisted.


If Bram set max heap to more than 1.5 GB and immediately received the 
message “Could not reserve enough space …”, this could be an indication 
that he is running a 32-bit Java on his 64-bit machine.


Bram, what is the output of 'java -version'? Mine is:

java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

If it does not contain '64-Bit', it’s 32 bit, and you need to install a 
different binary from Oracle’s site.


Gerrit

On 16.08.2017 07:53, Kirsten, Dirk wrote:

Hi Bram

How did you set the java heap space? Usual way would be by using the 
command line and setting e.g.


   java -Xmx2g

However, the object which throws the error is already nearly 1,5GB in 
size, so if other stuff has to be allocates as well it could simply be 
that it is still not enough. Try increasing it to at least 4GB.


Indicating the memory leak: Well, it could certainly be, but I 
personally doubt it. First of all, having millions of databases is a 
lot, so some performance impact is expected. And given the scale of your 
databases I wouldn't call 600ms "terribly slow", you just seem to have a 
lot of data. When you create a new database there is also stuff which 
has to be checked (I would guess for example that BaseX checks that the 
database not already exists, which runtime obviously depends on the 
number of databases you already have), so I would guess the performance 
is somewhat expected. Of course, it might be possible to optimize this 
and try to make BaseX more performant for your given scenario.


Cheers
Dirk

**
Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - 
Amtsgericht Frankfurt am Main - Reg.-Nr.: HRB 105546
Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: 
Daniel Grözinger



-Ursprüngliche Nachricht-
Von: basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] Im Auftrag von Bram 
Vanroy

Gesendet: Dienstag, 15. August 2017 22:22
An: basex-talk@mailman.uni-konstanz.de
Betreff: [basex-talk] Could not reserve enough space for object heap

Hi all

I'm running into an issue with many databases. I.e. one server instance 
with millions of databases. When creating all of these, I found that the 
more databases are included on the instance, the slower further database 
generation got. For instance, I could see in the logs that in the first 
< 10.000 databases the creation happened smoothly with around 50ms per 
file of 1-4kB. However, when having more and more databases for this 
server instance, things got very slow: for an XML file of 1-4kB the logs 
show ~600ms. This is terribly slow, as you can imagine.


At first I thought something was wrong with my hardware, but I checked 
on another system and the same issues arises. Then I thought maybe Java 
is doing something strange, so I figured I'd reboot and see if that 
cleared some stuff up. But now when I try to launch 'basex' or 
'basexserver', I get the following message:


  Could not reserve enough space for 1433600KB object heap

I googled the issue, and it was suggested that I added a JAVA option to 
my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating 
the memory it could use. I set that to 2048MB. But still the same issue 
persists.


I have contacted the list before, with issues of generating millions of 
database with the same server instance, and this seems another one 
related to the problem. I am no expert AT ALL, but isn't it possible 
there is some sort of micro memory leak that only becomes apparent when 
creating an amount of databases of this magnitude? If not, other ideas 
are welcome as well. At least on how to get rid of the Java error 
mentioned above.



Kind regards

Bram Vanroy



--
Gerrit Imsieke
Geschäftsführer / Managing Director
le-tex publishing services GmbH
Weissenfelser Str. 84, 04229 Leipzig, Germany
Phone +49 341 355356 110, Fax +49 341 355356 510
gerrit.imsi...@le-tex.de, http://www.le-tex.de

Registergericht / Commercial Register: Amtsgericht Leipzig
Registernummer / Registration Number: HRB 24930

Geschäftsführer: Gerrit Imsieke, Svea Jelonek,
Thomas Schmidt, Dr. Reinhard Vöckler