I can confirm that explicitly using a 64 bit JVM does not solve the issue I mentioned in my first email.
The problem persists; when working with a single server instance the creation of very small databases (~1-5kb) goes smoothly at the beginning but turns into a nightmare when reaching a certain threshold. Let's say around ~1M databases, the time to create subsequent databases increases dramatically. We're talking about a 10x duration increase. At first I thought I was doing something wrong, but for the life of me I can't figure out what! The java process doesn't utilize any more or less resources than it does when just starting, so that's not it either. Do you have any information that might help here? Does a server do anything to the databases when creating new ones? E.g. when one creates a new database, is a list of all databases re-indexed or something like that? This could explain the issue. But other inspiration is welcome as well, because I don't know where to look. Thanks in advance -----Oorspronkelijk bericht----- Van: Christian Grün [mailto:christian.gr...@gmail.com] Verzonden: woensdag 16 augustus 2017 11:31 Aan: Bram Vanroy <bram.van...@ugent.be> CC: BaseX <basex-talk@mailman.uni-konstanz.de> Onderwerp: Re: [basex-talk] Could not reserve enough space for object heap > I am running 64 bit, a x86-x64 JVM so I'd thought I could assign a > larger amount of memory to it. Hm. You could try to specify -d64 to ensure that the 64bit JVM is used. > Why does it need that much memory to just launch? Is it trying to load > some thing of all the existing databases in memory? As the error is returned by Java, not by BaseX, it’s most probably returnend before any databases are touched. Maybe it helps to consult some other Java forum pages and compare the configuration with yours? Hope this helps, Christian > Citeren Christian Grün <christian.gr...@gmail.com>: > > >> Hi Bram, >> >> As was indicated before, the JVM is restricted to appr. 1.4 -1.5, >> sometimes 1.6 GB on 32 bit Windows systems. >> >> Generally (and as you have impressively proven), it is indeed >> possible to have millions of databases in a single BaseX instance. In >> practice, it is recommendable to find a tradeoff between number of >> databases and number of documents/XML nodes per database. If there is >> no obvious way how to distribute your documents, an additional >> mapping database might be reasonable in order to look up where a document >> has gone. >> >> Hope this helps, >> Christian >> >> >> On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy <bram.van...@ugent.be> >> wrote: >>> >>> Hi all >>> >>> I'm running into an issue with many databases. I.e. one server >>> instance with millions of databases. When creating all of these, I >>> found that the more databases are included on the instance, the >>> slower further database generation got. For instance, I could see in >>> the logs that in the first < >>> 10.000 databases the creation happened smoothly with around 50ms per >>> file of 1-4kB. However, when having more and more databases for this >>> server instance, things got very slow: for an XML file of 1-4kB the >>> logs show ~600ms. This is terribly slow, as you can imagine. >>> >>> At first I thought something was wrong with my hardware, but I >>> checked on another system and the same issues arises. Then I thought >>> maybe Java is doing something strange, so I figured I'd reboot and >>> see if that cleared some stuff up. But now when I try to launch >>> 'basex' or 'basexserver', I get the following message: >>> >>> Could not reserve enough space for 1433600KB object heap >>> >>> I googled the issue, and it was suggested that I added a JAVA option >>> to my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) >>> indicating the memory it could use. I set that to 2048MB. But still >>> the same issue persists. >>> >>> I have contacted the list before, with issues of generating millions >>> of database with the same server instance, and this seems another >>> one related to the problem. I am no expert AT ALL, but isn't it >>> possible there is some sort of micro memory leak that only becomes >>> apparent when creating an amount of databases of this magnitude? If >>> not, other ideas are welcome as well. >>> At >>> least on how to get rid of the Java error mentioned above. >>> >>> >>> Kind regards >>> >>> Bram Vanroy >>> > > >
