Hi Bram, As was indicated before, the JVM is restricted to appr. 1.4 -1.5, sometimes 1.6 GB on 32 bit Windows systems.
Generally (and as you have impressively proven), it is indeed possible to have millions of databases in a single BaseX instance. In practice, it is recommendable to find a tradeoff between number of databases and number of documents/XML nodes per database. If there is no obvious way how to distribute your documents, an additional mapping database might be reasonable in order to look up where a document has gone. Hope this helps, Christian On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy <bram.van...@ugent.be> wrote: > Hi all > > I'm running into an issue with many databases. I.e. one server instance with > millions of databases. When creating all of these, I found that the more > databases are included on the instance, the slower further database > generation got. For instance, I could see in the logs that in the first < > 10.000 databases the creation happened smoothly with around 50ms per file of > 1-4kB. However, when having more and more databases for this server > instance, things got very slow: for an XML file of 1-4kB the logs show > ~600ms. This is terribly slow, as you can imagine. > > At first I thought something was wrong with my hardware, but I checked on > another system and the same issues arises. Then I thought maybe Java is > doing something strange, so I figured I'd reboot and see if that cleared > some stuff up. But now when I try to launch 'basex' or 'basexserver', I get > the following message: > > Could not reserve enough space for 1433600KB object heap > > I googled the issue, and it was suggested that I added a JAVA option to my > system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the > memory it could use. I set that to 2048MB. But still the same issue > persists. > > I have contacted the list before, with issues of generating millions of > database with the same server instance, and this seems another one related > to the problem. I am no expert AT ALL, but isn't it possible there is some > sort of micro memory leak that only becomes apparent when creating an amount > of databases of this magnitude? If not, other ideas are welcome as well. At > least on how to get rid of the Java error mentioned above. > > > Kind regards > > Bram Vanroy >
