[basex-talk] quick and dirty guide to increasing memory capacity?
I’m working on some XQuery code for language corpora and testing a function that reads a corpus and gathers information about its contents. The function works fine on the one-million word Brown Corpus, but it’s failing with an out-of-memory error on the five-million-word Hamburg Dependency Treebank (the XML for which is somewhat more verbose than Brown’s). Before I spend a lot of time trying to rewrite the code to reduce memory usage: Is there a quick way to give the BaseX GUI interface more memory? Once the code is more complete, it will make sense to try to improve its speed and lower its resource consumption (and at that point I’ll surely have lots of questions for the list), but for the moment I would prefer to postpone such considerations, and focus on writing code I understand. Thanks! C. M. Sperberg-McQueen Black Mesa Technologies LLC cms...@blackmesatech.com http://www.blackmesatech.com
Re: [basex-talk] Embedded BaseX
I am using BaseX in a multithreaded environment, which is highly transactional. I ended up using a singleton class as the controller and all other threads execute queries against this singleton class. I have successfully implemented an ELR (Electronic Laboratory Reporting) messaging system which is in production. This system runs under JBoss so you should not have any issues using something like Wildfly as an application server. Carl R Bondeson IT Analyst 3 Information Technology Connecticut Department of Public Health 410 Capitol Ave Hartford, CT 06134 Phone: 860-509-7434 carl.bonde...@ct.gov -Original Message- From: basex-talk-boun...@mailman.uni-konstanz.de [mailto:basex-talk-boun...@mailman.uni-konstanz.de] On Behalf Of Christian Grün Sent: Tuesday, July 11, 2017 8:02 AM To: George Sofianos Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Embedded BaseX Hi George, It’s recommendable indeed to only create one instance of the Context class. Context instances are lightweight, but operations like transactions are centrally controlled by this class. QueryProcessor are usually created anew for each query evaluation. Number of jobs… Just start in the Context class and follow the JobPool reference. Via the PARALLEL option [1], you can set a maximum limit of parallel database transactions. If you want to enforce that there are never more than transactions running, you can set FAIRLOCK [2] to true. I’ve just updated the description of the PARALLEL option to indicate what has changed since the introduction of the FAIRLOCK option. Hope this helps, Christian [1] http://docs.basex.org/wiki/Options#PARALLEL [2] http://docs.basex.org/wiki/Options#FAIRLOCK On Tue, Jul 11, 2017 at 11:36 AM, George Sofianos wrote: > Hello, > > I'm building a web service that along with some other things (XML > Validation, Saxon XQuery, etc) includes a BaseX processor for running > XQuery scripts. I'm wondering what is the best way to use an embedded > BaseX processor. Can some objects be shared (e.g Context, > QueryProcessor), or are they very lightweight and there is no need to share / > reuse ? > Can I get the number of the jobs running and the job status like in > the client / server mode? My intention is to make sure I can limit the > amount of the jobs running in parallel, and have more control over the > execution (maybe cancel jobs if necessary) > > Thanks, > > George >
Re: [basex-talk] Embedded BaseX
This is exactly what I was looking for, thanks! George On 07/11/2017 03:02 PM, Christian Grün wrote: Hi George, It’s recommendable indeed to only create one instance of the Context class. Context instances are lightweight, but operations like transactions are centrally controlled by this class. QueryProcessor are usually created anew for each query evaluation. Number of jobs… Just start in the Context class and follow the JobPool reference. Via the PARALLEL option [1], you can set a maximum limit of parallel database transactions. If you want to enforce that there are never more than transactions running, you can set FAIRLOCK [2] to true. I’ve just updated the description of the PARALLEL option to indicate what has changed since the introduction of the FAIRLOCK option. Hope this helps, Christian [1] http://docs.basex.org/wiki/Options#PARALLEL [2] http://docs.basex.org/wiki/Options#FAIRLOCK On Tue, Jul 11, 2017 at 11:36 AM, George Sofianos wrote: Hello, I'm building a web service that along with some other things (XML Validation, Saxon XQuery, etc) includes a BaseX processor for running XQuery scripts. I'm wondering what is the best way to use an embedded BaseX processor. Can some objects be shared (e.g Context, QueryProcessor), or are they very lightweight and there is no need to share / reuse ? Can I get the number of the jobs running and the job status like in the client / server mode? My intention is to make sure I can limit the amount of the jobs running in parallel, and have more control over the execution (maybe cancel jobs if necessary) Thanks, George
Re: [basex-talk] Embedded BaseX
Hi George, It’s recommendable indeed to only create one instance of the Context class. Context instances are lightweight, but operations like transactions are centrally controlled by this class. QueryProcessor are usually created anew for each query evaluation. Number of jobs… Just start in the Context class and follow the JobPool reference. Via the PARALLEL option [1], you can set a maximum limit of parallel database transactions. If you want to enforce that there are never more than transactions running, you can set FAIRLOCK [2] to true. I’ve just updated the description of the PARALLEL option to indicate what has changed since the introduction of the FAIRLOCK option. Hope this helps, Christian [1] http://docs.basex.org/wiki/Options#PARALLEL [2] http://docs.basex.org/wiki/Options#FAIRLOCK On Tue, Jul 11, 2017 at 11:36 AM, George Sofianos wrote: > Hello, > > I'm building a web service that along with some other things (XML > Validation, Saxon XQuery, etc) includes a BaseX processor for running XQuery > scripts. I'm wondering what is the best way to use an embedded BaseX > processor. Can some objects be shared (e.g Context, QueryProcessor), or are > they very lightweight and there is no need to share / reuse ? > Can I get the number of the jobs running and the job status like in the > client / server mode? My intention is to make sure I can limit the amount of > the jobs running in parallel, and have more control over the execution > (maybe cancel jobs if necessary) > > Thanks, > > George >
[basex-talk] Embedded BaseX
Hello, I'm building a web service that along with some other things (XML Validation, Saxon XQuery, etc) includes a BaseX processor for running XQuery scripts. I'm wondering what is the best way to use an embedded BaseX processor. Can some objects be shared (e.g Context, QueryProcessor), or are they very lightweight and there is no need to share / reuse ? Can I get the number of the jobs running and the job status like in the client / server mode? My intention is to make sure I can limit the amount of the jobs running in parallel, and have more control over the execution (maybe cancel jobs if necessary) Thanks, George
Re: [basex-talk] Data Access out of bounds error when running XQuery on a databse
> Tried the DB on the latest version and got the same error. Optimize all gave > me this "create statistics...:undeclared namespace prefix" Could you possibly add some more infos? Did you get this error with 7.9 or the latest version? Did the command complete successfully, or did it fail? Maybe a corrupt namespace hierarchy in your database causes these errors. You could additionally try to… 1. Run the INSPECT command and see if it reports any errors. 2. Export your database (via the EXPORT command) and create a new database from the resulting directory. > On Mon, Jul 10, 2017 at 2:59 PM, Christian Grün > wrote: >> >> Dear Jason, >> >> As you already indicated by yourself, version 7.9 is a bit older, so it >> would be interesting if you can reproduce the problem with a newer version. >> You could additionally try to optimize your database (using OPTIMIZE ALL or >> db:optimize(..., true()) ) and give us some feedback if the problem >> persists. >> >> Best, >> Christian >> >> >> >> Am 10.07.2017 8:49 nachm. schrieb "Jason Ricles" : >> >> I am running BaseX 7.9 (yes I know its old but for reasons beyond the >> scope of this we can't just update BaseX) on RHEL 6.9. >> >> The BaseX is not running any instance as a server but as a standalone >> application with its libraries being used with my java application. >> >> I am able to add new files to the database, but when I run an XQuery I get >> the following output: >> >> error running xquery: declare namespace tr = >> '[query]org.basex.core.BaseXException: Improper use? Potential bug? Your >> feedback is welcome: >> Contact: basex-talk@mailman.uni-konstanz.de >> Version: BaseX 7.9 >> Java: Oracle Corporation, 1.8.0_131 >> OS: Windows 7, amd64 >> Stack Trace: >> java.lang.RuntimeException: Data Access out of bounds: >> - pre value: 5813546 >> - #used blocks: 22710 >> - #total locks: 22710 >> - access: 22709 (22710 > 22709] >> at org.basex.util.Util.notExpected(Util.java:53) >> at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:508) >> at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:194) >> at org.basex.data.Data.kind(Data.java:284) >> at org.basex.query.value.node.DBNode$5.next(DBNode.java:353) >> at org.basex.query.value.node.DBNode$5.next(DBNode.java:345) >> at org.basex.query.path.IterStep$1.next(IterStep.java:39) >> at org.basex.query.path.IterStep$1.next(IterStep.java:31) >> at org.basex.query.path.MixedPath.iter(MixedPath.java:99) >> at org.basex.query.gflwor.GFLWOR$2.next(GFLWOR.java:78) >> at org.basex.query.MainModule$1.next(MainModule.java:98) >> at org.basex.core.cmd.AQuery.query(AQuery.java:96) >> at org.basex.core.cmd.XQuery.run(XQuery.java:22) >> at org.basex.core.Command.run(Command.java:329) >> at org.basex.core.Command.execute(Command.java:94) >> at org.basex.core.Command.execute(Command.java:117) >> at >> basex_debug.Corrupt_DB_Test.queryTestResultsUUID(Corrupt_DB_Test.java:114) >> at >> basex_debug.Corrupt_DB_Test.checkIfAlreadyAdded(Corrupt_DB_Test.java:48) >> at basex_debug.Corrupt_DB_Test.addToDatabase(Corrupt_DB_Test.java:64) >> >>at basex_debug.Corrupt_DB_Test.main(Corrupt_DB_Test.java:23) >> >> The query was running fine before so I know there is not a problem with >> the query, I have only read about the database getting corrupt with BaseX >> when ran in Server and Client mode. >> >> What may be causing this problem and what are some possible solutions? >> >> >> >
