Hi,
I made a lot of tests now and the inserts are much faster. I found out that the
database was not optimized. When I optimize the DB before calling the inserts
or deletes, they execute very fast.
But I have another problem now when I optimize the DB in Java code. I try to
insert the new nodes in chunks of 8000 nodes per command. As I already said: I
have to insert a lot of nodes (more than 100000) and I think that would be too
much for one XQuery command (???). So what I basically do now is this:
1. Gathering all the information I need for the insert commands
2. Creating a loop in Java that executes the insert commands every 8000th
iteration
3. After each iteration, I try to optimize the database with this Java command:
new XQuery("db:optimize('DB_Name', false())").execute(context);
The problem is that - when the second iteration of the loop tries to insert the
next 8000 nodes - I get this exception "TM: no context value bound". I add the
complete stack trace below. I already tried to create a new context, but that
didn't work. What could be the problem?
Thank you very much for your help in advance!
Michael
Here the complete stack trace:
org.basex.core.BaseXException: Stopped at ., 1/5847730:
[XPDY0002] TM: no context value bound.
at org.basex.core.Command.execute(Command.java:94)
at org.basex.core.Command.execute(Command.java:116)
at actions.basex.Test.run(Test.java:256) --> This line contains the
insert command: new XQuery([INSERT_QUERY]).execute(context);
at actions.basex.Test.<init>(Test.java:50)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at main.Main.main(Main.java:44)
Caused by: org.basex.query.QueryException: TM: no context value bound.
at org.basex.query.QueryError.get(QueryError.java:1392)
at org.basex.query.expr.path.Step.checkNode(Step.java:227)
at org.basex.query.expr.path.IterStep$1.next(IterStep.java:37)
at org.basex.query.expr.path.CachedPath.iter(CachedPath.java:72)
at org.basex.query.expr.path.CachedPath.nodeIter(CachedPath.java:51)
at org.basex.query.expr.path.AxisPath.iter(AxisPath.java:69)
at org.basex.query.expr.path.AxisPath.iter(AxisPath.java:40)
at org.basex.query.expr.constr.Constr.add(Constr.java:70)
at org.basex.query.expr.constr.CElem.item(CElem.java:93)
at org.basex.query.expr.constr.CElem.item(CElem.java:1)
at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:48)
at org.basex.query.expr.constr.Constr.add(Constr.java:70)
at org.basex.query.expr.constr.CElem.item(CElem.java:93)
at org.basex.query.expr.constr.CElem.item(CElem.java:1)
at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:48)
at org.basex.query.expr.List$1.iter(List.java:151)
at org.basex.query.expr.List$1.next(List.java:119)
at org.basex.query.QueryContext.next(QueryContext.java:397)
at org.basex.query.expr.constr.Constr.add(Constr.java:71)
at org.basex.query.up.expr.Insert.item(Insert.java:55)
at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:48)
at org.basex.query.expr.List$1.iter(List.java:151)
at org.basex.query.expr.List$1.size(List.java:141)
at org.basex.query.scope.MainModule.cache(MainModule.java:99)
at org.basex.query.QueryContext.iter(QueryContext.java:333)
at org.basex.query.QueryProcessor.iter(QueryProcessor.java:90)
at org.basex.core.cmd.AQuery.query(AQuery.java:100)
at org.basex.core.cmd.XQuery.run(XQuery.java:22)
at org.basex.core.Command.run(Command.java:257)
at org.basex.core.Command.execute(Command.java:93)
... 8 more<https://www.wie-soll-arbeit.at>
________________________________
Von: Christian Grün <christian.gr...@gmail.com>
Gesendet: Mittwoch, 8. August 2018 19:16
An: BIRKNER Michael
Cc: BaseX
Betreff: Re: [basex-talk] BaseX insert/delete node performance
Michael,
Welcome to the list.
One thing you could try immediately is to call OPTIMIZE – possibly followed by
the ALL flag, or db:optimize(..., true() – and see if performance improves.
Obviously, this doesn't make sense after each single update operation, but it
could be called before a bigger number of updates is to be performed.
> The problem is that in my case, I have to do about 150000 inserts and
> deletes, so it would take too much time.
If you define all the insert expression (or a bigger number than just 1 or 10)
in a single XQuery expression (via a FLWOR expression), you will benefit from
various bulk optimizations. Did you try that already?
Best,
Christian
BIRKNER Michael <michael.birk...@akwien.at<mailto:michael.birk...@akwien.at>>
schrieb am Mi., 8. Aug. 2018, 08:36:
Hello,
I asked this question in StackOverflow concerning some performance problems I
experienced when inserting nodes into a BaseX database:
https://stackoverflow.com/questions/51595210/basex-inserting-nodes-performance-problems
I already made some progress, especially when it comes to querying all data I
need for the updates. I work a lot with the indexes now.
But I still have problems with inserting - and also deleting - nodes. It
doesn't matter if I insert/delete nodes via a Java program or in the editor of
the BaseX GUI: Both is quite slow. Inserting just one node in the GUI with an
XQuery like this one takes up to 3 seconds:
insert node <related_record><title>Test title</title><author>Joe
Lastname</author></related_record> into db:open-id('Database_Name', 7947561)
Deleting a node with the following command takes up to 7 seconds:
delete node db:open-id('Database_Name', 88085737)
The problem is that in my case, I have to do about 150000 inserts and deletes,
so it would take too much time.
Maybe my database is just too big to be performant? Or some settings are wrong?
I'm very new to BaseX (and XML databases in general) so maybe there are just
some errors I don't see. I also give you some information on my database that I
copied from the info screen of the BaseX GUI:
Database Properties
NAME: Database_Name
SIZE: 2568 MB
NODES: 135607105
DOCUMENTS: 1
BINARIES: 0
TIMESTAMP: 2018-08-07T07:05:56.000Z
UPTODATE: true
Resource Properties
INPUTPATH: /path/to/file.xml
INPUTSIZE: 1774 MB
INPUTDATE: 2018-07-24T14:32:58.000Z
Indexes
TEXTINDEX: true
ATTRINDEX: true
TOKENINDEX: false
FTINDEX: false
TEXTINCLUDE:
ATTRINCLUDE:
TOKENINCLUDE:
FTINCLUDE:
LANGUAGE: English
STEMMING: false
CASESENS: false
DIACRITICS: false
STOPWORDS:
UPDINDEX: true
AUTOOPTIMIZE: false
MAXCATS: 100
MAXLEN: 96
SPLITSIZE: 0
Best regards,
Michael
Beachten Sie, dass Sie uns ab sofort unter einer geänderten Rufnummer
erreichen. Bitte speichern Sie gleich Ihren Kontakt zur AK Wien ein unter 501
65 1, gefolgt von der gewohnten Durchwahl.
Dieses Mail ist ausschließlich für die Verwendung durch die/den darin genannten
AdressatInnen bestimmt und kann vertrauliche bzw rechtlich geschützte
Informationen enthalten, deren Verwendung ohne Genehmigung durch den/ die
AbsenderIn rechtswidrig sein kann. Falls Sie dieses Mail irrtümlich erhalten
haben, informieren Sie uns bitte und löschen Sie die Nachricht. UID: ATU
16209706 I https://wien.arbeiterkammer.at/Datenschutz_(DSGVO).html
Beachten Sie, dass Sie uns ab sofort unter einer geänderten Rufnummer
erreichen. Bitte speichern Sie gleich Ihren Kontakt zur AK Wien ein unter 501
65 1, gefolgt von der gewohnten Durchwahl.
Dieses Mail ist ausschließlich für die Verwendung durch die/den darin genannten
AdressatInnen bestimmt und kann vertrauliche bzw rechtlich geschützte
Informationen enthalten, deren Verwendung ohne Genehmigung durch den/ die
AbsenderIn rechtswidrig sein kann. Falls Sie dieses Mail irrtümlich erhalten
haben, informieren Sie uns bitte und löschen Sie die Nachricht. UID: ATU
16209706 I https://wien.arbeiterkammer.at/Datenschutz_(DSGVO).html
