Re: [basex-talk] Bug merging maps with option duplicates:combine?
Thank you, Leo. Your quick analysis of the bug, and your efforts to fix it, are
much appreciated!
@Bridger: We have a number of fairly large (GBs) datasets that we are
processing with a number of diverse queries. In rare occasions this combination
appears to uncover a bug. :-)
Best regards,
Johannes
-Ursprüngliche Nachricht-
Von: Leonard Wörteler [mailto:leonard.woerte...@uni-konstanz.de]
Gesendet: Donnerstag, 9. Januar 2020 23:40
An: Bridger Dyson-Smith ; Johannes Echterhoff
Cc: BaseX
Betreff: Re: [basex-talk] Bug merging maps with option duplicates:combine?
Hi Bridger and Johannes,
wow, this is a really old and obscure bug in the `map:merge(...)` logic
you found there. It only occurs if both maps to be merged contain hash
collisions on the same key, and the two collision buckets are merged
together. This is not a common scenario. In a quick search I did not
find any BaseX versions that included maps but not this bug, so it could
well be that I introduced it in my original implementation -- whoopsie!
The AIOOB exception is only triggered when the second bucket is larger
than the first one, otherwise the result would just silently be wrong.
I am currently testing a proper fix for this, together with some
regression tests for all the merging strategies. Afterwards I'll open a
pull request.
Cheers, Leo
On 09.01.2020 21:23, Bridger Dyson-Smith wrote:
> Hi all,
>
> While I don't have any answer to this question, I can corroborate
> Johannes' finding.
>
> Also, Johannes, did you find a magic string? I tried some longer keys
> and they seem to work :)
> Best,
> Bridger
>
> On Thu, Jan 9, 2020 at 11:51 AM Johannes Echterhoff
> <mailto:echterh...@interactive-instruments.de>> wrote:
>
> Hello,
>
> When executing:
>
> __ __
>
> map:merge((
> map {
> "DENW21AL100077Hs": "a",
> "DENW21AL100076i5": "a"
> },
> map {
> "DENW21AL100077Hs": "b",
> "DENW21AL100076i5": "b",
> "DENW21AL100076hT": "b"
> }
> ), map {'duplicates': 'combine'})
>
> __ __
>
> in the BaseX GUI I get:
>
> __ __
>
> Error:
>
> Improper use? Potential bug? Your feedback is welcome:
>
> Contact: basex-talk@mailman.uni-konstanz.de
> <mailto:basex-talk@mailman.uni-konstanz.de>
>
> Version: BaseX 9.3.1
>
> Java: AdoptOpenJDK, 1.8.0_212
>
> OS: Windows 10, amd64
>
> Stack Trace:
>
> java.lang.ArrayIndexOutOfBoundsException: 2
>
> at
> org.basex.query.value.map.TrieList.add(TrieList.java:199)
>
> at
> org.basex.query.value.map.TrieList.addAll(TrieList.java:141)
>
> at
> org.basex.query.value.map.XQMap.addAll(XQMap.java:106)
>
> at
> org.basex.query.func.map.MapMerge.item(MapMerge.java:33)
>
> at
> org.basex.query.func.StandardFunc.optimize(StandardFunc.java:82)
>
> at org.basex.query.expr.Arr.compile(Arr.java:43)
>
> at
> org.basex.query.scope.MainModule.comp(MainModule.java:81)
>
> at
> org.basex.query.QueryCompiler.compile(QueryCompiler.java:114)
>
> at
> org.basex.query.QueryCompiler.compile(QueryCompiler.java:105)
>
> at
> org.basex.query.QueryContext.compile(QueryContext.java:312)
>
> at
> org.basex.query.QueryProcessor.compile(QueryProcessor.java:79)
>
> at org.basex.core.cmd.AQuery.query(AQuery.java:93)
>
> at org.basex.core.cmd.XQuery.run(XQuery.java:22)
>
> at org.basex.core.Command.run(Command.java:257)
>
> at org.basex.core.Command.execute(Command.java:93)
>
> at org.basex.gui.GUI.exec(GUI.java:422)
>
> at org.basex.gui.GUI.lambda$4(GUI.java:365)
>
> at java.lang.Thread.run(Thread.java:748)
>
> __ __
>
> Best regards,
>
> Johannes
>
[basex-talk] Bug merging maps with option duplicates:combine?
Hello,
When executing:
map:merge((
map {
"DENW21AL100077Hs": "a",
"DENW21AL100076i5": "a"
},
map {
"DENW21AL100077Hs": "b",
"DENW21AL100076i5": "b",
"DENW21AL100076hT": "b"
}
), map {'duplicates': 'combine'})
in the BaseX GUI I get:
Error:
Improper use? Potential bug? Your feedback is welcome:
Contact: basex-talk@mailman.uni-konstanz.de
Version: BaseX 9.3.1
Java: AdoptOpenJDK, 1.8.0_212
OS: Windows 10, amd64
Stack Trace:
java.lang.ArrayIndexOutOfBoundsException: 2
at org.basex.query.value.map.TrieList.add(TrieList.java:199)
at org.basex.query.value.map.TrieList.addAll(TrieList.java:141)
at org.basex.query.value.map.XQMap.addAll(XQMap.java:106)
at org.basex.query.func.map.MapMerge.item(MapMerge.java:33)
at
org.basex.query.func.StandardFunc.optimize(StandardFunc.java:82)
at org.basex.query.expr.Arr.compile(Arr.java:43)
at org.basex.query.scope.MainModule.comp(MainModule.java:81)
at org.basex.query.QueryCompiler.compile(QueryCompiler.java:114)
at org.basex.query.QueryCompiler.compile(QueryCompiler.java:105)
at org.basex.query.QueryContext.compile(QueryContext.java:312)
at
org.basex.query.QueryProcessor.compile(QueryProcessor.java:79)
at org.basex.core.cmd.AQuery.query(AQuery.java:93)
at org.basex.core.cmd.XQuery.run(XQuery.java:22)
at org.basex.core.Command.run(Command.java:257)
at org.basex.core.Command.execute(Command.java:93)
at org.basex.gui.GUI.exec(GUI.java:422)
at org.basex.gui.GUI.lambda$4(GUI.java:365)
at java.lang.Thread.run(Thread.java:748)
Best regards,
Johannes
Re: [basex-talk] How many QueryModule instances can be created?
Again, thank you, Christian. So by creating such an XQuery wrapper module, we would ensure that there is only a single import of the Java class (which represents module m1) - regardless of how many times the XQuery wrapper module is imported? If so, then indeed that sounds like an option. Best regards, Johannes -Ursprüngliche Nachricht- Von: Christian Grün [mailto:christian.gr...@gmail.com] Gesendet: Mittwoch, 18. Dezember 2019 17:48 An: Johannes Echterhoff Cc: BaseX Betreff: Re: [basex-talk] How many QueryModule instances can be created? Hi Johannes, > m1 is a Java module, with de.interactive_instruments.module.MyQueryModule > being the Java class that extends QueryModule. > … > We end up having multiple instances, i.e. Java objects, of class > MyQueryModule, … So the solution which I would recommend (and which I would generally recommend when importing Java classes) is to write an XQuery wrapper module for m1, and place the Java imports in that module (see [1] for an example). This module will only exist once in your query context. This approach has various other advantages: For example, you can work with XQuery data types in all other modules, and only the wrapper needs to ensure that the XQuery parameters will be correctly converted into and back from Java types. Does that sound like an option? Christian [1] http://docs.basex.org/wiki/Repository#Combined > -Ursprüngliche Nachricht- > Von: Christian Grün [mailto:christian.gr...@gmail.com] > Gesendet: Mittwoch, 18. Dezember 2019 15:59 > An: Johannes Echterhoff > Cc: BaseX > Betreff: Re: [basex-talk] How many QueryModule instances can be created? > > > It may also answer my second question, where I was referring to expath > > packaging and if that would make any difference - when compared to having a > > pure JAR with the Java classes (and required libraries). It sounds like > > expath packaging does not make a difference. Please correct me if I got > > this wrong. > > Right: EXPath is just another way of packaging the code. The XQuery parser > will handle all modules equally, no matter if they have initially been > packaged as XAR or via our own packaging mechanisms. > > After having read your initial mail for a second time, I noticed I may have > got your setup a little wrong. I think it will be easier to find a solution > if we manage to construct a little example that shows the behavior you > reported. Otherwise, there may be too many open questions to solve (how do > you import and initialize your Java Code? is initialization identical to > creating a class instance? is the initialization code embedded in global > variables? etc). > > I have attached a basic set of files that (I believe) simulates your setup. > Could you extend it for me, or comment back what I may have wrongly > understood?
Re: [basex-talk] How many QueryModule instances can be created?
Thank you for your patience and support, Christian.
Then my second question has been answered as well. So basically, when the
XQuery execution involves multiple imports of a Java module, regardless of
packaging, BaseX may create multiple instances of the according Java class.
I slightly updated your queries as follows (it is not executable, but it shows
our current setup):
testquery.xq:
-
import module namespace m1 = 'de.interactive_instruments.module.MyQueryModule';
import module namespace m2 = 'm2' at 'm2.xqm';
import module namespace m3 = 'm3' at 'm3.xqm';
m1:init(... some data ...),
m1:add('m1'),
m1:functionMno(),
m2:do(),
m3:do(),
m1:values()
-
m2.xqm:
-
module namespace m2 = 'm2';
import module namespace m1 = 'de.interactive_instruments.module.MyQueryModule';
declare function m2:do() {
... some xquery that uses a function from m1, e.g. m1:functionAbc() ...
};
-
m3.xqm:
-
module namespace m3 = 'm3';
import module namespace m1 = 'de.interactive_instruments.module.MyQueryModule';
declare function m3:do() {
... some xquery that uses a function from m1, e.g. m1:functionXyz() ...
};
-
m1 is a Java module, with de.interactive_instruments.module.MyQueryModule being
the Java class that extends QueryModule.
m1 is initialised, i.e. fed with some information, using methods defined by
class MyQueryModule.
Once that is done, expressions from testquery.xq call additional methods of
MyQueryModule, either directly or indirectly (by calling functions from the
XQuery modules m2 and m3).
We end up having multiple instances, i.e. Java objects, of class MyQueryModule,
- presumably due to the multiple imports of that module - and only one of them
has been initialised with the necessary information. What we would have wanted
is a single instance/object of MyQueryModule, which has the information given
to it by calling initialisation functions in testquery.xq, and which would
therefore have this information regardless of where one of the m1 functions is
called afterwards (be it in testquery.xq, or the modules m2 and m3).
So some kind of QueryModule singleton construct would be handy, but I doubt
that would work with BaseX as-is (assuming that a default constructor is used
to create an object of MyQueryModule when one of its methods is invoked). I
guess we could also explicitly create the MyQueryModule object (as outlined in
the FileWriter example on
http://docs.basex.org/wiki/Java_Bindings#Namespace_Declarations) and pass it
around as a function parameter, but that would increase the level of complexity
in our XQuery code quite a bit, so I doubt that this would be desirable for us.
Maybe having a QueryModule implementation that acts as a façade to a singleton
which does the actual work ... but I digress.
From what I've learned so far in this conversation, we'd better just ensure
that there is only a single import of module m1.
Best regards,
Johannes
-Ursprüngliche Nachricht-
Von: Christian Grün [mailto:christian.gr...@gmail.com]
Gesendet: Mittwoch, 18. Dezember 2019 15:59
An: Johannes Echterhoff
Cc: BaseX
Betreff: Re: [basex-talk] How many QueryModule instances can be created?
> It may also answer my second question, where I was referring to expath
> packaging and if that would make any difference - when compared to having a
> pure JAR with the Java classes (and required libraries). It sounds like
> expath packaging does not make a difference. Please correct me if I got this
> wrong.
Right: EXPath is just another way of packaging the code. The XQuery parser will
handle all modules equally, no matter if they have initially been packaged as
XAR or via our own packaging mechanisms.
After having read your initial mail for a second time, I noticed I may have got
your setup a little wrong. I think it will be easier to find a solution if we
manage to construct a little example that shows the behavior you reported.
Otherwise, there may be too many open questions to solve (how do you import and
initialize your Java Code? is initialization identical to creating a class
instance? is the initialization code embedded in global variables? etc).
I have attached a basic set of files that (I believe) simulates your setup.
Could you extend it for me, or comment back what I may have wrongly understood?
Re: [basex-talk] How many QueryModule instances can be created?
Hi Christian, Thank you. If I understand you correctly, that answers my first question. It may also answer my second question, where I was referring to expath packaging and if that would make any difference - when compared to having a pure JAR with the Java classes (and required libraries). It sounds like expath packaging does not make a difference. Please correct me if I got this wrong. You asked: Or should there only ever be a single such instance, regardless of how many times the module is imported? The answer is yes. That is because we call some specific functions first, to initialize the module, and we need the information conveyed by this initialization process throughout the whole execution of an xquery, regardless how many times the module is imported (by that xquery, or indirectly via modules that the xquery imports). Before we made an attempt of modularizing all our custom xquery functions into a set of modules, the Java module worked fine, presumably because it was only imported once. Now the Java module is imported by the xquery and some xquery modules (that are imported by the xquery). And we end up with multiple instances of the Java module. I was wondering if any packaging approach would ensure that only a single instance of the Java module is created, regardless of how many times the module is imported. From what I heard so far, it looks like that is not the case. Please confirm, or correct me if I got it wrong. If there is no guarantee that a Java module is instantiated multiple times if it is imported more than once, regardless of how the Java module is packaged, then we would simply go back to the original structure of our xquery (before the modularization attempt). I just need to understand what behavior to expect (from BaseX) when it processes imports of a Java module (provided as pure Java module, combined module, or using EXPath packaging). Best regards, Johannes -Ursprüngliche Nachricht- Von: Christian Grün [mailto:christian.gr...@gmail.com] Gesendet: Mittwoch, 18. Dezember 2019 11:07 An: Johannes Echterhoff Cc: BaseX Betreff: Re: [basex-talk] How many QueryModule instances can be created? Hi Johannes, > · Is it expected behavior that multiple instances of a Java > QueryModule (M1, in my scenario) may be created and used during an execution > of a query scenario like the one described above – or in general? Yes, this is currently expected behavior; the import mechanisms of XQuery and Java modules differ in various aspects. The customary way to proceed is to organize the Java calls in a single XQuery module. Both the Java class and the XQuery module can then optionally be bundled as JAR file (see [1]). Hope this helps, Christian [1] http://docs.basex.org/wiki/Repository#Combined Or should there only ever be a single such instance, regardless of how many times the module is imported? > > o Note: The functions used to initialize M1 are non-deterministic and > declared as such. Not entirely sure if that makes a difference regarding how > many times M1 would be created. > > · I have the same questions for the case that the Java QueryModule M1 > was packaged in a XAR (as described in > http://docs.basex.org/wiki/Repository#EXPath_Packaging). Would such a > packaging approach actually make any difference? > > Apologies that I do not have a small, self-contained example project to > demonstrate this. I hope that I explained the issue with sufficient detail > and clarity. If not, just let me know. > > Best regards, > > Johannes > > P.S.: If you have suggestions for a better approach of handling such a > scenario, where a QueryModule must be initialized before it can be used, and > there shall only be a single instance of this module within the execution of > an XQuery, let me know. > >
[basex-talk] How many QueryModule instances can be created?
Hello, We are struggling with a situation where we have built a custom Java QueryModule (as described in http://docs.basex.org/wiki/Java_Bindings) - let's call it M1, and that module is imported by two other XQuery modules (let's call them M2 and M3). When executing an XQuery that imports M1 and the other XQuery modules M2 and M3 within the BaseX GUI, multiple different instances of M1 are created. That is an issue, because we initialise the module M1 with some information (a custom geometry index) before calling any other function of M1. This initialization shall only occur once, because it is quite time-consuming. Also, the information from this initialization procedure must be available whenever the (non-initialization) functions of M1 are called - whether in the XQuery or the other modules M2 and M3. The situation is as follows: * testquery.xq imports the custom module M1 * testquery.xq also imports the XQuery modules M2 and M3, which are stored in other XQuery files; M2 and M3 both import M1 * the custom module is initialized in testquery.xq - not in M2 and M3 * non-initialization functions of M1 are called in testquery.xq, as well as by functions declared in M2 and M3 - but only after initialization functions of M1 have been called in testquery.xq We get exceptions when functions of M1 are called by functions declared in M2 and M3, apparently because multiple instances of M1 have been created by BaseX (tested with the BaseX GUI), and only one of them has been initialized (in testquery.xq). After this lengthy introduction of the issue at hand, here are my questions: * Is it expected behavior that multiple instances of a Java QueryModule (M1, in my scenario) may be created and used during an execution of a query scenario like the one described above - or in general? Or should there only ever be a single such instance, regardless of how many times the module is imported? o Note: The functions used to initialize M1 are non-deterministic and declared as such. Not entirely sure if that makes a difference regarding how many times M1 would be created. * I have the same questions for the case that the Java QueryModule M1 was packaged in a XAR (as described in http://docs.basex.org/wiki/Repository#EXPath_Packaging). Would such a packaging approach actually make any difference? Apologies that I do not have a small, self-contained example project to demonstrate this. I hope that I explained the issue with sufficient detail and clarity. If not, just let me know. Best regards, Johannes P.S.: If you have suggestions for a better approach of handling such a scenario, where a QueryModule must be initialized before it can be used, and there shall only be a single instance of this module within the execution of an XQuery, let me know.
[basex-talk] Optimization issue? - maps and count
All,
The following query:
let $lookupSequence := ('a','b')
let $map := map {'c': 'x'}
let $mapLookupEvaluation := $map?($lookupSequence)
return
(
'Evaluation of the map lookup is: ' || string-join($mapLookupEvaluation,", "),
'count($mapLookupEvaluation) is: ' || count($mapLookupEvaluation)
)
Results in:
Evaluation of the map lookup is:
count($mapLookupEvaluation) is: 2
I would have expected 0 instead of 2. Am I missing something?
Best regards,
Johannes
Re: [basex-talk] Optimization issue?
Dear Christian,
Tested the snapshot yesterday, and now the expression works as expected. Thank
you.
Do you have a target date for the release of BaseX 9.3?
Best regards,
Johannes
-Ursprüngliche Nachricht-
Von: Christian Grün [mailto:christian.gr...@gmail.com]
Gesendet: Donnerstag, 24. Oktober 2019 12:41
An: Johannes Echterhoff
Cc: BaseX
Betreff: Re: [basex-talk] Optimization issue?
Dear Johannes,
Thanks for the reproducible test case. The bug (a too eager merge of combined
range expressions) was fixed and a new snapshot is available [1,2].
Cheers
Christian
[1] https://github.com/BaseXdb/basex/issues/1744
[2] http://files.basex.org/releases/latest/
On Thu, Oct 24, 2019 at 10:54 AM Johannes Echterhoff
wrote:
>
> Hello,
>
> I’m experiencing some strange behavior when executing the following query in
> the BaseX 9.2.4 GUI (nevermind the awkward query structure – that is just to
> be able to inspect the results of the individual evaluation steps):
>
>
>
> let $t := 200.
>
> let $num := number($t)
>
> let $gt5000 := $num gt 5000
>
> let $ltMinus5000 := $num lt -5000
>
> return $gt5000 or $ltMinus5000
>
>
>
> The result is true, although it should be false.
>
>
>
> The compilation and optimization logs for this query in the GUI tell us the
> following:
>
> --
>
> Compiling:
>
> - rewrite > comparison to range comparison: ($num_1 > 5000) -> ($num_1
> >= 5000.0001)
>
> - rewrite 'gt' comparison to range comparison: ($num_1 gt 5000) ->
> ($num_1 >= 5000.0001)
>
> - pre-evaluate unary expression to xs:integer item: -5000
>
> - rewrite < comparison to range comparison: ($num_1 < -5000) ->
> ($num_1 <= -5000.0001)
>
> - rewrite 'lt' comparison to range comparison: ($num_1 lt -5000) ->
> ($num_1 <= -5000.0001)
>
> - inline $gt5000_2
>
> - inline $ltMinus5000_3
>
> - simplify or: (true())
>
> - pre-evaluate or to xs:boolean item: (true()) -> true()
>
> - remove unused variable $num_1
>
> - remove unused variable $t_0
>
> - simplify FLWOR expression: true()
>
> Optimized Query:
>
> true()
>
> --
>
>
>
> When the return statement is changed to “return
> ($num,$gt5000,$ltMinus5000,$gt5000 or $ltMinus5000)” then the result of the
> boolean operation (last element of the result sequence) is false, as expected.
>
> The GUI log then is:
>
> --
>
> Compiling:
>
> - rewrite > comparison to range comparison: ($num_1 > 5000) -> ($num_1
> >= 5000.0001)
>
> - rewrite 'gt' comparison to range comparison: ($num_1 gt 5000) ->
> ($num_1 >= 5000.0001)
>
> - pre-evaluate unary expression to xs:integer item: -5000
>
> - rewrite < comparison to range comparison: ($num_1 < -5000) ->
> ($num_1 <= -5000.0001)
>
> - rewrite 'lt' comparison to range comparison: ($num_1 lt -5000) ->
> ($num_1 <= -5000.0001)
>
> Optimized Query:
>
> let $t_0 := element x { ("200.") } let $num_1 := number($t_0) let
> $gt5000_2 := ($num_1 >= 5000.0001) let $ltMinus5000_3 :=
> ($num_1 <= -5000.0001) return ($num_1, $gt5000_2,
> $ltMinus5000_3, ($gt5000_2 or $ltMinus5000_3))
>
> --
>
>
>
> Am I missing something, or could this be a bug in BaseX 9.2.4? With BaseX
> 9.0.2, both queries (with the different return statements) returned the
> expected value false for the boolean operation.
>
>
>
> Best regards,
>
> Johannes
>
>
>
>
[basex-talk] Optimization issue?
Hello,
I'm experiencing some strange behavior when executing the following query in
the BaseX 9.2.4 GUI (nevermind the awkward query structure - that is just to be
able to inspect the results of the individual evaluation steps):
let $t := 200.
let $num := number($t)
let $gt5000 := $num gt 5000
let $ltMinus5000 := $num lt -5000
return $gt5000 or $ltMinus5000
The result is true, although it should be false.
The compilation and optimization logs for this query in the GUI tell us the
following:
--
Compiling:
- rewrite > comparison to range comparison: ($num_1 > 5000) -> ($num_1 >=
5000.0001)
- rewrite 'gt' comparison to range comparison: ($num_1 gt 5000) -> ($num_1 >=
5000.0001)
- pre-evaluate unary expression to xs:integer item: -5000
- rewrite < comparison to range comparison: ($num_1 < -5000) -> ($num_1 <=
-5000.0001)
- rewrite 'lt' comparison to range comparison: ($num_1 lt -5000) -> ($num_1 <=
-5000.0001)
- inline $gt5000_2
- inline $ltMinus5000_3
- simplify or: (true())
- pre-evaluate or to xs:boolean item: (true()) -> true()
- remove unused variable $num_1
- remove unused variable $t_0
- simplify FLWOR expression: true()
Optimized Query:
true()
--
When the return statement is changed to "return
($num,$gt5000,$ltMinus5000,$gt5000 or $ltMinus5000)" then the result of the
boolean operation (last element of the result sequence) is false, as expected.
The GUI log then is:
--
Compiling:
- rewrite > comparison to range comparison: ($num_1 > 5000) -> ($num_1 >=
5000.0001)
- rewrite 'gt' comparison to range comparison: ($num_1 gt 5000) -> ($num_1 >=
5000.0001)
- pre-evaluate unary expression to xs:integer item: -5000
- rewrite < comparison to range comparison: ($num_1 < -5000) -> ($num_1 <=
-5000.0001)
- rewrite 'lt' comparison to range comparison: ($num_1 lt -5000) -> ($num_1 <=
-5000.0001)
Optimized Query:
let $t_0 := element x { ("200.") } let $num_1 := number($t_0) let $gt5000_2
:= ($num_1 >= 5000.0001) let $ltMinus5000_3 := ($num_1 <=
-5000.0001) return ($num_1, $gt5000_2, $ltMinus5000_3, ($gt5000_2 or
$ltMinus5000_3))
--
Am I missing something, or could this be a bug in BaseX 9.2.4? With BaseX
9.0.2, both queries (with the different return statements) returned the
expected value false for the boolean operation.
Best regards,
Johannes
Re: [basex-talk] BaseX/GUI v9.1.2 memory use
For what it’s worth: in our developments we had a directory as part of the
project files where we stored all queries that were executed during development
and testing (these queries were dynamically created by other queries). Saving
these queries is just part of our process. However, after a while that resulted
in hundreds to thousands of query files. That resulted in a massive memory
demand, and we noticed increased CPU usage, both of which we could not explain
at first. Eventually we found out that the BaseX GUI automatically parsed all
these queries, each time initializing a custom module we wrote, where the
initialization of that module created a temporary directory with a number of
files. The solution for us was to simply move the directory where we save the
executed queries somewhere outside of the project files opened in the BaseX GUI.
Best regards,
Johannes
Von: BaseX-Talk [mailto:basex-talk-boun...@mailman.uni-konstanz.de] Im Auftrag
von Graydon Saunders
Gesendet: Mittwoch, 23. Januar 2019 14:19
An: Christian Grün
Cc: BaseX
Betreff: Re: [basex-talk] BaseX/GUI v9.1.2 memory use
Never had a problem with the GUI parsing project files. No issues with
symbolic links.
I have generally found basex very reliable but never try to update the
installed version; it's always move the old, unpack the new.
On Wed, Jan 23, 2019, 08:09 Christian Grün
mailto:christian.gr...@gmail.com>> wrote:
Thanks for your assessments. So maybe we should ensure that every file and
directory will only be parsed once.
Did anyone of you have problems with the automatic parsing of project files in
the BaseX GUI, or with symbolic links in particular?
Am Mi., 23. Jan. 2019, 09:12 hat Marco Lettere
mailto:m.lett...@gmail.com>> geschrieben:
Same for us over here. The ability to follow symlinks is a very powerful
feature that we use to externalize folders (data, restxq for instance). So
please don't remove it altogether!
M.
On 22/01/19 23:50, Graydon Saunders wrote:
I've been handling updates by making data/ a symbolic link to a data directory
that's a sibling of the basex directory. (Move the old, unpack the new, go
into new and replace data/ with a symbolic link up and over.)
Would hate to see that stop working.
On Tue, Jan 22, 2019, 17:36 Christian Grün
mailto:christian.gr...@gmail.com>> wrote:
Good to hear that! I can’t recollect that something particular has changed in
version 9.1.2, regarding the scanning of project files, but I’ll have some
thoughts how we can trace and interrupt such loops (or ignore symbolic links
instead).
Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith
mailto:bdysonsm...@gmail.com>> geschrieben:
Glad that helped :)
I see this when I start from a fresh install vs expanding the ZIP into the same
directory.
On Tue, Jan 22, 2019, 5:17 PM Rick Graham
mailto:rickhg1...@gmail.com> wrote:
Thanks Bridger!
Indeed, I quit basexgui and manually edited .basexgui to set the project
directory to a newly created empty directory. basexgui seems normal/stable
after that.
I rarely, as in almost never, use wine but I didn't have this issue with
previous versions of BaseX. Something seems unexpected here.
On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith
mailto:bdysonsm...@gmail.com>> wrote:
Hi Rick, et al,
I think (but am not 100% sure) that the GUI defaults to looking through your
home directory on startup. So, somewhere in `~/rick/.wine/dosdevices/...` you
have symbolic links that are looped.
I think you might be able to circumvent this problem by finding `.basexgui` -
it would probably be close to wherever you started the GUI from on your
filesystem. I think you can edit some of the PATHS there and that may help?
Again, I'm not sure. HTH!
Best,
Bridger
On Tue, Jan 22, 2019 at 4:56 PM Rick Graham
mailto:rickhg1...@gmail.com>> wrote:
The command-line seemed to be operating normally.
What exactly is/are my project directories?
I attached to the running GUI instance `strace -f -e trace=stat -p 13368` and
it has infinite repetitions of:
[pid 13436]
stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02",
0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links)
What's going on here?
On Tue, Jan 22, 2019 at 10:21 PM Christian Grün
mailto:christian.gr...@gmail.com>> wrote:
at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167)
at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)
at
Re: [basex-talk] determine index use in xquery:eval
Hi Christian, The tip regarding xquery:invoke was very useful. Looks like no indexes are applied. I'll have to investigate in more detail. Thank you for responding so quickly, Johannes -Ursprüngliche Nachricht- Von: Christian Grün [mailto:christian.gr...@gmail.com] Gesendet: Mittwoch, 24. Oktober 2018 15:22 An: Johannes Echterhoff Cc: BaseX Betreff: Re: [basex-talk] determine index use in xquery:eval Hi Johannes, As a single query may yield a large number of query plans if xquery:eval is repeatedly invoked, and as all of these plans will be created at runtime, we haven’t included the plans in the query optimization. However, if a query utilizes the index if it is run standalone, the index will also be used if xquery:eval is run, so you should be able to check some of your queries in advance. If your dynamic queries do not change, a good alternative is to use xquery:invoke; this might simplify your tests. Best, Christian On Wed, Oct 24, 2018 at 3:02 PM Johannes Echterhoff wrote: > > Hello, > > Is there a way to determine if the query that is executed by xquery:eval(…) > makes use of index information? Can we somehow obtain the optimized plan for > that query? > > Best regards, > > Johannes
[basex-talk] determine index use in xquery:eval
Hello, Is there a way to determine if the query that is executed by xquery:eval(...) makes use of index information? Can we somehow obtain the optimized plan for that query? Best regards, Johannes
Re: [basex-talk] Evaluating XPath within custom QueryModule
Hi Christian,
Thank you for investigating. I'll update the methods in my query module as you
suggested.
Best regards,
Johannes
-Ursprüngliche Nachricht-
Von: Christian Grün [mailto:christian.gr...@gmail.com]
Gesendet: Donnerstag, 25. Mai 2017 16:43
An: Johannes Echterhoff <echterh...@interactive-instruments.de>
Cc: BaseX <basex-talk@mailman.uni-konstanz.de>
Betreff: Re: AW: [basex-talk] Evaluating XPath within custom QueryModule
Hi Johannes,
To find the weak spot, I decided to further simplify your example (I have
attached the result).
It’s good to use the internal BaseX value types as Java function argument, such
as Value or ANode. If you use Object, the function input will be copied and
converted to a standard Java type, which is a DOM element in this case
(represented by BXElem). This node will be a full copy of the original node,
without its ancestors (otherwise, it would get huge if you work with large
databases).
Hope this helps,
Christian
On Tue, May 23, 2017 at 4:03 PM, Johannes Echterhoff
<echterh...@interactive-instruments.de> wrote:
> Hi Christian,
>
> Here you are (I hope this is sufficient, actual file content can be
> found further down below):
>
> · Test.java – A simple QueryModule
>
> · Test.xq – The query I execute from within a UnitTest
>
> · Test.xml – The XML that is queried (in my test, it was either
> located in C:/BaseXTest or directly accessible in Test.xq – see
> Test.xq for further details)
>
> · UnitTest – code I used to run the whole test (tested with BaseX
> Versions 8.5.3 and 8.6)
>
> What I noticed: If I do NOT load the XML document from an external
> file, everything works as expected. Then I get this result:
>
>
>
> indirectly: "4979"
>
> on element: "4979"
>
> on element: "4979"
>
> on element: "4979"
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> If the document IS loaded from an external file (which is what I tried
> first), I get the following – incorrect - result:
>
>
>
> empty: empty-sequence()
>
> on element: "4979"
>
> on element: "4979"
>
> empty: empty-sequence()
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Here, something appears to be wrong when evaluating parent elements.
>
>
>
> Best regards,
>
> Johannes
>
>
>
>
>
>
>
> -- Test.java
>
>
>
> import org.basex.api.dom.BXElem;
>
> import org.basex.core.Context;
>
> import org.basex.query.QueryException;
>
> import org.basex.query.QueryModule;
>
> import org.basex.query.QueryProcessor;
>
> import org.basex.query.value.Value;
>
> import org.basex.query.value.node.ANode;
>
> import org.basex.query.value.seq.Empty;
>
>
>
> public class Test extends QueryModule {
>
>
>
> public Value determineAttributeX(Value element) throws
> QueryException {
>
>
>
> String query = "declare namespace nsx
> = 'http://www.example.org/X';declare variable $elmt external; "
>
> + "let
> $elementWithAttributeX := $elmt/ancestor-or-self::*[@attributeX or
> nsx:boundedBy/*/@attributeX][1] "
>
> +
> "return if
> (empty($elementWithAttributeX)) then trace((),'empty: ') "
>
> +
> "else
> if($elementWithAttributeX/@attributeX) then
> trace($elementWithAttributeX/data(@attributeX),'on element: ') else "
>
> +
> "trace($elementWithAttributeX/nsx:boundedBy/*/data(@attributeX),'indirectly:
> ')";
>
>
>
> Context ctx = queryContext.context;
>
>
>
> try (QueryProcessor qp = new
> QueryProcessor(query, ctx)) {
>
>
>
> // Bind to context:
>
> qp.bind("elmt",
> element);
>
> qp.context(element);
>
>
>
> Value value =
> qp.value();
>
>
>
> return value;
>
> }
>
> }
>
>
>
> public Value parse(Object arg) throws QueryException {
>
>
>
>
Re: [basex-talk] Evaluating XPath within custom QueryModule
* @throws IOException if an error occurs while serializing the
results
* @throws QueryException if an error occurs while evaluating
the query
* @throws BaseXException if a database command fails
*/
public static void main(final String[] args) throws
IOException, QueryException {
String xq = "src/test/resources/test.xq";
// Evaluate the specified XQuery
String query = new
String(Files.readAllBytes(Paths.get(xq)));
XQuery xquery = new XQuery(query);
String result = xquery.execute(context);
System.out.println(result);
}
}
Von: Christian Grün [mailto:christian.gr...@gmail.com]
Gesendet: Dienstag, 23. Mai 2017 14:25
An: Johannes Echterhoff <echterh...@interactive-instruments.de>
Cc: BaseX <basex-talk@mailman.uni-konstanz.de>
Betreff: Re: AW: [basex-talk] Evaluating XPath within custom QueryModule
Could you please provide me with a little, seld-contained example?
Thanks in advance,
Christian
Am 23.05.2017 14:14 schrieb "Johannes Echterhoff"
<echterh...@interactive-instruments.de<mailto:echterh...@interactive-instruments.de>>:
Hi Christian,
Thank you. This is very useful.
One issue, though: in my test, it looks like the ancestors of the element that
is processed by the query are not available/accessible. What am I missing?
Best regards,
Johannes
-Ursprüngliche Nachricht-
Von: Christian Grün
[mailto:christian.gr...@gmail.com<mailto:christian.gr...@gmail.com>]
Gesendet: Dienstag, 23. Mai 2017 12:43
An: Johannes Echterhoff
<echterh...@interactive-instruments.de<mailto:echterh...@interactive-instruments.de>>
Cc:
basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>
Betreff: Re: [basex-talk] Evaluating XPath within custom QueryModule
Hi Johannes,
Welcome to the list.
> However, I wonder if it is possible to evaluate the expression also
> within the query module itself, i.e. when method “parse” is called. Is
> there a way to do so?
Absolutely. You find some examples in [1].
If you your class extends the QueryModule class – which is always recommendable
– a function could look as follows:
import org.basex.core.Context;
import org.basex.query.QueryModule;
import org.basex.query.QueryProcessor;
import org.basex.query.value.Value;
public class Module extends QueryModule {
public Value elementName(Value elem) throws Exception {
// Your query
String query =
"declare variable $elem external;" +
"name($elem), (: element bound to variable :)" +
"name(.) (: element bound to context :)";
Context ctx = queryContext.context;
QueryProcessor qp = new QueryProcessor(query, ctx);
// Bind to context:
qp.bind("elem", elem);
qp.context(elem);
Value value = qp.value();
return value;
}
}
Hope this helps,
Christian
[1]
https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/org/basex/examples/local/RunQueries.java
Re: [basex-talk] Evaluating XPath within custom QueryModule
Hi Christian,
Thank you. This is very useful.
One issue, though: in my test, it looks like the ancestors of the element that
is processed by the query are not available/accessible. What am I missing?
Best regards,
Johannes
-Ursprüngliche Nachricht-
Von: Christian Grün [mailto:christian.gr...@gmail.com]
Gesendet: Dienstag, 23. Mai 2017 12:43
An: Johannes Echterhoff <echterh...@interactive-instruments.de>
Cc: basex-talk@mailman.uni-konstanz.de
Betreff: Re: [basex-talk] Evaluating XPath within custom QueryModule
Hi Johannes,
Welcome to the list.
> However, I wonder if it is possible to evaluate the expression also
> within the query module itself, i.e. when method “parse” is called. Is
> there a way to do so?
Absolutely. You find some examples in [1].
If you your class extends the QueryModule class – which is always recommendable
– a function could look as follows:
import org.basex.core.Context;
import org.basex.query.QueryModule;
import org.basex.query.QueryProcessor;
import org.basex.query.value.Value;
public class Module extends QueryModule {
public Value elementName(Value elem) throws Exception {
// Your query
String query =
"declare variable $elem external;" +
"name($elem), (: element bound to variable :)" +
"name(.) (: element bound to context :)";
Context ctx = queryContext.context;
QueryProcessor qp = new QueryProcessor(query, ctx);
// Bind to context:
qp.bind("elem", elem);
qp.context(elem);
Value value = qp.value();
return value;
}
}
Hope this helps,
Christian
[1]
https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/org/basex/examples/local/RunQueries.java
[basex-talk] Evaluating XPath within custom QueryModule
Hello,
I am writing a custom QueryModule that can be used to parse information from a
given XML element. The module has a function like the following:
public ParsedType parse (Object arg) throws QueryException {
if (arg instanceof ANode) {
return utils.parse((ANode) arg);
} else if (arg instanceof BXElem) {
return utils.parse((Element) arg);
} else {
...
}
}
It turned out that parsing requires additional information that can be found
either on the XML element itself or - more importantly for my question - in one
of its ancestors.
The XPath expression to determine the information looks like this: /ancestor-or-self::*[@attributeX or nsX:SomeElement/*/@attributeX][1].
I can evaluate this expression in the XQuery before calling method "parse" from
my custom query module, and add the value of attributeX as an argument.
To do so, I actually have a function to determine the value of attributeX:
declare function local:determineAttributeX($elmt as element()) {
let $elementWithAttributeX := $elmt/ancestor-or-self::*[@attributeX or
nsX:SomeElement/*/@attributeX][1]
return
if (empty($elementWithAttributeX)) then
()
else if($elementWithAttributeX/@attributeX) then
$ elementWithAttributeX/data(@attributeX)
else
$ elementWithAttributeX/nsX:SomeElement/*/data(@attributeX)
};
However, I wonder if it is possible to evaluate the expression also within the
query module itself, i.e. when method "parse" is called. Is there a way to do
so? That would reduce the complexity of the XQuery that calls "parse" (and the
value of "attributeX" is always of interest when "parse" is called, so ideally
"parse" can determine the value itself).
I searched the internet (the archive of this mailing list as well as
stackoverflow) for quite some time but could not find an answer to this
question. Any help would therefore be greatly appreciated.
Best regards,
Johannes Echterhoff
