Re: [basex-talk] db:replace() in DBA resulting in duplicates
> > * Optimize DB > * The first db:replace() via RESTXQ creates a second file with the same > base-uri() > * The second db:replace() via RESTXQ replaces the earlier created second > file (duplicate) > Do you perform the next steps in the database instance that has been copied to another server, or does the problem also happen on the original instance? In the first case, do both servers use the same operating system (either Linux or Windows)? Should I try to optimize the database with 9.1.2 before copying and try > again? Definitely worth trying. If you export your data and create a new database with 9.2.x, and if the problem persists, we could have a look at it. this is a database-structure related issue? > > > -Ursprüngliche Nachricht- > > Von: Christian Grün [mailto:christian.gr...@gmail.com] > > Gesendet: Montag, 29. April 2019 16:00 > > An: Zimmel, Daniel > > Cc: BaseX > > Betreff: Re: [basex-talk] db:replace() in DBA resulting in duplicates > > > > > Still, when I browse the database, the /DB/path only exists once (as > far as I can > > see from the sorting), but querying is giving me two documents, > according to > > db:path() living in the same path. > > > This should never happen, should it? > > > > This is surprising indeed. > > > > Right now, db:add is still faster than db:replace, because the check > > for existing database paths can be completely skipped. This allows > > high-performance database insertions.In a future version of BaseX, I > > would love to get rid of db:add and db:replace and replace these > > functions with a db:update function, which will enforce that each > > document path exists only once in the database. This would surely > > reduce patterns like the one you encountered. > > > > If you still manage to get this reproduced, just come back to us. >
Re: [basex-talk] db:replace() in DBA resulting in duplicates
Follow-up, some details: * Create large (5GB) DB in BaseX GUI * Copy DB folder manually to another server * Optimize DB * The first db:replace() via RESTXQ creates a second file with the same base-uri() * The second db:replace() via RESTXQ replaces the earlier created second file (duplicate) I cannot reproduce with a small newly created db. The initial database was created with BaseX 9.0.2, update operation runs with BaseX 9.2.1 Should I try to optimize the database with 9.1.2 before copying and try again? Perhaps this is a database-structure related issue? > -Ursprüngliche Nachricht- > Von: Christian Grün [mailto:christian.gr...@gmail.com] > Gesendet: Montag, 29. April 2019 16:00 > An: Zimmel, Daniel > Cc: BaseX > Betreff: Re: [basex-talk] db:replace() in DBA resulting in duplicates > > > Still, when I browse the database, the /DB/path only exists once (as far as > > I can > see from the sorting), but querying is giving me two documents, according to > db:path() living in the same path. > > This should never happen, should it? > > This is surprising indeed. > > Right now, db:add is still faster than db:replace, because the check > for existing database paths can be completely skipped. This allows > high-performance database insertions.In a future version of BaseX, I > would love to get rid of db:add and db:replace and replace these > functions with a db:update function, which will enforce that each > document path exists only once in the database. This would surely > reduce patterns like the one you encountered. > > If you still manage to get this reproduced, just come back to us.
Re: [basex-talk] XQUnit Test
Freundliche Grüße Markus Schmid Am 17.05.2019 um 12:32 schrieb Christian Grün : >> my question was not about compile time but rather run time. It is about if >> calculation of X take place multiple times if X is accessed from multiple >> XQuery modules during the test run. > > Maybe you could start with some first tests, then we’ll see which > operations might need to be optimized. > Yes. I’ve started yesterday. I’m using v9.2.2 beta. It works fine. I only asked in order to optimize in advance. > > >> Am 17.05.2019 um 11:58 schrieb Christian Grün : >> If there are multiple XQuery Moduls (testsuites) with references to a module variable X in a helper xqy File, will the variable X be created for each XQuery Modul or only once when it is used the first time in any of the XQuery modules? >>> >>> If several of your modules import the same library module that >>> contains a variable declaration, the library module will be parsed and >>> compiled multiple times. Does this answer your question? >>> >>> On Fri, May 17, 2019 at 10:45 AM Markus Schmid wrote: I see. Thank you for the quick response. If there are multiple XQuery Moduls (testsuites) with references to a module variable X in a helper xqy File, will the variable X be created for each XQuery Modul or only once when it is used the first time in any of the XQuery modules? Thank you. Markus > Am 17.05.2019 um 09:16 schrieb Christian Grün : > > Hi Markus, > > Yes, that’s possible: If you specify a directory as argument, all > XQuery modules in that directory will be parsed, and a testsuite > element will be created for each module that contains tests. See [1] > for more information. > > Best, > Christian > > [1] http://docs.basex.org/wiki/Commands#TEST > > > >> On Fri, May 17, 2019 at 8:57 AM Markus Schmid >> wrote: >> >> Hi >> >> is it possible to run multiple testsuites with one TEST command call >> from the basex console? Or is there always only one element >> in the result? >> >> Markus >> >> __ >>
Re: [basex-talk] XQUnit Test
> my question was not about compile time but rather run time. It is about if > calculation of X take place multiple times if X is accessed from multiple > XQuery modules during the test run. Maybe you could start with some first tests, then we’ll see which operations might need to be optimized. > Am 17.05.2019 um 11:58 schrieb Christian Grün : > > >> If there are multiple XQuery Moduls (testsuites) with references to a > >> module variable X in a helper xqy File, will the variable X be created for > >> each XQuery Modul or only once when it is used the first time in any of > >> the XQuery modules? > > > > If several of your modules import the same library module that > > contains a variable declaration, the library module will be parsed and > > compiled multiple times. Does this answer your question? > > > > > >> On Fri, May 17, 2019 at 10:45 AM Markus Schmid > >> wrote: > >> > >> I see. Thank you for the quick response. > >> > >> If there are multiple XQuery Moduls (testsuites) with references to a > >> module variable X in a helper xqy File, will the variable X be created for > >> each XQuery Modul or only once when it is used the first time in any of > >> the XQuery modules? > >> > >> Thank you. > >> > >> Markus > >> > >>> Am 17.05.2019 um 09:16 schrieb Christian Grün : > >>> > >>> Hi Markus, > >>> > >>> Yes, that’s possible: If you specify a directory as argument, all > >>> XQuery modules in that directory will be parsed, and a testsuite > >>> element will be created for each module that contains tests. See [1] > >>> for more information. > >>> > >>> Best, > >>> Christian > >>> > >>> [1] http://docs.basex.org/wiki/Commands#TEST > >>> > >>> > >>> > On Fri, May 17, 2019 at 8:57 AM Markus Schmid > wrote: > > Hi > > is it possible to run multiple testsuites with one TEST command call > from the basex console? Or is there always only one element > in the result? > > Markus > > __ > >> >
Re: [basex-talk] XQUnit Test
Hi Christian my question was not about compile time but rather run time. It is about if calculation of X take place multiple times if X is accessed from multiple XQuery modules during the test run. Kind Regards Markus Am 17.05.2019 um 11:58 schrieb Christian Grün : >> If there are multiple XQuery Moduls (testsuites) with references to a module >> variable X in a helper xqy File, will the variable X be created for each >> XQuery Modul or only once when it is used the first time in any of the >> XQuery modules? > > If several of your modules import the same library module that > contains a variable declaration, the library module will be parsed and > compiled multiple times. Does this answer your question? > > >> On Fri, May 17, 2019 at 10:45 AM Markus Schmid wrote: >> >> I see. Thank you for the quick response. >> >> If there are multiple XQuery Moduls (testsuites) with references to a module >> variable X in a helper xqy File, will the variable X be created for each >> XQuery Modul or only once when it is used the first time in any of the >> XQuery modules? >> >> Thank you. >> >> Markus >> >>> Am 17.05.2019 um 09:16 schrieb Christian Grün : >>> >>> Hi Markus, >>> >>> Yes, that’s possible: If you specify a directory as argument, all >>> XQuery modules in that directory will be parsed, and a testsuite >>> element will be created for each module that contains tests. See [1] >>> for more information. >>> >>> Best, >>> Christian >>> >>> [1] http://docs.basex.org/wiki/Commands#TEST >>> >>> >>> On Fri, May 17, 2019 at 8:57 AM Markus Schmid wrote: Hi is it possible to run multiple testsuites with one TEST command call from the basex console? Or is there always only one element in the result? Markus __ >>
Re: [basex-talk] XQUnit Test
> If there are multiple XQuery Moduls (testsuites) with references to a module > variable X in a helper xqy File, will the variable X be created for each > XQuery Modul or only once when it is used the first time in any of the XQuery > modules? If several of your modules import the same library module that contains a variable declaration, the library module will be parsed and compiled multiple times. Does this answer your question? On Fri, May 17, 2019 at 10:45 AM Markus Schmid wrote: > > I see. Thank you for the quick response. > > If there are multiple XQuery Moduls (testsuites) with references to a module > variable X in a helper xqy File, will the variable X be created for each > XQuery Modul or only once when it is used the first time in any of the XQuery > modules? > > Thank you. > > Markus > > > Am 17.05.2019 um 09:16 schrieb Christian Grün : > > > > Hi Markus, > > > > Yes, that’s possible: If you specify a directory as argument, all > > XQuery modules in that directory will be parsed, and a testsuite > > element will be created for each module that contains tests. See [1] > > for more information. > > > > Best, > > Christian > > > > [1] http://docs.basex.org/wiki/Commands#TEST > > > > > > > >> On Fri, May 17, 2019 at 8:57 AM Markus Schmid wrote: > >> > >> Hi > >> > >> is it possible to run multiple testsuites with one TEST command call from > >> the basex console? Or is there always only one element in the > >> result? > >> > >> Markus > >> > >> __ > >> >
Re: [basex-talk] XQUnit Test
I see. Thank you for the quick response. If there are multiple XQuery Moduls (testsuites) with references to a module variable X in a helper xqy File, will the variable X be created for each XQuery Modul or only once when it is used the first time in any of the XQuery modules? Thank you. Markus > Am 17.05.2019 um 09:16 schrieb Christian Grün : > > Hi Markus, > > Yes, that’s possible: If you specify a directory as argument, all > XQuery modules in that directory will be parsed, and a testsuite > element will be created for each module that contains tests. See [1] > for more information. > > Best, > Christian > > [1] http://docs.basex.org/wiki/Commands#TEST > > > >> On Fri, May 17, 2019 at 8:57 AM Markus Schmid wrote: >> >> Hi >> >> is it possible to run multiple testsuites with one TEST command call from >> the basex console? Or is there always only one element in the >> result? >> >> Markus >> >> __ >>
Re: [basex-talk] Optimizer and JSON/XML response
Hi Christian,
I'll try to answer your questions inline ...
On 16/05/19 19:38, Christian Grün wrote:
Hi Marco,
We haven’t experienced such a behavior by ourselves, even in very
complex applications. Is the faulty behavior reproducible?
Only with the exact configuration of our current applciation...
Does it
make a difference if you change the value of the PARSERESTXQ option
[1]?
No, it doesn't. we tried higher values (wrt the default 3) and also 0
and -1 ...
And is it already the http:send-request call that returns the
wrong result, or is it the actual RESTXQ function in which the
http:send-request call is contained?
The http:send-request returns the second element of the response in map
format. We discovered it actually because a following XPath expression
raised an error because it hit a map instead of a document as expected...
You could additionally output the
result of the function call to standard error (via fn:trace) or to a
file (via file:write) and compare the output in both of your setups.
We did all sort of tests and it's the http:send-request without doubts
that already returns a map...
If there’s any chance to reproduce this (e.g. by creating hundreds or
thousands of dummy RESTXQ functions), that’d be fine.
I've tried to generate #FILES * #FUNCT number of RESTXQS with
increasing number of files and functions (up to OOM exceptions with
default -Xms=2G) but I'm unable to reproduce... [A][B]
So I suspect that the problem might be in he file structure we use
softlinks to hook up restxq folders, a REPO full of implementation
modules so may be the resolution of all this relations might be the issue?
Clueless at the moment ...
M.
[A] python script to generate extra RESTXQS
#!/bin/python3
FILES = 100
RESTXQS = 1000
qtempl = open("template.txt").read()
for f in range(FILES):
outf = open("file" + str(f) + ".xqm", "w")
qout = 'module namespace _ = "urn:dummy:' + str(f) + '";\n\n';
for q in range(RESTXQS):
qout += "\n" + qtempl.format(path=(str(f)+"_"+str(q)))
outf.write(qout)
[B] template query stored in template.txt
declare
%rest:path("test/{path}")
%rest:GET
function _:f{path}() {{
"{path}"
}};
Cheers,
Christian
[1] http://docs.basex.org/wiki/Options#PARSERESTXQ
On Thu, May 16, 2019 at 1:27 PM Marco Lettere wrote:
Hello everyone,
we came accross an inconsistency which we are not able to find an
explanation for and giving up after having spent on it the whole morning...
From a restxq we call an external REST service (InfluxDB) like shown in
[1]. This usually works as expected returning a response forged as in [2].
When the same RESTXQ is deployed in a much more complex application
(with several RESTXQ modules and REPO modules) the first call (which
takes a lot of time because it incurs the initial compiling delay)
returns [3] instead of [2]. From the second call onwards the result
stabilizes correctly to [2].
If I call any other RESTXQ function as a sort of warm-up (spending there
all the initial delay) [1] behaves correctly returning always [2].
Do you please have a clue on why this happens?
We noticed it for the first time because we started to use service jobs
which cause this access pattern ...
Thanks for any help.
[1]
module namespace test = 'test';
declare
%rest:path("test")
%rest:GET
function test:test() {
let $url :=
"http://localhost:8086/query?pretty=false&chunked=false&db=nestway&q=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27";
let $results := http:send-request(, $url)
return $results
};
[2]
http://expath.org/ns/http-client";
status="200" message="OK">
<_ type="object">
<_ type="object">
frontoffice
<_>time
<_>consumable
<_>customer
<_>guests
<_>type
<_>user
<_ type="array">
<_>2019-05-10T21:55:00Z
<_>access
<_>c9a83ce3-c938-4677-a0df-72bc96354509
<_ type="number">0
<_>starttimer
<_>d50de945-23ed-4b33-84e8-8ef533a96fb7
[3]
http://expath.org/ns/http-client";
status="200" message="OK">
map {
"results": [map {
"series": [map {
"values": [["2019-05-10T21:55:00Z", "access",
"c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer",
"d50de945-23ed-4b33-84e8-8ef533a96fb7"]],
"name": "frontoffice",
"columns": ["time", "consumable", "customer", "guests", "type",
"user"]
Re: [basex-talk] XQUnit Test
Hi Markus, Yes, that’s possible: If you specify a directory as argument, all XQuery modules in that directory will be parsed, and a testsuite element will be created for each module that contains tests. See [1] for more information. Best, Christian [1] http://docs.basex.org/wiki/Commands#TEST On Fri, May 17, 2019 at 8:57 AM Markus Schmid wrote: > > Hi > > is it possible to run multiple testsuites with one TEST command call from the > basex console? Or is there always only one element in the > result? > > Markus > > __ >
