Re: [basex-talk] Optimizer and JSON/XML response
Hi Christian,
I'll try to answer your questions inline ...
On 16/05/19 19:38, Christian Grün wrote:
Hi Marco,
We haven’t experienced such a behavior by ourselves, even in very
complex applications. Is the faulty behavior reproducible?
Only with the exact configuration of our current applciation...
Does it
make a difference if you change the value of the PARSERESTXQ option
[1]?
No, it doesn't. we tried higher values (wrt the default 3) and also 0
and -1 ...
And is it already the http:send-request call that returns the
wrong result, or is it the actual RESTXQ function in which the
http:send-request call is contained?
The http:send-request returns the second element of the response in map
format. We discovered it actually because a following XPath expression
raised an error because it hit a map instead of a document as expected...
You could additionally output the
result of the function call to standard error (via fn:trace) or to a
file (via file:write) and compare the output in both of your setups.
We did all sort of tests and it's the http:send-request without doubts
that already returns a map...
If there’s any chance to reproduce this (e.g. by creating hundreds or
thousands of dummy RESTXQ functions), that’d be fine.
I've tried to generate #FILES * #FUNCT number of RESTXQS with
increasing number of files and functions (up to OOM exceptions with
default -Xms=2G) but I'm unable to reproduce... [A][B]
So I suspect that the problem might be in he file structure we use
softlinks to hook up restxq folders, a REPO full of implementation
modules so may be the resolution of all this relations might be the issue?
Clueless at the moment ...
M.
[A] python script to generate extra RESTXQS
#!/bin/python3
FILES = 100
RESTXQS = 1000
qtempl = open("template.txt").read()
for f in range(FILES):
outf = open("file" + str(f) + ".xqm", "w")
qout = 'module namespace _ = "urn:dummy:' + str(f) + '";\n\n';
for q in range(RESTXQS):
qout += "\n" + qtempl.format(path=(str(f)+"_"+str(q)))
outf.write(qout)
[B] template query stored in template.txt
declare
%rest:path("test/{path}")
%rest:GET
function _:f{path}() {{
"{path}"
}};
Cheers,
Christian
[1] http://docs.basex.org/wiki/Options#PARSERESTXQ
On Thu, May 16, 2019 at 1:27 PM Marco Lettere wrote:
Hello everyone,
we came accross an inconsistency which we are not able to find an
explanation for and giving up after having spent on it the whole morning...
From a restxq we call an external REST service (InfluxDB) like shown in
[1]. This usually works as expected returning a response forged as in [2].
When the same RESTXQ is deployed in a much more complex application
(with several RESTXQ modules and REPO modules) the first call (which
takes a lot of time because it incurs the initial compiling delay)
returns [3] instead of [2]. From the second call onwards the result
stabilizes correctly to [2].
If I call any other RESTXQ function as a sort of warm-up (spending there
all the initial delay) [1] behaves correctly returning always [2].
Do you please have a clue on why this happens?
We noticed it for the first time because we started to use service jobs
which cause this access pattern ...
Thanks for any help.
[1]
module namespace test = 'test';
declare
%rest:path("test")
%rest:GET
function test:test() {
let $url :=
"http://localhost:8086/query?pretty=false&chunked=false&db=nestway&q=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27";
let $results := http:send-request(, $url)
return $results
};
[2]
http://expath.org/ns/http-client";
status="200" message="OK">
<_ type="object">
<_ type="object">
frontoffice
<_>time
<_>consumable
<_>customer
<_>guests
<_>type
<_>user
<_ type="array">
<_>2019-05-10T21:55:00Z
<_>access
<_>c9a83ce3-c938-4677-a0df-72bc96354509
<_ type="number">0
<_>starttimer
<_>d50de945-23ed-4b33-84e8-8ef533a96fb7
[3]
http://expath.org/ns/http-client";
status="200" message="OK">
map {
"results": [map {
"series": [map {
"values": [["2019-05-10T21:55:00Z", "access",
"c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer",
"d50de945-23ed-4b33-84e8-8ef533a96fb7"]],
"name": "frontoffice",
"columns": ["time", "consumable", "customer", "guests", "type",
"user"]
Re: [basex-talk] Optimizer and JSON/XML response
Hi Marco,
We haven’t experienced such a behavior by ourselves, even in very
complex applications. Is the faulty behavior reproducible? Does it
make a difference if you change the value of the PARSERESTXQ option
[1]? And is it already the http:send-request call that returns the
wrong result, or is it the actual RESTXQ function in which the
http:send-request call is contained? You could additionally output the
result of the function call to standard error (via fn:trace) or to a
file (via file:write) and compare the output in both of your setups.
If there’s any chance to reproduce this (e.g. by creating hundreds or
thousands of dummy RESTXQ functions), that’d be fine.
Cheers,
Christian
[1] http://docs.basex.org/wiki/Options#PARSERESTXQ
On Thu, May 16, 2019 at 1:27 PM Marco Lettere wrote:
>
> Hello everyone,
>
> we came accross an inconsistency which we are not able to find an
> explanation for and giving up after having spent on it the whole morning...
>
> From a restxq we call an external REST service (InfluxDB) like shown in
> [1]. This usually works as expected returning a response forged as in [2].
>
> When the same RESTXQ is deployed in a much more complex application
> (with several RESTXQ modules and REPO modules) the first call (which
> takes a lot of time because it incurs the initial compiling delay)
> returns [3] instead of [2]. From the second call onwards the result
> stabilizes correctly to [2].
>
> If I call any other RESTXQ function as a sort of warm-up (spending there
> all the initial delay) [1] behaves correctly returning always [2].
>
> Do you please have a clue on why this happens?
>
> We noticed it for the first time because we started to use service jobs
> which cause this access pattern ...
>
> Thanks for any help.
>
> [1]
>
> module namespace test = 'test';
>
> declare
>%rest:path("test")
>%rest:GET
> function test:test() {
>let $url :=
> "http://localhost:8086/query?pretty=false&chunked=false&db=nestway&q=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27";
>let $results := http:send-request(, $url)
>return $results
> };
>
> [2]
>
> http://expath.org/ns/http-client";
> status="200" message="OK">
>
>
>
>
> value="95b729c7-77ca-11e9-8065-"/>
>
>
>
>
>
> <_ type="object">
>
> <_ type="object">
> frontoffice
>
> <_>time
> <_>consumable
> <_>customer
> <_>guests
> <_>type
> <_>user
>
>
> <_ type="array">
> <_>2019-05-10T21:55:00Z
> <_>access
> <_>c9a83ce3-c938-4677-a0df-72bc96354509
> <_ type="number">0
> <_>starttimer
> <_>d50de945-23ed-4b33-84e8-8ef533a96fb7
>
>
>
>
>
>
>
>
>
> [3]
>
> http://expath.org/ns/http-client";
> status="200" message="OK">
>
>
>
>
> value="7957030a-77cc-11e9-8066-"/>
>
>
>
> map {
>"results": [map {
> "series": [map {
>"values": [["2019-05-10T21:55:00Z", "access",
> "c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer",
> "d50de945-23ed-4b33-84e8-8ef533a96fb7"]],
>"name": "frontoffice",
>"columns": ["time", "consumable", "customer", "guests", "type",
> "user"]
> }]
>}]
> }
>
Re: [basex-talk] Optimizer and JSON/XML response
Just to add up ...
BaseX version is 9.0.2.
Moreover when running from a schedule service job the only way to avoid
the issue described is to always first call another RESTXQ and then [1].
Which is somehow what I expect if services are run in their own
process/jvm...
Despite the subject of the mail it looks like an issue in the RESTXQ
parser/compiler rather than an optimizer issue...
M.
On 16/05/19 13:27, Marco Lettere wrote:
Hello everyone,
we came accross an inconsistency which we are not able to find an
explanation for and giving up after having spent on it the whole
morning...
From a restxq we call an external REST service (InfluxDB) like shown
in [1]. This usually works as expected returning a response forged as
in [2].
When the same RESTXQ is deployed in a much more complex application
(with several RESTXQ modules and REPO modules) the first call (which
takes a lot of time because it incurs the initial compiling delay)
returns [3] instead of [2]. From the second call onwards the result
stabilizes correctly to [2].
If I call any other RESTXQ function as a sort of warm-up (spending
there all the initial delay) [1] behaves correctly returning always [2].
Do you please have a clue on why this happens?
We noticed it for the first time because we started to use service
jobs which cause this access pattern ...
Thanks for any help.
[1]
module namespace test = 'test';
declare
%rest:path("test")
%rest:GET
function test:test() {
let $url :=
"http://localhost:8086/query?pretty=false&chunked=false&db=nestway&q=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27";
let $results := http:send-request(, $url)
return $results
};
[2]
http://expath.org/ns/http-client";
status="200" message="OK">
value="95b729c7-77ca-11e9-8065-"/>
<_ type="object">
<_ type="object">
frontoffice
<_>time
<_>consumable
<_>customer
<_>guests
<_>type
<_>user
<_ type="array">
<_>2019-05-10T21:55:00Z
<_>access
<_>c9a83ce3-c938-4677-a0df-72bc96354509
<_ type="number">0
<_>starttimer
<_>d50de945-23ed-4b33-84e8-8ef533a96fb7
[3]
http://expath.org/ns/http-client";
status="200" message="OK">
value="7957030a-77cc-11e9-8066-"/>
map {
"results": [map {
"series": [map {
"values": [["2019-05-10T21:55:00Z", "access",
"c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer",
"d50de945-23ed-4b33-84e8-8ef533a96fb7"]],
"name": "frontoffice",
"columns": ["time", "consumable", "customer", "guests", "type",
"user"]
}]
}]
}
[basex-talk] Optimizer and JSON/XML response
Hello everyone,
we came accross an inconsistency which we are not able to find an
explanation for and giving up after having spent on it the whole morning...
From a restxq we call an external REST service (InfluxDB) like shown in
[1]. This usually works as expected returning a response forged as in [2].
When the same RESTXQ is deployed in a much more complex application
(with several RESTXQ modules and REPO modules) the first call (which
takes a lot of time because it incurs the initial compiling delay)
returns [3] instead of [2]. From the second call onwards the result
stabilizes correctly to [2].
If I call any other RESTXQ function as a sort of warm-up (spending there
all the initial delay) [1] behaves correctly returning always [2].
Do you please have a clue on why this happens?
We noticed it for the first time because we started to use service jobs
which cause this access pattern ...
Thanks for any help.
[1]
module namespace test = 'test';
declare
%rest:path("test")
%rest:GET
function test:test() {
let $url :=
"http://localhost:8086/query?pretty=false&chunked=false&db=nestway&q=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27";
let $results := http:send-request(, $url)
return $results
};
[2]
http://expath.org/ns/http-client";
status="200" message="OK">
value="95b729c7-77ca-11e9-8065-"/>
<_ type="object">
<_ type="object">
frontoffice
<_>time
<_>consumable
<_>customer
<_>guests
<_>type
<_>user
<_ type="array">
<_>2019-05-10T21:55:00Z
<_>access
<_>c9a83ce3-c938-4677-a0df-72bc96354509
<_ type="number">0
<_>starttimer
<_>d50de945-23ed-4b33-84e8-8ef533a96fb7
[3]
http://expath.org/ns/http-client";
status="200" message="OK">
value="7957030a-77cc-11e9-8066-"/>
map {
"results": [map {
"series": [map {
"values": [["2019-05-10T21:55:00Z", "access",
"c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer",
"d50de945-23ed-4b33-84e8-8ef533a96fb7"]],
"name": "frontoffice",
"columns": ["time", "consumable", "customer", "guests", "type",
"user"]
}]
}]
}
