Re: [basex-talk] Optimizer and JSON/XML response
Hi Christian, I'll try to answer your questions inline ... On 16/05/19 19:38, Christian Grün wrote: Hi Marco, We haven’t experienced such a behavior by ourselves, even in very complex applications. Is the faulty behavior reproducible? Only with the exact configuration of our current applciation... Does it make a difference if you change the value of the PARSERESTXQ option [1]? No, it doesn't. we tried higher values (wrt the default 3) and also 0 and -1 ... And is it already the http:send-request call that returns the wrong result, or is it the actual RESTXQ function in which the http:send-request call is contained? The http:send-request returns the second element of the response in map format. We discovered it actually because a following XPath expression raised an error because it hit a map instead of a document as expected... You could additionally output the result of the function call to standard error (via fn:trace) or to a file (via file:write) and compare the output in both of your setups. We did all sort of tests and it's the http:send-request without doubts that already returns a map... If there’s any chance to reproduce this (e.g. by creating hundreds or thousands of dummy RESTXQ functions), that’d be fine. I've tried to generate #FILES * #FUNCT number of RESTXQS with increasing number of files and functions (up to OOM exceptions with default -Xms=2G) but I'm unable to reproduce... [A][B] So I suspect that the problem might be in he file structure we use softlinks to hook up restxq folders, a REPO full of implementation modules so may be the resolution of all this relations might be the issue? Clueless at the moment ... M. [A] python script to generate extra RESTXQS #!/bin/python3 FILES = 100 RESTXQS = 1000 qtempl = open("template.txt").read() for f in range(FILES): outf = open("file" + str(f) + ".xqm", "w") qout = 'module namespace _ = "urn:dummy:' + str(f) + '";\n\n'; for q in range(RESTXQS): qout += "\n" + qtempl.format(path=(str(f)+"_"+str(q))) outf.write(qout) [B] template query stored in template.txt declare %rest:path("test/{path}") %rest:GET function _:f{path}() {{ "{path}" }}; Cheers, Christian [1] http://docs.basex.org/wiki/Options#PARSERESTXQ On Thu, May 16, 2019 at 1:27 PM Marco Lettere wrote: Hello everyone, we came accross an inconsistency which we are not able to find an explanation for and giving up after having spent on it the whole morning... From a restxq we call an external REST service (InfluxDB) like shown in [1]. This usually works as expected returning a response forged as in [2]. When the same RESTXQ is deployed in a much more complex application (with several RESTXQ modules and REPO modules) the first call (which takes a lot of time because it incurs the initial compiling delay) returns [3] instead of [2]. From the second call onwards the result stabilizes correctly to [2]. If I call any other RESTXQ function as a sort of warm-up (spending there all the initial delay) [1] behaves correctly returning always [2]. Do you please have a clue on why this happens? We noticed it for the first time because we started to use service jobs which cause this access pattern ... Thanks for any help. [1] module namespace test = 'test'; declare %rest:path("test") %rest:GET function test:test() { let $url := "http://localhost:8086/query?pretty=false&chunked=false&db=nestway&q=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27"; let $results := http:send-request(, $url) return $results }; [2] http://expath.org/ns/http-client"; status="200" message="OK"> <_ type="object"> <_ type="object"> frontoffice <_>time <_>consumable <_>customer <_>guests <_>type <_>user <_ type="array"> <_>2019-05-10T21:55:00Z <_>access <_>c9a83ce3-c938-4677-a0df-72bc96354509 <_ type="number">0 <_>starttimer <_>d50de945-23ed-4b33-84e8-8ef533a96fb7 [3] http://expath.org/ns/http-client"; status="200" message="OK"> map { "results": [map { "series": [map { "values": [["2019-05-10T21:55:00Z", "access", "c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer", "d50de945-23ed-4b33-84e8-8ef533a96fb7"]], "name": "frontoffice", "columns": ["time", "consumable", "customer", "guests", "type", "user"]
Re: [basex-talk] Optimizer and JSON/XML response
Hi Marco, We haven’t experienced such a behavior by ourselves, even in very complex applications. Is the faulty behavior reproducible? Does it make a difference if you change the value of the PARSERESTXQ option [1]? And is it already the http:send-request call that returns the wrong result, or is it the actual RESTXQ function in which the http:send-request call is contained? You could additionally output the result of the function call to standard error (via fn:trace) or to a file (via file:write) and compare the output in both of your setups. If there’s any chance to reproduce this (e.g. by creating hundreds or thousands of dummy RESTXQ functions), that’d be fine. Cheers, Christian [1] http://docs.basex.org/wiki/Options#PARSERESTXQ On Thu, May 16, 2019 at 1:27 PM Marco Lettere wrote: > > Hello everyone, > > we came accross an inconsistency which we are not able to find an > explanation for and giving up after having spent on it the whole morning... > > From a restxq we call an external REST service (InfluxDB) like shown in > [1]. This usually works as expected returning a response forged as in [2]. > > When the same RESTXQ is deployed in a much more complex application > (with several RESTXQ modules and REPO modules) the first call (which > takes a lot of time because it incurs the initial compiling delay) > returns [3] instead of [2]. From the second call onwards the result > stabilizes correctly to [2]. > > If I call any other RESTXQ function as a sort of warm-up (spending there > all the initial delay) [1] behaves correctly returning always [2]. > > Do you please have a clue on why this happens? > > We noticed it for the first time because we started to use service jobs > which cause this access pattern ... > > Thanks for any help. > > [1] > > module namespace test = 'test'; > > declare >%rest:path("test") >%rest:GET > function test:test() { >let $url := > "http://localhost:8086/query?pretty=false&chunked=false&db=nestway&q=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27"; >let $results := http:send-request(, $url) >return $results > }; > > [2] > > http://expath.org/ns/http-client"; > status="200" message="OK"> > > > > > value="95b729c7-77ca-11e9-8065-"/> > > > > > > <_ type="object"> > > <_ type="object"> > frontoffice > > <_>time > <_>consumable > <_>customer > <_>guests > <_>type > <_>user > > > <_ type="array"> > <_>2019-05-10T21:55:00Z > <_>access > <_>c9a83ce3-c938-4677-a0df-72bc96354509 > <_ type="number">0 > <_>starttimer > <_>d50de945-23ed-4b33-84e8-8ef533a96fb7 > > > > > > > > > > [3] > > http://expath.org/ns/http-client"; > status="200" message="OK"> > > > > > value="7957030a-77cc-11e9-8066-"/> > > > > map { >"results": [map { > "series": [map { >"values": [["2019-05-10T21:55:00Z", "access", > "c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer", > "d50de945-23ed-4b33-84e8-8ef533a96fb7"]], >"name": "frontoffice", >"columns": ["time", "consumable", "customer", "guests", "type", > "user"] > }] >}] > } >
Re: [basex-talk] Optimizer and JSON/XML response
Just to add up ... BaseX version is 9.0.2. Moreover when running from a schedule service job the only way to avoid the issue described is to always first call another RESTXQ and then [1]. Which is somehow what I expect if services are run in their own process/jvm... Despite the subject of the mail it looks like an issue in the RESTXQ parser/compiler rather than an optimizer issue... M. On 16/05/19 13:27, Marco Lettere wrote: Hello everyone, we came accross an inconsistency which we are not able to find an explanation for and giving up after having spent on it the whole morning... From a restxq we call an external REST service (InfluxDB) like shown in [1]. This usually works as expected returning a response forged as in [2]. When the same RESTXQ is deployed in a much more complex application (with several RESTXQ modules and REPO modules) the first call (which takes a lot of time because it incurs the initial compiling delay) returns [3] instead of [2]. From the second call onwards the result stabilizes correctly to [2]. If I call any other RESTXQ function as a sort of warm-up (spending there all the initial delay) [1] behaves correctly returning always [2]. Do you please have a clue on why this happens? We noticed it for the first time because we started to use service jobs which cause this access pattern ... Thanks for any help. [1] module namespace test = 'test'; declare %rest:path("test") %rest:GET function test:test() { let $url := "http://localhost:8086/query?pretty=false&chunked=false&db=nestway&q=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27"; let $results := http:send-request(, $url) return $results }; [2] http://expath.org/ns/http-client"; status="200" message="OK"> value="95b729c7-77ca-11e9-8065-"/> <_ type="object"> <_ type="object"> frontoffice <_>time <_>consumable <_>customer <_>guests <_>type <_>user <_ type="array"> <_>2019-05-10T21:55:00Z <_>access <_>c9a83ce3-c938-4677-a0df-72bc96354509 <_ type="number">0 <_>starttimer <_>d50de945-23ed-4b33-84e8-8ef533a96fb7 [3] http://expath.org/ns/http-client"; status="200" message="OK"> value="7957030a-77cc-11e9-8066-"/> map { "results": [map { "series": [map { "values": [["2019-05-10T21:55:00Z", "access", "c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer", "d50de945-23ed-4b33-84e8-8ef533a96fb7"]], "name": "frontoffice", "columns": ["time", "consumable", "customer", "guests", "type", "user"] }] }] }
[basex-talk] Optimizer and JSON/XML response
Hello everyone, we came accross an inconsistency which we are not able to find an explanation for and giving up after having spent on it the whole morning... From a restxq we call an external REST service (InfluxDB) like shown in [1]. This usually works as expected returning a response forged as in [2]. When the same RESTXQ is deployed in a much more complex application (with several RESTXQ modules and REPO modules) the first call (which takes a lot of time because it incurs the initial compiling delay) returns [3] instead of [2]. From the second call onwards the result stabilizes correctly to [2]. If I call any other RESTXQ function as a sort of warm-up (spending there all the initial delay) [1] behaves correctly returning always [2]. Do you please have a clue on why this happens? We noticed it for the first time because we started to use service jobs which cause this access pattern ... Thanks for any help. [1] module namespace test = 'test'; declare %rest:path("test") %rest:GET function test:test() { let $url := "http://localhost:8086/query?pretty=false&chunked=false&db=nestway&q=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27"; let $results := http:send-request(, $url) return $results }; [2] http://expath.org/ns/http-client"; status="200" message="OK"> value="95b729c7-77ca-11e9-8065-"/> <_ type="object"> <_ type="object"> frontoffice <_>time <_>consumable <_>customer <_>guests <_>type <_>user <_ type="array"> <_>2019-05-10T21:55:00Z <_>access <_>c9a83ce3-c938-4677-a0df-72bc96354509 <_ type="number">0 <_>starttimer <_>d50de945-23ed-4b33-84e8-8ef533a96fb7 [3] http://expath.org/ns/http-client"; status="200" message="OK"> value="7957030a-77cc-11e9-8066-"/> map { "results": [map { "series": [map { "values": [["2019-05-10T21:55:00Z", "access", "c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer", "d50de945-23ed-4b33-84e8-8ef533a96fb7"]], "name": "frontoffice", "columns": ["time", "consumable", "customer", "guests", "type", "user"] }] }] }