Hi Christian,

Thank you for your reply, you always help me.

*- Can you share the query with us?*. Yes, of course.

The query is pretty much the same that I've sent to Fabrice:

*(# db:copynode false #) {*
*  for $case in
doc('file_A.xml')/trademark-applications-daily/application-information/file-segments/action-keys/case-file[case-file-header/status-code=(410,413,616,620,624,625,630,631,638,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,663,664,665,666,667,668,672,680,681,682,686,688,689,690,692,693,694,700,701,702,703,704,705,706,707,708,717,718,719,720,721,722,724,725,730,731,732,733,734,739,740,744,745,746,748,752,753,756,757,760,762,763,764,765,766,771,772,773,774,775,777,778,779,780,790,794,800,801,802,803,804,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,969,973)]
 *
*  return insert node $case as last into doc("US")*
*}*


And you're totally right about the security concerns: I don't offer to
anyone the access to the RESTXQ endpoint, it's only for me but from another
app ( on this case an Azure Web job instance ).

It's easy for me to do some maintenance tasks from a C# app, using the
RESTXQ, instead of directly on some BaseX GUI.

At the beginning I tried to do this:

*let $path:="A:\sources\xml\"*

*let $files:=*
*let $parts:=("US00","US01")*
*for $part in $parts*
*let $dir:= $path || $part*
*for $file in file:list($dir)*
*return $path || $part || "\" || $file*

*return*
*(# db:copynode false #) {*
*  for $file in $files  *
*  for $case in
doc($file)/trademark-applications-daily/application-information/file-segments/action-keys/case-file[case-file-header/status-code=(410,413,616,620,624,625,630,631,638,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,663,664,665,666,667,668,672,680,681,682,686,688,689,690,692,693,694,700,701,702,703,704,705,706,707,708,717,718,719,720,721,722,724,725,730,731,732,733,734,739,740,744,745,746,748,752,753,756,757,760,762,763,764,765,766,771,772,773,774,775,777,778,779,780,790,794,800,801,802,803,804,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,969,973)]
 *
*  return insert node $case as last into doc("US")*
*}*

And, of course, this ran out of memory in a matter of seconds. ( there are
more than 100 files )

So, I thought:

-* "Maybe, if I execute each of them using the RESTXQ the memory will be
released after each execution..." *

But seems I'm still missing something because the result is the same. But,
at least, I can finish the process restarting the HTTP server when it fails.

I've uploaded two of the XML files in case you want to test something with
them.[1]


Por supuesto que ayuda!
Regards,
Sebastian.

[1] https://easyupload.io/sd2ilu

On Fri, May 29, 2020 at 8:38 AM Christian Grün <christian.gr...@gmail.com>
wrote:

> Hi Sebastian,
>
> In general, artifacts of updating queries should be cleaned up after
> the execution.
>
> The stack trace indicates that a very large main-memory database
> instance was created by one of your queries that exceeded the memory
> limits. Can you share the query with us?
>
> As XQuery is more powerful than pure query languages, it may be risky
> to allow the execution of arbitrary client code. First of all, you
> should ensure that the code is run with limited user permissions;
> otherwise, your system can be wiped out by a single file:delete('/',
> true()) call. Next, you could try to limit memory usage and execution
> time via the 'memory' and 'timeout' parameters [1]. However, as it’s
> close to impossible to reliably control the memory consumption of
> single threads in Java, I would rather suggest providing predefined
> user queries.
>
> Espero que esto ayude,
> Christian
>
> [1] https://docs.basex.org/wiki/XQuery_Module#xquery:eval
>
>
>
> On Thu, May 28, 2020 at 8:39 PM Sebastian Guerrero <chap...@gmail.com>
> wrote:
> >
> > Hi BaseX team!
> >
> > A quick question.
> >
> > Is there some known bug/common setting missing for RESTXQ and memory
> problems?
> >
> > I have this simple module ( into /webapp, a .xqm file ) :
> >
> >
> > module namespace exe = 'http://site.com/execute';
> > declare
> > %updating
> > %rest:path("update")
> > %rest:consumes("application/x-www-form-urlencoded")
> > %rest:POST function exe:update() {
> >
> > xquery:eval-update(request:parameter("query"))
> >
> > };
> >
> > I use it to execute some updates against some databases from different
> clients.
> >
> > Everything works fine by a while, but after some time I get this error
> [1]:
> >
> > java.lang.OutOfMemoryError: Java heap space
> >
> > I noticed that every call to update() the memory grows and grows until
> it reaches the OutOfMemoryError. [2]
> >
> > If I stop the HTTP server, the memory is released immediately. [3]
> >
> > What I'm doing wrong?
> > Is there some command to execute the GC?
> > Is this a problem with "xquery:eval-update()"?
> > Am I using it in the wrong way?
> >
> > Best regards,
> > Sebastian.
> > [1] https://imgur.com/DrcbwQg
> > [2] https://imgur.com/fonmrhm
> > [3] https://imgur.com/SYFBFK8
>

Reply via email to