Re: [basex-talk] RESTXQ - java.lang.OutOfMemoryError: Java heap space
Hi Christian,
Thank you for your reply, you always help me.
*- Can you share the query with us?*. Yes, of course.
The query is pretty much the same that I've sent to Fabrice:
*(# db:copynode false #) {*
* for $case in
doc('file_A.xml')/trademark-applications-daily/application-information/file-segments/action-keys/case-file[case-file-header/status-code=(410,413,616,620,624,625,630,631,638,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,663,664,665,666,667,668,672,680,681,682,686,688,689,690,692,693,694,700,701,702,703,704,705,706,707,708,717,718,719,720,721,722,724,725,730,731,732,733,734,739,740,744,745,746,748,752,753,756,757,760,762,763,764,765,766,771,772,773,774,775,777,778,779,780,790,794,800,801,802,803,804,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,969,973)]
*
* return insert node $case as last into doc("US")*
*}*
And you're totally right about the security concerns: I don't offer to
anyone the access to the RESTXQ endpoint, it's only for me but from another
app ( on this case an Azure Web job instance ).
It's easy for me to do some maintenance tasks from a C# app, using the
RESTXQ, instead of directly on some BaseX GUI.
At the beginning I tried to do this:
*let $path:="A:\sources\xml\"*
*let $files:=*
*let $parts:=("US00","US01")*
*for $part in $parts*
*let $dir:= $path || $part*
*for $file in file:list($dir)*
*return $path || $part || "\" || $file*
*return*
*(# db:copynode false #) {*
* for $file in $files *
* for $case in
doc($file)/trademark-applications-daily/application-information/file-segments/action-keys/case-file[case-file-header/status-code=(410,413,616,620,624,625,630,631,638,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,663,664,665,666,667,668,672,680,681,682,686,688,689,690,692,693,694,700,701,702,703,704,705,706,707,708,717,718,719,720,721,722,724,725,730,731,732,733,734,739,740,744,745,746,748,752,753,756,757,760,762,763,764,765,766,771,772,773,774,775,777,778,779,780,790,794,800,801,802,803,804,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,969,973)]
*
* return insert node $case as last into doc("US")*
*}*
And, of course, this ran out of memory in a matter of seconds. ( there are
more than 100 files )
So, I thought:
-* "Maybe, if I execute each of them using the RESTXQ the memory will be
released after each execution..." *
But seems I'm still missing something because the result is the same. But,
at least, I can finish the process restarting the HTTP server when it fails.
I've uploaded two of the XML files in case you want to test something with
them.[1]
Por supuesto que ayuda!
Regards,
Sebastian.
[1] https://easyupload.io/sd2ilu
On Fri, May 29, 2020 at 8:38 AM Christian Grün
wrote:
> Hi Sebastian,
>
> In general, artifacts of updating queries should be cleaned up after
> the execution.
>
> The stack trace indicates that a very large main-memory database
> instance was created by one of your queries that exceeded the memory
> limits. Can you share the query with us?
>
> As XQuery is more powerful than pure query languages, it may be risky
> to allow the execution of arbitrary client code. First of all, you
> should ensure that the code is run with limited user permissions;
> otherwise, your system can be wiped out by a single file:delete('/',
> true()) call. Next, you could try to limit memory usage and execution
> time via the 'memory' and 'timeout' parameters [1]. However, as it’s
> close to impossible to reliably control the memory consumption of
> single threads in Java, I would rather suggest providing predefined
> user queries.
>
> Espero que esto ayude,
> Christian
>
> [1] https://docs.basex.org/wiki/XQuery_Module#xquery:eval
>
>
>
> On Thu, May 28, 2020 at 8:39 PM Sebastian Guerrero
> wrote:
> >
> > Hi BaseX team!
> >
> > A quick question.
> >
> > Is there some known bug/common setting missing for RESTXQ and memory
> problems?
> >
> > I have this simple module ( into /webapp, a .xqm file ) :
> >
> >
> > module namespace exe = 'http://site.com/execute';
> > declare
> > %updating
> > %rest:path("update")
> > %rest:consumes("application/x-www-form-urlencoded")
> > %rest:POST function exe:update() {
> >
> > xquery:eval-update(request:parameter("query"))
> >
> > };
> >
> > I use it to execute some updates against some databases from different
> clients.
> >
> > Everything works fine by a while, but after some time I get this error
> [1]:
> >
> > java.lang.OutOfMemoryError: Java heap space
> >
> > I noticed that every call to update() the memory grows and grows until
> it reaches the OutOfMemoryError. [2]
> >
> > If I stop the HTTP server, the memory is released immediately. [3]
> >
> > What I'm doing wrong?
> > Is there some command to execute the GC?
> > Is this a problem with "xquery:eval-update()"?
> > Am I using it in the wrong way?
> >
> > Best regards,
> > Sebastian.
> > [1] https://imgur.com/D
Re: [basex-talk] RESTXQ - java.lang.OutOfMemoryError: Java heap space
Hi Sebastian,
In general, artifacts of updating queries should be cleaned up after
the execution.
The stack trace indicates that a very large main-memory database
instance was created by one of your queries that exceeded the memory
limits. Can you share the query with us?
As XQuery is more powerful than pure query languages, it may be risky
to allow the execution of arbitrary client code. First of all, you
should ensure that the code is run with limited user permissions;
otherwise, your system can be wiped out by a single file:delete('/',
true()) call. Next, you could try to limit memory usage and execution
time via the 'memory' and 'timeout' parameters [1]. However, as it’s
close to impossible to reliably control the memory consumption of
single threads in Java, I would rather suggest providing predefined
user queries.
Espero que esto ayude,
Christian
[1] https://docs.basex.org/wiki/XQuery_Module#xquery:eval
On Thu, May 28, 2020 at 8:39 PM Sebastian Guerrero wrote:
>
> Hi BaseX team!
>
> A quick question.
>
> Is there some known bug/common setting missing for RESTXQ and memory problems?
>
> I have this simple module ( into /webapp, a .xqm file ) :
>
>
> module namespace exe = 'http://site.com/execute';
> declare
> %updating
> %rest:path("update")
> %rest:consumes("application/x-www-form-urlencoded")
> %rest:POST function exe:update() {
>
> xquery:eval-update(request:parameter("query"))
>
> };
>
> I use it to execute some updates against some databases from different
> clients.
>
> Everything works fine by a while, but after some time I get this error [1]:
>
> java.lang.OutOfMemoryError: Java heap space
>
> I noticed that every call to update() the memory grows and grows until it
> reaches the OutOfMemoryError. [2]
>
> If I stop the HTTP server, the memory is released immediately. [3]
>
> What I'm doing wrong?
> Is there some command to execute the GC?
> Is this a problem with "xquery:eval-update()"?
> Am I using it in the wrong way?
>
> Best regards,
> Sebastian.
> [1] https://imgur.com/DrcbwQg
> [2] https://imgur.com/fonmrhm
> [3] https://imgur.com/SYFBFK8
Re: [basex-talk] Eclipse editor support and authentication
Hi Ben, Over the time, I have encountered all kinds of development environments for BaseX projects, including Emacs, Vim, Sublime, oXygen, etc. One project was completely developed with Windows Notepad; I was surprised to learn that it encompassed thousands of lines of code ;) The BaseX GUI is definitely used by many developers, also for more complex web applications. Its main advantage is that project files are always compiled in the background. The number of available features and shortcuts is growing with each version, they are listed in our documentation [1]. A special gimmick is the »Sort Lines« feature, it’s very efficient even for texts with millions of lines. In an upcoming version, we’ll add Search & Replace for multiple files (including regex operations), this will facilitate refactorings. But I completely agree that the editing facilities cannot cope with full-fledged feature sets of Java IDEs. In earlier days, the XQuery Eclipse plugins were somewhat ok; today, Reece H. Dunn’s support for IntelliJ is definitely the better choice [2]. Regarding safe authentication, you have probably already followed Steve’s links. Salted SHA256 hashes are currently used for passwords. Hope this helps, Christian [1] https://docs.basex.org/wiki/Shortcuts [2] https://docs.basex.org/wiki/Integrating_IntelliJ_IDEA On Mon, May 25, 2020 at 3:37 AM Ben Pracht wrote: > > Hi Folks, > > Perhaps I'm using too new of an Eclipse (Photon), but I'm not able to > integrate Eclipse with BaseX. I installed the trial Oxygen plugin, though I > don't want to pay for it. Even then, it would not offer a BaseX datasource, > only one for ExistDB. I tried to install XQDT, but it could not be installed > without downgrading many components, which I did not want to do. > > As an Eclipse person, using IntelliJ would not be very productive as I've > gotten used to the Eclipse keyboard shortcuts. > > I'd like to create a server program based in BaseX with RESTful xquery based > services. Initially, it would be a CRUD type application. It doesn't need > high performance, just not terrible, and I'd like to be able to authenticate > into it. I'm confident BaseX can handle the task, it's just getting an > editor capable of handling it. Using the GUI that comes with BaseX is good > for small tasks, but not so good for larger applications. > > It doesn't appear that there's much in the way of XQuery support in Eclipse > anymore, and even the XQDT if it could be installed doesn't support the newer > 3.1 type xquery (I believe I read). Can someone recommend a good editor? > I'm using GVim for many small projects, though it takes work to get it to > work with larger projects efficiently (CTags helps for C/C++ code, but I > dont' think it works with XQuery). > > Separately, would it be possible to develop a secure BaseX based application > that users could log in, and authenticate into, store delicate personal > information and have it be sufficiently secure? Could it be at least as > secure as, say a Tomcat application? Sorry to ask this question, I do app > development, but usually my role comes in after the user has been > authenticated using something like OAuth or other. > > > Best Regards, > Ben Prachtcom
