Re: [basex-talk] Retrieve large amount of data from XML-resource
Hello Christian,
Am 15.02.2017 um 12:14 schrieb Christian Grün:
> Hi Jan,
>
>> For example for a xml-file with the size of 79167874 ("FACTOR", as shown by
>> the web-gui) "session.execute(new XQuery("doc('db/file.xml')"));" will
>> result in an exception[0].
> First of all, we have recently improved logging output for the
> client/server architecture with BaseX 8.6.1. Maybe you get some more
> helpful output with the latest snapshot [1]?
Excuse me.. In Germany we say "zu frueh gefreut"..
When running the query WITH ADMIN PERMISSIONS, it works.
When running the query with read/write permissions the same (not more
detailed) "interupted"-Exception shows up.
Jan
>
> To decrease memory consumption, you can specify an output stream in
> your client (via setOutputStream). Do you really want to return the
> full XML document and process it in main memory? Returning 80 MB of
> data should not cause serious problems, though, even with 1GB of RAM.
> Did you manage to successfully parse this resource in the BaseX GUI
> (or is the document already stored in a database)?
>
>> My assumption is that a timeout is exceeded somewhere.. so my next question
>> is how to properly set the timeout
> For non-admin users, you can set a global server timeout via the
> TIMEOUT option [2]. If I remember correctly, the timeout will not be
> checked while a resource is parsed, it could be that it does not
> necessarily stop your query in time.
>
> Cheers,
> Christian
>
> [1] http://files.basex.org/releases/latest/
> [2] http://docs.basex.org/wiki/Options#TIMEOUT
> .
>
Re: [basex-talk] Retrieve large amount of data from XML-resource
> Excuse me.. In Germany we say "zu frueh gefreut".. > When running the query WITH ADMIN PERMISSIONS, it works. > > When running the query with read/write permissions the same (not more > detailed) "interupted"-Exception shows up. So this indicates pretty clearly that the TIMEOUT was exceeded. I guess your document is not stored in a database yet, right? Feel free to give me feedback on the questions of my initial mail.. >> To decrease memory consumption, you can specify an output stream in >> your client (via setOutputStream). Do you really want to return the >> full XML document and process it in main memory? Returning 80 MB of >> data should not cause serious problems, though, even with 1GB of RAM. >> Did you manage to successfully parse this resource in the BaseX GUI >> (or is the document already stored in a database)? >> >>> My assumption is that a timeout is exceeded somewhere.. so my next question >>> is how to properly set the timeout >> For non-admin users, you can set a global server timeout via the >> TIMEOUT option [2]. If I remember correctly, the timeout will not be >> checked while a resource is parsed, it could be that it does not >> necessarily stop your query in time. >> >> Cheers, >> Christian >> >> [1] http://files.basex.org/releases/latest/ >> [2] http://docs.basex.org/wiki/Options#TIMEOUT >> . >> >
Re: [basex-talk] Retrieve large amount of data from XML-resource
Hi Jan, > btw: Is there an issue with the latest basex web-app? I am not able to > log in (the field to specify the port is missing as well). With 8.6, the remote connections feature was removed from the DBA [1]. As it seems that this was used by more people than we thought, we’ll probably reintroduce it in a rewritten version in future. Until then, simply use the 8.5.x DBA code. A question to everyone on the list: Who else is missing this feature? Christian [1] http://docs.basex.org/wiki/DBA
Re: [basex-talk] Retrieve large amount of data from XML-resource
Hello Christian,
Am 15.02.2017 um 12:14 schrieb Christian Grün:
> Hi Jan,
>
>> For example for a xml-file with the size of 79167874 ("FACTOR", as shown by
>> the web-gui) "session.execute(new XQuery("doc('db/file.xml')"));" will
>> result in an exception[0].
> First of all, we have recently improved logging output for the
> client/server architecture with BaseX 8.6.1. Maybe you get some more
> helpful output with the latest snapshot [1]?
I can't figure out why but both of my problems are gone after updating
to version 8.6.1. - thank you.
(It would be helpful to know why to verify it is not an arbitrary
problem with my system..).
btw: Is there an issue with the latest basex web-app? I am not able to
log in (the field to specify the port is missing as well).
regards,
Jan
Re: [basex-talk] Retrieve large amount of data from XML-resource
Hi Jan,
> For example for a xml-file with the size of 79167874 ("FACTOR", as shown by
> the web-gui) "session.execute(new XQuery("doc('db/file.xml')"));" will
> result in an exception[0].
First of all, we have recently improved logging output for the
client/server architecture with BaseX 8.6.1. Maybe you get some more
helpful output with the latest snapshot [1]?
To decrease memory consumption, you can specify an output stream in
your client (via setOutputStream). Do you really want to return the
full XML document and process it in main memory? Returning 80 MB of
data should not cause serious problems, though, even with 1GB of RAM.
Did you manage to successfully parse this resource in the BaseX GUI
(or is the document already stored in a database)?
> My assumption is that a timeout is exceeded somewhere.. so my next question
> is how to properly set the timeout
For non-admin users, you can set a global server timeout via the
TIMEOUT option [2]. If I remember correctly, the timeout will not be
checked while a resource is parsed, it could be that it does not
necessarily stop your query in time.
Cheers,
Christian
[1] http://files.basex.org/releases/latest/
[2] http://docs.basex.org/wiki/Options#TIMEOUT
[basex-talk] Retrieve large amount of data from XML-resource
Hello,
i am getting an BaseXException when trying to get large amount of data from a
XML-resource with a JAVA-Client.
For example for a xml-file with the size of 79167874 ("FACTOR", as shown by the
web-gui) "session.execute(new XQuery("doc('db/file.xml')"));" will result in an
exception[0].
My assumption is that a timeout is exceeded somewhere.. so my next question is
how to properly set the timeout (Or in general, how to run specific commands
via java-client, I can't find it in the wiki.
"session.query("timeout[0]");" seems to not have any influence.
Query large files in general, for example query its number of instances, seems
to not be the problem.
Increasing the -Xmx in the basex-script did not help.
# Options for virtual machine (can be extended by global options)
BASEX_JVM="-Xmx10G $BASEX_JVM"
Increasing the -Xmx of the java-application did not help as well.
I hope you can help me.
best regards,
Jan
[0] org.basex.core.BaseXException: Interrupted.
at org.basex.api.client.ClientSession.receive(ClientSession.java:191)
at org.basex.api.client.ClientSession.execute(ClientSession.java:160)
at org.basex.api.client.ClientSession.execute(ClientSession.java:165)
at org.basex.api.client.Session.execute(Session.java:36)
at net.ddns.janstapel.datamanagement.Main.main(Main.java:37)
